Regression analysis to the umpteenth degree. My father has never used a computer and gets this.
Imagine you are a trading company that has a stock trading algorithm (bot) that uses reinforcement learning to continuously improve it's performance. It's making good money but you can't understand exactly how it works so it makes you nervous. Now imagine it starts making more money than all your other algorithms. You internalize your nervousness and use it exclusively. Now imagine it's not imaginary and happening all around us.
You can only explain it because you have oversimplified the problem. You invest only looking at profits (and in some timeframe too). Deep learning is applied to everything from art to justice, handwaving its implications away.
Look at this other scenario:
In a language course the teacher looks for any subject that help people talk, even if somewhat polemic. One day the teacher brings up the subject of self driving cars.
One of the students claims to be a test pilot for new prototype cars (at a mainstream car company trying to catch up with market disturbers), testing cars intended for mass market, just before they're there so that they may still be modified. He pretends to know, but he's apparently no engineer.
The teacher says how should a self driving car behave when the cars ha to decide whether to hit an old man or a child in a critical situation. Meaning letting philosophical questions to machines is troublesome.
The student says he's guessing it would hit the child, since it's less pixels on cameras, so hitting a smaller obstacle would seem safer.
The rest of the class is flabbergasted that someone might construe self-driving to just crashing to the smaller object whenever one can't avoid the crash at all.
Nobody is too worried whether the self proclaimed test pilot is right. The problem is they realize they hadn't even though what the problem was, they were simply assuming someone had solved it. Well, maybe someone has solved a problem and has called it the same as someone else. But the problem is that those claiming to have solved it probably can't say what would the car do in that scenario. They could say something like "well, it's trained from so many hours of coverage form real people driving cars, so it would most likely do what most people do in that situation.". But nobody knows whether a particular situation has happened in the coverage, whether it has been overriden by different situations, whether the real situation will or will not match... It's just some statistic "we've had fewer accidents than human driven cars". So what? Humans are assumed a right to exist and take decisions, which one can afterward judge and punish if found wrong. Machines don't even have a right to exist, let alone take decisions, so people assume the humans designing it are responsible and must be able to explain the design decisions. With deep learning the engineer just can't really explain the decisions embedded in the model, just how the training material was selected, the neural network topology, etc. Yes, but why did it hit the child ?