It is not known how the individual neurons work together to arrive at the final output in deep learning. It is also not known what any specific neuron is doing on its own. This is called black box.

Are deep learning neural networks black boxes? Each neuron in such a network receives inputs from other neurons. It multiplies each by a weight learnt during training. Then an output is generated, Mathematically, it is equivalent to a boundary in the space of previous layer of neurons. Black box emerges when multiple neurons work together. It exists owing to bizarre decisions by intermediate neuros on the way before making network’s final decision. It is complex non-linear mathematics. Black box is a result of non-intuitive intermediate decisions.

In the training, each neuron learns, but the shallower neurons learn to extract features from inputs and map them into useful latent space. All neurons are being trained simultaneously. To begin with, they start randomly. Shallower neurons develop latent space which is useful for deeper neurons. Deeper neurons, at the same time, must learn how to map this latent space to proper outputs.

There is poor performance of some neurons crippling the learning of each neuron. They must learn how to compensate the deficiencies of other neurons. It is a tangled web of interdependence.

The nature of learning is entangled. Each neuron thus does not have a cear function. There is no clarity as to the task it is trying to learn. This leads to a black box.

It is an un-openable black box. We have to find an alternative training method for deep learning. It should resemble human reasoning. Gradient-based deep learning will never achieve real AI.

An alternative training method is called essence neural networks (ENNs). Each neuron takes specialised function, still it can scale to larger problems. Black box vanishes when we throw away gradient descent.

Black box is not necessary feature of neural networks. We have to go beyond back propagation and gradient descent. Then deep learning mimics human reasoning.

In complex models, parameters or weights considered could run into thousands or millions or billions. These are considered black boxes. Many clinicians are wary of ML because of the concerns about black box. Internal working of such a system is not known. The internal logic of CNN is not known. Even NLP has black box nature.