internal covariate shift
Internal Covariate shift
Umut : "Fancy way of saying change (shift) in the input distribution (covariate) of hidden layers (internal)"
In order to understand why this is a problem, you can imagine a neural network trained on a dataset with mean 0 and standard deviation 1. When we suddenly start feeding it data with a completely different distribution, the accuracy of the network's predictions drops.
Something similar applies during the training of the network: if you update a layer with back propagation based on updates from one distribution, then it has a negative impact on the layer's performance if that input distribution changes in the meantime.
This is an innate problem of doing gradient descent, where the last layer, call it n is updated first based on the activation outputs of layer n-1. (And layer n-1 is updated using activations from outputs of layer n-2 ). Now, since the weights of layer n-2 will have changed, the input distribution into layer would have been different if we now again do a forward pass. But layer n-1 was trained on the old input distribution, i.e. the output distribution of layer n-2.
This problem is always present to some degree, but becomes a problem if the shifts become too big, for example when certain gradients start vanishing or exploding.
The ideal scenario is where your input distribution stays the same throughout the network, so that the network can just focus on learning the right weights to solve the problem given this input distribution, without having to constantly adapt to changing inputs.
Note that whereas it is often said that batch normalization eliminates internal covariate shift, this is not entirely accurate. It minimizes the problem by estimating the shift with learnable parameters; but these parameters themselves can of course themselves be subject to covariate shift!
Also see Foster 2019, Generative Deep Learning, p.52.
You scale the input to facilitate a good start of the training procedure, to avoid that random weights do not immediately lead to huge activations and huge errors, i.e. exploding gradients.
But another matter is whether the activations through layers remain stable. However, during training the weights are being updated. It can happen that the weights in one layer become relatively big or small. Foster calls this "runaway weight values" that can cause an "overall collapse of the network" (52).
This breaks the assumption being made by networks during training, namely that the distribution of input activations from previous layers is more or less consistent across layers.
Why does this slow down training? From original batch normalization paper:
This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities.
Foster uses an analogy to explain internal covariate shift, which I find kind of nice to record here:
Imagine you’re carrying a tall pile of books, and you get hit by a gust of wind. You move the books in a direction opposite to the wind to compensate, but in doing so, some of the books shift so that the tower is slightly more unstable than before. Initially, this is OK, but with every gust the pile becomes more and more unstable, until eventually the books have shifted so much that the pile collapses. This is covariate shift. (52)