Batch normalization
From original batch normalization paper. Internal variate shift ...
slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities.
Batch normalization mediate problems caused by internal covariate shift . As such, it allows a bit more liberalism with higher learning rates and can speed up training.
But batch normalization does not simply eliminate internal covariate shift, and neither is this straightforwardly the explanation for its success.
Normalizing inputs can also help prevent that for example sigmoid activations become saturated.
Intuition
The basic idea behind batch normalization is to apply the very common practice of data normalization in neural networks as well. In machine learning applications such as kNN, data normalization is a must because otherwise the used distance metrics are dominated by features that just happen to have a larger scale.
In a way, this is finishing the job we start with proper intialization. We wanted to have zero-mean activations with a manageable variance. Then why don't we enforce that at each layer?
Procedure:
- Calculate mean and standard deviation for each input channel across the (mini-)batch
- Another way of saying this is that you normalize everything but channels (i.e. you do the normalization computation per channel).
- Normalize inputs by subtracting the mean and dividing the standard deviation of the mini-batch.
The basic idea is thus to convert the input distribution to a standard Gaussian distribution. However, this may not be the most appropriate for a given dataset and problem. So instead, the common approach is to make the mean and variance parameters that the network can learn and adjust.
For each channel, you learn two extra parameters:
- The scale (gamma)
- The shift (beta)
Where you take the normalized input $x_{norm}$ and scale it and shift it.
$y_i \leftarrow \gamma x_{norm} + \beta$
These scale and shift parameter are learnable.
It may be the case that batch normalization is actually unneeded and hurting performance. If that's the case, we just want the network to learn an identity mapping such that $\hat{x}^{(k)} = x^{(k)}$. You'll see that the identify mapping can be retrieved by the network learning $\gamma^{(k)} = \sqrt{Var[ x^{(k)}]} $ and $\beta^{(k)}= E[ x^{(k)}$.
Summary:
- Compute the expectation over mini-batch $x^{(k)}$ (with k samples)
- Compute variance over $x^{(k)}$
- Normalize: $\hat{x}^{(k)} = \frac{ x^{(k)} - E[ x^{(k)} ]}{\sqrt{Var[ x^{(k)}]}}$
- Compute $y^{(k)} = \gamma^{(k)} \hat{x}^{(k)} + \beta^{(k)}$
Mini-batch size
Q: How does batch normalization work when you don't have a batch but only one observation?
A: Well, computing statistics over a single sample doesn't make any sense, and the computed statistic will be very unstable. So don't do this if you have a mini-batch size of 1 (i.e. no batches at all)!
It may of course happen that you end up with a too small mini-batch.
Solution with momentum: instead of only using the mean and standard deviation of the current input, include the moving average of the mean and standard deviation of each channel and store this moving average at the layer. Then you always have an available estimate.
You can weigh the current value and the previous value when computing the running average using the momentum parameter.
Parameters
So each layer has two trainable/learnable parameters, namely the aforementioned gamma and beta. Additionally, there is the non-trainable moving average and standard deviation. We need these four parameters per channel in the preceding layer.
Regularization
Interestingly, batch normalization can also be seen to apply a form of regularization.
Because the mean and variance are computed over mini-batches that in turn are sampled randomly, there will be variation in the mini-batch estimates relative to the true dataset mean and variance. This gives the network more robustness against changes in the input activations while still reaping the benefits of more stable learning.
This variation will be smaller if your mini-batch size is larger, so this subtle regularization effect is only if your mini-batch size is sufficiently small.
The covariate shift strikes back
Batch normalization tries to minimize internal covariate shift by normalizing and also learning the shift and variance. This does not entirely eliminate internal covariate shift insofar as the learned parameters may also not be entirely accurate.
Additionally, to some extent the scale and shift parameters will themselves be subject to internal covariate shift. Since they are used for manipulating the inputs, this reintroduces internal covariate shift between layers! (But if I'm not mistaken, this is what has the regularization effect mentioned above.)
However, batch normalization is still likely to help optimization because internal covariate shift is, in that case, no longer caused by all parameters of the preceding layer, but only by two parameters.
This likely results in a smoother error landscape that facilitates the stability of the learning procedure.
So the answer why batch normalization works is not just that eliminates internal covariate shift, but instead:
The important property of batch normalization is that the mean and variance are not decided by complex interactions between multiple layers, but rather by two simple parameters. This simplification of the learning dynamic is the key property of batch normalization and why it improves performance.
Source is this great blog.
Alternative: Layer normalization
In batch normalization, you normalize over the batch dimension of a mini-batch. As said before, this introduces problems when you have extremely small batches or even use single observations (e.g. in online learning).
Instead of normalizing over the batch dimension, you can also normalize over the channel/feature dimension. This makes the normalization procedure independent of the batch size, but unlike batch normalization it is not a trivial reparametrization. By normalizing over feature maps, you change what the network does.