Batch Normalization

Also see batch_normalization.md

Generally in machine learning normalizing input data facilitates the training procedure.

  • E.g. in kNN, it makes features more "comparable" in terms of a distance-metric
  • Of in SVM it makes sure some feature is not extremely important just due to having a larger scale

In NNs each layer takes the activations of previous layers as inputs. They may be all over the place. If we normalize them as well, learning also becomes more efficient!

In a way, this is finishing the job we start with proper intialization. We wanted to have zero-mean activations with a manageable variance. Then why don't we enforce that at each layer?

General procedure

We can do this without problem, because normalization is differentiable! This means that batch normalization does not block gradient flow.

  1. Compute the expectation over mini-batch $x^{(k)}$ (with k samples)
  2. Compute variance over $x^{(k)}$
  3. Normalize: $\hat{x}^{(k)} = \frac{ x^{(k)} - E[ x^{(k)} ]}{\sqrt{Var[ x^{(k)}]}}$
  4. Compute $y^{(k)} = \gamma^{(k)} \hat{x}^{(k)} + \beta^{(k)}$

Why is step 4 handy? It may be the case that batch normalization is actually unneeded and hurting performance. If that's the case, we just want the network to learn an identity mapping such that $\hat{x}^{(k)} = x^{(k)}$. If you look at equation 4, you'll see that the identify mapping can be retrieved by the network learning $\gamma^{(k)} = \sqrt{Var[ x^{(k)}]} $ and $\beta^{(k)}= E[ x^{(k)}$.

During training

  1. Learn $\gamma$ and $\beta$
  2. Store the average mini-batch expectation and variance because this needs to be used at test time for making predictions
    • Note that this is an attempt to approximate the mean and variance over the whole dataset (but mean of means is not the same as mean over whole dataset )
    • This way we can avoid computing the mean and variance over the whole dataset, which is too laborious and slow and probably requires too much memory
    • This is not exactly the same, but this in fact is considered to be a useful property because it serves as a form of regularization (see below)

What does it do?

  • Normalized activations speeds up training
  • Makes weights of deep layers more robust to changes of weights in previous layers because the mean and variance of the incoming activations stay stable
  • Has some regularization effect
    • Mean and variance are computed per mini-batch
    • Mini-batches are randomly sampled
    • Which introduces variance in the normalized activation values
    • So if we use small enough mini-batches, the mini-batch mean will have more variance w.r.t. the true dataset mean.
    • But mini-batches obviously should not be too small as well, because then the computed statistics are not representative
    • In particular, it does not make any sense to use batch normalization when using a mini-batch of a single sample