Weight initialization in neural networks

@Xavier_initialization @neural_network

Weight initialization has a large impact of learning speed.

If your weights are too small or too big respectively, the signal either shrinks or explodes respectively during a forward pass, to the point where it's not very usable anymore. During back propagation this will either cause the network to barely update (extremely slow convergence) or too roughly (potentially preventing convergence altogether). Additionally, exploding activations may cause memory-issues and decrease computation efficiency.

E.g. consider very large activations going into a sigmoid activation function: this saturates the function and makes it completely unresponsive to differences in the learning signal.

Sketch of the problem

A forward pass consists of n matrix multiplications, between an input vector (e.g. activations) and your weight matrix.

Why does initializing weights from a standard distribution not suffice?

Observation:

  • assuming we do not have activation functions (big assumption!)
  • assuming we initialize our inputs according to a normal distribution
  • then if you initialize weights from a standard normal distribution, the standard deviation of the output activations (so after the whole forward pass with n matrix multiplications) will be close to $\sqrt{#inputs}$.
  • N.B. in this scenario the weights are of the same scale as the inputs.

Reasoning:

In a single matrix multiplication, for a single node in the next (fully connected) layer you effectively multiply n activations with their weights. Because each activation is sampled along a normal distribution, the distribution of that node in the single layer will have mean 0, but potentially a variance of 512. I.e. if all weights happen to be initialized as +1, the activation of a node in the next layer could be 512; similarly -512 when all weights are initialized as -1.

The standard deviation of a node in the next layer could approximately be $\sqrt{#inputs}$.

We can prevent this explosion by preemptively dividing the weights by $\sqrt{#inputs}$.

However, we made some big assumptions here. In deep neural networks, we have various non-linear activation functions in between that also influence the size of values passing between layers. E.g. common functions such as the hyperbolic tangent squash the values into [0,1] again. If we then apply above trick, suddenly the weights will cause low activations.

So a basic weight initialization scheme would sample from some distribution, either uniform or standard, in the range [0,1] or some smaller range (default MxNet samples from [-0.7, 0.7]), and then divide by the square root of the number of inputs. In short, it samples weights uniformly between [-1/sqrt(n), 1/sqrt(n)]. But this leads to vanishing gradients.

Xavier initialization is nowadays a standard solution to this issue.

N.B. in some libraries (e.g. keras) this is called Glorot initialization instead (Xavier Glorot was the name of the researcher)

Xavier initialization

Paper that proposes Xavier initialization: Understanding the difficulty of training deep feedforward neural networks - Xavier Glorot & Yoshua Bengio.

Goal of the paper: "to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks" (p. 249)

Observations: the recent (cf. 2006-2010) success in training deep nets is due to new training mechanisms and better weight initialization schemes.

Solution to the sketched problem you still sample from a uniform distribution, but you make sure it's bounded by the following formula:

$\frac{2}{\sqrt{n_{in} + n_{out}}}$

These are also called "fan-in" and "fan-out" apparently.

To ensure information flow both through forward passes and back propagation, you want to meet two requirements (p. 253):

  1. $\forall i, n_i Variance[W^i]=1$
  2. $\forall i, n_{i+1} Variance[W^i]=1$

So at each given node you want to have a variance of 1 over the incoming activations, and you want to maintain this variance when passing to other nodes.

Combining these requirements, we get the following if we assume that the two relevant layers are the same width n:

$(n_i + n_{i+1})Var[W^i] = 2$

So: $Var[W^i] = \frac{2}{(n_i + n_{i+1})}$

(The paper still does some magic with a square root of 6, which I don't follow).

The new procedure is to sample from a uniform distribution with mean 0 and variance $s^2 = \frac{2}{(n_i + n_{i+1})}$.

There is a new problem: Xavier initialization still leads to vanishing gradients when used in combination with non-symmetric non-linear activation functions. The Kaiming Initialization procedure tries to remedy this.

Kaiming Initialization

TODO

He initialization

He initialisation is specifically tailored to be used with ReLu activation, which is de facto the default activation function for CNNs right now.

See ISMI-4d-ReLu_with_He_initialization