Rectified Linear Unit (ReLu)
Nowadays ReLu is the default activation function for CNNs. This goes well with the He 2015 initialization scheme. This note explains why.
What problem does it solve?
The problem with a sigmoid (or similar) activation function:
- Sigmoid gets saturated quickly
- This means that the derivative will be practically 0
- This prevents backpropagation of gradients, because it will multiply all previous gradients with 0
- This problem is manageable in shallow networks, but practically makes learning in deeper networks extremely inefficient (and was thus a huge innovation that boosted the deep learning revolution)
Improvements
- Greatly accelerates SGD
- Very inexpensive to compute (because no expontential function etc.)
- No vanishing gradient problem
Leaky ReLu fixes the "dying ReLu" problem.
Relation to initialization
There is a correlation between the distribution of activations and the initialization method.
The saturation of the sigmoid is useful for visualizing the potential problem. If you initialize your weights such that many of your layers activations fall in the saturated area of the sigmoid function, then all these activations are essentially thrown away and useless.
So instead during initialization you want to ensure that most of the activations are in the area where the sigmoid is sensitive (i.e. in the "linear part")
This means that we want to initialize such that the incoming activation will be centered around 0 with a controllable variance.
Glorot 2010
Glorot observed however that if you have more inputs to a neuron, the variance of the incoming activations will also be higher, meaning that the above problem is more likely to occur.
Glorot 2010 suggested to correct for that by downscaling the weight vector by the square root of the fan-in (i.e. the amount of incoming activation values).
Weight initialization: Gaussian with $N(0, \sqrt{\frac{1}{n_{in}}})$.
He 2015
Glorot2010 is a general approach that does not factor in differences between activation functions.
He2015 is instead designed specifically for ReLu and is a slight adjustment of Glorot2010.
Weight initialization: Gaussian with $N(0, \sqrt{\frac{2}{n_{in}}})$.
Also see neural_network_weight_initialization.md .