Residual or skip connections
From CGCV-Lecture_Notes.md .
Make training easier by preventing vanishing gradients.
It makes it easier to fit the identity function.
A NN layer can be written as:
$y = f(x * w)$
I.e. the output a layer is the input convolved with weights, passed through some non-linear function.
A residual layers adds the input to the output (either before or after the non-linearity). This can be done after each layer, but in principle this is used to skip layers (so the input x that is added is the input from a few layers back). This construction is called a residual block.
$y = f(x * w + x)$ or $y = f(x * w) + x$.
Umut said that both versions are used. To me the second is more intuitive, but Umut uses the first formulation and most visualizations of residual blocks also place the addition operation before the non-linearity (typically ReLu).
What does this do? By making the identity function easy to fit, we ensure that if a given block does not learn anything useful (too bad) it at least can still learn the identity function so that this does not hurt optimization.
- Premise: f(xw)=0 is easy to fit, but f(xw)=x (identity function) is hard to fit
- Assume that for some layer, the best thing to achieve is just pass the input through the non-linearity, so that it doesn't hurt the network. So y = ReLU(x * w ) = x, i.e. fit the identity transformation.
- This requires the network to fit y = ReLU(x * w ) = x, which we just said was hard to do.
- Now consider y = ReLU(x * w) + x. This must fit ReLU(x * w) = 0, which we just said was easy!
This solves vanishing gradients, because during back propagation we keep multiplying them with small numbers the more we go towards the shallow end of the network. But now gradients can be passed through the simple additive connection. The vanishing gradient problem specifically occurs due to multiplication.
