Regularization in CNNs
The amount of neurons increases the capacity of networks to fit data. We need enough neurons to fit very complex patterns in input data, for which image data is a good example. As a direct result, a high capacity also leads to the risk of overfitting.
Discourage large weights
We can modify the loss function to discourage large weights with regularization strength $\lambda$ initially set by the user and can be tuned:
$$L = L_{data}(x,W) + \lambda R(W)$$
L1- and L2-regularization are typical regularization schemes for punishing large weights. L1 increases the loss by the sum of all weights. L2 increases the loss by the sum of squares of all weights.
L1 -> $R(W) = \sum_i |w_i|$
L2 -> $R(W) = \sum_i w_i^2$
Dropout
Creative schema, also used (for the first time?) in AlexNet, that is a different form of regularization and usually improves performance.
- Dropout randomly drops neurons for each training sample
- This means that de facto each training sample is processed by a different network configuration
- In this sense, dropout can be seen as a subtle form of ensemble learning where the network combines all sub-networks created by dropping specific neurons.
- Less parameters, less risk of overfitting!
Further rationale: don't place all your bets on one horse!
If you imagine a situation without dropout, it may be a case that over time the neuron becomes most sensitive to an input of a single other neuron, because this happens to be a very good feature for minimizing the loss. Likewise, the weights over the other connections may decay. During dropout, it may be the case that this neuron is dropped, which over time will cause the importance of weights to be more "spread out" over more incoming neurons.
This effect is comparable to L2 regularization because spreading the weights reduces the L2-norm of the weights!
Dropout is typically combined in dense layers, because we have the most parameters there with the highest risk of overfitting.
It can however also be used in convolution layers.
Batch Normalization
If your mini-batch size is small enough, batch normalization can also have a regularizing effect.
See ISMI-4f-Batch_normalization and batch_normalization.md .