Convolutional Neural Network
@neural_network @convolution
Common data augmentations
The main goal of data augmention is to reduce overfitting. Data augmentation has the advantage of learning more invariant features (e.g. also independent of rotation) and has to additional effect of providing more training examples.
- Image translations and reflections
- Lower the intensity of RGB channels in training images
Why do we do the latter?
One of the reasons for scaling input data into a neural network is to ensure a stable start to training over the first few iterations. Since the weights of the network are initially randomized, unscaled input could potentially create huge activation values that immediately lead to exploding gradients. For example, instead of passing pixel values from 0–255 into the input layer, we usually scale these values to between –1 and 1. (Foster 2019, 52)
Convolution
.
Convolution of signal f and filter g:
$$(f * g)[x,y] = \sum_{u=-M}^{M}\sum_{v=-M}^{M} f[x-u, y-v]g[u,v]$$
Where the filter is in this case a convolution kernel.
For multi-channel input (e.g. RGB image), "one" filter application will internally be the application of three filters, each to one channel, but still with a single response (by summing the responses to each input channel):
$$C(i,j) = \sum_{k=0}^{l} \sum_{u=-M}^{M}\sum_{v=-M}^{M} f_k[x-u, y-v]g_k[u,v]$$
A regular convolution without padding will shrink the feature maps. With filter size $f$:
$o=i-f+1 = i-(f-1)$
Which is a special case of $\frac{i + 2p - f}{s}+1 = i$.
Deep learning libraries generally offer two types of convolution:
- 'valid': only convolve the input signal; but then filter application of any filter larger than 1 will not use border pixels as the filter's anchor, thus reducing the output size
- 'same': to keep the same size after convolution, you have to pad the input signal
- The required amount of padding depends on the convolution filter size and stride
'Same' padding
To ensure the output size stays the same, you need to apply 'same' padding with $(f-1) / 2$ pixels, where f is the filter size.
(This can be easily derived from the general formula $\frac{n + 2p - f}{s}+1 = n$)
Hyperparameters and their effects
- Depth/channels: the number of channels
- Padding: padding of input around the border (default: zeros)
- Stride: how much do you slide the filter per step (default: 1)
Assume input of shape i x i and filter of shape f x f.
- For padding p, output: $(i + 2p - f + 1) * (i + 2p - f + 1)$
- For stride s, output $\frac{i+2p -f}{s}+1$ x ibid.
Convolving over multiple channels makes my brain hurt.
With stride 1 and padding 1, 6x6x3 input convolved with 3x3x3 filter.
Then for each application of the filter you end up with 4x4x2 output which is called a feature map.
(6 + 21 - 3)/1 + 1 = 4 and (3+21-3) / 1 + 1 = 2.
Does this mean that for n applications you'll end up with 4 x 4 x 2n output, so with 2n channels and 2n feature maps?
N.B. when doing 2D convolution, you end up with 4x4x#filters.
You thus see that while reducing the original vertical and horizontal dimensions of the input image (usually with 1 or 3 channels), you'll end up with many more channels after doing multiple convolutions.
Amount of parameters
See example here.
Question on filters and kernel size
Filters and the amount channels (=amount of feature maps)
Q: when we define a neural network using a deep learning framework, we only specify the kernel size but not explicitly the depth of used filters. Why is that?
A: Answer is below in the example. The depth of the filter matches the amount of channels of the preceding layer, so that each filter application produces one response. The amount of output channels of is thus determined by the amount of used filters.
Note however that it is a very good convention to increase the amount of channels for deeper layers.
Example
Taken from David Foster, Generative Deep Learning, p.49-50.
REVIEW THIS. Seems F*cked.
Define a simple network:
- RGB Image Input (32,32,3)
- Conv 1: 4x4 kernel, stride 2, padding 'same', 10 filters,
- Conv 2: 3x3 kernel, stride 2, padding 'same', 20 filters
- Flatten
- Dense with softmax, 10 outputs
Let's ignore batch size for now.
The filters of the first layer have dimension 4x4x3, because the input has three channels.
The depth of the filters is always the same as the number of channels in the preceding layer.
TODO: seems everything below is actually about 'valid' convolution, not 'same', since obviously the size of the feature maps is reduced. But even then not quite... output 16x16 assumes padding=1 I think.
With stride 2 and padding 'same' we halve the width and height of the input to 16x16. We apply 10 filters, so our output looks like 16x16x10. As said above, the depth of the filter matches the amount of channels of preceding layer. This implies that each filter application will result in a single response. This is why the resulting depth/amount of channels after applying 10 filters is 10.
Next filter will be 3x3x10, again matching the amount of channels from the previous layer. If we apply 20 3x3x10 filters, again with stride 2 and padding 'same', on 16x16x10, then we get 8x8x20.
Advantage (w.r.t dense layers)
Why use convolutional layers instead of fully connected layers? Dense layers:
- Inefficient: Dense layer has a lot of weights especially for the first layer, which will be HUGE
- A small 128x128 layer will already have 16384 x n + 1 parameters for n neurons in the dense layer.
- So dense layers hugely increase the capacity of the network, giving a large risk of overfitting
- To tune this very large amount weights without overfitting, you need a lot of training data
- You also need a huge amount of memory to store all weights and gradient computations
- No robustness: because of the fixed 1-to-1 connectivity on the level of individual pixels there is no robustness against shifts in input, i.e. no translation invariance
- Does not take spatial ordering into account
- But image data has a clear 2D topology
Convolution instead ...
- Takes the 2D topology into account because filters are 2D
- Translation invariance for feature recognition
- Better generalization performance as result of feature invariance and the hierarchical layout of feature extraction
- I.e. later layers have more global receptive fields (cf. vision in biology)
- See receptive fields
- Less free parameters due to weight sharing
- In convolution you only have the kernel weights which are convolved over all regions of the image (vs. having separate weights for each input pixel and each output neuron as in dense layers)
- This expresses that whatever feature the filter captures may be present anywhere in the input image
Pooling
See pooling.
Weight initialization
See weight initialization schemas for neural networks
Regularization
See regularization
Internal Covariate shift
See internal coveriate shift .
Batch normalization
See batch normalization .