Popular Convolution Neural Network Architectures
VGG-net (a.k.a. OxfordNet) 2014
- 19 layers (this was deep back then)
- Fixed filter size 3x3 for all filters (no attempt to optimize this)
- After pooling, double channels: 64 -> 128 -> 256 -> 512
- Different versions tested with more repetitions
- Final config of conv layers: 2x64, 2x128, 4x256, 4x512, 4x512
GoogLeNet 2015
- Deeper, 22 layers
- New layer architecture with the "inception module".
- The basic idea of the inception module is to replace a single convolution layer with a concatenation of multiple filters
- 1x1 convolution
- 3x3 convolution
- 5x5 convolution
- 3x3 max pooling
This was problematic:
- Concatenation of feature maps leads to a very large weight matrix
- Max pooling affects the x- and y-direction but not the "depth", i.e. it does not change the amount of feature maps
Final inception module
So an additional measure was taken to reduce the amount of feature maps
- before 3x3 convolution (because this is expensive with high amounts of feature maps) apply a 1x1 convolution
- before 5x5 (for the same reason) apply a 1x1 convolution
- after 3x3 max pooling (because we want pooling to effectively reduce the size of the weight matrix) apply a 1x1 convolution
Why do 1x1 convolutions reduce dimensionality?
- Input: NxNxM feature map (so with M channels)
- Filter: apply 1x1xM filter
- Output is a NxN feature map
- This means that if you apply K 1x1xM filters, your output will be NxNxK!
- Effectively allowing you to arbitrarily modulate the amount of channels in your feature maps
- And thus modulate the computational cost of the inception module
Q: Name several issues with the basic idea of the inception module of GoogLeNet.
Q: Mention how dimensionality reduction in the inception module is achieved.
The final architecture has many layers, with about 9 of these "inception modules". Various versions of the "Inception network" are used in medical imaging.
Extra classifications
GoogLeNet applied another trick. At two intermediate moments in the architecture intermediate predictions were made. These intermediate gradients were then also used during backpropagation, the idea being to stimulate discrimination in earlier layers as well. This trick was applied because they noticed that in this deep architecture it was hard to effectively backpropagate gradients through all layers. (No ResNets back then).
Average pooling
Another innovation was the usage of average pooling to replace the fully-connected layers that were typical at the end of the network. This "squeezes" and flattens the input. In case of GoogLeNet, from 7x7x1024 to 1x1x1024 with 7x7/1 average pooling.
What's the obvious benefit of this approach?
FC layers are very expensive as they have a huge amount of parameters. Average pooling instead uses zero parameters.
Q: What is a clear advantage of using average pooling to replace fully-connected layers?
How to go deeper? He et al. 2015
Experiments in "blindly" increasing network depth showed that just adding more layers in fact increases both train- and test error.
If you have ... -> x -> y -> z, with dz/dy and dy/dz, the further you backpropagate the smaller the gradients become, making it very hard to train earlier layers.
ResNet 2015
The gradients could not "flow" back to the input signal. To facilitate this flow, residual nets introduced skip connections.
Skip connections make it easy to learn the identity function so that even if a block does not learn anything (e.g. due to vanished gradients ), this does not hurt optimization in other layers of the network.
Typical block looks like:
- x -> BN -> ReLu -> weight -> BN -> ReLU -> weight -> addition
- where parallel you have x -> addition
Using residual connections, the error of deeper networks was suddenly lower than that of more shallow networks (slides contain image of comparing 18 with 34 layers). Increase in performance was shown even still when increasing the amount of layers to +100. But at some point this advantage stopped. Resnet-1001 showed that the error increased compared to ResNet-100.
ResNet 2016
A year later ResNet was slightly re-engineered and then indeed ResNet-1001 performed better.
TODO: unclear from the lecture what exactly changed.
ResNet is still very popular more or less in the form of the 2016 paper.
DenseNet 2016, Huang et al. 2016
Inspired by skip connections, but took this idea to the limit. Do not only shortcut the feature maps between convolution blocks, but introduce skip connections coming in from all previous layers, including the input signal.
ResNet takes output of convolutional layer and does a element-wise feature summation with the output from the previous feature map.
DenseNet's connectivity is described as "feature concatenation" in the 2016 paper. Think back to inception: this will lead to many features. DenseNet therefore also uses K 1x1 convolutions to control the amount of channels per layer.
The claims are that this is beneficial because:
- Strong gradient flow: it's way easier to get gradients from deep layers
- Computationally efficient: due to controlling the amount of K channels with 1x1 convolutions; in DenseNet apparently K can be quite low.
- Low complexity features: final classifier sees features from all layers at all granularities of representation
- So also simple features are leveraged due to the classification layer also being connected to input and early layers
Using pre-trained architectures
All these networks are available (trained on ImageNet), which is helpful when you have limited training data.
You can use the pre-trained network as a feature extractor and then retrain a new classifier on the extracted features, using your limited training data.
When a medium-sized dataset is available, you can freeze a selection of layers and fine-tune the others.