CNN Network Parameters
You should be able to compute how many parameters are used in convolutional neural networks.
This is good to understand the capacity of the network, i.e. it's ability to fit (and thus overfit). This in turn is good to understand the degree of regularization that is needed. It is also good to understand and tweak the memory footprint of the network.
- Input: 32x32x3
- 10 classes
- valid convolutions
Compute the parameters for the following network:
- 3x3x3 filter, with 32 filters
- Params: 3x3x3 x 32 + 32 = 896
- Output Shape: 30x30x32
- 2x2 pooling
- Params: 0
- Output Shape: 15x15x32
- 3x3x32, 64
- Params: 3x3x32 x 64 + 64 = 18.496
- Output Shape: 13x13x64
- 2x2 pooling
- Params: 0
- Output Shape: 6x6x64
- 3x3x64, 128
- Params: 3x3x64x128 + 128 = 73.856
- Output Shape: 4x4x128
- 2x2 Pooling
- Params: 0
- Output Shape: 2x2x128
- FC, 256
- Params: 2x2x218 x 256 + 256 = 131.328
- Output Shape: 1x256
- Softmax:
- Params: 10 x 256 + 10 = 2570
- Output Shape: 1x10
Total parameters: 227246
I initially made a mistake in how bias is applied on multi-channel filter. You have a bias term per filter, where a multi-channel filter (i.e. stacked filters) count as a single filter.
You see that even in such a simple network, the parameter explosion really happens in the dense layers. Even with a single layer in the above example, over half of the parameters is in that single dense layer.
- LeNet (1998) had 60.000 parameters.
- AlexNet (2012) had 60.522.856 parameters.
- Modern networks have an insane amount of params.
Visualising Learned filters
AlexNet's learned filters in early layers look like Gabor filters. But filters at deeper layers are very hard to interpret.
There is another technique to visualize filters.
You can compute the gradient for each filter w.r.t. the input, which will show you to which input type the filter will respond maximally.
Nice example in Zeiler and Fergus, 2013. Due to feature hierarchy, you see dog faces pop up in later layers.