Convolutional Neural Networks

Discussion of the three terms: convolution, neural, network.

Also see my extensive related note on CNN_convolutional_neural_network.md

Convolution

Convolution of signal A and filter f:

$(f * g)[x,y] = \sum_{u=-M}^{M}\sum_{v=-M}^{M} A[x-u, y-v]f[u,v]$

Two typical options:

  • 'valid': only convolve the input signal; but then filter application of any filter larger than 1 will not use border pixels as the filter's anchor, thus reducing the output size
  • 'same': to keep the same size after convolution, you have to pad the input signal
    • The required amount of padding depends on the convolution filter size and stride

Padding

  • 'Same' padding: $(f-1) / 2$

Stride

  • decreases output size
  • if you also use 'same' convolution you end up going over a larger input area

Multi-channel filters

  • For each filter application you get one output channel
  • For multi-channel input (e.g. RGB image), "one" filter application will internally be the application of three filters, each to one channel, but still with a single response (by summing the individual responses)

For k input channels:

$$C(i,j) = \sum_{k=0}^{l} \sum_{u=-M}^{M}\sum_{v=-M}^{M} A_k[x-u, y-v]f_k[u,v]$$

A convolution layer is precisely that: a number of multi-channel filters, where the amount of filter channels depends on the amount of channels at the previous layer, and where the number of output channels depends on the amount of applied filters.

This means that

1) The filter depth is initially driven by the input image channels; later it is determined by the amount of channels in the previous layers 2) But you can control the amount of channels by choosing the amount of filter applications; so this is a design choice.

A single filter application can be written as $w^T x + b$. So don't forget the bias parameter!

Practical examples

  • Input: 32 x 32 x 3
  • Filter: 5 x 5 x 3

Q: What's the output with valid convolution without stride?

32 - (5-1) -> 28x28x1 activation map.

Neural

The output of a "neuron" is determined by the weights of the kernel filter, but also by the bias term which is not visible in the convolution operation as such.

So for filter size f=3, we need to train (3x3)+1=10 parameters for a single filter application.

Earlier in the course we also convolved filters over input. The main difference introduced by CNNs is that now we learn the best filters.

Convolution vs. dense layers

Why use convolutional layers instead of fully connected layers?

  1. Inefficient: Dense layer has a lot of weights especially for the first layer, which will be HUGE
    • A small 128x128 layer will already have 16384 x n + 1 parameters for n neurons in the dense layer.
    • So dense layers hugely increase the capacity of the network, giving a large risk of overfitting
    • To tune this very large amount weights without overfitting, you need a lot of training data
    • You also need a huge amount of memory to store all weights and gradient computations
  2. No robustness: because of the fixed 1-to-1 connectivity on the level of individual pixels there is no robustness against shifts in input, i.e. no translation invariance
  3. Does not take spatial ordering into account
    • But image data has a clear 2D topology

Convolution instead ...

  • takes the 2D topology into account because filters are 2D
  • does weight sharing, because the filter weights are convolved over the different regions of an input image (vs. separate regions having separate weights)
    • This expresses that whatever feature the filter captures may be present anywhere in the input image

Network

Typical CNN structure:

  1. Input
  2. Convolution
  3. Bias
  4. Non-linearity
  5. (Max) pooling
  6. Feature maps
  7. Repeat (...)
  8. Fully-connected layers
    • The convolution layers extract features
    • The FC layers use these features e.g. for classification

Pooling

  • Introduces additional translation invariance
    • The example in the slides shows nicely that the output of two pooling steps is identical if you shift the window over the input image by a single pixel
  • Reduces size of feature maps while keeping highest activations
  • Smaller feature maps leads to a better memory footprint
  • Increases receptive field

Also see pooling .

Non-linearity

Problem with older activation function is vanishing gradients. For tanh and sigmoid there is only a narrow range where the gradient is large, and it's practically 0 outside of that range. This means that after a certain point there is no sensitivity left for changes in input, i.e. there is saturation. This makes learning slower.

ReLu:

  • Avoid saturation
  • Avoid vanishing gradient
  • Makes learning faster (because no vanishing gradient)
  • Sparse representation, because it "kills" negative signals.

Variants:

  • PReLu additionally includes a slope for the negative input region and this slope can be learned
  • Weakly ReLu: does not kill negative inputs but has a fixed slope in negative part (i.e. this slope is not tuned!)