Receptive Fields

The receptive field of a neuron is the area of the input image that is seen by that neuron in a layer of the network.

If you do a 3x3 convolution, what's the receptive field of a neuron in the next convolutional layer?

A: 3x3

If you add a second convolutional layer with the same filter size, what's then the receptive field of a neuron in that layer?

A: 5x5

Main insight: a main effect of convolution is that the receptive field at deeper layers increases!

This is also why generally you want to increase the amount of channels (i.e. the amount of feature maps) when the network becomes deeper. Because the receptive field increases, the more complex features are there to be learned. Whereas usually we decrease the amount of neurons in dense layers to create bottlenecks, with convolution layers we don't because the complexity of the features increases (i.e. they are combinations of combinations of features etc.).

Prefer repeated smaller convolutions over a single large one

But if the receptive field of two applications of 3x3 is equal to one time 5x5, why do we not just apply 5x5?

Convolution operations are expensive and larger filters require more parameters than two applications with a smaller filter, despite having the same receptive field.

  • (5x5): 5x5+1 = 26 parameters
  • 2x(3x3): 2x(3x3+1) = 20 parameters

Max Pooling

The main effect of max pooling is to reduce the data size at subsequent layers, which also means doubling the receptive field (with (2x2)-pooling)! So in this case you increase the receptive field through a form of downsampling.

The downside of pooling is that you

  • Decrease resolution
  • Lose some spatial information

Why do we care about receptive fields

We want a single prediction in the output to be informed by a sufficiently large input region. This means that the network will involve contextual information for each prediction.

This is very important for getting robust predictions, for example when you are classifying single pixels during segmentation.

Effective receptive field

Research has shown that what is theoretically the receptive field, i.e. according to reasoning about filter sizes and pooling etc., is not always the effective receptive field. It turns out that the effective receptive field generally follows a Gaussian distribution, which is only a fraction of the theoretical receptive field.

The effective receptive fields depends on several factors, including:

  • Used initialization strategy
  • Types of used non-linearities
  • Type of layers used

Other

See CNN convolutional neural network for remarks on the output sizes of convolution operations.