Upscaling feature maps

A fully convolutional net without pooling is very inefficient and has too low receptive fields.

Instead, we can counter the downsampling of pooling layers by upsampling again, bit like a encoder-decoder network with bottleneck layers.

There have been several proposals for up-sampling.

Later architecture added skip connections between the encoder and decoder part of this network. See Unet (next section).

Deconvolution network

Instead of convolution layers we introduce deconvolution layers where we learn the filters to do up-sampling. Additionally, to do the reverse of pooling we can do unpooling which generates a sparse feature map.

Max unpooling

To do this operation, we have to remember the positions of the max elements in the pooling operation. Normally you don't do this, so this is good to realize.

During unpooling, you take the activation map and then propagate the activation values to the locations of the max values during pooling. So note that you just take the activations of the previous layer, and not the values from the original pooling. Fill the rest with zeros, so the resulting feature map is very sparse. From the original pooling you only use the locations of the max values.

Up-/De-/Transposed convolution

Consider a typical convolution with a 3x3 filter on 4x4 input.

Stride determines ratio inputsize / outputsize.

E.g. with 4x4 input and a 3x3 filter with pad 1 and stride 2, the output size is 4/2 in both dimensions -> 2x2.

Learn filters:

  1. You know the desired filter size because you want to undo the corresponding convolution operation in the encoder part of the network
  2. Randomly initialize the filters
  3. Multiple the input value with all filter coefficients
    • We now have as input value what would be the output value in a regular convolution
  4. Slide filter by stride
  5. If outputs overlap, sum them
  6. Take the part of the output where all filters overlap

In case of a deconvolution of stride 2 and pad 1 on 2x2 input, this will again result in 4x4 output.

Checkerboards

Deconvolution typically produces checkerboard artefacts.