Fully convolutional nets

Conv nets can also be used for segmentation tasks.

The basic idea is to classify for each pixel to which class it belongs (e.g. artery vs. background) and often also to group different objects together with the same class label.

Classification vs segmentation

  • Very low resolution output vs. full-resolution output
  • output is one value vs. output is a 2D map
  • Global prediction vs. local prediction
  • Image level vs. pixel level

ConvNets for segmentation

ConvNets can classify patches and label the central pixel. To classify all pixels, you'll have to slide the patches over the image, i.e. convolve. But it's easy to see that this method is

  • very inefficient, because we do an amount of convolutions for classifying a patch, but then repeat almost all of those convolutions if we shift the patch by a single pixel
  • and very slow, because you have an enormous amount of classification steps per image, namely one for each image pixel,

So how can we use ConvNets efficiently? We need to "re-think" fully connected laters.

ConvNets work so well due to translation invariance of the convolution blocks, that use the following main ingredients:

  • Convolution
  • Pooling
  • Activation functions

Q The lecture slides stress that these do not depend on the input size. Why is that a relevant remark here? It's fairly obvious that they produce some activation in a feature map independent of the input size; but why is this an important point here?

When you have a ConvNet with these type of blocks and then dense layers at the end for classifying, the ConvNet overal computes a very complex function, e.g. the likelihood of being a cat. But if we strip the dense layers away, we do something similar but instead end up with a deep filter, i.e. a sequencing of filters that ultimately compute a single very complex filter or "translation" function (because the output will be also some sort of image instead of a class label).

Such a deep filter is also called a fully-convolutional network.

Q: Explain translation invariance

Replacing dense layers with convolutions

One of the bottlenecks making typical ConvNets costly are the dense layers (is this the main point?)

A fully connected layer has fixed dimensions and throws away spatial coordinates by flattening everything and making a vector. So normally we have a dense layer with K neurons, where each neuron is fully connected to each feature map produced by the previous layer.

So when the previous convolutional layer produces WxHxC, then each neuron of the K neurons has WxHxC parameters (+ bias parameters).

The output of a fully connected layer is flat, so 1xK. So the dense layer has WxHxCxK parameters (again leaving out bias parameters for convenience).

We can re-conceptualize the 1xK output of the dense layer as a 1x1xK activation map. How do we get this using convolutions? The input was WxHxC, so if we apply a filter of WxHxC, we'll only get a single response. So if we use K WxHxC filters, we'll get K single outputs.

So using the dense layer as well as the fully convolutional approach, we had WxHxCxK parameters: the amount of parameters for both approaches stays the same!

So what's the benefit? We got rid of "hardwired" connections between the dense layer and the previous convolution layer. With larger inputs, the previous network has too little weights.

The fully convolutional approach just requires your filter to be WxHxCxK, which doesn't make any assumptions about the input image size in the sense that it keeps on working for bigger inputs. It's just that in that case the output will not be 1x1xK but larger.

Q: What happens when you apply a fully-convolutional net on bigger input images?

Example: if you train a fully convolutional net on patches (e.g. of cats) such that the output is 1xK, and then apply this on a larger input size with pictures containing a cat somewhere, then the deep filter will produce a sort of heatmap (so obviously no longer the 1D output since the input image was larger).

Training with patches

The previous means that we can train a network with patches, and then still be able to use it on a larger input image!

Pros:

  • We do not need "dense annotiations" of all pixels
  • When using small patches we need less memory and training is faster
  • We can use patches from different sources
    • Increase diversity
    • Faster convergence
  • We can do patch sampling to compensate for class imbalance
  • We can sample difficult to classify locations/patches more often to let the network focus on problematic cases, e.g. to boost recall

Cons:

  • Using patches, borders between objects are not explicitly defined
    • So this method will probably not do well on borders
    • since you usually only give a single label per patch
      • But notice that U-net uses dense annotations
  • Sometimes we can only use a label for the central pixel
    • Isn't this the same point as the previous bullet point?
  • If you train a network on patches and then use it for segmenting larger input images, the output will be an activation map that in typically has a lower resolution than the input image
    • This is mainly due to the pooling layers

Countering lower resolution after training on patches

Lowering of resolution is mainly due to the pooling.

In general ConvNets pooling is very good because it increases the receptive field of neurons in the next convolutional layer.

But when using ConvNets for segmentation specifically, the con of pooling is that is reduces the size of the feature maps, leading to lower resolution output activation/feature maps.

To pool or not to pool?

Strategies to get back to full resolution

1) Combine multiple low-resolution results * "Shift and stitch" * Repeat process on shifted inputs * Interlace resulting images * Not used anymore in practice, because it requires multiple forward passes instead of a single forward pass 2) Avoid low resolution in the first place by removing pooling * This leads to extremely slow performance * Because you barely reduce the size of input maps, you have to keep doing convolution on feature maps practically the size of the input image * This means that each neuron in the output activation map has a very small effective receptive field. * Due to this, performance is low because pixel predictions do not effectively use context i.e. they are too local 3) Upscaling feature maps * Interpolation strategies (kNN, linear, etc.) * Learning de-convolution filters