U-Net

This is currently the state-of-the-art network for image segmentation. Using U-Net, we can predict full resolution maps.

Loss function with full resolution segmentation

When our output is full resolution, we can compare individual pixels!

Before we had a single label per patch. Now each pixel counts as one sample (i.e. each pixel is labeled) so we can compute the loss per pixel. So the final cost is the average of losses over each pixel.

  • Cross-entropy loss
  • Log-loss: -1/#pixels $\sum log(y_{pred} * y_{target})$
  • Diced score
  • ...

Architecture

  • Like a fully-convolutional net, we first have a contracting path were the input is downsampled.
  • Then we have an expanding path where we up-sample again
  • There are skip-connections between the similarly structured components of the contracting and expanding path.
    • These are not the same as in ResNets! They are in fact concatenations.

The contracting path is very similar to VGG:

  • two identical "valid" convolution layers
  • 2x2 max pooling while doubling the amount of filters

The expanding path:

  • counters the max pooling with 2x2 up-convolution
  • applies the same convolution, i.e. 3x3 with ReLu

Concatenation:

  • The expanding path "reconstructs" the corresponding feature maps from the contracting path (so after a bottleneck)
  • The original feature map is concatenated to the reconstruction

Output:

  • To end up with a segmentation map, use 1x1xC convolutions
  • For classifying foreground and background, end with 2 of those 1x1xC convolutions

Properties

  • Because every pixel can be used for training, U-net works well with little training data (since each image produces so many samples)
  • U-net is fast! 512x512 segmentation under <1 sec GPU (in 2015)
  • Heavy data augmentation was used in the original paper
    • Elastic deformation

Trick for instance separation: weights map

Trick: add a pixel-wise loss weight to the loss to get really clear separation between segmented objects (instance segmentation). You can visualize these weights as a map where pixels on borders are manually annotated with very high weights.

So as a loss function they use categorical cross-entropy but now at the pixel level.

$$ E = \sum_{x \in \Omega} w(x) log(p_{l(x)}(x))$$

Why subscript $l_{(x)}$? Is the right hand side here a shorthand for the cross-entropy? I think it is a weird formulation

$$w(x) = w_c(x) + w_0 \dot exp( - \frac{( d_1(x) + d_2(x) )^2}{2 \sigma^2})$$

Where $w_c$ (c for class) is class-balancing, i.e. the minority class will have a higher weight to make it weight more in the loss computation.

The RHS here is a decreasing exponential function that starts from $w_0$. $d(x)$ is the distance from x to an edge (two distance terms for x- and y-direction) So '0' because it's an intercept. $w_0$ determines how large the weight is when the pixel separates two instances, because then the distances will be closer to 0, so the exponential will be closer to 1, resulting in $w_0 x 1 = w_0$.

$\sigma$ is a hyperparameter that "dampens" the descent of the function.

3D Convolutions

A 3D version of U-net was also developed. If you have 3D input and apply a 3D filter, you also get a single output, just like in the 2D case.

3D convolutions are relevant for the various types of 3D image data in medical imaging:

  • CT
  • MRI
  • OCT: Optical Coherence Tomography

3D feature maps

If we apply a filter of size HxW to a 2D image, we end up with (H,W,F) shaped output. If we have 3D data and apply a 3D filter, we get (H,W, D, F) output. We cannot visualize this anymore!

3D U-net

Network:

  • 3x3x3 valid convolutions
  • 2x2x2 pooling
  • 2x2x2 up-convolution
  • 1x1x1 convolutions at the end

Input:

  • 132 x 132 x 116
  • 3 channels (markers)

Output:

  • 44 x 44 x 28
  • 3 labels

Details:

  • Because you cannot really densely annotate all pixels in 3D volumes, the used sparse annotations
  • i.e. they just labels several 2D slices
  • input was only 3 images (but in 3D, so a lot of voxels)!
  • So they heavily relied on data augmentation

Surprisingly (at least to me, this still resulted on very good 3D dense segmentation.)

Improvements to U-Net in 2015-2019

In the original architecture, they start with input 572 (unconventional!) and ended with 388 (unconventional!). Why?

They selected the input size such that before each pooling operation the feature map size was even. This allows "a seamless tiling of the output segmentation map". I.e. if you pool an uneven layer, you throw a tiny bit of spatial information away (i.e. some activations in the feature map), which can misalign the original feature maps and the reconstructed feature maps.

During the concatenation step you then do not correctly concatenate the corresponding regions. This will create artefacts in the output. Note that we use pixel-wise loss, so this one pixel misalignment does matter!

Cropping

If you want to crop paths from the concatenating path to the expanding path, you need both to be even otherwise the alignment will be tricky.

Q: Why do we have to crop in the first place?

I'm not sure, but it does allow flexibility in the architecture, because then you can for example crop a feature map of size 136 from the contracting path to 104 in the expanding path since 136-104=32.

Actually, I didn't notice before, but despite the up-convolutions, the feature maps on the expanding path are smaller. So what you do is you take the center of the corresponding feature map in the contracting path and concatenate it. This is the step that is trivial for even feature maps, but where you have to make a choice for uneven ones.

Using same convolutions instead of valid

The original authors didn't use 'same' convolutions, because they wanted to avoid artefacts.

But with 'same' convolutions, we don't have to pick such a weird input size, which really simplifies the architecture. Also, concatenating is simple then.

Up-convolution

So called "up-convolution" consists of:

  1. Upsampling (possibly with nearest neighbour interpolation)
  2. Followed by 2x2 convolution which halves the number of feature channels.

The second step is actually really important, because this cancels out the concatenation of feature maps that is so characteristic of U-net. So, example:

  • On the contracting path, you may have 512 filters 64x64.
  • You go down to 1024 feature maps of size 32x32 with 2x2 pooling
  • Regular convolutions, 32 -> 30 -> 28 (still 1024 feature maps)
  • Now we do up-convolution and get 512 56x56 feature maps
    • So amount of channels halved!
    • And size of feature maps doubled due to up-sampling
  • The halving of the amount of feature maps (from 1024 to 512) is important because now we concatenate the 512 64x64 filters from the contracting path! (the ones we started with at 1), i.e. we are right back at 1024 feature maps!
  • We "combine" the "reconstructed" feature map and with higher resolution by applying regular convolutions and now again halving the amount of feature maps.

Improving the upsampling

If you do the nearest neigbour upsampling, you essentially repeat several activation values until you get the right shape. This of course does not lead to a very smooth activation pattern, so a first improvement is to do a "same" convolution afterwards to smoothen the feature maps out.

Replace pooling and up-convolution with bilinear interpolation

Another variant replaces the pooling and up-convolution with bilinear interpolation. This produces output with less artefacts.

Because it's linear you can easily do backpropagation. Also, it still doesn't have any parameters, so nice and efficient.

Replace "bottom" convolutions with dense layers

Dense layers in the bottom part maximize the field of view of the network. (Note that this does assume that you use a specific input size for training and testing).

What's the rationale here? Well, by using a dense layer in the transition from the contracting to the expanding path, you make sure that each feature at the start of the expanding path has the largest possible receptive field w.r.t. to the previous layer, and as a result has the whole input image as its receptive field!