Insights from assignments

A1: Basic Techniques

Histogram matching is used in this assignment for:

  • Enhancing relevant information in raw image data (with a target processed example)
  • Cancel out unwanted variability in images

Breast tomography

First:

  • Logaritmic transformation
  • Intensity inversion

Then options:

  • Contrast stretching
    • Check histogram to determine p0 and pk
  • Histogram matching
    • Returns a LUT: look-up-table that maps each input intensity to an output intensity such that the distribution of the target is mimicked.

Stain normalization with histogram matching

  • You do histogram matching per channel
  • Always normalize input

Trachea detection

  • Blob detector: Gaussian 2D kernel
  • Pay attention of the pixel/voxel size!
    • Also depending on whether you define sigma in pixels or millimiters
  • LoG: Lacplacian of Gaussian
  • Convolve image with LoG
  • Very slow with large filters!
  • Review in notes: convoluation with LoG in frequency domain is way faster because it reduces to a simpler operation.
  • You can combine several filters: take max response, average response etc.

A2: Segmentation using filters

Feature extraction

Define a feature extractor based on several Gaussian kernels that are applied at certain pixel locations.

  • Use different scales
  • Derivatives, at different scales and angles
  • Second derivatives

Creating and preprocessing training data:

  • Feature vectors per pixel
  • Define for each pixel if it's foreground or background
  • Taking all pixels is probably to slow, so pixel n samples per class to create a balanced training set
  • Normalize the feature vectors feature-wise

Classification:

  • Slides use kNN, which is nice during training because it's stateless, but is very expensive during testing because distances to all other points in feature space need to be computed

Post-processing

  • Thresholding
  • Erosion
  • Opening
  • Closing

A3: Nodule detection

  • Axial, coronal, sagittal
    • Axial: "flat" horizontal slice
    • Corona: tussen je oren! : - D
    • Sagittal: from nose to back
  • Waarom leveren axial slices een beter beeld op?
    • We wisten dit niet echt.

Create Validation Set

  • In this assignment we had imbalanced data
  • Take a 30% split for validation data
  • But to ensure balanced validation data, take 30% of the class with least samples

Training data

  • Why do we shuffle the training data when using minibatch gradient descent?
    • In mini-batch gradient descent the "step" is based on the gradients over the current batch of examples. If you shuffle your dataset, each next batch of examples will look differently in terms of class distribution w.r.t. to the previous one. This helps gradient descent getting "unstuck" from local minima.

Classification

kNN:

  • First try a kNN classifier again
  • Does very badly because it always predicts the majority class!
  • Note that if we didn't have balanced validation data, we'd have a very high accuracy score (which would be misleading)

Fully-connected network:

  • Still uses image features that were provided (unknown what they are)
  • Categorical cross-entropy
  • SGD
  • lr = 0.001
  • metric accuracy

Still pretty sensitive to the majority class! Changes:

  • More layers, early stopping, dropout, L2 regularization
  • We weighted our classes in our loss function to make the minority class weigh more during training.

  • We tried a convnet on the raw data but something was fucked.

A4: CIFAR10 image classification

Standard stuff.

A5: Fully convolutional nets

  • Extract patches as training samples on-the-fly
    • So also take matching label patches
  • Now we can also apply data augmentation on the fly
    • This is nice, because we don't have to recreate a whole static training set for trying out new augmentations
  • Sample such that half of samples are foreground and half background (but within the fundus mask!)
  • Classifying individual pixels with an NN (see A3) is infeasible for segmenting full larger images, so dedicated segmentation techniques are needed.

A trick to obtain a label for all pixels efficiently is to define a network architecture that does not include a fully connected (or dense) part. As the network only contains convolutional filters, the spatial structure is kept intact. Therefore the whole network can be thought of as a single large convolutional filter that can be applied to the whole image at once. This speeds up the classification and allows full image segmentation in milliseconds!

Labels:

  • One-hot encoded
  • Before we only labeled a central pixel of a patch (for classification purposes)
  • Our fully-convolutional net still does! Why?
  • But because now we take the fully convolutional approach, we can train on all labels simultaneously, like U-net does!

We did we still use patches anyways instead of full image input?

  • Doesn't require all pixels in the whole image to be labeled (so useful in more contexts)
  • Less memory needed, faster training
  • On-the-fly augmentations
  • Allows smarter sampling strategies

Fully convolutional

  • In the assignment we had to indicate how much our output is downsampled w.r.t. input; we do pooling in our fully convolutional network
  • so to compare input and output we need to compensate for the difference in scale by:
    • downscaling the label maps to the output size
    • downscaling the masks as well
  • A typical measure for segmentation is the Dice score, which is the size of the overlap of the two segmentations divided by the total size of the two objects
    • REVIEW DICE
  • Accuracy is not a good measure for segmentation
    • does not factor in true negatives, which is also relevant for a good segmentation
    • bad for imbalanced data

Architecture that works with input (25,25) during training.

  • input layer 25x25
  • 32 filters of 4x4 22x22 x32
  • 32 filters of 3x3 20x20 x32
  • pooling (2x2) 10x10 x32
  • 64 filters of 3x3 8x8 x64
  • 64 filters of 3x3 6x6 x64
  • pooling (2x2) 3x3 x64
  • 128 filters of 3x3 1x1 x 128
  • 64 filters of 1x1 1x1 x 64
  • 2 filters of 1x1 1x1 x 2

How do we get the final segmentation? We can throw in the whole image, right? Then the output will be bigger but still two feature maps: one background, one foreground. Guess we take argmax then to determine the value for each pixel.

If we now apply this to the original input images, what will the output be?

Input: 584 x 565 Output: 140 x 136

So the output is about 4 times smaller. As you can see, we are better of padding the input image first with a multiple of the patch size.

Fully convolutional with dilation

Our patch size is quite small and ideally the filter we learn takes a larger context from the input into account. This can be achieved while avoiding the downsampling by replacing pooling with dilated convolution.

  • input layer
  • 32 filters of 3x3
  • 32 filters of 3x3
  • 64 filters of 3x3, dilation rate 2
  • 64 filters of 3x3, dilation rate 4
  • 128 filters of 3x3, dilation rate 8
  • 64 filters of 1x1
  • 2 filters of 1x1

Now we can use input (33, 33)

The convolutions reduce to (25,25)? But receptive field is?

TODO:

Dilation: f + f x r?

U-net

  • Use fully annotated patches instead
  • This means we do not care about balancing the central pixel class anymore
  • We use 'same' convolutions in this variant, so output should have the same size as input
  • The only problem is if the input size is not divisible by the maximum downsampling rate
    • Q: Why?

Input patch size 88 x 88: