ConvNets in practice: Tips & Tricks

Data augmentation

If your dataset is small, image augmentation is a good idea to both increase diversity in the training samples as well as increase the amount of samples:

  • Mirroring
  • Distorting / Blurring
  • Rotations
  • Color changes
  • etc.

Also see https://github.com/mdbloice/Augmentor

Also see https://github.com/albumentations-team/albumentations

Affine

Affine translations may modify distances and angles.

  • Translation
  • Rotation
  • Scaling
  • Shear
  • Zoom

Rigid

Rigid transformations instead preserve distances and angles. This is a subset of affine transformations.

  • Translation
  • Rotation

3D

For 3D transformations we use a 4x4 transformation matrix. The fourth dimension is for storing offset values.

Transformations can then be concatenated by chaining matrix multiplications.

Data balancing

Imbalanced datasets are problematic because:

  • The network usually minimizes the loss for the high-occurrence class because that's the quick 'n dirty thing to do.
  • However, the under-represented class is almost always absolutely the most important in medical imaging tasks

How to improve:

  • Sampling techniques; under- or oversampling
  • Cost sensitive techniques: heavily penalize errors on the minority class
  • One-class learning: only learn the minority class and combine results later.

Things to check during training

  • Check the learning curve
  • Filters should look "meaningful"
    • i.e. they should not mostly have zeros
  • Activations ibid.
    • You should not have too many zero activations
    • Compute histogram of activation and monitor it

Hands-on tricks

  • Normalize input data to make training easier
  • Use dropout
  • Use batch normalization (or a good alternative)
  • Use ReLU
  • Initialize with He15
  • Use ADAM (or RMSProp)
  • Use adaptive learning rate

To check your network is bug-free, always first overfit on a very small training set, e.g. 5 samples. You should get close to perfect performance. If not, you probably have a bug in your code, a problem with your labels, too few parameters, or something like that.

To get a baseline performance, compute accuracy on the validation set.