ConvNets in practice: Tips & Tricks
Data augmentation
If your dataset is small, image augmentation is a good idea to both increase diversity in the training samples as well as increase the amount of samples:
- Mirroring
- Distorting / Blurring
- Rotations
- Color changes
- etc.
Also see https://github.com/mdbloice/Augmentor
Also see https://github.com/albumentations-team/albumentations
Affine
Affine translations may modify distances and angles.
- Translation
- Rotation
- Scaling
- Shear
- Zoom
Rigid
Rigid transformations instead preserve distances and angles. This is a subset of affine transformations.
- Translation
- Rotation
3D
For 3D transformations we use a 4x4 transformation matrix. The fourth dimension is for storing offset values.
Transformations can then be concatenated by chaining matrix multiplications.
Data balancing
Imbalanced datasets are problematic because:
- The network usually minimizes the loss for the high-occurrence class because that's the quick 'n dirty thing to do.
- However, the under-represented class is almost always absolutely the most important in medical imaging tasks
How to improve:
- Sampling techniques; under- or oversampling
- Cost sensitive techniques: heavily penalize errors on the minority class
- One-class learning: only learn the minority class and combine results later.
Things to check during training
- Check the learning curve
- Filters should look "meaningful"
- i.e. they should not mostly have zeros
- Activations ibid.
- You should not have too many zero activations
- Compute histogram of activation and monitor it
Hands-on tricks
- Normalize input data to make training easier
- Use dropout
- Use batch normalization (or a good alternative)
- Use ReLU
- Initialize with He15
- Use ADAM (or RMSProp)
- Use adaptive learning rate
To check your network is bug-free, always first overfit on a very small training set, e.g. 5 samples. You should get close to perfect performance. If not, you probably have a bug in your code, a problem with your labels, too few parameters, or something like that.
To get a baseline performance, compute accuracy on the validation set.