TIL project

Notes from introduction video

TIL score: Estimate percentage of stroma (connective tissue) covered by lymphocytes.

Breast Cancer segmentation challenge Track 1:

  • 148 slides
  • Multi-class dense annotations
    • Includes stroma and tumor, but also fat, necrotic ets.
    • So use some multi-class segmentation architecture, e.g. end-to-end segmentation network such as Mask-RCNN
  • Test set of 46 whole-slide images
    • Tumor (bulk) segmentation to whole slide
    • Stroma segmentation in specific ROI
    • Dice score evaluation
    • To be determined whether there is a single leaderboard for tumor and stroma detection, or two separate ones.

Guidelines

Guideline for standardized evaluation of TILs in breast cancer

  1. Define the area for TIL evaluation
    • TIL within borders of invasive tumors (edge included, but not reported separately)
    • Exclude areas with necrosis and fibrosis
  2. Focus on stromal TILs
    • Stromal area
  3. Determine type of infiltrate
    • Exclude immune cells in necrotic areas
    • Include only mononucleides , lympohics and plasma cells
  4. Place tumor in one of three groups and assess percentage stromal TILS
    • A) no/minimul cells, up to 10% TILS
    • B) intermediate/heterogeneous infiltarte, 10-40%s
    • C) High immune infiltrate, 40-90%
    • Area of stromal tissue used for percentage calculation, not the amount of cells
  5. Report percentage of TILs (on average, don't focus on hotspots)

Introduction presentation

Postdoc at Radboud.

Segmentation exercise. This project is part of his own project. He already did some work on this dataset.

TILs is a nice biomarker that correlates with survival rates.

In his project, Witali is trying to predict response to chemotherapy based on TILs. If there's no predicted response, we can skip chemo and go straight to surgery.

Density of lympocytes.

Lymphocytes are usually small and roundish.

  • Tumor
  • Lymphocytes
  • Stroma
  • Fat
  • Necrosis

Minimal goal:

  • Predict stroma+lymphocytes vs. rest
    • The lymphocytes are always in the stroma?
    • So we do not segment stroma without lymphocytes?
    • This shouldn't be too difficult
  • We are in the segmentation track
    • The other track predicts nuclei types
  • Later tracks can be combined to determined TIL score/percentage.

Would be interesting to find lymphocytes in the stroma. But his network already predicts this project goal. (Unet network)

To make it more interesting: also segment the other classes in "rest", i.e. try to replicate the results from the paper, i.e. segment everything apart.

We have regions with dense annotations. They are annoying to make, so most annotation is sparse.

You will never really find a pixel-perfect score, but Witali thinks that 0.95 would be the max, or 0.9 on the diagonal in the confusion matrix. Depends on the data.

There's dense annotation for nuclei in some regions. For segmentation, everything is... segmentated?

In the shared paper, there's a problem because annotations were overlapping. And there was no clear hierarchy, so unclear which one should be used.

So a first task is also to check whether the annotations are still overlapping. Is the data still the same as the original data from paper? (E.g. is the dataset of the file size exactly the same? Would be bad)

We can choose to do 5 or 6 classes. But it never hurts to have more classes (given enough training data for the classes), since you can put classes together in the end. So have a look at the distribution of classes! Also useful for class weighting.

Balanced patch sampling based on central pixel (cf. exercise) is ... not possible? Bla bla, for some reason only class based weighting possible.

TODOs:

  • Explore data, data distribution (same as two years ago? Otherwise use Witali's data set)
  • Overlapping annotations
  • Create pipeline (rather script than notebook!)
    • Pre-extract patches instead of whole TIFFs (128x128)
    • Convert notebooks to scripts
      • Cartesius has a time limit on interactive sessions
      • So for experiments script is better
    • patch sampling
    • augmentation (find good transforms and thresholds)
      • Very important in this task! Look for papers
      • David Ciompi Mitosis: paper on augmentation settings
      • Rotation, contrast, color augmentation to fix problem below
      • Problem: different scanners and scanner settings, as well as data variability, lead to slightly different colorings!
    • training
    • evaluation (acc, f1, confusion matrix, training progress)
    • unet++, class weighting

Tools:

  • asap for visualization (but problem using the library on Cartesius, so you can't use this on Cartesius)
    • the group doesn't have time to fix this issue
  • Reading slides: openslide, pyvips
    • Snippets for reading in TIFFs (openslide-based)
    • 0.5 micrometer on a pixel is highest possible zoom; spacing param?
    • spacing of 0.5 or 1 should do. 0.5 is default choice.
    • at 0.5 spacing, min. patch size should be ... ?
    • TODO play around with patch size
    • TIFFs are HUGE so it's best to have code to only load the relevant patches / ROIs
  • Create GitHub repo
    • Create github repo and add Witali so he can commit snippets
  • wandb for tracking experiments
    • Can also be done locally

Group usually used valid padding. Also relevant for determining patch size.

With 4 or 5 max pools you should see enough context.

For now we can locally start with the data exploration etc. until we have access to Cartesius.

  • Doodle

  • Q: is TIL it's own class? (part of inflammatory infiltrates?)

Meeting 2

Questions:

  • Can we make the repo public? If no, why not?
    • Witali will ask Francesco
  • Why is dataset only 6Gb when a single slide can already be 60Gb?
    • The new data set only has the ROI; since we don't have ground truth anyways outside ROI
    • small margin on edges are preserved to avoid artefacts
    • usually we have sparse annotations, but we have the luxury of dense annotations
  • Info about libraries? Connection pyvips and openslide. What's their connection? Downside to using Tifffiles? Zar.
    • pyvips can't read batch by batch, apparently
    • openslide: main advantage is that openslide allows grabbing of patches without having to load the whole tiff image (e.g. of 60Gb size) in memory
    • easiest: use pyvips to read whole image into memory, extract patch from relatively small image on the fly
    • tifffile is also fine: whatever works.

Thijs question about architecture:

  • Architecture & Multi-task pathology paper: Can we use DenseNet with a segmentation head? Or would you recommend Unet?
    • VGG-16 used in original paper, we are not
    • Witali: generally speaking training from scratch with Unet should give good results.
    • Because we have densely annotated tissue and thus plenty of training data, the benefit of pretraining is limited in terms of performance.
  • How to transfer DenseNet encoder to Unet architecture?
    • fast.ai PyTorch has a dedicated function to turn any encoder into a Unet!
  • Problem GPU on Cartesius is very old!
    • PyTorch 1.2 works (pip installable)
    • Or build PyTorch 1.7 yourself from scratch; but is laborious
    • Using docker files doesn't really work unfortunately
    • Witali is pushing instructions for building from scratch
  • There may still be a problem with the spacing of the train set we have now: slides supposed to be extracted at 0.5, but it seems spacing still at 0.25 for some slides.
    • Convenient of openslide is that you can extract at a particular spacing
    • TODO: Can we also do this with tifffile etc.?
  • Witali recommended albumenations for data augmentation, but we are already using it :-D.
    • Thijs is doing an awesome demo
  • TODO patch balancing
    • Q: what's the best approach here?
    • Witali: in segmentation it's hard to do perfect balancing when using the center pixel label
    • Witali: combine with class weighting in loss function works well enough; and DL deals with imbalance a bit more naturally anyways than traditional ML, so maybe you don't even need both.
      • TODO this can be a small experiment
  • Thijs: what's the best option for patch sampling in terms of speed and memory?
    • 1) pre-extract patches and save them in memory
    • 2) extract patches randomly at runtime
    • Witali: Problem with 1 is that the network sees the same patches, the benefit of 2 is to introduce more variety in training data
    • So keep 2, unless you run into problems
    • num_patches now determines size of epoch in our current code (verify)
    • We can probably increase it to way higher than 500 ; but what's most important is to correctly combine it with learning decay, which is defined on epochs. So make sure that this fits with the chosen size of the epoch.
    • PyTorch lightning should have some inbuild method for this ... missed that? @Thijs do you recall this?
  • Thijs: what's the recommended patch size?
    • For Unet, it doesn't really matter
    • Always use fully convolutional network, because at runtime you don't necessarily know the input patch size.
    • Two options: patches or whole ROI as input.
    • Rule of thumb for patch approach, not smaller than 128.
    • For ROI input: we should be able to fit in the whole ROI as long as its < 2000;
      • if it doesn't you have to divide up the input and combine back the outputs, which is not ideal.
    • But KISS (Keep it Simple Stupid!), try first with same padding
  • Isabel: do we have a pre-trained net?
    • Witali: we don't know of a suitable one.
    • Most will be pre-trained on ImageNet and not on medical data, so dubitable whether it's very useful
    • TODO experiment whether pretrained on ImageNet works well
      • Witali: "one small interesting experiment here would be to compare unet from scratch, unet with imagenet and unet with the pretrained decoder in that paper. surpisingly, imagenet-pretrained networks work quite well for medical images (despite having been trained on very different data)"
    • Is it a problem that pretrained nets don't use dilated convolutions?
      • Witali doesn't like them anyways because they result in artefacts

TODO:

  • Document your code!
  • List sources if you take code from elsewhere
  • Choose LICENSE?
    • Previous point also has influence on which LICENSE we choose
  • TODO implement patch balancing
  • TODO implement class weighting in loss function
  • +experiment compare patch balancing with class weighting and patch balancing+class weighting
  • +experiment Compare Unet vs. pretrained DenseNet + segmentation
  • +experiment Find pre-trained Unet and compare with Unet trained from scratch, or possible also with unet with pretrained decoder (e.g. DenseNet)
  • +experiment trying different patch sizes or possibly even take whole ROI as input (not recommended)
  • TODO use fast.ai for converting encoders to Unet
  • TODO does our stuff run on pytorch 1.2
  • TODO Play around with augmentations
  • TODO

  • pip -r requirements.txt >> README.md