Region Proposal Networks
Again, this is about using CNNs to perform the task of object detection.
R-CNN (2013)
- AlexNet showed that ConvNets are great for classification
- But can they also be used for detection?
- R-CNNs <- "Regions with CNN features"
Approach:
- Input training images with given bounding boxes (original paper took Pascal VOC challenge with 20 classes)
- Selective search algorithm produces about 2000 region proposals i.e. bounding boxes
- Use a downstream CNN classifier on each proposal
- Proposal is resized to required input for the CNN (i.e. it is warped)
- Then processed by AlexNet pre-trained on ImageNet, strip away the last dense layer with 1000 neurons (ImageNet has 1000 classes)
- The second to last layer of AlexNet then instead gives a vector of 4096 features
- Note that all these feature vectors need to be stored in memory because they are required for the classification step.
- Classify the 4096 feature vector with a SVM
- One SVM per class is trained! SVM is a binary classification method, so for multiclass classification you need to combine several binary classification results or train one SVM per class.
- So samples are organized per class
- As input positive and negative samples, you use IoU > 0.5 to determine which samples are positive.
Outputs:
- Label
- Bounding Box (each sample already was/had a bounding box from the proposal phase)
Pros:
- This is the first and creative method using ConvNet features for object detection
Cons:
The obvious downside is that this algorithm is very very slow.
- Region proposal is slow because you have to make so many proposals
- Using the ConvNet for feature extraction requires you to store many GB of feature data after forward passes
- Since each image easily produces ~2000 proposals, you have to do 2000 forward passes per image!
- You have to train a 20-class SVM classifier
- So essentially every aspect of this algorithm is quite slow.
Fast R-CNN (2015)
In R-CNN you first crop the proposal patch so it is the expected input size of AlexNet (or some other CNN). Fast R-CNN doesn't use AlexNet but VCC-16 (I thought).
Fast R-CNN instead just runs the CNN on the whole input image so you get a feature map as output.
The difference is that R-CNN has to do many forward passes per image, whereas Fast R-CNN only does a single forward pass. So R-CNN is also extremely slow not just during training but also during test time.
In R-CNN the bounding box proposals are defined on the input image though, so you need to project these bounding boxes onto the feature map.
Q: Is the region proposal ran on the input image and then projected onto feature maps? Or is it run directly on the feature maps themselves?
I guess this projection isn't too hard because there's a deterministic relation between input and output with convolution; but I'm not sure there is even an explicit projection since you can also define the proposal step directly on the acquired feature map. A: I think, actually, that the region proposal algorithm is directly run on the feature maps.
The downstream layers (e.g. FC layers) however do need inputs in the correct size. So what Fast R-CNN does instead is to "crop" the feature map (that is a projection of a proposal bounding box) by using a "ROI pooling layer", which pools only a subsection of the feature map.
Q: Will the output of the ROI pooling layer always be of the same size?
On the later FC layers, you define two outputs:
- A softmax classifier
- A bounding box regression that refines the output bounding box
- So is this necessary because before the bounding box size is fixed instead of random, as in the regular R-CNN? No doesn't look like it.
Pro:
- Everything is done in a single network with end-to-end training
- And you don't use an additional classifier like an SVM
- Way faster than R-CNN; for comparable performance reduction from 84 hours training to 8.75
Con:
- You still use the very slow region proposal (Aha!)
- In fact most of the computation is in the region proposal step; if you leave it out you have a significant speed-up at testing time.
Faster R-CNN (2015)
This network is yet faster because it strips away the external region proposal step and instead includes this step in the network itself.
I find this model very complex and the slides are only discussing this network on a very vague abstract level; not sufficient at least for me to understand what's going on.
So we have a single network for:
- Region proposal
- Classification
- Bounding box refinement
This is both faster and better performing than Fast R-CNN.
Components:
- Fully-convolutional network
- (New) Region Proposal Network, which is also a bit like Fast R-CNN with a softmax and bbox regressor after convolution layers.
- Last part including classifier, ROI pooling and bbox refinement, i.e. basically a Fast R-CNN network
Region proposal network
- Input: feature maps coming from fully-convolutional layers
-
Output: rectangular objects with an "objectness" score
- Objectness is defined as the likelihood of containing an object or background
-
For each anchor pixel in the image, apply the anchor boxes, resulting in tens of thousands of anchor boxes
- Prune boxes that overlap with high scoring boxes
- Extract about 2000 boxes during training phase, about 300 in testing
- Further reduce the 2000 by sampling to 256 (this is still per image)
- These are first classified as "foreground" or "background" (or potentially "ignore")
- Bounding box regression tries to tighten the anchor boxes around the target
- Why regression? Prediction on bounding box coordinates.
- Each of these remaining 256-d feature vectors, processed in a sliding window fashion
- Q: Snap hier weer niks van. Hoe kan dat nou op een vector? Of zijn het 256 feature maps? Maar dan is het geen vector.
- 2*K scores: (p_object, p_background) for K anchor boxes
- 4*K coordinates: (x,y,h,w) for K anchor boxes
Anchor boxes
For each location of the feature map (during a sliding window approach) you have K anchor boxes, because
1) more than one object could be present e.g. overlapping (a cycling person will have regions with both part of the cyclist and the bicycle) 2) we don't know the shape and aspect ratio of the object.
The best anchor is learned via backpropagation.
Detection network
- Features are cropped and scaled to 14x14 and later max-pooled to 7x7
- For each cropped feature, the final dense layers output ...
- the score for each class
- the bounding box for each class
- i.e. ( 256 x K, 256 x 4 x K) with K classes, in one-hot encoding.
- If I understand correctly, at this point the classes are just "foreground" vs "background" as outputted from the RPN
- Labels determined using IoU
Each layer produces a 256-feature vector. (How?)