You Only Look Once-model (Yolo; 2015)
Paper: You Only Look Once: Unified, Real-Time Object Detection by Joseph Redmon et al.
Single-stage detector models based on CNNs. Another approach to object detection.
Difference with Faster R-CNN and other region based models is that the former are called two-stage detector models and YOLO instead is a single stage detector model.
(So the internal region proposal network of the Faster R-CNN is still considered as a separate network)
A single network does:
- Bounding box prediction
- Class prediction for each bounding box
By treating this as a big regression problem, meaning that the whole image will be predicted at once. Because this whole network is designed for detection instead of repurposing a classification network, in a single network, you can optimize detection end-to-end only with YOLO. For example in R-CNN you have different components that need to be trained separately.
- Divide image in grid of cells
- Within each grid assign a set of possible bounding boxes
- For each bounding box we want to compute an
- offset w.r.t. the true location
- Cf. paper: "We parametrize the bounding box x and y coordinates to be offsets of a particular grid cell location so they are also bounded between 0 and 1" (p.3)
- a class probability for those bounding boxes
- These outputs are included in a single giant tensor that predicts everything for the whole image (instead of running seperate models on separate regions first)
From Redmon's website:
Prior detection systems repurpose classifiers or localizers to perform detection. They apply the model to an image at multiple locations and scales. High scoring regions of the image are considered detections.
We use a totally different approach. We apply a single neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities.
Our model has several advantages over classifier-based systems. It looks at the whole image at test time so its predictions are informed by global context in the image. It also makes predictions with a single network evaluation unlike systems like R-CNN which require thousands for a single image. This makes it extremely fast, more than 1000x faster than R-CNN and 100x faster than Fast R-CNN. See our paper for more details on the full system.
1. Grid cells
The grid design enforces spatial diversity in the bounding box predictions. Often it is clear which grid cell an object falls in to and the network only predicts one box for each object. (p.4)
- Divide image into SxS grid.
- Each grid cell predicts
- B bounding boxes
- Confidence score for those boxes (for a particular class?). Needed because of following point
- A single grid cell may contain bounding boxes that cover different objects;
- "If the center of an object falls into a grid cell, that grid cell is responsible for detecting that object"
- Bounding boxes here are the same concept as anchor boxes in Faster R-CNN
2. Bounding boxes
Each bounding/anchor box contains 5 predictions: (x,y,h,w, confidence).
- (x,y) center of bounding box relative to the bounds of the grid cell
- (w,h) predict width and height relative to the whole image, so a proportion 0 <= (w,h) <= 1
- Confidence C: $Pr(Object) * IoU$ between predicted and any ground truth box
- C is not "class", but confidence for foreground/background because it is only about the presence/absence of an object!
- So is really about capturing whether an object is present.
- This is new w.r.t. faster R-CNN
Why x Pr(object)?
- If there's no object, confidence should be 0.
- If there is an object, we want the confidence to be equal to the IoU with the ground truth bbox.
Limitation:
YOLO imposes strong spatial constraints on bounding box predictions since each grid cell only predicts two boxes and can only have one class. This spatial constraint limits the number of nearby objects that our model can predict. Our model struggles with small objects that appear in groups, such as flocks of birds. (p.4)
3. Classification
Each grid cell predicts C conditional class probabilities $P(class_i | object)$ per grid cell.
The following step was not explained in the lecture!
Notice that confidence scores etc. are computed over bounding boxes, that are relative to some responsible grid cell. Here instead the class probabilities are computed per grid cell:
These probabilities are conditioned on the grid cell containing an object. We only predict one set of class probabilities per grid cell, regardless of the number of boxes B.
To then get to scores for the individual bounding boxes belong to that grid cell, you multiple the class probabilities of the grid cell (that count for all related bounding boxes) with the confidence scores for the individual bounding boxes. This gives individual confidence scores for classes for bounding boxes:
$$Pr(Class_i | Object ) * Pr(Object) * IOU = Pr(Class_i) * IOU$$
So you can see this factors in:
- the probability of that class occurring in the box (taking into account of the box capturing an object in the first place )
- and how well the predicted box fits the true bounding box of the object
4. Output as a tensor
The output is a large tensor with multiple components:
- The cells of the grid cell ( S x S )
- (x, y, h, w, foreground/background confidence) for each box belonging to the grid cells
- C class probabilities
So output 3D tensor is $( S x S x (B x 5 + C))$.
The right hand side could be split into multiple dimensions by looking at C as a separate dimension, but this was not done in the original paper.
In the paper with Pascal VOC data this looks like:
- Input: 448x448
- 20 classes (C=20)
- Grids with S=7
- Each grid cell is 64 pixels
- B=2 so two bounding boxes per cell
So output is 7x7x30 tensor, which is predicted in one go by YOLO.
5. CNN architecture
24 layer ConvNet pretrained on ImageNet.
6. Custom loss function of Yolo-v1!
The custom loss function is built up from several components, because we want to predict multiple things.
1. First goal is to predict the center (x, y) of the bounding box with a low error.
A first component is thus the sum-squared error for $x$ and $y$. The sum is actually two sums per grid (I think),
- one over each individual cells (so from 0 to SxS)
- then for each cell, sum over the bounding boxes
There's an additional term $\mathbb{1}_{ij}^{obj}$:
Read: obj -> "object"
- the j-th bbox that is responsible for prediction in the i-th cell
- does this notation mean it is only 1 for the bbox that is responsible for the i-th cell, and 0 otherwise?
- Q: How was this determined?
Paper:
YOLO predicts multiple bounding boxes per grid cell. At training time we only want one bounding box predictor to be responsible for each object. We assign one predictor to be “responsible” for predicting an object based on which prediction has the highest current IOU with the ground truth. This leads to specialization between the bounding box predictors. Each predictor gets better at predicting certain sizes, aspect ratios, or classes of object, improving overall recall. (p.3)
Purpose:
It also only penalizes bounding box coordinate error if that predictor is “responsible” for the ground truth box (i.e. has the highest IOU of any predictor in that grid cell). (p.4)
2. We want to predict the size (w,h) of the bounding box with a low error
The problem here is that the effect of the error term depends on the size of a bounding box. A small error on a large bounding box covering a large object is unproblematic. But the same error on a small bounding box is bad news and this should be factored in.
From paper:
Sum-squared error also equally weights errors in large boxes and small boxes. Our error metric should reflect that small deviations in large boxes matter less than in small boxes. To partially address this we predict the square root of the bounding box width and height instead of the width and height directly. (p.3)
This is factored in by computing the SSE but now using the prediction and true value for $\sqrt{w}$ and $\sqrt{h}$. This is said to partially address this issue.
Rationale: this assumes that w and h are between 0 and 1, which means that the square root will actually be bigger than the input value. The square root of small values is "boosted" more than values closer to 1, which also that the loss will be relatively higher for smaller boxes.
Error in a small box has a much greater effect on the IOU, which is why you want the network to do extra well for small boxes:
our loss function treats errors the same in small bounding boxes versus large bounding boxes. A small error in a large box is generally benign but a small error in a small box has a much greater effect on IOU. Our main source of error is incorrect localizations. (p.4)
3. We want confidence to be high when there is an object in the ground truth
- Confidence high: When there is an actual object in the bounding box that is predicted by the network
Now we take the SSE over $C_i - \hat{C_i}$.
- But for grid cells without objects we want confidence to be very small!
So yet another SSE term is added, but now specifically for situations where no object is present in the grid cell.
Again we take the SSE over $C_i - \hat{C_i}$, but now instead of $\mathbb{1}{ij}^{obj}$ we multiply each term with $\mathbb{1}{ij}^{noobj}$.
- Extra coefficients $\lambda_{coord}$ and $\lambda_{noobj}$ for stability of gradient descent
The added term for "no object" apparently led to instability so they added a weighing term $\lambda_{noobj}$ and tuned it.
Also, in every image many grid cells do not contain any object. This pushes the “confidence” scores of those cells towards zero, often overpowering the gradient from cells that do contain objects. This can lead to model instability, causing training to diverge early on. (p.3)
Similarly, the SSE terms of 1 and 2 are also weighed with $\lambda_{coord}$.
Read: "bounding box coordinates" vs. "no object"
The loss for bounding box coordinates is boosted with $\lambda_{coord}=5$ and to deal with the above-mentioned instability the loss for confidence of "no object" cases is dampened with $\lambda_{noobj}=0.5$.
- Finally, a SEE for classification error
Now SSE between $p_i(c)$ and $\hat{p_i(c)$, which now is actually the classification score of cell $i$ for each class $c$.
Now $\mathbb{1}_{i}^{obj}$ (notice the single index instead of two) is used.
Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). (p.4)
Overview of CNN object detection methods
We covered more or less from 2014 to 2016. Note that from YOLO-v1 to YOLO-v2 there is actually a performance drop. This is probably because they were focusing on speed instead of performance.
Non-max suppression
NMS is used in many object detection approaches that using bounding boxes.
For any object we have almost certainly generated multiple anchor boxes that cover it. Each anchor box has its own probability of containing an object.
For the final detection, we only want to keep one box.
- Pick a probability threshold
- Remove all boxes with probability < threshold
- While there are predicted boxes
- Pick the box with the largest probability
- Discard all boxes with IoU >= 0.5 of the max probability box
Note that of course you can have multiple object. This procedure only removes overlapping boxes, so if you have two boxes for two objects they'll stay.