Explainable Object Detection (blog)

Source: https://www.steadforce.com/blog/explainable-object-detection

Object detection models are not supported out of the box by existing explainability tools.

While explainable AI is not a new topic in computer vision, there are no explainable object detection tools yet available to analyze an object detection model down to a specific input. Therefore, we will now show one approach to a detailed instance-based explanation.

  • Closest analogue to object detection is image classification
    • Similar input
    • DNN architecture
  • Object detection output is however very different
    • Show multiple objects of differing size
    • NMS filters output in order to get to final prediction
    • NMS is non-differentiable

Dedicated explainability tools for DNNs use gradients. But there is no direct non-differentiable connection from input to output because: 1) multiple outputs, 2) NMS is non-differentiable.

Proposed solution here: use SHAPs model-agnostic KernelExplainer and fit YoloV5 such that it can be wrapped in it.

  • Visual explanations
  • Confidence scores
  • Simplify the problem: consider only one prediction and then ask "what part of the image contributes to this particular detection and how much?"

SHAP works by changing the input and then computing the contribution of each feature to the output. However, a 160x160 image already has 25,600 pixel values, so it's infeasible to change all of those. Also, the contribution of a single pixel is not going to be very meaningful.

This is actually a typical approach for using SHAP with images. From https://christophm.github.io/interpretable-ml-book/shap.html :

Shapley values tell us how to fairly distribute the “payout” (= the prediction) among the features. A player can be an individual feature value, e.g. for tabular data. A player can also be a group of feature values. For example to explain an image, pixels can be grouped to super pixels and the prediction distributed among them. One innovation that SHAP brings to the table is that the Shapley value explanation is represented as an additive feature attribution method, a linear model. That view connects LIME and Shapley Values.

This way you can compute the contribution of super pixels to the output and plot this as a heatmap over superpixels.

  • We want high contribution super pixels to be within the bounding box of the output we wanted to explain.
  • Helps us identify which image components tend to influence the model output
    • This could be useful for debugging
    • Maybe this could help in determining preprocessing steps that improve performance?
    • Could be useful for finding failure modes? E.g. if it's a type of clothing that determines the model output, would this prevent detection of humans?

Research ideas:

  • How to pick super pixels? Maybe not on a fixed grid? Is this beneficial?
  • How about
  • Experiment with different super pixel scales, appropriate for the application
    • Small super pixels will take a lot of computation power
  • How can this be used to debug erroneous detections?
  • How to deal with debugging false negatives?
    • And identify the problem?
    • E.g. problematic angles
    • Partially visible people?

Another limitation here is the explanation of false negatives: We could modify the scoring procedure to look for a target that is only very weakly predicted. But to determine what input the model would need to see for a hit, we’d have to add information to the image. Say, a person is not detected because the head is obscured. To check if the occlusion is the problem, we’d have to placea head into the image in the expected position. Not only is this more difficult to implement, the space of possible imputations is also much larger for a given missed detection