Why do we want explainable AI?

Why do we want explainability or interpretability of machine learning models?

  • Model debugging: Why did the model make this mistake?
  • Feature Engineering: How can we improve our model?
  • Detecting issues with fairness: e.g. does the model discriminate?
  • Human-AI interaction and cooperation: In order to trust the model's decisions, it is important that I understand them.
  • Compliance: Can we demonstrate that the model satisfies certain legal requirements?

The above reasons are more urgent in high-risk applications such as healthcare, finance, judicial etc.

Explainability is thus not necessarily an intrinsic ethical value, but instead an instrumental value for other values.

From Lipton - The mythos of model interpretability :

The demand for interpretability arises when there is a mismatch between the formal objectives of supervised learning (test set predictive performance) and the real world costs in a deployment setting. (2)

A typical loss function only minimizes some real-valued loss function. But... desiderata such as productivity, ethics, legality cannot be encoded as a real-value we can optimize.

The main idea of interpretability is that understanding the model allows us to factor in these desiderata that we cannot directly optimize and define formally, but that we nevertheless must take into account when any AI system is deployed. So again, interpretability serves an instrumental value.

Example

In this video around 9:50 an example is given of using Shapley values for explaining a model for credit allocation.

It turns out that the model is identifying retirement-age individuals based on their long credit history! Age is a protected class, but even though age is not actually in the dataset, the model uses the length (in months) of credit history as a proxy!

Use cases for explainability

Model development:

  • Debugging
  • Data exploration
  • Monitoring of errors
  • Encoding prior beliefs (how? could be a RQ?)

Human/AI collaboration:

  • Customer retention
  • Decision support
  • Human risk oversight

Regulatory compliance:

  • Consumer explanations
  • Anti-discrimination
  • Risk management

Scientific discovery:

  • Population subtyping
  • Pattern discovery
  • Signal recovery