The Mythos of Model Interpretability | Lipton
People agree on the usefulness of interpretability of models, but in practice the notion of interpretation is underspecified:
- "diverse and sometimes non-overlapping motivations for interpretability" (1)
- "myriad notions of what attributes render models interpretable" (1)
- "Despite this ambiguity, many papers proclaim interpretability axiomatically" (1)
Goal of this paper is to unravel these points.
1) Survey of motivations for interpretability 2) Model properties and techniques for interpretability * transparency to humans and post-hoc explanations as competing notions. 3) Discussion of these notions of explainability and challenge the idea that linear models are in themselves interpretable whereas neural networks are not.
Introduction
Main question: "What is interpretability and why is it important?" (1)
Main claim: "interpretability is not a monolithic concept, but in fact reflects several distinct ideas." (1)
Some different possible takes on interpretability from the literature (p.1):
- Interpretability to engender trust (this is a big one!)
- But what's trust in this context?
- Faith in model performance?
- Robustness?
- Low-level mechanistic understanding?
- Does it apply to features, parameters, models, or training algorithm?
- Relation to uncovering a causal structure in data?
- N.B. in philosophy of science explanation has a stronger causal aspect
- Also relevant here: supervised ML minimizes a loss function, which can be done "in a purely correlative fashion" (2)
- Right to explanation: doesn't really require mechanistic interpretability so is yet another viewpoint.
- It is sometimes assumed human decisions are explainable. But this does not imply a mechanistic account of decision-making in the human brain. So what notion of "interpretability" do we use here? (2)
- Communication of "useful information of any kind" (2)?
Which properties render models interpretable?
- Does interpretability equal understandability or intelligibility, i.e. understanding how models work?
- Is "transparency" required for understanding?
- But what's transparency?
- Of algorithm, parameters?
- Related to model complexity?
A clear conflict in two common notions of interpretability can be shown in the case of post-hoc interpretability. This is because post-hoc explanations are perfectly possible for systems of which the internals are not interpretable. Examples are the human brain as well as black boxes that are usually said to not be interpretable. The notion of post-hoc interpretability challenges this idea! (cf. 2)
Desiderata of Interpretability Research
When do we require interpretability?
The demand for interpretability arises when there is a mismatch between the formal objectives of supervised learning (test set predictive performance) and the real world costs in a deployment setting. (2)
A typical loss function does not factor in extra desiderata such as interpretability. But... desiderata such as productivity, ethics, legality cannot be encoded as a real-value we can optimize.
The main idea of interpretability is that understanding the model allows us to factor in these desiderata that we cannot directly optimize and define formally, but that we nevertheless must take into account when any AI system is deployed.
This section lists some important desiderata and their relation with interpretability.
Trust
Interpretability as prerequisite for trust. This sections asks... but what's trust?
- High trust when the model performs well? Then interpretability is superfluous
- Yet, the understanding facilitated by interpretability may just serve to make people feel more at ease; i.e. trust obviously has a subjective dimension
- More interesting: "when the training and deployment objectives diverge, trust might denote confidence that the model will perform well with respect to the real objectives and scenarios." (3)
- This may also mean investigating for which type of examples the model does well or fails (which is not factored in e.g. in an accuracy score)
Causality
Supervised ML works with correlations, not causation. But ideally, finding correlations is a nice way of generating hypotheses that then can be further tested in more controlled experiments, ideally uncovering causal relations.
The idea is that this "hypotheses generation" step can be facilitated by making models more interpretable.
Some techniques, such as Bayesian (neural) networks, have better hopes of capturing (clues about) causal relations. But they make (strong) assumptions about prior knowledge.
Transferability
Humans have an incredible capacity to generalize knowledge to new and unfamiliar situations. The transferability of ML systems is limited and typically only assessed by train-test splits.
TODO make a note about generalization and how train-test splits are not enough.
Three examples are given that I all find interesting, so I summarize them.
Example 1:
A model that predicts probability of death from pneumonia assigned a lower probability of death for asthma patients. This doesn't generalize to the real world and the predictions are invalid, because these patients obviously have a higher risk. The thing is that these high-risk patients also receive way more aggressive and I guess preventative treatment, a causal factor influencing the data that's unaccounted for. Notice that in a train-test split the model may still "generalize" well, but not to the real world.
Example 2:
CNNs are shown to be susceptible to adversial input images (pertubations that are not visible to the human eye). Again, CNNs may have very good generalization to a test set, but these adversial cases show failure in an adversial environment. Also important: humans wouldn't be susceptible to this strategy, so additionally the model fails here in cases where a human wouldn't.
Example 3:
For credit scoring one commonly used model is logistic-regression, precisely because it is interpretable. But in this case interpretability also allows for gaming of the credit scores!
These rating improvement strategies do not fundamentally change one’s underlying ability to pay a debt. The fact that individuals actively and successfully game the rating system may invalidate its predictive power. (p.3)
Informativeness
I found that the description of this point could be clearer.
However, in another common use paradigm, the supervised model is used instead to provide information to human decision makers, a setting considered by Kim et al. (2015); Huysmans et al. (2011). While the machine learning objective might be to reduce error, the real-world purpose is to provide useful information. The most obvious way that a model conveys information is via its outputs. However, it may be possible via some procedure to convey additional information to the human decision-maker. (4)
I think the main point here is that interpretability could make model outputs more informative, in the sense that you can combine whatever you learn from it with your own insights, versus just getting an output decision or some number.
Fair and ethical decision-making
It is now a commonly held idea that models need to be interpretable in order to make sure that the decision-making process in which ML models are nowadays increasingly involved are fair and ethical.
Typical ML evaluation metrics such as accuracy or AUC are not sufficient for this.
Legally speaking, the GDPR right to explanation also emphasizes a need for interpretability. Also, decisions must be able to be contested. Two requirements we can then distill for explanations are that explanations:
1) "present clear reasoning based on falsifiable propositions" (4) 2) "offer some natural way of contesting these propositions and modifying the decisions appropriately if they are falsified." (4)
Properties of interpretable models
The authors distinguishes two types of model properties/techniques for interpretability:
1) Transparency: How does the model work? 2) Post-hoc explanations: What else can the model tell me? * Interesting way to phrase post-hoc!
The division is not strict, as e.g. most post-hoc explanations strive for a form of transparency, e.g by being able to interpret parameters.
Transparency
"some sense of understanding the mechanism by which the model works" (4).
I guess transparency has this connotation of seeing "through" the model. Glass box vs. black box.
Transparency distinguished on three levels:
- Entire model: simulatability
- Individual components (e.g. parameters): Decomposability
- Training algorithm: algorithmic transparency.
Simulatability
Simulatability implies that a person can "contemplate the entire model at once"(4) and also do so in reasonable time (5). This in practice always implies the transparent/interpretable model is a very simple model. What constitutes "reasonable" time is also subjective to a degree.
Usually people talk about sparse linear models, because they have less parameters. In the case of decision trees the model size increases very quickly, but one may still argue this is fine because inference is still very fast and easy with decision trees. But notice that the meaning of simple is slightly different: model complexity in terms of parameters vs. inference time.
The author contests the claim that linear models etc. are inherently interpretable as such:
In this light, we suggest that neither linear models, rule-based systems, nor decision trees are intrinsically interpretable. Sufficiently high-dimensional models, unwieldy rule lists, and deep decision trees could all be considered less transparent than comparatively compact neural networks. (5)
Decomposability
Are the individual components intelligible?
Interesting point: interpretability is not so easy as it depends on many decision steps e.g. in preprocessing and feature engineering. Example: weights in linear regression may be coined as feature strengths.
Note that this notion of interpretability requires that inputs themselves be individually interpretable, disqualifying some models with highly engineered or anonymous features. While this notion is popular, we shouldn’t accept it blindly. The weights of a linear model might seem intuitive, but they can be fragile with respect to feature selection and pre-processing. For example, associations between flu risk and vaccination might be positive or negative depending on whether the feature set includes indicators of old age, infancy, or immunodeficiency. (5)
Algorithmic Transparency
One could for example show that a training algorithm will converge to a unique solution, which may incur some form of confidence. Modern ML has limited algorithmic transparency, but the authors also ask: how bad is that actually?
On the other hand, modern deep learning methods lack this sort of algorithmic transparency. While the heuristic optimization procedures for neural networks are demonstrably powerful, we don’t understand how they work, and at present cannot guarantee a priori that they will work on new problems. Note, however, that humans exhibit none of these forms of transparency.
What's meant exactly with that "humans" don't exhibit this form of transparency? It's not very common to talk about the transparency of humans.
Post-hoc interpretability
Post-hoc explanations do not try to "look into" the model, but extract and provide useful information about its outputs.
For all we know, the processes by which we humans make de- cisions and those by which we explain them may be distinct. One advantage of this concept of interpretability is that we can interpret opaque models after-the-fact, without sacrificing predictive performance. (5)
Text explanations
Sections describes the use of other models to generate textual explanations for the output of another model. Concrete example: network that generates captions to images. Another concrete example is using topic modelling and then using top-words in topic as explanations. All up for debate whether those are good explanations (if explanations at all).
Visualization
I wasn't familiar with t-SNE yet:
One popular approach is to visualize high-dimensional distributed rep- resentations with t-SNE (Van der Maaten & Hinton, 2008), a technique that renders 2D visualizations in which nearby data points are likely to appear close together. (6)
I'm familiar with this approach though:
Mordvintsev et al. (2015) attempt to explain what an image classification network has learned by altering the input through gradient descent to enhance the activations of certain nodes selected from the hidden layers. (6)
Another strategy for CNNs is that from intermediate layers, through gradient descent, you can still reconstruct inputs with decent fidelity.
Local explanations
Describe what a neural network depends on locally. (Cf. a linear model instead models a global relationship between inputs and outputs).
One example: compute a saliency map that shows what a network is "focusing on".
LIME by Ribeiro et al. is another: "explain the decisions of any model in a local region near a particular point, by learning a separate sparse linear model to explain the decisions of the first." (6)
Explanations by example
Report example data points that the model considers to be similar. Sort of like reasoning by analogy.
In DNNs "we can use the activations of the hidden layers to identify the k-nearest neighbors based on the proximity in the space learned by the model.
Example: show similar words (in word2vec) based on the learned latent word vectors.
Example: Bayesian methods are also usable as generative models and can thus also create examples.
Discussion
Are linear models strictly more interpretable than deep neural networks?
Usually people point out a trade-off between explainability and accuracy in AI . This section more specifically points out a trade-off between algorithmic transparency and decomposability.
Due to their flexibility, DNNs "tend to operate on raw or lightly processed features. So if nothing else, the features are intuitively meaningful, and post-hoc reasoning is sensible." (7)
To get comparable performance with linear models you need to very heavily engineer features. These features are then arguably not that meaningful anymore, i.e. in this scenario you may still have algorithmic transparency (because e.g. linear regression is easy enough to explain), but you'll lose decomposability because the features themselves are arguably not that meaningful anymore.
However, I'm not sure I'm convinced that the "raw" features of DNNs would be interpretable. In an image, the first features often are pixels, OK, that's clear enough, but is that really meaningful? Their weighting and combination still becomes increasingly complex.
Claims about interpretability must be qualified
Because we've seen "interpretation" is not univocal.
If the model satisfies a form of transparency, this can be shown directly. For post-hoc interpretability, papers ought to fix a clear objective and demonstrate evidence that the offered form of interpretation achieves it.
Transparency may be at odds with broader objectives of AI
See basically the trade-off between explainability and accuracy in AI .
Post-hoc interpretations can potentially mislead
We caution against blindly embracing post-hoc notions of interpretability, especially when optimized to placate subjective demands. In such cases, one might - deliberately or not - optimize an algorithm to present misleading but plausible explanations. As humans, we are known to engage in this behavior, as evidenced in hiring practices and college admissions. Several journalists and social scientists have demonstrated that acceptance decisions attributed to virtues like leadership or originality often disguise racial or gender discrimination (Mounk, 2014). In the rush to gain acceptance for machine learning and to emulate human intelligence, we should be careful not to reproduce pathological behavior at scale. (7)
Contributions
Not all aspects of interpretability are covered by the research community.
Moreover, we identify the incompatibility of most presently investigated interpretability techniques with pressing problems facing machine learning in the wild. For example, little in the published work on model interpretability addresses the idea of contestability. (8)
Algorithms won't help if you have no understanding of what the actual problem is!
We argue that the paucity of critical writing in the machine learning community is problematic. When we have solid problem formulations, flaws in methodology can be addressed by articulating new methods. But when the problem formulation itself is flawed, neither algorithms nor experiments are sufficient to address the underlying problem. (8)