Explaining Explanations in AI
@XAI
What makes an "explanation" in XAI, where techniques are developed to promote the interpretability? What is the difference with how explanations are understood in other fields?
Introduction
Main problem formulation: black box models may have an internal state of millions of interdependent values. As a consequence, the underlying functions may be too difficult to understand, as well as the overall decision-making process.
But what type of explanation would remedy this problem?
one of the most striking aspects of research into explainable AI (xAI) is how many different people, be they lawyers, regulators, machine learning specialists, philosophers, or futurolo- gists, are all prepared to agree on the importance of explainable AI. However, very few stop to check what they are agreeing to, and to find out what explainable AI means to other people involved in the discussion (Lipton, 2016). (p. 279)
In fact, the working definition of "explanation" within xAI is very specific and only a minor fraction of the broader literature on explanation, e.g. within philosophy.
The type of explanation used in XAI is typically another model that is easier to interpret, operating on the model to be explained:
Work on xAI currently occupies only one or two small branches of this diverse research landscape. Specifically, the vast majority of work in xAI produces simplified approximations of complex decision-making functions. We argue that these approximations function more like scientific models than the types of scientific and ‘everyday’ explanations considered in philosophy, cognitive science, and psychology. (p. 279)
This type of ML explanation is more of a "scientific model". So, there is a gap between "explanations" in this ML context and our everyday explanations. This paper:
- Explores this gap between explanation in xAI and broader more common conceptions
- Claim: "explanation" in the XAI sense is "generally more akin to scientific modelling than explanation giving." (279)
- Hypothesis: "If this comparison holds, it follows that the majority of currently available methods will, at best, produce locally reliable but globally misleading explanations of model functionality." (279-280)
Recommendation:
We then examine research on explanations in philosophy, cognitive science, and the social sciences which suggests that ‘why-questions’ (e.g. ‘why did the model exhibit that behaviour?’) require explanations that are contrastive, selective, and socially interactive. On this basis, we argue that, if xAI is to produce methods that make algorithmic decision-making systems more trustworthy and accountable, the field’s attention must shift to the development of interactive methods for post-hoc interpretability that make it easier to contest algorithmic decisions, and facilitate informed dialogue between users, developers, algorithmic systems, and other stakeholders. (p. 280)
Explanations in philosophy
Which type of explanation is currently produced by XAI and are they actually helpful?
The xAI actually investigates two aspects that are related but different:
- Interpretability
- Burrell, 2016, p.1: from the perspective of the recipient of a decision, there is not really an idea of how that output came to be, given the inputs
- This is often what is meant with "explainability"
- Explanations of algorithmic models and decisions
- This is not quite the same as interpretability; with "explanation" we instead refer to a set of ways to exchange information about a phenomenon to someone
Explanations serve many functions (see refs on p. 280):
- necessary to comply with legislation
- verify/improve system functionality
- aid learning from the system
Rough distinctions in philosophy:
- degree of "completeness", degrees to which causal chains are explained
- which roughly distinguishes "everyday" from "scientific" explanations
- both are causal but scientific explanations require a more complete account of causality
- everyday explanations focus more on "why" instead of general scientific relationships.
Based on this distinction, xAI deals more with "everyday" explanations, because it has to account for the "why" of a particular thing, e.g. "a specific decision, event, trained model, or application." (280)
No scientific laws; at best causal relations between model variables.
XAI
- Transparency: Addresses how a model functions internally and understands various targets mechanistically:
- Simulatability: "functioning of the model"
- Decomposability: functioning of individual components
- Algorithmic transparency: the training algorithm
- Post-hoc interpretation: concerns how the model behaves
Transparency
Three levels of model explanation for transparency (Lepri et al., 2017)
- entire model
- individual components e.g. parameters
- particular training algorithm
A model is transparent on a level when a person can understand this aspect in its entirety. p.280, also see ref to Lipton 2016.
Post-hoc Interpretable explanations
See p. 280 for refs.
- verbal explanation
- visualisations and interactive inferfaces
- local explanations / approximations
- fit a simpler local model around a particular decision
- perturb variables to measure how a prediction changes
- case-based explanations (for non-cased based machine learning) Q: what does that mean?
- "using the trained model as a distance metric to determine which cases in the training data are most similar to the case or decision to be explained." (281)
- These training cases can then be shared.
xAI right now is focused on approximations, which can be used both local and global. Two main approaches:
Broadly speaking there are two widely used classes of model (i) Linear or Gradient-based approximations that assign a single importance weight to each feature (be it someone’s age, or a particular pixel in an image) and (ii) Decision tree-based methods that use nested sets of yes/no decisions to approximate classifiers. (p. 3)
How do these relate to existing approaches to explanation e.g. in philosophy or social sciences?
Scientific Modelling and Explainable AI
Claim: "the closest analogue for the bulk of methods currently occupying xAI researchers lies in the use of scientific modelling, or the building of approximate models that are not intended to capture the full behaviour of physical systems but rather to provide coarse approximations of how the systems behave."
Further claim: this is useful for experts but can be misleading for the layman, as an explanation of how the model works.
Analogy is made to physics, particularly to how Newtonian physics, general relativity and quantum physics are all accurate on different levels.
Although any physical system can be understood in terms of the emergent properties of subatomic particles, such descriptions are neither human comprehensible nor computationally feasible. Instead, scientists deal in local approximations that provide accurate descriptions of the phenomena they are interested in, but which may prove inaccurate in a larger domain. (281)
Point here is that:
- whether an explanation is appropriate should be related to it's "level"
- the type of "explanation" in xAI is more akin to this scientific approach
See Box's maxim that all models are wrong, but some useful.
But when approximations are supposed to be explanations that help trust the behavior of a complex model, then of course the question arises... is the approximation actually reliable?! An approximation can be seen as a sort of analogy where we can ask whether the analogy holds. This is the crux!
This characterisation captures many of the challenges when offering approximations of models as explanations. It is not enough to simply offer a human interpretable model as an explanation. For an individual to be able to trust such a model as an approximation, they must know over which domain a model is reliable and accurate, where it breaks down, and where its behaviour is uncertain. If the recipient of a local approximation does not understand its limitations, at best it is not comprehensible, and at worst misleading. (p. 281)
Local approximations only have explanatory power insofar as "their limitations are clearly documented and understood by recipients" (281-282)!
This has some possible implications:
- It then seems that the explanations of XAI are more useful for tech people who are already sufficiently familiar with the model etc.
- The converse of that is that this type of explanation is probably less useful towards let's say "customers" or "users" of the system; specifically those affected by its decisions.
- There's again a lot of emphasis on documenting limitations; but this is actually quite useful and applicable guideline! :-)
Also, this makes me think of an interesting research project: How reliable are local approximations (i.e. 'explanations') in terms of the global system behavior? Without guarantees here, what explanatory power do these explanations really have? Do measures already exist that express the reliability of the explanations?
Linear Approximations
three-way trade-off between the simplicity of the approximated model, the size of the domain it describes, and the accuracy of this description (p. 282)
Linear models in continuous spaces
Two typical uses:
- Importance measure: "give a single measure of the importance of each variable to the classifier they are approximating"
- This can be a number that indicates how much the classifier response changes when you vary a feature
- local approximation
Suffer from two issues:
- "the sensitivity of a classifier to a change in a particular variable may vary with the amount the variable changes." (282)
- This is just a complex way of saying that a relation approximation with a linear classifier may in fact not be linear, right?
- "the second issue is one of dependencies between variables and how to capture the relationships between them." (282)
Gradient sensitivity versus binarization
Two answers to above problems of sensitivity and scale.
- "The first notes that the sensitivity of a classifier
is only well defined when the size of in the previous section tends
towards zero." (p. 282)
- Then sensitivity equals gradient
- Q: I do not understand this.
- LIME: Locally Interpretable Model-Agnostic Explanations
- Binarize by switching features on or off; what's the influence on the prediction?
- Question: contribution of feature compared to what? What's a valid baseline?
And how does LIME work in computer vision? You cannot remove pixels from an image. So setting a baseline implies some choice, that has implications. Some options:
- you can define a baseline by graying out a region (suggested that LIME does this)
- or blurring the image (Deep Taylor Decomposition).
- Set to a user-specified value (DeepLift)
- Set internal network values to 0 (layerwise relevance propagation)
But these choices can have strong implications on feature importance. E.g. when you choose grey pixels as a baseline, you assume grey pixels have "no effect" (not so true when classifying elephants, I guess!)
Linear models in high dimensionality
Even after binarizing, you can ask: which value to approximate?
For example, should we only pay attention to contrasting solutions closest to the original solution as suggested by DeepLift (illustrated by Figure 1 centre; (Shrikumar et al., 2017, 2016)); uniformly weight over all possible values, suggested by SHAPE (Lundberg and Lee, 2017) (Figure 1 right); or pay as much attention to values close to the data point as to those close to the alternate solution (e.g. a solid grey image); or a weighted mixture of the two approaches (e.g. Figure 1 left LIME; (Ribeiro et al., 2016))? (p. 283)
Alternatives to scientific modelling
Issues of local approximations (p. 283):
- issues with generalizability
- "arbitrariness in choice of domain"
- "mislead recipients unless the domain and epistemic limitations of the approximation are known" (p. 283)
Main point: the explanations offered by XAI are of limited usefulness for non-experts:
The utility of local approximations is dependent upon the knowledge of the recipient regarding the approximations limitations, including conditions under which it will break down and provide a misleading explanation of the decision-making model. Local approximations can be useful as a type of ‘explanation kit’ or causal chain that allows expert users to explore slices of a model for prototyping or debugging (Miller, 2017, p.17)(Poulin et al., 2006). However, their ultimate utility and reliability for non-experts, including individuals subject to decisions made by the system, is highly questionable. (p. 283)
Contrastive explanations
To explore which type of explanation is also useful for non-experts, have a look at other sciences.
Main point: "contrastive theories argue that causal explanations inevitably involve appeal to a counterfactual case, be it a cause or event, which did not occur." (p. 283)
Reference is made to Lipton's inference to the best explanation .
There's some discussion about whether the contrastive aspect is essential to explanation, but in any case it's a useful perspective to take into account for XAI.
This paper focuses on three points, from empirical work of Miller, 2017. Everyday explanations are:
- Contrastive
- Selective
- Social
Contrastive
Often an explanation is sought for abnornmal events. In this case, these explanations tend to be naturally contrastive. Because why this this abnormal case occur, rather than this normal case? Causal reasoning involves reference to counterfactuals.
Selective
In Miller's words:
“Explanations are selected – people rarely, if ever, expect an explanation that consists of an actual and complete cause of an event. Humans are adept at selecting one or two causes from a sometimes infinite number of causes to be the explanation” (Miller, 2017, p.5) (p. 284)
Instead, we select a few "why's" that are 1) informative (not be entirely reducible to presuppositions), 2) relevant to asked questions / epistemic relevance.
What does this imply for xAI? The epistemic relevance of an explanation should, ideally, not just depend on "statistical weight":
In xAI, selection often takes the form of key features or evidence being emphasised in explanation interfaces based upon their relative weight or influence on a given prediction or output (Biran and McKeown, 2014; Poulin et al., 2006). As the observations above suggest, the relevance of features (and explanations addressing them) would be based not only on ’statistical weight’, but also the explainee’s subjective interests and expectations. (p. 284)
Social
Explanations also involve interaction between explainer and explainee.
For machine learning models it may be wise to take this into account:
- Interactive: "treat explanation generation as an interactive process, initially involving a mix of human and automated actors, at a minimum an inquirer (e.g. a developer, user) and the model or system (Kayande et al., 2009; Martens and Provost, 2013)."
- Iterative: "insofar as they must be selected and evaluated on the basis of shared presuppositions and beliefs. Relevance is key, and iteration may be required to communicate effectively or clarify points of confusion on the path towards a mutually understood explanation." (p. 284)
So in short, everyday explanations are really not just the positing of "causes".
Contrastive explanations in xAI
Alternative to model-based explanations you can also offer contrastive explanations in terms of your data.
Rather than explicitly generating a model that approximates functional values over a restrictive domain, and relying on the user to interpret this, contrastive explanations directly offer an alternative data point: “If your data had looked like this, you would have been given this classification score instead.” These alternative data points can be computed exactly. As such, many of the challenges facing ’modelling’ approaches to generating explanations, such as the quality of the approximation or the limits of a chosen domain, do not arise to a comparable degree. (p. 285)
Some approaches and problems are discussed, but not in depth. Worth it to look into this alternative approach more later.
Apparently these contrastive methods only work on single datapoints But ultimately the relevance of the output is what is at stake. So you also need to show that the contrast applied on the input data point is relevant for the output decision.
Towards communicative contrastive explanations
This section identifies several contextual factors. It discusses the possibility that the explainer chooses the explanation type such that it may lead to acceptance of a decision but is de facto misleading:
Lipton (2016) has urged for caution in adopting post-hoc interpretation methods due to this potential to mislead recipients, for example by falsely attributing a decision to an irrelevant feature, or a more acceptable feature (e.g. post code, ’leadership’) that serves as a proxy for a legally protected feature (cf. Barocas and Selbst (2016); Kim (2016)). (285)
So ideally, explanations take place within a communicative practice that allows the explainee to dialogically and critically interact with the explanation.
This section also discusses the relation between explanation and justification. A main point here is that justification is underrepresented in xAI, probably because it focuses more on algorithmic solutions. Sometimes, however, there's a legitimate conflict that requires broader (democratic) processes in order to resolve them.
Conclusion:
Finally, very little work in xAI addresses the link between interpretability and contestability of models or decisions (Lipton, 2016). Going forward, explanations of specific algorithmic decisions should allow the justification of a black-box model or decision to be debated and contested.. If we use justificatory discourse as a framework for explanation requirements, we need to determine what sort of records algorithmic systems must retain in order to allow for contesting and post-hoc auditing of abnormal events (Mittelstadt, 2016; Sandvig et al., 2014), for instance through identification of classification errors or inaccurate input data (Poulin et al., 2006). Where approximations are used to provide explanations, information should also be provided to affected parties detailing the limitations and relative resilience of the approximation, including the domain addressed and why it has been chosen. Further, meaningful, critical dialogue can be achieved between user, developer, and model by ensuring explanations are contrastive, selective, and social. We thus need to ensure xAI aims to develop methods for producing explanations of model functionality and specific decisions that embody these characteristics. Reliable methods for producing contrastive explanations and explanation by approximation are both required.
Further reading
Some references I found interesting:
- Zachary C. Lipton. 2016. The Mythos of Model Interpretability. arXiv:1606.03490 [cs, stat] (10 6 2016). http://arxiv.org/abs/1606.03490
- Stakeholders involved in algorithmic-decision making: Berendt and Preibusch (2017)
(Lepri et al., 2017) - Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2017. Fair, Transparent, and Accountable Algorithmic Decision-making Processes: The Premise, the Proposed Solutions, and the Open Challenges. Philosophy & Technology (15 8 2017). https://doi.org/10.1007/s13347-017-0279-x
- Peter Lipton. 1990. Contrastive explanation. Royal Institute of Philosophy Supplements 27 (1990), 247–266.