Taxonomy of explainability methods in machine learning
There are several ways to classify these. Here's an oversight collected from multiple sources.
This note is a bit too ambitious. This problem is quite hard, because there's so many different angles from which to classify the XAI field. I'm keeping track of papers on XAI taxonomies here: index XAI explainable AI
Source:
- Interpretable Machine Learning: A Guide for Making Black Box Models Explainable
- Barredo Arrieta - XAI Concepts taxonomies opportunities and challenges
Intrinsic vs. post hoc interpretability
Intrinsic: Some machine learning models have limited complexity and interpretable internals, e.g. shallow decision trees or sparse linear models. In this case we speak of intrinsic interpretability.
Intrinsic interpretation is by definition model-dependent.
Post hoc: An interpretation method is applied after the model is trained (? Weird actually: we always train the model before we try to explain it right?). For example, computing permutation feature importance is post hoc.
Post hoc interpretability can both be applied to models that are intrinsically interpretable as well as to 'black box' models.
Model-agnostic techniques are always post-hoc approaches.
As a rule of thumb, all state-of-the-art models do not have intrinsic interpretability.
Outputs of interpretation methods
- Feature summary statistic(s)
- feature importance
- pairwise feature interaction strength
- Feature summary visualization
- For some types of features, visualization is necessary for human interpretation
- Model internals ( by definition model-specific )
- e.g. learned weights
- only meaningful for intrinsically interpretable models
- e.g. in linear models the model internals are at the same time summary statistics
- learned tree structure, i.e. rules for the splits
- e.g. learned weights
- Data points (in the case of example-based interpretations)
- E.g. counterfactual explanations: closest data point for which the prediction changes
- prototypical data points for predicted classes
- this all assumes the data points themselves are interpretable!
- E.g. a pixel output is understandable, but a 100-dimensional feature vector not so much.
- Another intrinsically interpretable model that locally or globally approximates a black box model.
- This second model itself is interpretable in terms of model internals or a feature summary statistic.
Model-specific vs. model-agnostic
Interpretation of model internals is of course always model specific. Some interpretation methods only work for neural-networks and are thus also model-specific.
Model-agnostic tools instead do not refer to any model internals, meaning that they typically analyze combinations of (feature) inputs and the model outputs. This in turn means that they actually need access to the model outputs, i.e. they are all post-hoc and applied after training.
Scope of interpretability: Local vs global vs everything in between
Interpretation methods can:
- Explain a single prediction/classification
- Explain the entire model behavior
- Or some scope in between, at different moments in a machine learning pipeline.
Q: Under which circumstances can local explanations be more accurate than global approximations? Why?
Algorithmic Transparency
Transparency about 1) how an algorithm learns a model from data, 2) what kind of relationships a particular algorithm is able to learn (and which ones not).
Notice an important difference (and limitation in terms of explanation): algorithmic transparency is only about the algorithm and thus not about the model nor about the used data.
If an algorithm is transparent, it is easier to explain, but this transparency in itself does not constitute a proper explanation.
Global (holistic) interpretability
Can you comprehend the whole model at once?
Explaining global model output:
- have the whole trained model
- have knowledge of the algorithm
- have knowledge of the data
- then understand the importance of all features and how they interact
If this is possible, you get an understand of the distribution of outputs (so not just one output, but the whole range of possible outputs).
Arguably, this is already impossible when you have a simple linear model with 5+ features, as humans can only really properly understand and visualize in 3D.
Also in the case of models with weights... we cannot even keep all weights in memory when trying to understand their influence on the model output.
Modular interpretability
You can take an approach similar to the global interpretation, but then apply it to particular modules of machine learning methods. Some parts of some methods can be interpreted relatively well.
Examples:
- Linear models: interpretable module is the weights
- Decision trees: interpretable part are splits or leaf node predictions
Even in those simple cases, the interpretability is limited. For example, interpreting the weight of one feature in a linear regression is dependent on the interpretation of all other weights as well (that you would take into account with a fully global approach) unless features are perfectly uncorrelated.
Local interpretability (for a single prediction)
Zoom in on a single input instance and explain why the prediction is what it is.
For a global understanding of the model outputs, we always need to consider all features and all their complex interactions.
If you only feed a single instance instead, this prediction may only be affected by a more limited set of features and even have simpler relations (e.g. linear or through some monotonic function).
For example, the value of a house may depend nonlinearly on its size. But if you are looking at only one particular 100 square meters house, there is a possibility that for that data subset, your model prediction depends linearly on the size. You can find this out by simulating how the predicted price changes when you increase or decrease the size by 10 square meters. Local explanations can therefore be more accurate than global explanations.
This is crucial!
More than one, but less than all predictions
If you have multiple instances, you can either:
- Treat them as the whole dataset and apply a global method
- Apply a local method to each of them and aggregate results
Globally speaking, the output always depends on all features.