SHAP: Shapley additive explanations

Local vs global

A nice thing of SHAP is that your local and global explanations share the same conceptual foundation, because global SHAP interpretations are combinations of the local SHAP explanations over many explained samples.

KernelSHAP is very slow though, so you probably have to compromise on the global explanation by computing them on a subset of the data set.

Background data set and computational complexity.

A common pattern is to provide the train dataset as the background data set and then generate predictions on the test set.

# explain all the predictions in the test set
explainer = shap.KernelExplainer(dtree.predict_proba, data=X_train)
shap_values = explainer.shap_values(X_test)
shap.force_plot(explainer.expected_value[0], shap_values[0], X_test)

I did not understand clearly what was the role of the data argument, which is explained as a "background data set". This is the description from KernelExplainer docs:

The background dataset to use for integrating out features. To determine the impact of a feature, that feature is set to “missing” and the change in the model output is observed. Since most models aren’t designed to handle arbitrary missing data at test time, we simulate “missing” by replacing the feature with the values it takes in the background dataset. So if the background dataset is a simple sample of all zeros, then we would approximate a feature being missing by setting it to zero. For small problems this background dataset can be the whole training set, but for larger problems consider using a single reference value or using the kmeans function to summarize the dataset. Note: for sparse case we accept any sparse matrix but convert to lil format for performance.

Some examples, like the one above, pass the whole training data set as a background data set, but this is prohibitively computationally expensive for anything other than small data sets. What happens if you pass the whole data set as a background data set e.g. to the KernelExplainer?

Computational complexity

To indicate the effect of the background data set size on the amount of model evaluations per explained instance, see the following example from the docs:

explainer = shap.KernelExplainer(f, X.iloc[:50,:])
shap_values = explainer.shap_values(X.iloc[299,:], nsamples=500)
shap.force_plot(explainer.expected_value, shap_values, X_display.iloc[299,:])

The background data set has 50 examples here. For each sample in the background dataset, the KernelExplainer will perturb each single input example 500 times for estimating the SHAP value. In this example, this leads to 50 * 500 = 25000 model evaluations. If that's a large neural network, this will take a while!

Choosing background data

In many coding examples, I find that people use the median for the background data set for the kernel explainer.

med = X_train_norm.median().values.reshape((1,X_train_norm.shape[1]))
explainer = shap.KernelExplainer(f, med)

Another alternative is all zeros:

# use Kernel SHAP to explain the network's predictions
explainer = shap.KernelExplainer(f, np.zeros((1,50)))
shap_values = explainer.shap_values(np.ones((1,50)), nsamples=1000) # runs VGG16 1000 times

So what's exactly the difference between provided a larger background data set and a simple vector of dimension 1 x n_features?

Important information is that KernelSHAP is the happy marriage between Shapley values and the LIME interpretation method. My current understanding of this combination is as such (this explanation assumes familiarity with Shapley values):

  • Sample $K$ binary vectors ${0,1}^M$ (see the nsamples parameter) where $M$ is the max length of the vector
    • Christoph Molnar calls these binary vectors "coalition vectors" to stay closer to game-theory.
    • Interpretation: 1 indicates the presence of a coalition member, 0 its absence.
    • Shapley values work with all permutations. SHAP does a sampling approximation, which is obviously more efficient and allows finetuning of efficiency vs. variance.
    • SHAP prioritizes binary vectors with very few 1s and many 1s (see the SHAP Kernel) and fills out nsamples in this order.
  • In practice $M$ will often be equal to the amount of features, but for example in image data the features could be image segments rather than individual pixels.
    • So if we have $M$ features, 1 indicates the presence of a feature, 0 its absence.
  • Generate model predictions on the coalition vectors, which requires a wrapper function h(x) that translates the binary coalition vectors to model input, i.e. the original feature space of the training/test data.
  • We now have a new data set with K coalition vectors and corresponding model predictions.
  • Fit a weighted linear regression model on this new data set
    • LIME puts more weighs on permuted instances that are closer to the example we are explaining
    • The SHAP kernel is a function that puts the most weight on coalition vectors with very few 1s and those with many 1s.
    • It's due to mathematical properties of this kernel that the outcome coefficients of the LIME approach are Shapley values.
  • The coefficients of the weighted linear model indicate the additive contribution to the model output (since the inputs are normalized to {0, 1}), in other words we have Shapley values (OK this requires more mathematical proof)

OK, so given this process, I'm starting to understand the role of the background data set. The key concept here is the function h(x) that maps the binary coalition vectors to the feature space of the trained model. How? This function maps the 1s from the coalition vector to the corresponding feature value from the instance that we are explaining. The 0s in the coalition indicate "missing" features, but since models don't run on missing features, by default a random feature value is sampled from another instance in the background data set.

So this is what happens when you provide a whole dataset as the background dataset to the explainer!

If you provide a single vector as the background data set, there's only a single value to sample and this will be used to generate a model prediction for the test instance.

For the 50*500 computation in the example above (from the docs) I'm trying to describe the process in natural language (I'm really not sure I understand this correctly):

  • for a single test explanation instance ...
  • you create nsamples binary vectors
  • for each of those vectors you fill in the missing values by sampling from the background data set, which results in multiple vectors (amount is the size of background data set)
  • we then want the expected model value $f(h'(z`)) = f(z)$ i.e. we need a prediction value for our binary vectors, so that we can train our weighted regression model
  • but we have missing values in $z'$. What's the expected value of this function given that we only know the 1s by definition of the simplified input, i.e. $E[f(z) | z_S]$?
  • We can solve this expectation under the assumption that we can sample from the conditional distribution $z_{\overline{S}} | z_{S}$ where $z_{\overline{S}}$ is the part of $z$ with missing features (0s in $z'$).
    • i.e. $E_{z_{\overline{S}} | z_{S}}[f(z)]$
  • The assumption of feature independence makes sampling easier. We can then instead sample from $E_{z_{\overline{S}}}[f(z)]$
    • This is where the background data set comes in! We can sample vector ${z_{\overline{S}}}$ (conceptually: filling in the 0s) by sampling a value for each of the features.
    • Again, the assumption of feature independence allows us to decompose the distribution whole vector into the marginal distributions for each feature.
  • I think it's at this point that, given samples for missing values from the background set, you can integrate out the missing features, which is also easier given the independence assumption. This amounts to computing the model prediction for $z$ as the expectation under sampling of the missing values, i.e. the weighted average prediction over the vectors with sampled missing values!
    • The more samples, the better your approximation of the marginal distribution for each feature will be.
    • When using a constant value, we have P(X=x)=1 which makes the process trivial. This is where a big speed up can be achieved, at the cost of approximation accuracy.
    • Is this why many examples use the median or mode for this constant value, since this tends to be the most likely feature value?
  • If no single constant value seems appropriate (e.g. for categorical variables), perhaps it's most suitable to "summary" the data set such that more likely values in the training data set will also be more likely sampled in the SHAP process.
    • You can be pretty creative in how to "summarize" your data set. Some examples run shap.kmeans to create a background data set.

For label encoded categorical variables, is it an idea to provide all categories once in the background data? This effectively means that the marginal distribution for that categorical variable is uniformly distributed, which it most likely is not, but at least it takes all discrete values into account during sampling instead of picking one arbitrary label as indicating "missing" (which it effectively is not).

Tweaking performance

Some explainers can efficiently extract the background data set from the fitted model, e.g. the TreeExplainer.

The KernelExplainer does not. The following remark is interesting: "Note that KernelExplainer does a sampling approximation for large values of M, but for small values it is exact." (Here $M$ is the length of the coalition vectors and is typically equal to the amount of features. If it's large it becomes prohibitively expansive to marginalize out the whole feature space of "missing values", but apparently if M is sufficiently low the KernelExplainer goes for standard analytical solution of Shapley values.)

For the latter, the following warning is interesting: "Using 120 background data samples could cause slower run times. Consider using shap.kmeans(data, K) to summarize the background as K weighted samples". The SHAP documentation mentions shap.explainers.Sampling as an alternative that does work with big background data sets because it samples this data set.

KernelExplainer is prohibitively slow with a large data set as a background data set, because then the whole "sampling + marginalizing out features" described above kicks in.

The solution is to either provide a background data set that is as small as possible while still "summarizing" / "approximating" the marginal distribution of the features in the training data.

See for example this interesting remark from this tutorial "In problems with more features we would want to pass only the median of the training dataset, or weighted k-medians." In these cases, the median or weighted k-medians are effectively a way to "summarize" the data set.

An obvious optimization is to limit the amount of samples. By default this scales with the amount of features plus a constant term. The constant term is relatively high if you have few features, which will probably exhaustively sample your distributions and lead to an exact solution. The shap_values argument accepts the nsamples kwargs:

Number of times to re-evaluate the model when explaining each prediction. More samples lead to lower variance estimates of the SHAP values. The “auto” setting uses nsamples = 2 * X.shape[1] + 2048.

Also, perhaps don't explain on the whole data set but on a certain amount of samples, e.g. 10000 samples. The question is, of course, how many samples is sufficient to get a stable explanation (this depends on the variance of the samples I guess). I don't have a fundamental answer for this, but you could determine this empirically with experimentation.

Dealing with categorical values

There's no support for categorical variables.

See discussion here.

Helpful answer in Github issue discussion

If you have a categorical variable that you one-hot encoded and you want to get a single variable for that categorical variable, then you can use a Shap utility called DenseData to group columns together under a single variable name. You can get a single SHAP value by summing the SHAP values of the dummy columns.

Code example:

from shap.common import DenseData

group_names = ['height', 'width', 'hair_colour']
# assume first two columns are the continuous variables and the next the enc. categorical var
groups = [[0], [1],  [2, 3, 4, 5]]
background_data = DenseData(my_data, group_names, groups)

NOTE I suspect shap.common is from an older package version. I haven't tried to run this code.

For label encoded categorical variables, is it an idea to provide all categories once in the background data? This effectively means that the marginal distribution for that categorical variable is uniformly distributed, which it most likely is not, but at least it takes all discrete values into account during sampling instead of picking one arbitrary label as indicating "missing" (which it effectively is not).

Plots

Waterfall

E[f(X)] is the average prediction over the whole (background) data set.

Waterfall plot for a single explanation list how each feature has contributed to the difference between the prediction f(x) and the expectation E[f(X)]. On the left you will see feature values.

Force plot

You can see this as a "condensed" version of the waterfall plot. You can use it for a single data point: shap.plots.force(shap_values[0]).

You can also create an interactive stacked force plot over many data points:

shap.plots.force(shap_values[0:100]).

This rotates the plot 90% vertically and stacks them horizontally so that the horizontal axis shows all samples.

You can also filter on a specific feature so that you see the force of that feature for all individual samples at once.

You can also filter which specific shap effect to display on the y-axis.

Bar plot (Absolute Mean SHAP per feature)

The shap.plots.bar(shap_values) show the absolute mean SHAP value per feature in a bar plot (mean over all samples).

In this way SHAP can be used as a type of feature importance.

Bee swam plot

shap.plots.beeswarm(shap_values)

The bee swarm plot visualizes all SHAP values per feature.

This summary plot is more informative than the mean SHAP value because this plot shows the distribution.

For example, you can start to see that higher values for a feature start having a disproportionate (in the sense of non-linear) large effect on the model output.

Dependence plots

If the bee swarm plot shows that some features might have a very non-linear relation with the target variable, then the dependency plot may be very useful.

There is no separate function for this, a dependence plot is basically a scatter plot of all shap values for a given feature.

shap.plots.scatter(shap_values[:, "feature_name"])

Scatter plot of SHAP values vs. feature values for a single feature.

Turns out, there is a default function (example; also shows how SHAP values help identifying a problem with KNN without normalization):

shap.dependence_plot("Education-Num", shap_values, X_valid.iloc[0:1000,:])

Interactions

By default, SHAP values are computed under the assumption of feature independence or, more precisely, for additive models.

By inspecting the bee swarm and dependence plot, you may find that this assumption does not hold.

In that case, have a look at Shap interaction values.