Rationale Discovery and Explainable AI | Cor Steging, Silja Renoij, Bart Verheij 2021 Legal Knowledge and Information Systems

Abstract:

The justification of an algorithm’s outcomes is important in many domains, and in particular in the law. However, previous research has shown that machine learning systems can make the right decisions for the wrong reasons: despite high accuracies, not all of the conditions that define the domain of the training data are learned. In this study, we investigate what the system does learn, using state-of-the-art explainable AI techniques. With the use of SHAP and LIME, we are able to show which features impact the decision making process and how the impact changes with different distributions of the training data. However, our results also show that even high accuracy and good relevant feature detection are no guarantee for a sound rationale. Hence these state-of-the-art explainable AI techniques cannot be used to fully expose unsound rationales, further advocating the need for a separate method for rationale evaluation. (225)

The crux of ML in AI & Law: "These systems had high accuracy scores, but they performed well for the wrong reasons, as their decision making was unsound." (226)

Goal: "Knowing whether a system learned what it is supposed to learn is essential in pursuing responsible AI. In this paper we investigate whether state-of-the art explainable AI techniques provide relevant insight into the evaluation of the rationale of a machine learning system." (226)

Background

Domain & data set: welfare benefit domain from from "Bench-Capon T. Neural networks and open texture. In: Proceedings of the 4th International Conference on Artificial Intelligence and Law. ICAIL 1993. ACM, New York; 1993. p. 292-7."

Receiving of benefit depends on 6 conditions that are formalized. A neural network was trained on artificial data sets that were created using these conditions. The question is then: did the neural network "internalize" the six conditions?

Rationale evaluation was based on a previous paper: Steging C, Renooij S, Verheij B. Discovering the rationale of decisions: Towards a method for aligning Learning and reasoning. In: Proceedings of the 18th International Conference on Artificial Intelligence and Law. ICAIL ’21. ACM, New York; 2021. p. 235-9.

This rationale evaluation method designs test sets such that the neural net can only perform well when it has learned a "particular element of the rationale".

For the welfare benefit domain, to measure how well any given condition Ci is learned, a dedicated test set is created in which every condition is satisfied except for Ci. The values of the features that define Ci are then generated across their full range of values. That way, the eligibility in this dedicated test set is determined solely by condition Ci: condition Ci is satisfied iff the instance is eligible. Networks are only able to classify the instances from this dedicated test set correctly if they have learned condition Ci. (227)

RQ is investigated using a simple multi-layered perceptron. This MLP is trained on two types of data sets that are designed for the following two descriptions to hold:

  • Type A: ineligible instances may fail due to multiple conditions and "it is possible to achieve a theoretical accuracy of 98.95% while only knowing four out of the six conditions that define the domain" (227)
  • Type B: "ineligible instances only fail due to a single condition." (227)

These data sets have 52 noise variables and 12 relevant variables. Including the noise variables means that the network should learn to discriminate which variables are relevant from those that aren't.

Previous results: when training the NNs on type A data set, the performance on a type B dataset drops significantly, but drops to chance level when testing on the test sets that require the learning of a single rationale (Age-Gender condition; and Patient-Distance condition). This shows that when training on the more realistic type A data set, the network fails to learn the underlying rationale despite having very high accuracy on the type A test set.

However, when training on the type B data set, performance is better across the board on all data sets.

XAI & Results

The question now is: are XAI methods - SHAP and LIME - useful for assessing whether a neural network has correctly learned a rationale in situations where the accuracy is high?

SHAP explanations are local but additive, meaning you can assess the effect of features over the whole data set by adding the effects of individual instances to assess the average impact per feature. This doesn't work for LIME, which only works locally i.e. on instance level. The study (manually?) evaluated 120 LIME instances (10 each for 12 configurations) and results did not contradict the SHAP (global) summary.

Main finding:

Based solely on the SHAP and LIME explanations, we would not be able to know that the rationale is unsound for the networks trained on a large type A dataset. Therefore, even though the XAI methods can expose a faulty rationale, they cannot guarantee a sound rationale. Previously research claimed that it is possible to make the right decisions without knowing why [19,20]. These results expand upon that notion, and suggest that it is also possible to know what is important without knowing why. In other words, we find that systems that have both a high accuracy and assign high importance to the correct features are still not guaranteed to use a sound rationale. (233)

In some cases the SHAP results showed that not all relevant features were important for the classifications, which in this case is a good thing because it means SHAP shows that the model does not use the correct rationale (which was already established). But particularly when using a very large (type A) data set, the SHAP results show all relevant features as having an impact, even though previous results have shown that the correct rationale was not used.

Conclusion:

However, for some conditions, the XAI methods did not detect an unsound rationale when the rationale was known to be unsound. Neither high accuracies, nor acceptable explanations from XAI techniques therefore can guarantee a sound rationale. This finding further supports the need for a rationale-evaluation method in order to obtain responsible AI. (233)

To read:

  • Górski Ł, Ramakrishna S. Explainable artificial intelligence, lawyer’s perspective. In: Proceedings of the 18th International Conference on Artificial Intelligence and Law. ICAIL ’21; 2021. p. 60-8.
  • Mumford J, Atkinson K, Bench-Capon T. Machine learning and legal argument. In: Proceedings of the 21st Workshop on Computational Models of Natural Argument; 2021. p. 47-56.
    • "emphasizes that explaining the outcome using a list of important input features is insufficient in a legal setting" (226)