Explainable AI Lawyer's Perspective | Gorski 2021
User evaluation on XAI is lagging behind in general; also specifically in the legal domain. This is a pilot user evaluation study of Grad-CAM, LIME and SHAP in the legal domain.
- "assesses the performance of different explainability methods (Grad-CAM, LIME, SHAP), in explaining the predictions for a legal text classification problem; those explanations were then judged by legal professionals according to their accuracy. " (60)
- what are the desired qualities of a legal decision aid system w.r.t. explainability
- and what is the general understanding of XAI by legal experts?
Introduction
Motivation: "Due process of law and fair trial requirements make an explanation of AI-based decision models a must. However, the legal and technological discourses pertaining to the creation of explainable AI seem to be separated. In particular, the methods used to create explainable systems seem to favour the needs of AI-engineers." (60)
Related work
Because XAI for law is connected to regulatory frameworks on the use of AI, it is important that AI experts and legal experts engage in dialogue. But there's
Case-based reasoning (has been a main focus in AI & Law) is more focused on common law tradition. Should XAI for the civil law tradition move in a similar direction?
As far as the legal perspective is concerned, in [7] it was noted that the judiciary will play an important role in the production of different forms of XAI, especially because judicial reasoning, using case-by-case consideration of the facts to produce nuanced decisions, is a pragmatic way to develop rules for XAI [7]. Whilst this way of thinking sounds very attractive and reminds how seminal works on case-based-reasoning in AI originated in the AI&Law community [24], on the other hand, the cited passage clearly alludes to common law judge mode of thinking and civil law judge may be more concerned with general rules. (60)
The general tenet of the discussed work in this paper is that decisions requiring explainability in the legal domain should simply not use machine learning. Rule-based and case-based, prevalent in this fielt, are inherently interpretable.
Conceptual framework for a user's mental model of an AI system providing an explanation (p. 61):
- Understanding of the domain
- Knowledge of the AI system's implementation
- knowledge of how an explanation strategy works with the system
Methods
Tested methods:
- Grad-CAM (gradient based)
- "The class activation map gives information on how strongly the particular tokens present in the input sentence influence the prediction of the CNN" (p.3)
- SHAP (shapley value based)
- additive, so can provide feature importance scores both with local and global consistency
- LIME (local surrogate)
- local only
... for a CNN-based text-based classification pipeline.
Task is text classification on sentence level.
The PTSD data set offers annotated sentences from 50 decisions by the Board of Veteran's Appeals of the US Department of Veterans Affairs: "Finding Sentence", "Evidence Sentence", "(Evidence-based) Reasoning Sentence", "Legal-Rule Sentence", "Citation Sentence", "Other Sentence".
All XAI methods are able to highlight the importance of input tokens. Evaluation is done both qualitatively (judgement by legal experts) and quantitatively:
- fraction of elements above threshold; measures how much of the input string is considered
- intersection over union between two binarized heatmaps
- This is to see to which degree two XAI methods agree on which parts of the input are important
Qualitative evaluation is only done on correctly classified sentences, because the heatmap of incorrectly classified sentences is harder to understand. I think this is a conceptual mistake. Consider the following remark from Rudin 2019:
Demonstrating a method using explanations only for the correct class is misleading. This practice can instill a false sense of confidence in the explanation method and in the black box. Consider, for instance, a case where the explanations for multiple (or all) of the classes are identical. This situation would happen often when saliency maps are the explanations, because they tend to highlight edges, and thus provide similar explanations for each class. These explanations could be identical even if the model is always wrong. Then, showing only the explanations for the image’s correct class misleads the user into thinking that the explanation is useful, and that the black box is useful, even if neither one of them are. (4-5)
Results and conclusion
See paper for various observations on user survey. Several of these observations seem to be contradicting each other.
Essentially, the metrics vary quite a bit where user evaluations are relatively stable. Also some users score a method with the maximum score whereas others give a very low/minimum score. Maybe some quantitative report on user agreement would have been appropriate here.
Also, I find the conclusion to be a bit too inconclusive:
In this paper, we have presented a comparison of different explainability methods when applied to legal classification neural network (CNN) and provided an assessment of explainable artificial intelligence as understood by lawyers. Different XAI methods were graded similarly by the users, though certain variances can be spotted. The metrics presented herein offer software engineers an option to quantify explanations given by the system. When a more general point of view is taken, one which comes from prospective users, it should be noted that the lawyers are generally looking forward to the implementation of (explainable) artificial intelligence systems and solutions that allow them to be more efficient in their use of time. Users are thus waiting for results of AI - and in particular XAI research. (65)