Evaluating XAI: A comparison of rule-based and example-based explanations | Van der Waa, Nieuwburg, Cremers, Neerincx, Artificial Intelligence 291 (2021)
Abstract
there is an overall lack of valid and reliable evaluations of the effects on users’ experience of, and behavior in response to explanations. New XAI methods are often based on an intuitive notion what an effective explanation should be. Rule- and example-based contrastive explanations are two exemplary explanation styles. In this study we evaluate the effects of these two explanation styles on system understanding, persuasive power and task performance in the context of decision support in diabetes self-management. (1)
Findings:
- "rule-based explanations have a small positive effect on system understanding" (1)
- "both rule- and example-based explanations seem to persuade users in following the advice even when incorrect" (1)
- " Neither explanation improves task performance compared to no explanation" (1)
- Probably because of the local scope, instead of global scope where underlying rationale is explained
Introduction
Premise: explanations are user-focused but nevertheless there is a large lack of user evaluation in the field of XAI.
Recommendations w.r.t. user evaluations:
1) "how to construct a theory of the effects that explanations are expected to have" (2) 2) "how to select a use case and participants to evaluate that theory" (2) 3) "which types of measurements to use for the theorized effects" (2)
This paper evaluates two types of contrastive explanation:
- rule-based
- example-based
- no explanation (control condition)
Two experiments:
- Effect on system understanding
- Effect on persuasive power and task performance
- Note that a system's explanation might be wrong yet persuasive
- "Task performance is defined as the decision accuracy of the combination of the system, explanation and user." (2)
These are all (intermediate) factors in trust calibration.
Hypothesis:
- Rule-based is expected to improve system behavior because the decision boundary is explicitly given, leading to better task performance
- "we expected participants to be able to identify the most important feature used by the DSS in a given situation, replicate this feature’s relevant decision thresholds and use this knowledge to predict the DSS’s behavior in novel situations" (2)
- Rule-based expected to have little persuasive power because they are very factual and provide little information about its correctness
- Opposite effects for example-based
Background
Goals of XAI:
- system understanding
- better predictability of behavior
- calibrate system trust
- system debugging
- verification
- justification
These goals are often not explicit, while they may impose very different requirements on explanations.
- Many papers acknowledge that explanations are user-centered, but do not actually evaluate on this user aspect
- The papers that do typically focus on subjective measures:
- user satisfaction
- goodness of explanation
- acceptance of system advice
- trust in system
- What is missing is how these subjective aspects correlate with behavioral effects; i.e. these things measure user belief but not the improvement w.r.t. understanding, trust or a specific task.
- One study that does measure behavioral effects, measures whether an explanation helps making inference
- Needed: construct validity and reliability.
- Construct validity: do you accurately measure the intended constructs (e.g. trust)?
- Reliability: internal consistency and reproducability of results
Two types of contrastive explanation:
Contrastive explanations can vary depending on the way the advice is contrasted with a different advice, for example using rules or examples. Within the context of a DSS advising an insulin dose for DMT1 self-management, a contrastive rule-based explanation could be: “Currently the temperature is below 10 degrees and a lower insulin dose is advised. If the temperature was above 30 degrees, a normal insulin dose would have been advised.” This explanation contains two rules that explicitly state the differentiating decision boundaries between the fact and foil. Several XAI methods aim to generate this type of “if... then...” rules, such as [35–38]. (4)
An example-based explanation refers to historical situations in which the advice was found to be true or false: “The temperature is currently 8 degrees, and a lower insulin dose is advised. Yesterday was similar: it was 7 degrees and the same advice proved to be correct. Two months ago, when it was 31 degrees, a normal dose was advised instead, which proved to be correct for that situation”. Such example- or instance-based explanations are often used between humans, as they illustrate past behavior and allow for generalization to new situations [39–42]. Several XAI methods try to identify examples to generate such explanations, for example those from [43–47]. (4)
Interesting observation:
Most of the existing research focused on exploring how users preferred a system would reason, by rules or through examples. For example, users prefer an example-based spam-filter over a rule-based [48], while they prefer spam-filter explanations to be rule-based [49]. (4))
Recommendations for setting up user studies
- On defining constructs and relations
- On picking a use case and setting up the experimental context
- On choosing appropriate measurements for the constructs
Useful, but see paper.
Use case
Simplified task for a real-world application, which allows the study to be done with healthy patients who don't have prejudices about diabetes as well:
In this use case, both the advice and the explanations are simplified. This study therefore falls under the human grounded evaluation category of Doshi-Velez and Kim [4]: a simplified task of a real-world application. The advice is binary (higher or lower), whereas in reality one would expect either a specific dose or a range of suggested doses. This simplification allowed us to evaluate with novice users (see Section 6.3), as we could limit our explanation to the effects of a too low or too high dosage without going into detail about effects of specific doses. (6)
Another main simplification is that the explanations did not follow from XAI methods but were hand-made.
There are several limitations caused by these two simplifications. First, we imply that the system can automatically select the appropriate foil for contrastive explanations. Second, we assume that the XAI method is able to identify only the most relevant factors to explain a decision. Although this assumes a potentially complex requirement for the XAI method, it is a reasonable assumption as humans prefer a selective explanation over a complete one [2]. (6)
2 -> Miller (2019) - Explanation in AI insights from social sciences
Constructs, expected relations, measurements
Constructs:
- System understanding (intermediate construct for trust)
- Recall what system considers important factor: "capacity to correctly identify the decisive factor of the situations in the system's advice" (7)
- Accurateness of user mental model: "ability to accurately predict the advice in novel situations" (7)
- Self-reported perceived system understanding to assess over- or underestimation
- Persuasive power
- "number of times participants copied the advice, independent of its correctness"(7)
- Task performance (intermediate construct for trust)
- Number of correct decisions
- Self reported perception of the correctness in predicting the advice
Checking confounds:
- usability questionnaire
- perceived system accuracy
"The combination of self-reported and behavioral measurements enabled us to draw relations between our observations and a participant’s own perception. Finally, by measuring a single construct with different measurements (known as trian- gulation [65]) we could identify and potentially overcome biases and other weaknesses in our measurements." (7)
Experiments.
Skipped; see paper
Discussion
Both methods do not really help participants predict system behaviour (globally):
First, rule-based explanations indeed seem to allow participants to more accurately identify the factor from a situation that was decisive in the system’s advice. However, rule-based nor example-based explanation allowed participants to learn to predict system behavior. The rule-based explanations however, did cause to participants to think that they better understood the system compared to example-based and no explanations. The example-based explanations only showed a small and insignificant increase in perceived system understanding. It is important to note that there was no correlation between the self-reported measurement of understanding and the behavioral measurements of understanding. This shows that participants had a perception of understanding that differed from the understanding as measured with factor identification and advice prediction. (15)
- Both contrastive methods had more persuasive power than the control, but example-based not significantly more persuasive
- In both experiments, no correlation between reports of participant's self-perception of predicting advice correctness and the actual amount of correct decisions
- These measures seem to measure different constructs
- Which could imply that people are not good at self-assessing their capability in this regard
- "Two common styles of explanations answering a contrasting question did not appear to increase task performance, an effect often attributed to such explanations within the field." (16)
- This is honestly a bit of a bombshell! This suggests that all these XAI methods which makes claims, are really not that effective when the actual user is involved in the evaluation.
Conclusion
Summary of results:
Results showed that contrastive rule-based explanations allowed participants to correctly identify the situational factor that played a decisive role in a system’s advice. Neither example-based or rule-based explanations enabled participants to correctly predict the system’s advice in novel situations, nor did they improve task performance. However, both explanation styles did cause participants to follow the system’s advice more often, even when this advice was incorrect. This shows that both rules and examples that answer a contrastive question are not sufficient on their own to improve users’ understanding or task performance. We believe that the main reason for this is that these explanations lack a clarification of the underlying rationale of system behavior. (17)