How People Explain Action (and Autonomous Intelligent Systems Should Too) | De Graaf & Malle
There is a need to make Autonomous Intelligent Systems (AIS) explainable.
"Our thesis is that people will regard most AIS as intentional agents and apply the conceptual framework and psychological mechanisms of human behavior explanation to them."
If we use this "folk psychology" and build it into AIS so that they can explain their own behavior, then:
1) "people are more likely to form correct mental models of such a system" 2) "and calibrate their trust in the system."
Explanation: (1) someone explains (2) something to (3) someone.
The success of an explanation thus also depends on the audience, i.e. you need to explain things in a so and so way for a particular audience.
This paper focuses on regular people as an audience.
The most critical audience factor is "the conceptual and linguistic framework within which people explain human behavior." So an AI that wants to be explainable needs to do so within that framework.
"Robots and virtual agents typically have numerous features that trigger human inferences of intentional agency (e.g., eyes, self-propelled movement, contingent interaction) (Johnson 2000; Premack 1990)."
Main claim:
"Our analysis will there-fore focus on AIS that people clearly treat as agents and that perform actions people consider intentional (e.g., making decisions, offering suggestions). For those intentional agents, we hypothesize, people will apply the same conceptual framework of behavior explanation that they apply to humans; and they will expect AIS to apply this frame-work as well."
If some systems do not trigger those intentional explanations, a purely mechanical framework suffices.
Human Conceptual Framework of Behavior Explanation
Folk psychology main concepts, that are closely related:
- Agent
- Intentionality
- Mind
What is hold to be an agent?
Objects that are:
- Self-propelled
- Behave contingently with the perceiver (non-predicability)
With a special sensitivity to face, gaze, motion, we quickly ascribe intentionality.
Intentional behavior is thought to be caused by key mental states.
It has been empirically shown humans tend to ascribe these three factors to AIS, and thus it is reasonable to hypothesize that they explain AIS behavior in these terms as well.
Making Sense of Autonomous Intelligent Systems
Humans use "human" models for AIS and this has risks:
"However, people’s mental models of AIS stem from their interactions with living beings. Thus, people easily run the risk of establishing incorrect or inadequate mental models of artificial systems, which could result in self-deception or even harm (Wortham, Theodorou, and Bryson 2016a)."
When robot behavior cannot be explained in terms of intentions, empirical research shows that people do not trust AIS decisions, or find robots creepy and untrustworthy.
Conversely, people can also over-trust be tricked into going along with faulty behavior when that seems to be intentional, due to an incorrect model.
"People sometimes over-trust artificial agents, such as when they comply with a faulty robot’s unusual requests (Salem et al. 2015) or follow the lead of a potentially dysfunctional robot (Robinette et al. 2016)."
This tendency is stable, is not bothered by knowledge of the robot:
" Moreover, a long-term study (Tullio et al. 2007) showed that initially established (incorrect) mental models of an intelligent information system remained robust over time, even when details of the system’s implementation were given and initial beliefs were challenged with contradictory evidence."
Ordinary Behavior Explanation
People divide events into intentional or non-intentional, and languages all over the world have linguistic constructs to express intentions.
How does ascribing intention work?
"Agents intentionally perform action A when they have an intention to A, have the skill to A, and have awareness of A-ing while A-ing. In addition, they form the intention to A on the basis of a de-sire for an outcome O and the belief that A leads to O. All other events are “unintentional,” lacking the critical role of an intention as mediating between the mind and the action (Heider 1958)."
How do People Explain Intentional Behaviors?
Intentional behavior is explained with reason explanations: reference to desires and (background) beliefs
Whereas unintentional explanations are given in causal terms.
This implies that AIS needs to explain their behavior in terms of reasons rather than causes!
1) Reasons emerge from a reasoning process that uses beliefs and desires (Cf. BDI architectures). Rationality to action. 2) Reasons have a subjective element, i.e. tied to what made sense for this particular agent, not another. * Subjective aspect is most obvious in explanations from false beliefs. * This does not imply a requirement of self-awareness.
Why are explanations sought for?
The either wonder themselves or expect someone else to wonder about some observed behavior.
"Case (a) leads to “private explanations,” and case (b) leads to “communicative explanations.”"
- Aware of the event (it is observable)
- Do not have the feeling that the event is understood
- Event is personally relevant.
If these conditions are met, AIS need to provide an explanation
Challenges for implementation
1) The language of explanation * Natural language * From the intentional stance 2) Distinct classes of behavior * Intentional vs unintentional 3) Select distinct types of explanation * Reason explanation vs cause explanation
BDI agents:
"The agent therefore needs to have a predictive forward model of the likely outcomes in the world if the intentional action succeeds; outcomes that don’t match the prediction are unintentional (either failures or undesired side-effects)."
Particular issues, how many reasons/causes are enough to explain something? There must be some sort of relevance criterion.
"In the psychological literature, this is sometimes called the “causal selection problem”—the difficulty of selecting a small number of causes/reasons that sufficiently explain the event in question."
"We have seen earlier that reason explanations are based on structural relations be-tween desires/goals, beliefs, and resulting intentional actions. These relations are often characterized as “practical reasoning arguments” (Walton 2015) and as conforming to a rationality principle (Malle 1999)."
One option is thus "filling in gaps" so to say of this structure.
But to know the content of these structures, we need a huge amount of common sense:
"Knowing that an agent had some desire that he believed could be fulfilled by some action does not satisfactorily explain that action; what that desire was is critical for understanding the action. The proper relationship among contents of beliefs, desires, and actions requires there to be a large knowledge structure of associative, social, and causal relations (the dreaded “common sense”)."