Meaningful Human Control over Autonomous Systems: A Philosophical Account | Santoni de Sio and Van den Hoven, 2018

Introduction

However, policy-makers and technical designers lack a detailed theory of what “meaningful human control” exactly means. In this paper, we lay the foundation of a philosophical account of meaningful human control, based on the concept of “guidance control” as elaborated in the philosophical debate on free will and moral responsibility.

Following the ideals of “Responsible Innovation” and “Value-sensitive Design,” our account of meaningful human control is cast in the form of design requirements. We identify two general necessary conditions to be satisfied for an autonomous system to remain under meaningful human control: first, a “tracking” condition, according to which the system should be able to respond to both the relevant moral reasons of the humans designing and deploying the system and the relevant facts in the environment in which the system operates; second, a “tracing” condition, according to which the system should be designed in such a way as to grant the possibility to always trace back the outcome of its operations to at least one human along the chain of design and operation. (1)

We then briefly introduce the distinction between incompatibilist and compatibilist theories of moral responsibility, and we explain why we consider the compatibilist approaches to moral responsibility most suitable to ground a theory of meaningful human control over autonomous weapon systems. (2)

Autonomous systems and the problem of meaningful human control

"In the legal–political debate on autonomous weapon systems of the past few years, these ethical concerns have been synthesized in the following principle:

Principle of meaningful human control future weapons systems must preserve meaningful human control over the use of (lethal) force, that is: humans not computers and their algorithms should ultimately remain in control of, and thus morally responsible for relevant decisions about (lethal) military operations. (Article 36, 2015)

This principle has attracted a wide consensus among scholars and policy-makers (Knuckey, 2014; Article 36, 2014; Horowitz and Scharre, 2015; Ekelhof, 2017), as “it offers more precision (control versus the somewhat ambiguous conceptual “loop” or the more passive “judgment”), it explicitly emphasizes the quality of control (“meaningful”), and it implicitly accords responsibility to human agents for decisions concerning each individual attack” (Vignard, 2014: 3)." (whole quote on page 3)

But what does meaningful human control exactly mean?

There is a need to define clear guidelines.

Toward a theory of MHC

David Mindell 2015: 10: move away from myths about full autonomy and look at situated autonomy in real systems. (3)

Goals of the paper:

  • Define control in terms of "guidance control" (Fischer and Ravizza 1998), and hope to extend the discussion to other autonomous systems such as self driving cars.
  • Propose concrete guidelines and ethical guidance to police makers.

Methods:

  • Responsible Innovation
  • Value-sensitive design
    • or "Design for Values"
    • Naast Van den Hoven 2007 and 2013, see Friedman and Kahn, 2003.

Three main points of innovation:

1) Control * To what extent do humans remain morally responsible for the behavior of intelligent automated systems? 2) Compatibilism: individuals retain responsibility een if their actions can be explained in causal or even deterministic terms. * Addresses the challenges from contemporary neuroscience * But has not been explicated clearly with respect to actions mediated by relatively autonomous systems * "May maintain control and moral responsibility for actions mediated by the use of robotic intelligent systems with high levels of autonomy." * E.g. Dennett 1984, Vincent 2013 3) Clear utility: "we demonstrate how one might design for meaningful human control, thereby extending the approach of value-sensitive design in ethics of technology to a new domain of cases." (4)

MHC overcomes three obstructions, also pointed out by Van den Hoven 2013.

  1. Keeping a human in the loop is not a sufficient condition for being in control. See paper or Van den Hoven 2013 for details.
  2. "controlling in the sense of being in the position of making a substantive causal contribution to a (military) activity through one’s intentional actions might not be a sufficient condition for meaningful control either, for instance, if one does not have the psychological capacity to respond appropriately under the circumstances and/or they are not in the position to appreciate the real capabilities of the system they are interacting with." (4)
  3. Forms of criminal responsibility require more than simple causal control over events, but stricter control conditions such as "knowledge, intention, capacity and opportunity" (4). This may be hard and unfair to enforce on individual humans, part of the overall complicated system.

How to avoid accountability gaps?

  • Commanders should receive appropriate training, see Saxon 2016.
  • Programmers should be made aware of their moral responsibility, Leveringhaus 2016.

What is the relation between control and moral responsibility?

Incompatibilism:

Incompatibilists believe that humans can be in control of and morally responsible for their actions if and only if they possess a special power to make decisions and carry out actions that escape the causal influence of genetic (neuro)biological, sociopsychological, and circumstantial factors. (4)

With this position, you can go two ways.

You can either depart from the given that humans are nevertheless morally responsible, and as a consequence, one must posit a contra-causal power and autonomy to make decisions; this metaphysical position is defended by what is called in this article libertarians in a Kantian tradition. E.g. Van Inwagen 1983.

On the other hand, you can deny free will and consequently hold that from a philosophical viewpoint no individual can be morally responsible. This free will scepticism can be grounded in various things, including general metaphysical causal determinism, neurobiological reductionism, etc.

Compatibilism:

humans may be morally responsible for (some of) their actions even if they do not possess any special metaphysical power to escape the causal influences on their behavior. (5)

Traditional compatibilism, such as Hobbes and Hume:

According to traditional compatibilism in order for agents to be morally responsible their actions need to be free only in the sense of being the causal product of internal motivational factors—desires, intentions, traits of characters, values—as opposed to being the product of “external” forces, i.e., the product of physical or psychological coercion. (5)

Present-day compatibilists:

"while also rejecting the idea of strong contra-causal free will as a necessary condition for moral responsibility, rely on a more complex view of human mind and action." (5) Namely that mental causation is not a sufficient ground for moral responsibility. There is such a thing as rational control on actions (while not denying these actions have causes, I presume).

The authors argue that these present-day compatibilists offer the best hopes for a philosophical foundation of MHC. (cf. 5)

Unlike libertarians, the delegation of decision-making to "non-human agents" does not necessarily imply a full loss of moral responsibility and autonomy.

Unlike free will skeptics, they do recognize a difference between "human actions and other natural events"

Unlike traditional compatibilists, the rational control over actions is not simply causation but then from within (the causal effects of desires and intentions).

Approach of this article:

Approach = "guidance control" (Fischer and Ravizza, 1998) + Nozick's theory of knowledge.

"The ideas developed here apply equally to automated decision-making on the basis of advanced AI working via Machine Learning in a data environment."

Conditions for Guidance Control

Two conditions:

"the decisional mechanism leading up to X should be

(1) “moderately reason-responsive” and (2) the decisional mechanism should be “the agent’s own.” " (5)

Reason responsiveness

  • Agent can recognize strong reasons to act or not to act
  • and bring himself to act or not to act in a "sufficiently broad range of circumstances"

The point of this definition is to distinguish situations in which actors can be exempted of moral responsibility due to excusing factors, such as brain manipulation, brain lesions, neurological disorders, phobias etc.

In other words, in those cases that person's decisional mechanism is not responsible (enough) to moral reasons.

Example given (5): an agoraphobic that does not run out in the street to help someone in need, even though that person might recognize that as the morally right thing to do.

This is not a point about the extent to which an agent can manipulate external or internal (motivational) circumstances, cf. 5. "Simply" concerns the flexibility of the relevant decisional mechanism.

Decisional mechanism should be integral part of agent

The agent considers and recognizes the decisional mechanism as its own.

Cf. this seems to be the core of the tracking condition.

That requires:

1) observing the effects of actions in the world 2) and that that elicits moral reactions in others 3) and "the views specified in the first two conditions are based on the agent’s evidence in an appropriate way" (5); somewhat enigmatic formulation.

But the example the authors give is clear:

If you consciously decide to settle an important matter by the toss of a coin, you are aware that you have decided to rely on what you perfectly know to be a random decision-making system and you have to accept that others will hold you accountable for the outcome of the choice (you cannot blame the system). (5-6)

The same does not hold for decisional mechanisms that an agent does not hold as his own.

Tracking and tracing conditions

Fischer and Ravizza's theory of "guidance control" concerns "intra-personal decision mechanisms, i.e., the mechanisms of an individual human mind or brain" (6).

To be applicable to control autonomous systems, these systems must be considered as a part of the relevant decision making mechanisms.

In other words, the approach taken here point back to theories of extended, embedded and embodied cognition, cf. Clark and Chalmers, 1998; Alfano, 2013. I.e. the idea that "human decisions and actions and even human personality traits are not represented in any brain mechanism" (6). This description is general, and I suspect very inprecise.

In any case, the crux remains the same:

in fact, if we consider autonomous (weapon) systems as part of the decision-making mechanisms through which human agents carry out actions in the world, then Fischer and Ravizza’s conditions for guidance control on everyday actions may provide the basis for an account of meaningful human control over (military) actions mediated by autonomous (weapon) systems. (6)

N.B. mediated by autonomous systems. Cf. Verbeek's Materializing Morality.

Tracking

Cf. Fischer and Ravizza's reason responsiveness:

What seems required is that behavior of the system (the human operators, and the complex system, including interfaces that support decision-making) covary with moral reasons of a human agent for carrying out X or omitting X. (6)

+

a so-called tracking relation between human moral capacities to respond to relevant moral reasons and (military) systems actions. (6)

Cf. Nozick's "truth-tracking" account of knowledge.

Nozick’s definition is known as a truth-tracking account of knowledge, because mental states in human minds should track the states of affairs in the world for them to constitute knowledge, in pretty much the same way a so-called tracker fund at the stock market just follows what the stock exchange is doing and the mercury column in a thermometer tracks the temperature in the room in a lawlike manner. (6)

Has four conditions (6):

(i) P is true (Federer is the Wimbledon champion). (ii) S believes that P (I believe that Federer is the Wimbledon champion). (iii) If it were not the case that P, then S would not believe that P (if Federer was not the Wimbledon champion, I would not believe that he is the Wimbledon champion). (iv) If it were the case that P, then S would believe that P (if Federer was the Wimbledon’s champion, then I would believe that he is the Wimbledon’s champion).

Uses the "subjuncitve conditional", if it were the case.

Interesting remark: probabilistic and connectionist models are not excluded from the tracking condition due to the fact that you cannot program in such a subjunctive conditional (which you could do with symbolic AI). See:

By characterizing tracking in terms of subjunctive conditionals, it is not implied that those forms of AI and machine-learning tools that are probabilistic in nature would not be per se eligible candidates for satisfying the tracking relations. It just means that whatever its nature or functioning, the system should be able to respond to the world’s features in a satisfactory way; this also means that, in practice, we may need to set a reasonable threshold for judging “how much” reliability in the system responding to the relevant features of the world qualifies as “tracking” for a particular purpose. (6)

In the case where a decision mechanisms is extendend an included both a human actor and an autonomous system, we require a dual-tracking relation.

1) System should allow human agent's beliefs to track the relevant states in the world 2) System should "also track the (relevant) moral reasons of the relevant agents deploying the method and it should effectively implement them." (7)

How does this look like? Given a scenario where there is a military target, but there might be children present:

If the moral reasons of the human agents were different (there were no children after all, but instead young men) in a morally relevant sense (they were carrying explosives, and so they could be in principle considered as a legitimate target of a military attack), the mechanism would accommodate them so as to justify a change of plans, then the system would change its behavior accordingly. If on the other hand the world were different in a morally relevant sense, the mechanism would also track those altered states of affairs and represent them accordingly. (7)

So the system tracks both the environment for morally relevant changes and informs the human actor adequately - based on which the human actor might have different moral reasons for various possible actions - and the system is responsive to the moral reasons of the human operator, and includes them somehow in the proposed plan.

So the definition of the tracking condition is:

First necessary condition of meaningful human control. In order to be under meaningful human control, a decision-making system should demonstrably and verifiably be responsive to the human moral reasons relevant in the circumstances—no matter how many system levels, models, software, or devices of whatever nature separate a human being from the ultimate effects in the world, some of which may be lethal. That is, decision-making systems should track (relevant) human moral reasons. (7)

This requirement is indeed necessary:

Systems that do not display such a twofold epistemic and moral tracking reason-responsiveness, no matter how efficient they may be in performing specific tasks and even in achieving some broad and morally worthy human goals, would not qualify as being under meaningful human control. They would be like human actions carried out under the influence of potent drugs, phobias, or neurological disorders: a behavior that is clearly under the causal control of the human mind, but not under the right kind of rational control that grounds moral responsibility.(7)

Examples

When systems misrepresent the state of affairs, human moral reasoning will be bound to be erroneous. The article gives three very clear examples:

1:

A famous example would be the Goalkeeper system used by the British Navy in the Falkland war, which misclassified an incoming Exocet rocket as friendly (i.e., French), when it was in fact deployed by the Argentinian enemy. In this case, the system demonstrated its inability to track some morally relevant state of affairs: a rocket being “friendly” as opposed to just being of the kind usually used by allies; and because of that it was also unable to track the relevant moral reasons of the human commanders: targeting enemy rockets rather than just targeting rockets with certain material features. (7)

2:

More recently, the use of machine-learning systems has allegedly led to misclassification of enemy and friendly tanks because the training set had many images of enemy tanks with clouds and many of friendly tanks with cloudless skies or tracking higher or lower resolution (Yudowsky, 2006). These systems were also tracking irrelevant properties in the training set.

3:

It eventually learns how to do that and unfailingly classifies new pictures as either huskies or wolves. It turns out, however, that the system just looks at presence of snow in the background, since the majority of wolves’ picture had snow in the background, whereas the husky photos did not. (8)

Here the subjunctive conditionals of the tracking condition clearly show their value. If, in the last example, that system would be presented with a dog in the snow, it would have classified it as a wolf due to the snow in the background. cf. Ribeiro et al. 2016 So this classification system tracks an irrelevant property.

These are relevant conditions, given this example:

" (i) When presented with a wolf, the System classifies object as “Wolf” (ii) If it were to be presented (in a great variety of possible worlds, e.g., snow and green vegetation) with a non-wolf (e.g., husky), then it would not classify the object as “Wolf” but as “Husky” or “non-Wolf.” (iii) If it were to be presented (in a great variety of possible worlds) with a wolf, then it would classify it as “Wolf.” (8) "

N.B. The "tracking" relation only specifies this tracking of moral reasons, but does not protect against malintent of the human operator! If there is malintent, however, it would be clear who is morally to blame, given that the tracking condition holds.

The only absolutely hard constraint is that the system tracks a human agent and not another artificial system.

However, the tracking condition is also not completely morally neutral. For a given system, it must be decided what the relevant moral principles, norms, values is supposed to track/reflect. This is obviously a normative exercise that will lead to difference in assessing whether the tracking condition applies.

Tracing

The idea of tracing is often used by moral responsibility theorists and according to Manuel Vargas tracing “is one of a few things to which nearly all parties in the debate about free will [and moral responsibility] appeal to with equal enthusiasm” (Vargas, 2005 quoted by Timpe, 2011). (8)

This is the core of the concept of tracing:

The concept of tracing tries to capture the basic intuition that a human agent may be responsible for an outcome even if she does not satisfy the conditions for responsibility in situ at the time of her action, provided that she was responsible at an earlier time for finding herself later in the position of not satisfying those conditions. (8)

Classical example: a drunk driver causes an accident while intoxicated at a morally incapable moment, but obviously bears full moral responsibility because of the prior decision to drink before driving. The incapacitation was anticipated.

Another example:

Jim is presented with a dilemmatic choice by the captain to select a prisoner to be shot. If he refuses then all prisoners will be shot. According to Kamm, the threat originates with (and therefore the responsibility traces back to) the captain, not with Jim. The captain is the choice architect, who creates Jim’s dilemma in which there are only tragic choices to be made by Jim. (8)

The main challenge is to extend this tracing condition to sitatuations with

1) more human agents 2) the involvement of non-human (sub)systems in the final outcome (8)

Suggestion of Van den Hoven of how tracing could apply to for example a plane with automated subsystems, e.g. CAS for automatic collision avoidance.

According to Van den Hoven the pilot is supposed to check his work environment before he enters into it and is supposed to establish whether it will allow him to do what he ought to do. Or to have it inspected by others on his behalf. If he fails to do so when he can and finds himself later “locked-in” in a system’s environment that does not allow him to discharge his obligation to avoid a collision, then he may be held morally responsible for a negative outcome as his responsibility can be traced back to his failing at an earlier moment to do a proper check. (9)

And:

In case it turns out that conditions for operator or user first-order moral responsibility are not satisfied as a result of no fault of the operator, then the responsibility for the accident may trace back to other agents upstream, or to the designers of the system, who fail to exemplify proper understanding of the system. (9)

In other words, those in the loop here have a "meta-task responsibility" (9)

an obligation to check whether the system is responsive to the dynamic moral reasons of relevant moral agents that obtain and that apply to them in situ (van den Hoven, 1998). They have an obligation to check whether the system allows them to do what they ought to do in situ. (9)

Accordingly, there is a responsibility for designers to facilitate the analyses required for such a meta-task responsibility.

The tracing condition is formulated as follows, based on Fischer and Ravizza's "ownership condition, which requires the agent has to properly understand and endorse the mechanism of moral decision-making leading to her action." (9)

Second necessary condition of meaningful human control: in order for a system to be under meaningful human control, its actions/states should be traceable to a proper moral understanding on the part of one or more relevant human persons who design or interact with the system, meaning that there is at least one human agent in the design history or use context involved in designing, programming, operating and deploying the autonomous system who (a) understands or is in the position to understand the capabilities of the system and the possible effects in the world of the its use; (b) understands or is in the position to understand that others may have legitimate moral reactions toward them because of how the system affects the world and the role they occupy.

If the tracing condition does not hold, then: "They would be like human actions carried out under psychological manipulation, subliminal persuasion, brainwashing, and indoctrination; here, the agent’s behavior is clearly responsive to someone’s reasons, but not to the agent’s reasons." (9)

Important note (cf. 11): the tracing condition needs to be satisfied in one agent, and not in a distributed manner, i.e. it cannot be the case that one involved human actor is aware of responsibility for a subpart of the decision system, whereas another feels responsible for another subpart. There must be at least one agent who is able to recognize the limits and capabilities of the system and simultaneously realizes this implies responsibility for the final outcome of the decision proces.

MHC, tracking, tracing implications on autonomous weapon systems

Current systems do not satisfy the tracking condition, and thus require human supervision.

(Detailed discussion provided in the article).

A clear obstacle for the tracing condition is that making decisions from a distance through computer systems can lead to a reduced sense of responsibility.

Important nuance:

So, tracing as we defined it is not only meant, as it were, to help finding someone to blame after an accident has occurred; tracing is much more than that. Tracing is an essential component of meaningful human control over a system, because it requires that there always are individual humans along the chain who are capable and motivated to take active steps to prevent unwanted outcomes to occur in the first place. (10)

Broader applications of MHC

Claim and ambition: MHC "can be one of the central notions of thinking about Responsible Innovation in robotics and AI." (11)

Two design implications for the tracing condition:

1) "we need to understand what the ideal distribution of tasks between humans and robots is, from a functional point of view; for instance, in a vehicle equipped with assisted cruise control, which driving operations should be delegated to the computer and which should remain with the human driver." (12) 2) we "need to engage in a social and psychological investigation to understand under which circumstances the human drivers are in practice able and motivated to do their part when requested; in the case of assistive cruise control, this means acquiring sociopsychological data to assess the reasonableness of the normative expectations attributed to the driver to perform certain tasks and supervise certain operations; it may also mean filling possible psychological gaps by an appropriate design of new systems of training and licensing for users." (12)

Very concrete recommendation. The required transformation is way more than a legalistic story about attributing liability. The bigger moral story requires:

In this perspective, in order to implement the tracing condition, to enhance meaningful human control, and to reduce the risks of “responsibility gaps” we not only need—as it is often claimed—to design appropriate new normative systems, for instance, new legal rules for attributions of liability in the event of accidents involving autonomous systems. We also need to design educational and training systems to improve the understanding of the functioning of these systems and the risks and responsibilities associated with designing and operating them. (12)