What makes human control over weapon systems “meaningful”? | Daniele Amoroso, Guglielmo Tamburrini, 2019

Executive summary

First, AWS would be unable to comply with the principles of distinction, proportionality, and precaution embedded into International Humanitarian Law (IHL). Second, AWS are likely to result in an accountability gap. Third, AWS would run counter to the principle of human dignity. These claims provide guidelines to distinguish perfunctory from truly meaningful human control, by pinpointing functions that one must prescriptively assign to human control. (2)

What are the functions that humans must perform to ensure MHC over weapon systems? Both ethically and legally:

First, human control must afford a fail-safe mechanism, preventing IHL breaches from occurring. Second, it must serve as a catalyst for accountability. Third, it must ensure that it is a moral agent, and not an artificial one, that makes decisions affecting the life, physical integrity and property of people. (2)

Argues against a one-fits all solution, that is either too permissive or too restrictive. Report gives various examples of both.

In favor of differentation:

The application of these overarching principles in concrete situations must be facilitated and given concrete operational content by the formulation of a set of rules. These rules take the form of “if-then” rules expressing the fail-safe, accountability, and moral agency conditions for exercising in context a genuinely MHC over weapon systems. (3)

Levels of automation, cited:

  1. "L1. A human engages with and selects targets, and initiates any attack;"
  2. "L2. A program suggests alternative targets and a human chooses which to attack;"
  3. "L3. A program selects targets and a human must approve before the attack;"
  4. "L4. A program selects and engages targets, but is supervised by a human who retains the power to override its choices and abort the attack;"
  5. "L5: A program selects targets and initiates attack on the basis of the mission goals as defined at the planning/activation stage, without further human involvement."

L1 and L2 can reasonably be exerted. L5 should be always exempted for MHC.

The authors sketch minimal requirements for a MHC Protocol.

Point 4 especially interests me. Control by design:

The Section on “Control by design” must crucially include provisions prescribing that the design and development of weapon systems always comply with interpretability and explainability requirements: to achieve situational awareness for MHC purposes, human operators should be put in a position to get a sufficient amount of humanly understandable information about machine data processing (interpretability requirement), and to obtain an account of the reasons why the machine is suggesting or going to take a certain course of action (explainability requirement)

Main article

Main arguments related to necessary conditions for MHC that are not fulfilled.

  1. "First, AWS would be unable to comply with the principles of distinction, proportionality, and precaution embedded into International Humanitarian Law (IHL). The development of AWS fulfilling distinction and proportionality requirements at least as well as a competent and conscientious human soldier presupposes the solution of many profound research problems in artificial intelligence (AI) and advanced robotics" (7)
  2. "AWS are likely to determine an accountability gap.20 One cannot exclude that AWS will make targeting decisions that, were they taken by human agents, would trigger individual criminal responsibility. Who will be held responsible for this conduct?" (7)
  3. "AWS would run counter to the principle of human dignity, which would dictate that decisions affecting the life, physical integrity and property of human beings involved in an armed conflict should be entirely reserved to human operators and cannot be entrusted to an autonomous artificial agent." (7-8)

These arguments are dominantly deontological.

From this three functions are at least required:

First, human control must afford a fail-safe mechanism, which is meant to prevent a malfunctioning of the weapon from resulting in a direct attack against the civilian population and objects, or in excessive collateral damages.24 Second, it must serve as a catalyst for accountability, insofar as it secures the legal conditions for responsibility ascription in case a weapon follows a course of action that is in breach of international law.25 Third, it must ensure that it is a moral agent, and not an artificial one, that takes decisions affecting the life, physical integrity and property of people (including combatants) that are involved in an armed conflict.26 To be ethically and legally sound, therefore, rules aimed at determining the obligations flowing from the MHC requirement should guarantee that all these functions are properly fulfilled.