The responsibility gap: Ascribing responsibility for the actions of learning
automata | Andreas Matthias, 2004
This is a very good primer on the issue of a responsibility gap in contemporary AI systems. It does not offer a road to solutions, which is especially understandable given this is a 2004 paper.
Abstract:
Traditionally, the manufacturer/operator of a machine is held (morally and legally) responsible for the consequences of its operation. Autonomous, learning machines, based on neural networks, genetic algorithms and agent architectures, create a new situation, where the manufacturer/operator of the machine is in principle not capable of predicting the future machine behaviour any more, and thus cannot be held morally responsible or liable for it. The society must decide between not using this kind of machine any more (which is not a realistic option), or facing a responsibility gap, which cannot be bridged by traditional concepts of responsibility ascription.
Introduction
How do humans ascribe responsibility?
- should be able to offer an explanation of intentions and beliefs behind action
- person must be rightly subject to range of specific reactive attitudes (resentment, gratitude, censure, praise) cf. Strawson 1962, Oshana 2002: 263.
- person must have control over behavior and resulting consequences in a "suitable sense", see Fischer and Ravizza 1998:13.
Summarizing:
That means that the agent can be considered responsible only if he knows the particular facts surrounding his action, and if he is able to freely form a decision to act, and to select one of a suitable set of available alternative actions based on these facts. (175)
Responsibility w.r.t. machines:
- If the machine works according to the specification, responsibility lies with the operator to use the machine as specified.
- If not, responsibility lays with the manufacturer of the machine.
This shows the importance of control: the machine specification transfers responsibility from the manufacturer to its operator. If the machine does not work according to the specification, this transfer is invalid and consequently it is the manufacturer that has to take responsibility.
If there is a partial loss of control due to accidental circumstances, both outside of the control of the manufacturer and operator, there is a third option: nobody is blamed and costs of an incident are collectively shared.
This sets the scene for the main conflict addressed in this paper:
In the following sections of this paper we will see how certain recent developments in the way of manufacturing computerised, highly adaptive, autonomously operating devices, inevitably lead to a partial loss of the operator’s control over the device. At the same time, the degree in which our society depends on the use of such devices is increasing fast, and it seems unlikely that we will be able or willing to abstain from their use in the future. Thus, we face an ever-widening responsibility gap, which, if not addressed properly, poses a threat to both the consistency of the moral framework of society and the foundation of the liability concept in law. (176)
Applications of Learning Automata
Problems specific to learning automata:
- An accident may occur due to a decision made not by an explicit command, nor by a preprogrammed algorithm, but partly based on data that was learning by a machine in the course of its operation.
Example: adaptive elevator that leaves director waiting for 30 min, so that he is late to a very important meeting. Financial damage. Programmer will deny responsibility: undesirable behavior caused because elevator adapted parameters in the learning process.
Interesting remark here: you can often offer mathematical proofs about convergence to an optimal solution! That slightly changes things, whether such a proof can be completed, and has been done before putting the machine out in the wild.
Another example: algorithm for lung cancer detection is designed to prevent false negatives, which would be life threatening. As a consequence, the change on false positives is raised. Who is responsible for false positives?
Remark in footnote 7 about the question: who can be held responsible?:
But even where no legal liability can be identified (for example in German law), it is possible to ask who should (on moral grounds) be considered responsible for the damage done by the machine. (176)
Once again, the issue of the responsibility gap:
Now it can be shown that there is an increasing class of machine actions, where the traditional ways of responsibility ascription are not compatible with our sense of justice and the moral framework of society because nobody has enough control over the machine’s actions to be able to assume the responsibility for them. These cases constitute what we will call the responsibility gap. (177)
Types of relevant learning automate and their engineering
We can distinguish four primary types of learning automata which are of interest in the present context: symbolic systems, connectionist architectures (including reinforcement learning systems), genetic algorithms (including genetic programming), and autonomous agents (mobile and immobile).
Some notes:
The rules of symbolic systems can be inspected and corrected, and properties can be verified.
Neural networks are, owing to the principles of their architecture, black boxes. We can evaluate their behaviour by applying test patterns and observing their output, but we cannot
– have a look at the information that is stored inside the network, and, even more importantly; – see what information is not represented inside it. (178-179).
The latter point refers to the fact that these networks learn by example and that the learned representations are not understandable nor meaningful for humans.
Footnote 16:
All the social procedures that humans and animals have developed to evaluate other individuals (rituals of introduction, teaching by demonstration, examinations and psychological tests) are attempts to solve this problem: that we cannot look into the heads of others and see the knowledge, intentions and beliefs stored therein. Instead we are forced to infer them indirectly through observation of the other’s behaviour and the application of test patterns.
Reinforcement learning: required in highly dynamic environments.
Very interesting remark. For the first time, errors are a feature of the systems design:
And since learning is done by trial and error, we find that errors are unavoidable features of any reinforcement learning system. This is quite contrary to the common, traditional understanding that technology, done correctly, must operate free of errors, and that errors are always the errors of the programmer, not of the programmed machine. Reinforcement learning presents us, for the first time, with the necessity of errors as we know it from living systems: as a system feature, the precondition for learning and adaptive behaviour, and not merely a product flaw.
Genetic algorithms and genetic programming are also mentioned.
W.r.t. autonomous agents.
- "First, that agents are per definitionem designed to act, and that, in the course of their operation, they must inevitably interact with other things, people, and social entities (laws, institutions and expectations)." (180)
- "Second, that agents which have a physical presence constitute a new category of machines: such that can learn from the direct interaction with a real environment and that can in return directly manipulate this same environment." (180)
Responsibility gap
Research question:
To what extent is it possible to hold the manufacturer/ programmer/operator of an autonomous, learning automaton responsible for the actions of the machine? (181)
Development of programmers increasingly losing control over program execution (181-182). From coder to creator:
- Coder, traditional software, full control.
- Logic-programming, declarative, no control over execution flow
- Artificial Neural Networks, no symbolic representation of information, no control flow, resistant to interpretation.
- Reinforcement learning: same issues, but also takes away distinction between training and production phase, even lower predictability, errors cannot be avoided but are integral to learning.
- Genetic programming. The code itself is machine-generated, another layer of obscurity.
- Could also be an internet agent, alongside autonomous vehicles and military targeting systems etc.