Ethics and self-driving cars a white paper on responsible innovation in automated driving systems | Filippo Santoni de Sio
Methodology: Responsible Innovation and Value-Sensitive Design
Main concept: meaningful human control
Relevant factors: individual rights to safety, access to mobility, privacy.
Three levels of ethical issues that need to be integrated
1) Operational: how should ADS be programmed to deal with difficult situations 2) Tactical: How should road traffic be regulated? (E.g. interaction of ADS with traditional vehicles) 3) Strategic: How should the broader societal impact of ADS be anticipated, guided, regulated.
Methodology
Responsible innovation
Cf. Van den Hoven 2013 on "Responsible Innovation"
Ethics needs to be pro-active
We must prevent a situation where there is a disconnect between abstract moral discussions and the real world of engineering and policy. The basic idea is that ethical constraints and aims should become the very shapers of innovations at the relevant point in time and a place, where they still can make a difference, instead of fuelling political and academic discussions after the fact. (6)
This has two requirements:
1) Interdisciplinarity 2) Anticipatory analysis
Value-Sensitive Design
Use moral values as requirements in the design of technologies.
Added requirements, to the ones already mentioned:
3) Systematic ethical analysis * Identify as many relevant values as possible 4) Embed values in design * This requires a translation step of values into guidelines for design.
Beyond ethical dilemmas
Much research on ADS tries to show how ethics are involved in ADS by focusing on tragic futuristic dilemmas. These do not however provide an ethics of ADS that applies to non-futuristic and current scenarios, with semi-automatic ADS, mixed traffic, multiple programmers with no specific individual in full control etc.
Also: how do we come from one specific ethical issue to a general framework?
1) Do not take futuristic scenarios for granted. They might not realize. 2) Include broader societal and normative issues, not only specific dilemmas 3) Take a pro-active attitude in trying to prevent moral dilemmas instead.
Ethical Issues with ADS
Main question:
how can designers and policy makers comply with the moral obligation to make traffic safer, more sustainable and efficient via ADS while at the same time avoiding unreasonable risks, maintaining human accountability and respecting competing vital interests and individual rights like the right to life and physical integrity and the right to privacy.
Consider the following components:
Safety
- There may be a moral obligation to introduce ADS to increase safety and reduce travel accidents due to human error
- While avoiding the introduction of unpredictable risks to human life
Meaningful Human Control and Responsibility
Advice: always maintain meaningful human control. What is that?
Either an appropriate design for a:
1) Partial automation system 2) Supervised automation system
The option is left open whether meaningful human control can be achieved with a fully automated system.
These designs should be made in parallel with legal innovations for avoiding responsibility gaps.
Rights
- Right to physical integrity (the "new victims" problem)
- Right to access basic transport services
- Right to privacy
The transition towards automation
Advice is to move away from futuristic scenarios where cities are swarmed with fully automated vehicles.
Instead, focus on realistic short-term transition scenarios:
1) Dual mode vehicles / Partial Automation * Automated mode only in suitable designated areas 2) Supervised single mode "pods"; low speed and separated from other traffic * Transition towards mixed traffic under human supervision (possibly remote)
These thus both involve some form of human control. How do we design these systems and under what conditions do we use them?
Ads, ethical issues, design challenges
All concerns related to "killer robots" (AWS) should be taken into account, since even though ADS are obviously not designed for injuring or killing people, this could still be the result of ADS on the road.
1) It may be morally wrong to give a technical system full control over potentially lethal activities 2) The use of ADS might lead to responsibility gaps.
What is Meaningful human control?
1) Having a human "in the loop" is not a sufficient condition for meaningful control. * Due to lack of information * Due to limited control over all relevant routines * Due to the control being irrelevant (a human pressing a button reflexively without thinking) 2) Simply having a causal influence does not yet guarantee the process is redirected in a way that is morally right. So causal influence is perhaps not a sufficient condition either. 3) Different forms of legal responsibility assume different levels of causal control over events
W.r.t. 3:
Thirdly and relatedly, whereas some forms of legal responsibility (tort liability, strict liability) require only that the agent have relatively simple forms of causal control over events, other forms of legal responsibility (typically criminal responsibility) usually require stricter control conditions of knowledge, intention, capacity and opportunity; therefore no matter how strong the political will to keep some human responsible or accountable for the behaviour of autonomous systems, attributions of (criminal) responsibility that are not grounded in the relevant control conditions may turn out to be not only morally unfair but also ultimately difficult to enforce in tribunals.
Meaningful control is not necessarily direct
Direct control over all aspects of the operating system is not required.
Instead, it is argues that meaningful human control is close to the concept of ecological control. That is a more top-down exercise of control over a system without micro-managing specific components. In daily life we also control actions by distributing tasks to various unconscious mechanisms or tools.
As applied to the behaviour of ADS the idea of ecological control entails that: a human agent A may be in control of an action performed by an autonomous system S provided that S is part of a system Y whose general functioning is guided by the moral and practical reasons of A; in particular, the distribution of tasks to different parts of the system is such as to allow the system to be responsive to the relevant moral and practical aims of A (cf. Di Nucci and Santoni de Sio 2014, Santoni de Sio and van den Hoven manuscript).
Summary:
1) A human is in control over an action taken by an ADS 2) if the ADS is part of a system guided by the moral and practical reasons of that human. 3) This assumes that tasks are distributed in the ADS such that the overall system is responsive to the relevant moral and practical aims of the controlling human.
(Cf. my idea from a few years back for a "morality knob" )
The overall ADS system includes, apart from the vehicle, the larger technical infrastructure and all relevant social and legal institutions.
In other words, the delegation of tasks can preserve human control, under these conditions.
Main take-away
The white paper:
1) explores partial and supervised automation. 2) and investigates under which conditions meaningful human control is preserved.
Partial Automation
Under meaningful human control (clear division of control conditions)
1) Clear division of tasks * only automated mode in special conditions, e.g. well mapped roads 2) In both modes the ADS responds to motivations of some human; either the driver, or the designers and road planners. 3) This results in a relatively clear division of responsibility attribution
Under shared control
If the human driver and the ADS share control over the vehicle, some issues might arise:
1) Trust calibration * Humans can overestimate the capacity of the vehicle and overdelegate * Or underestimate, and keep in control where it's safer to give away control 2) Meaningful human control is not guaranteed. * "the human driver is not equipped with the motivational capacity to realize this distribution of tasks (for instance the driver tends to use the “autopilot” mode also when not recommended)." (15) 3) Bad design of the transition between machine and human control. * E.g. give control away too quickly so the driver does not have time to adjust and be appropriately aware of all circumstances.
Technical and normative challenges
1) Understand what the ideal distribution of tasks is * Technically * Socially/psychologically: ensure that the human is motivated enough
Technical:
1) Design a transparent and "accountable" interface * Clear picture of circumstances and what driver is supposed to do, and how
Normative:
1) How much freedom should a user have in complying with indications from the system? * Can the ADS forcibly override control? (vs individual freedom) * Neutral inform to leave control? (not motivating enough?) * Nudging techniques
Supervised automation
Under human control
When pods are on designated tracks and supervised, then human and ADS share control in a way, which caused issues with partially automated ADS.
However, the human control is now done by trained professionals under remote control, which takes away time and emotional pressure.
Potential insights from civil aviation
Civil aviation could be very relevant because it combines elements relevant to ADS:
- Human control onboard
- Automated pilots
- Remote traffic control
Full automation
Inappropriate ethics of trial-and-error; Risk of machine learning
Current machine learning approaches need to "learn" from many examples before reaching a desired level. However, this method is not socially responsible, because we cannot train ADS in real life situations if that desired level is not already guaranteed. (So, we have a typical "cold start" problem).
Using machine learning poses a risks: machines could learn from anything and from anyone. (E.g. chatbot turning racist).
This also makes it possible to manipulate ADS systems by abusing their learning mechanisms.
Mixing intelligences
In real life scenarios ADS need to deal with humans with various driving styles and mental states.
Various trade-offs, e.g.:
- Follow traffic rules to the letter
- Conflicts with dynamic, reasonable and flexible adaptation to human drivers
- Instead make ADS learn from actual human driving
- See risk of machine learning above.
Non-verbal human communication
How do ADS communicate with regular human drivers? Can ADS comply e.g. with police signalling you to stop?
1) Implement message system 2) Human drivers instead need to adapt to the presence of and interaction with ADS
Special Testing Zones for Responsible Innovation
One possible answer to the "cold start problem" of ADS in real-life situations is to offer designated and controlled zones in real cities for testing techniques that have already proved safe under "laboratory conditions".
Responsibility gaps
"circumstances in which a serious accident happens and nobody can be reasonably held responsible or accountable due to the unpredictability or opaqueness of the process leading to the accident." (20)
"the American judge LJ Hale once wrote, cars are “potentially dangerous weapons” (Eagle v Chambers), whose reckless or negligent use sometimes amounts to a crime." (20)
Depending on the distribution of control, different humans can be held responsible, but in any case some human must be held responsible to avoid a responsibility gap.
Tort liability: who pays for damages?
Three legal issues:
1) Law does not see ADS as agents but as tools 2) The driver/owner is not always responsible in ADS 3) "liability for road accidents concern also “extra-contractual” third parties, that is parties not bound by any contractual relationship with the owner/driver" * E.g. an unknown pedestrian * I think this is insufficiently explained
When is someone liable?
1) Fault liability * Direct cause of damage by irregular behavior 2) Strict liability * E.g. like a parent is liable for a child * Includes product liability, responsibility for a defective product.
Applied to ADS, currently law would only consider fault liability and product liability.
Issues:
1) Fault liability is fair but does not cover situations where malfunctioning was unpredictable. * User relatively happy * Plaintiff not so happy 2) Strict liability: * Plaintiff is happy * Huge burden on programmers and designers * Discourages innovation
Open Question: what is we consider ADS as agents instead, and apply responsibility like a parent for its child.
Similar to the issue for machine learning of having limited real-life data, is the issue for legal experts of having no legal precedents to make up laws.
Insight from Roman Law
"In The Human Use of Human Beings (1950), the father of cybernetics, Norbert Wiener, wrote that “the automatic machine … is the precise equivalent of slave labor.” (cited by Pagallo 2013: 102)." (23)
"This comparison may be appropriate also from a legal point of view, since in ancient Rome slaves were considered as things that still could perform some social activities. The peculium was a limited amount of money assigned to these slaves to do some activities on behalf of their master. It aimed to strike a balance between the claim of the masters not to be dilapidated by their slaves’ businesses and commercial activities (as the slave responsibility was limited by the amount of their available peculium) and the interest of the slaves’ counterparties to safely transact with them (Pagallo 2013: 103). It has been suggested that a “digital peculium” may be created to give robotic agents, including ADS, a limited capacity to pay for damages that cannot be traced to any fault or liability of their human “masters”." (23)
This suggestion is somewhat comic to me, and very typical for a legalistic mindset.
Rights
Utilitarian cost-benefit analyses (CBA) seems very appropriate for ADS, but has limits that should be taken into account.
There are some deontological constraints that should be applied, and which are in there core non-utilitarian:
1) Respect moral duties 2) Respect individual rights 3) Ensure a fair distribution of benefits over different social groups
"New Victims"
Will introduction of ADS change the distribution of risks across different societal groups? E.g. increase the chance at incidents for pedestrians, whereas the overall death toll would be lower? Is it okay to knowingly increase the incident change for pedestrians?
Possible objections based on:
1) Normative positions of involved actors (rights and responsibilities) 2) Long-term effects
The right to not be intentionally injured w.r.t. public authority
it is usually legally forbidden to intentionally injure or kill an innocent person P in order to save other people A, B, C from a threat which is not related to the behaviour of P. This prohibition is grounded in the individual right to life and physical integrity. So, it may be thought that this prohibition applies also to the developers of a policy on road traffic (Santoni de Sio, manuscript): a reduction of the losses of motor vehicle drivers may not be achieved, for instance, via an expected increase in the losses among pedestrians or cyclists. On the other hand, unlike private citizens, the public authority does have the legitimate power to decide to penalize one specific group of citizens in order to pursue what it considers to be the “public good”, so that looking at individual rights may not be enough to decide on this issue (Christie 1999). (25)
Special responsibilities of private motor vehicles users
ADS have a higher potential for causing losses to third parties in case of a crash, despite being on average safer.
"Therefore, the enhanced ability for crash avoidance brought by artificial intelligence may put an even higher duty of care on those who want to introduce new vehicles on the public road." (26)
"Spirit" and long-term effects of ADS policies
"We should avoid the paradoxical scenario in which the presence of ADS on the road is perceived as threatening the safety of those who do not use motor vehicles, as this scenario might eventually create an incentive to use motor vehicles also for those who would be able and willing to not do so; and so reducing (or even eliminating) one or more of the beneficial goals of the policy." (26)
Access to mobility
Even though ADS would be very nice for example for disabled people, it is a question how accessible ADS will be, e.g. economically (too expensive) or technically (not all users can operate it).
Privacy and Data Protection
ADS will record much data of behaviour of vehicles and their drivers/passengers, including others around the car through various sensors and cameras.
- ADS can be target of hacking
- Collection and storage of personal data might threaten moral autonomy 1) Creates information asymmetry: passengers vs those controlling transport infrastructures. 2) Creates power imbalance, if the government starts using this information (despite claiming it's for safety)