Human behaviour with automated driving systems: a quantitative framework for meaningful human control | Heikoop, Hagenzieker, Mecacci, Calvert, Santoni de Sio, Van Arem, 2019.

Introduction

Automation levels: SAE standards.

Issues with the human factors of ADS:

  • out-of-the-loop problems (Kaber and Endsley, 1997)
  • mode confusion (Stanton and Marsden 1996)
  • behavioral adaptation (Rudin-Brown and Parker)

Most of these have to do with a transfer of control between the human driver and the ADS. So we need a safe and meaningful way transfer of control that avoids these issues.

The notion of meaningful human control (MHC) comes from political debates on autonomous weapon systems. See: http://www.article36.org/weapons/weapons-review/autonomous-weapons-meaningful-human-control-and-the-ccw/

Meaningful control ideally for all forms of control (cf. Michon 1995, think this is also addressed either in Van den Hoven 2013 or Santoni de Sio))

1) Operational 2) Tactical 3) Strategical

Santoni de Sio and Van den Hoven (2018): two conditions for MHC.

1) Tracking condition:

The first condition, ‘tracking’, regards the capacity of a system to always respond to human reasons to act or refrain from acting. In order to be under meaningful human control, a system should always be able to adjust its behaviour in accordance to a human controller’s (or designer’s) intentions. (2)

2) Tracing condition

The second condition, ‘tracing’, prescribes the presence of at least one human agent (e.g., driver, operator, system designer, etc.) that can both (1) fully understand the real capabilities of the system and (2) wilfully bear the moral consequences of the system’s behaviour (Santoni de Sio and Van de Hoven 2018).

The study of human behavior is crucial for realizing these conditions.

  • W.r.t. tracking:
    • To what extent can and should drivers express their reasons to act in relation to an ADS? (Cf. 2)
    • How would that communication look like?
  • W.r.t. tracing:
    • " behavioral science is important to determine the quality and quantity of the understanding required from a controller regarding the system’s workings"

To get those into sight, there is a prior requirement, namely that we know how to "assess what behaviour is involved in driving a vehicle (and with ADS), from a human-oriented perspective" (2)

The taxonomy of Rasmussen (1983) is used.

The theoretical assumption of his approach is that humans are:

1) goal-oriented 2) thus no input-output systems that strictly adhere to given commands, such as computers 3) and accordingly, have reasons for their behavior (actions have a particular meaning)

These are also compatible with the theoretical assumptions of the tracking condition. The tracking condition assumes similarly the intentionality of human behavior (BDI etc.), and then requires a correspondence between human intentionality and ADS behavior.

A qualitative approach to assessing human behavior is as of yet missing. Method: formulate such a method through literature review.

Once a quantitative method for assessing driving behavior from a human perspective is available, this can serve further research into the influence of ADS on human driving behavior. This can inform particular interventions to ensure MHC.

(Example that I can think of right now: you need to find ways in which people stay engaged, otherwise you can't assume them to be able to take over control at any point)

Development of MHC framework for ADS

Methods:

Combine:

1) SAE taxonomy * SAE International. 2018. Taxonomy and Definitions for Terms Related to Driving Automation Systems for on-road Motor Vehicles. Washington, DC: SAE International. * https://www.sae.org/standards/content/j3016_201806/ 2) Classification of human behavior by Rasmussen (1983)

Into a 6x3 grid with 18 fields.

Each field is filled with the amount of driver tasks that are subject to that field. This is based on

  • literature research
  • inferential reasoning in case literature does not exist yet.

SAE levels of automation

See https://www.sae.org/standards/content/j3016_201806/

N.B. this section is cited from the paper (p.3-4), but I added list syntax!

  • Level 0: ‘The performance by the driver of the entire [dynamic driving task], even when enhanced by active safety systems’.
  • Level 1: ‘The sustained and [operational design domain]-specific execution by a driving automation system of either the lateral or the longitudinal vehicle motion control subtask of the [dynamic driving task] (but not both simultaneously) with the expectation that the driver performs the remainder of the [dynamic driving task]’.
  • Level 2: ‘The sustained and [operational design domain]-specific execution by a driving automation system of both the lateral and longitudinal vehicle motion control subtasks of the [dynamic driving task] with the expectation that the driver completes the [object and event detection and response] subtask and supervises the driving automation system ’.

N.B. from here on I gave up on italicizing everything. Refer to the original paper.

  • Level 3: ‘The sustained and [operational design domain]-specific performance by an [automated driving system] of the entire [dynamic driving task] with the expectation that the [dynamic driving task] fallback-ready user is receptive to [automated driving system]-issued requests to intervene, as well as to [dynamic driving task] performance-relevant system failures in other vehicle systems, and will respond appropriately’. Level 4: ‘The sustained and [operational design domain]-specific performance by an [automated driving system] of the entire [dynamic driving task] and [dynamic driving task] fallback without any expectation that a user will respond to a request to intervene’.
  • Level 5: ‘The sustained and unconditional (i.e., not [operational design domain]-specific) performance by an [automated driving system] of the entire [dynamic driving task] and [dynamic driving task] fallback without any expectation that a user will respond to a request to intervene’.

Rasmussen's classification of human behavior

  • Skill based behaviour: "acts or activities which take place without conscious attention or control, and which is automated and highly integrated" (4)
  • Rule-based behaviour: "Rule-based behaviour is defined as routinely executed acts or activities that follow a stored rule or procedure, often from instruction or preparation." (4)
    • N.B. difference with skill is dependent on level of training and required amount of attention; unconcious vs explicit recollection of facts.
  • Knowledge-based behavior
    • "Knowledge-based behaviour is the performance of an act or activity during unfamiliar situations, and is goal-controlled. Here, a person needs to plan his/her actions, evaluate those, and consider the best response by functional reasoning. Usually, this is done by selecting from (parts of) previous similar experiences (and thus other rule- or skill-based behaviour), and piecing together a novel reaction to a novel situation." (4)

Filling in the framework (p. 4-9)

For a detailed description of how many road rules are relevant for skill-, rule- and knowledge based behavior, see the paper.

The general trend is that with increasing level of automation increasingly less skill- and rule based capabilities are required.

Interestingly, with increasing automation (up until the unknowns of SAE 4+), the amount of required knowledge does not decrease but rather increase. That's a really interesting result that emphasizes the importance of a good interaction between human driver intentionality and ADS.

Some citations:

SAE level 1-2.

Lastly, the introduction of novel systems, such as advanced driver assistance systems, inadvertently introduces novel situations. Thus, in contrast with skill- and rule-based behaviour, these systems will add drivers knowledge-based behaviour more than they take over. Although little is known about what situations may occur when driving automated vehicles, several knowledge-based behaviours are expected to be requested by driving with such systems, such as coordinating, cooperating and collaborating with the activated system, but also understanding the distribution of tasks between the driver and the system, as well as knowing when it is safe to engage in secondary tasks (Spulber 2016). (7)

SAE level 3

W.r.t. Increase in knowledge-based behavior: "Within this stage, driver skill- and rule-based behaviour may already deteriorate to knowledge-based, adding up to a driver’s required knowledge-based behaviour." (5)

Up to SAE level 3 automation, the SAE defines that the human driver is expected to serve as a fall-back to perform the dynamic driving task in case of an emergency, like a system malfunction (SAE International 2018) (see Table 1, bold line; see also ‘Updated: Autonomous driving levels 0 to 5: Understanding the differences’, 2016). This means that for all these levels (SAE levels 0 to 3), the human driver is expected to be able to perform as if they were driving a manual vehicle. Given the considered ironies of automation discussed above, this appears to be misplaced. (8)

How can you verify the reduction of knowledge-based behavior due to deterioration?

Implications

Decline of skill- and rule based behavior

Decline of skill and rule based behavior can be expected, because increasing automation implies those skills and rule are applied less and less. Skills that are not used tend to deteriorate.

There are some paradoxical situations here.

To counter such deterioration of skills, the human driver would need to practice them. But this defeats the whole point of why we are automating those related tasks.

It seems that automation replaces based skills and rule-based behavior. But those are the ones that humans are perfectly capable of doing themselves.

Driver assistance systems thus seem to assist mostly where people do not really need assistance. (e.g. accelerating)

Rise and fall of knowledge-based behavior

Until and up to SAE actually an increase in required knowledge-based behavior.

The introduction of advanced driver assistance systems appears to introduce more novel situations than they dissolve. Behavioural changes such as, but certainly not limited to, becoming complacent and having to supervise an automated system will have to be accounted for in order to ensure safe driving with such systems (e.g., Brookhuis, De Waard, and Janssen 2001; Lindgren and Chen 2007; Stanton and Young 2005). (10)

SAE 3 requires human intervention option.

To what extent a request to intervene requires knowledge-based behaviour is yet to be determined, but quick regeneration of awareness of the situation at hand is considered to be one of the requirements (e.g., De Waard et al. 1999; Eriksson and Stanton 2017; Lu, Coster, and De Winter 2017). (10)

The human driver as a fallback mechanism should be reconsidered

Reclaiming control for SAE3 might be futile

  • "the majority of skills, rules and knowledge necessary for safe driving have not been mobilised in this time" (10)
  • Given more time, these required skills are likely deteriorated
  • Behavioral adaptations might make meaningful control unlikely
    • Carryover effects with platooning
    • Loss of task engagement

SAE 4 and 5 are the big unknown

Likely no human role whatsoever anymore.

Unclear what a meaningful fallback is in case of systems failure.

For a vision on what the future might bring:

Chipchase, J. 2014. Twelve concepts in autonomous mobility. https://medium.com/studio-d/ concepts-in-autonomous-mobility-80732bc4a44d

Chipchase, J. 2016. Driver behaviours in a world of autonomous mobility. https://medium.com/studio-d/15-more-concepts-in-autonomous-mobility-8fd1c794e466

Implications for meaningful human control

Following is a paragraph quoted in full, with interspersions by me. All text on page 12.

The presented framework provides an assessment of the human component of the human– machine interaction with ADS, presenting a quantification of human behaviour in relation to driving tasks and levels of automation. Its contribution to the study of meaningful human control is twofold. Firstly, it helps in understanding the extent to which humans (driver, operators, etc.) are able to express certain reasons through behaviour. This understanding is of paramount importance in designing systems that are meant to respond to human reasons to remain under meaningful human control.

This is actually not that clear to me. How did this paper show that?

Secondly, by providing an all-encompassing list of human behaviours during (automated) driving, the framework contributes to determine the extent to which humans can understand an ADS’s mechanisms.

They do indeed offer an extensive list of relevant behaviors, but is it really addressed to which extent humans can understand ADS mechanisms?

This is vital to assess whether and to what extent the tracing condition is fulfilled, namely (in this case) whether some human driver/controller can legitimately be held responsible for the behaviour of the system, and is sufficiently aware and willing to endorse this responsibility.

The tracking condition is not fulfilled:

On the one hand, it is clear from the presented framework that the current transition of levels of automation does not fulfil the tracking condition, since the covariance between the human driver’s intentions and the ADS behaviour is violated over the various levels of automation.

This seems to be the reasoning:

Especially for SAE level 3 automation, the ADS requires more from the human driver than the human driver can intend, let alone perform.

The tracing condition is more a matter of a good transition of responsibility onto the ADS rather than control, because control becomes increasingly less relevant for higher SAE levels. If the driver has little to no control, this transition may imply a shift of responsibility to other people such as supervisors:

On the other hand, the framework clearly shows that in order for the tracing condition to be met, a transition of responsibility is required—rather than a transition of control —since with higher levels of automation it becomes more important how the ADS is designed to execute its tasks and less important how the human driver ought to execute its tasks. Since drivers’ knowledge of the system and their capacity to perform tasks in general tend to decrease at higher levels of automation (see Table 1), other people ought to be considered more suitable to fulfil the tracing condition. Rather than the driver, designers or possibly control room operators might be good candidates for a meaningful role as controllers and thus bearers of (moral) responsibility. (12)

Limitations

  • "does not provide answers about the effects on the quality of human behaviour. It can be argued that certain skill-, rule- or knowledge- based behaviours have more weight than others in the driving task."
  • SAE 4+ largely unknown
  • Framework is not empirically tested
    • Perhaps write a research proposal on this?

Other possible frameworks of human behavior

See page 13:

  • Markov dynamic model of driver action (Pentland and Liu 1999)
  • conceptualisation of a driver's task (De Winter et al. 2014)
  • the human-machine cooperation framework of Hoc, Young, and Losseville (2009)
  • the hierarchical structure of the road user task (Michon 1985)

Mismatch supply and demand

The apparent mismatch between the availability of skills, rules and knowledge at especially the higher levels of automation, and what is requested from the driver (e.g., acting as a fallback) suggests that the current transfer of control within an ADS needs an overhaul, and, more importantly, a human-oriented transfer of control. A suggestion for this type of transfer would be a human-oriented approach, in which the ADS first takes over control of the tasks humans are notoriously bad at, such as parallel parking or reversing (“Mirror – Signal – Panic” 2018), and with higher levels of automation taking over progressively easier tasks. However, this elicits the paradox where the tasks that are difficult for human drivers to perform are difficult for an ADS designer to implement in its algorithms, which apparently calls for a change of perspective in the automation design domain. (13)

Also: mode errors for the transition of control.

Future research

  • Develop new human-oriented framework for transition of control
  • Empirically testing those models
  • "Lastly, a qualitative approach could be made regarding a framework that assesses the effects of automated driving on human behaviour that, for instance, adds weights to the specific tasks in relation to their importance or occurrence during driving with and without ADS, in order to assess for which tasks the most (qualitative) gain is to be made to automate, and to assess which tasks are most important to remain under human control to maintain meaningful human control over automated systems." (14)

The latter could be a topic for a research proposal with a quantitative approach.