Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

In this paper, we introduce a framework for algorithmic auditing that supports artificial intelligence system development end-to-end, to be applied throughout the internal organization development life- cycle. Each stage of the audit yields a set of documents that together form an overall audit report, drawing on an organization’s values or principles to assess the fit of decisions made throughout the process. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale artificial intelligence systems by embedding a robust process to ensure audit integrity. (33)

Introduction

A main downside from existing external auditing, is that the audit often happens after model deployment.

This paper instead presents "internal algorithmic audits" (my emphasis, 33). Audit process is called SMACTR.

The audit process is necessarily boring, slow, meticulous and methodical—antithetical to the typical rapid development pace for AI technology. However, it is critical to slow down as algorithms continue to be deployed in increasingly high-stakes domains. By considering historical ex- amples across industries, we make the case that such audits can be leveraged to anticipate potential negative consequences before they occur, in addition to providing decision support to design mitiga- tions, more clearly defining and monitoring potentially adverse out- comes, and anticipating harmful feedback loops and system-level risks [20].

Governance, Accountability and Audits

Def. accountability:

We use accountability to mean the state of being responsible or answerable for a system, its behavior and its potential impacts. (34)

Definition comes from: Nitin Kohli, Renata Barreto, and Joshua A Kroll. 2018. Translation Tutorial: A Shared Lexicon for Research and Practice in Human-Centered Software Systems. In 1st Conference on Fairness, Accountability, and Transp arancy. New York, NY, USA. 7.

Algorithms aren't legal agents and cannot be held accountable. Instead, an organization should be held accountable through governance structures:

If the responsible development of artificial intelligence is a core purpose of organizations creating AI, then a governance system by which the whole organization is held accountable should be established. (34)

Following is a great example. Existing engineering standards are not sufficient!

For example, a power plant can be consistently productive while causing harm to the environment through pollution [42]. Similarly, an AI system can be found technically reliable and functional through a traditional engineering quality assurance pipeline without meeting declared ethical expectations. (34)

The latter ethical process also optimizes for values. This may touch existing procedures, e.g. for reliability (value: safety), but is usually not (fully) covered by it.

What is an audit?

Def. audit from Jie Liu. 2012. The enterprise risk management and the risk oriented internal audit. Ibusiness 4, 03 (2012), 287:

Audits are tools for interrogating complex processes, often to determine whether they comply with company policy, industry standards or regulations (Raji 2020, 34)

IEEE definition:

The IEEE standard for software development defines an audit as “an independent evaluation of conformance of software products and processes to applicable regulations, standards, guidelines, plans, specifications, and procedures” [32]. (p.2)

AI Principles as Customized Ethical Standards

Problem statement:

Important values such as ensuring AI technologies are subject to human direction and control, and avoiding the creation or reinforcement of unfair bias, have been included in many organizations’ ethical charters. However, the AI industry lacks proven methods to translate principles into practice [49], and AI principles have been criticized for being vague and providing little to no means of accountability [27, 82].

  • 49: Brent Mittelstadt. 2019. AI Ethics: Too Principled to Fail? SSRN (2019).
  • 27: Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019. Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In Proceedings of the 52nd Hawaii International Conference on System Sciences.
  • 82: Jess Whittlestone, Rune Nyrup, Anna Alexandrova, and Stephen Cave. 2019. The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions. In Proceedings of the AAAI/ACM Conference on AI Ethics and Society, Honolulu, HI, USA. 27–28.

How can we use these general ethical principles in an organisation?

Thus, in the absence of more formalized and universal standards, they can be used as a North Star to guide the evaluation of the development lifecycle, and internal audits can investigate alignment with declared AI principles prior to model deployment. We propose a framing of risk analyses centered on the failure to achieve AI principle objectives, outlining an audit practice that can begin translating ethical principles into practice. (35)

What interests me particularly here is that a risk analysis-approach is taken! I've been struggling myself with how to frame assessment analogously to existing risk assessments, e.g. in information security.

Audit Integrity and Procedural Justice

In order to maintain the integrity of the audit process itself it is important to have a clear audit methodology. You don't want there to be constant doubts about whether the audit itself is valid; this would make the audit largely ineffective.

The Internal Audit

Advantage of external audit is that they are unaffected by internal organizational considerations and thus are more easily considered to be credible.

But a clear limitation of external audits is that often they do not have access to intermediate models, training data etc. which may be protected as trade secrets. Instead, they can only audit based on model outputs and interact through given APIs. Internal audits do have direct access to these things, which is a great advantage.

The goal of internal audits:

The goals of an internal audit are similar to quality assurance, with the objective to enrich, update or validate the risk analysis for product deployment. Internal audits aim to evaluate how well the product candidate, once in real-world operation, will fit the expected system behaviour encoded in standards.

  • the internal audit also allows for proactive ethical intervention because it can be done during development instead of only after deployment
  • audit outcomes can be ambiguous, so it is important to identify the relevant stakeholders that can act on audit outcomes
  • because internal auditors work at the organization, the audit can also be used to motivate internal organizational changes to align with ethical standards
  • internal audits also further facilitate external audits, by creating more reporting on internal processes; this is useful for transparency and communication to other (external) stakeholders.

Lessons from auditing practices in other industries

Interesting remark:

Complex systems tend to drift toward unsafe conditions unless constant vigilance is maintained [42]. It is the sum of the tiny probabilities of individual events that matters in complex systems—if this grows without bound, the probability of catastrophe goes to one. The Borel-Cantelli Lemmas are formalizations of this statistical phenomenon [13], which means that we can never be satisfied with safety standards. (35)

Insights from aerospace

Design checklists can be handy. Some remarks:

  • Goal: "assisting designers in having a more informed view of important questions, edge cases and failures" (35)
  • Cautions:
    • "risk of blind application" (35)
    • not considering "the broader context and nuanced interrelated concerns" (35-36)
    • "avoid yes/no questions to reduce the risk that the checklist becomes a box-ticking activity" (36)
      • "for example by asking designers and engineers to describe their processes for assessing ethical risk" (36)
  • "Checklist use should also be related to real-world failures and higher-level system hazards"
    • Q: I get the relation to failures, but what about the higher-level system hazards? What does that mean?

Traceability relates product requirements and system design. In software engineering traceability is part of requirements engineering.

Later in the article: "trace model output back to requirements" (37). I haven't really thought about traceability before as being so coupled to the requirements.

Traceability in AI is difficult:

However, in AI research, it can often be difficult to trace the provenance of large datasets or to interpret the meaning of model weights—to say nothing of the challenge of understanding how these might relate to system requirements. Additionally, as the complexity of sociotechnical systems is rapidly increasing, and as the speed and complexity of large-scale artificial intelligence systems increase, new approaches are necessary to understand risk [42] (36)

Failure Modes and Effects Analysis:

The main purpose of a FMEA is to define, identify and eliminate potential failures or problems in different products, designs, systems and services. Prior to conducting a FMEA, known issues with a proposed technology should be thoroughly mapped through a literature review and by collecting and documenting the experiences of the product designers, engineers and managers. Further, the risk exercise is based on known issues with relevant datasets and models, information that can be gathered from interviews and from extant technical documentation.

Take-away: useful question for developers is whether the dataset (or a very similar one) is already used in another application. Are there known issues and risks associated with this data usage?

FMEA approach is very common, but not yet (sufficiently?) present in the field of AI:

To our knowledge, however, the FMEA method has not been applied to examine ethical risks in production-scale artificial intelligence models or products.

Insights from medical devices

Design controls ensure that "the end product matches the intended use, and that potential risks from using the technology have been anticipated and mitigated" (36)

Important insight here: intended use determines risk (but possible unintended uses also do so!), and risk in turn determine design requirements:

The intended use of a “device” (or, increasingly in medicine, an algorithm—see [60] for more) determines the level of design control required: for example, a tongue depressor (a simple piece of wood) is the lowest class of risk (Class I), while a deep brain implant would be the highest (Class III). The intended use of a tongue depressor could be “to displace the tongue to facilitate examination of the surrounding organs and tissues”, differentiating a tongue depressor from a Popsicle stick. This may be important when considering an algorithm that can be used to identify cats or to identify tumors; depending on its intended use, the same algorithm might have drastically different risk profiles, and additional risks arise from unintended uses of the technology. (36)

Design history file:

Evidence that medical device designers and manufacturers have followed design controls must be kept in a design history file (DHF), which must be an accurate representation and documentation of the product and its development process. Included in the DHF is an extensive risk assessment and hazard analysis, which must be continuously updated as new risks are discovered. Companies also proactively maintain “post-market surveillance” for any issues that may arise with safety of a medical device. (36)

Structural vulnerability:

In medicine there is a deep acknowledgement of socially determinant factors in healthcare access and effectiveness, and an awareness of the social biases influencing the dynamic of prescriptions and treatments. This widespread acknowledgement led to the framework of operationalizing structural vulnerability in healthcare contexts, and effectively the design of an assessment tool to record the anticipated social conditions surrounding a particular remedy or medical recommendation [61]. Artificial intelligence models are equally subject to social influence and social impact, and undergoing such assessments on more holistic and population- or environment-based considerations is relevant to algorithmic auditing. (36)

Fully agreed, but it's not yet as clear to me how this looks like in practice!

I found the insights from finance not as interesting.

Discussion and challenges

Waterfall vs. agile development:

Current software development practice in general, and artificial intelligence development in particular, does not typically follow the waterfall or verification-and-validation approach [16].

Agile development is more fast-pased and iterative and posed challenges to audits. Idea: also do agile auditing.

Internal audits and (bottom-up) risk-assessment and risk register:

Most internal audit functions outside of heavily regulated industries tend to take a risk-based approach. They work with product teams to ask "what could go wrong" at each step of a process and use that to build a risk register [59]. This allows risks to rise to the surface in a way that is informed by the people who know these processes and systems the best. Internal audits can also leverage relevant experts from within the company to facilitate such discussions and provide additional insight on potential risks [3].

Additional problem: AI is a bit of a general purpose tech and is developed in many different contexts. An additional lack of standardization makes effective governance very hard.

SMACTR: An Internal Audit Framework

  1. Scoping
  2. Mapping
  3. Artefact Collection
  4. Testing
  5. Reflection

Each phase has documentation requirements.

See top page 38 for chart.

 SMACTR

Hypothetical case study:

Company X has decided to pilot the SMACTR internal audit framework to fulfill a corporate mandate towards responsible innovation practice, accommodate external accountability and operationalize internal consistency with respect to its identified AI principles. (38)

Second case: smile detection in photo booths by Happy-Go-Lucky, Inc. Templates for this case study is available as supplementary materials.

Ref: responsible innovation: Jack Stilgoe, Richard Owen, and Phil Macnaghten. 2013. Developing a framework for responsible innovation. Research Policy 42, 9 (2013), 1568–1580.

Point out elements of responsible innovation:

  • Anticipation
  • Reflexivity
  • Inclusion
  • Responsiveness

Insight:

Risk-based assessments can be limited in their ability to capture social and ethical stakes, and they should be complemented by anticipatory questions such as, “what if...?”. The aim is to increase ethical foresight through systematic thinking about the larger sociotechnical system in which a product will be deployed [50]. (38)

Also note: "Though not covered here, an equally important process is determining what systems to audit and why." (38)

Scoping

The goal of the scoping stage is to clarify the objective of the audit by reviewing the motivations and intended impact of the investigated system, and confirming the principles and values meant to guide product development. This is the stage in which the risk analysis begins by mapping out intended use cases and identifying analogous deployments either within the organization or from competitors or adjacent industries. The goal is to anticipate areas to investigate as potential sources of harm and social impact. At this stage, interaction with the system should be minimal. (39)

Pre-requisite documents:

  • Declaration or confirmation statement of the ethical principles, standards and AI principles.
  • Product Requirements Document (PRD) or something similar, like a project planning

Key artefacts of auditers:

  1. Ethical review of system use case
  2. Social Impact Assessment

1. Ethical Review of System Use Case

Stress on standpoint diversity:

Algorithm development implicitly encodes developer assumptions that they may not be aware of, including ethical and political values. (39)

  • To identify problematic background assumptions it is beneficial to include independent experts as well as marginalized groups in the review process.
  • "ask people to reflect on their preliminary assessment and then ask whether they might have reason to regret the action later on" (39)
  • also have a (diverse) internal ethics board to review and document all proposals

2. Social Impact Assessment

This impact assessment should inform the ethical review.

Two steps:

  1. "an assessment of the severity of the risks"
  2. "and an identification of the relevant social, economic, and cultural impacts and harms that an artificial intelligence system applied in context may create" (39)

I myself have seen PIAs and BIAs, i.e. Privacy- and Business Impact Assessments. E.g. BIAs (on information security) are scored on CIA (confidentiality, integrity, accessibility). If the impact on a disruption on either of these axis is severe, this triggers are set of more extensive security measures and auditing processes (like additional pen testing).

Similarly, such a "SIA" could help determine the type of ethical scrutiny that is required. TODO: Have a look at the actual documents to see how the authors propose this happens.

Mapping

Not yet a testing phase, but relating what's already there from the previous phase to those stakeholders that will be involved in the actual testing/reviewing.

  • map internal stakeholders
    • "enables an internal record of individual accountability with respect to participation towards the final outcome" (39)
    • "enables the trace of relevant contacts for future inquiry" (39)
  • identify key collaborators for the internal audit
  • "orchestrate the appropriate stakeholder buy-in required for execution" (39)
  • initiate FMEA and risk prioritization for later testing
    • identify failure modes: what's the impact of false negatives and false positives?

Key artefacts:

  1. Stakeholder map and collaborator contact list
  2. System map of product development lifecycle and engineering system overview, especially when multiple models inform the end product
  3. Design history file review of past product versions
  4. "Report or interview transcripts on key findings from internal ethnographic fieldwork involving the stakeholders and engineers." (40)

  5. Stakeholder Map

Pretty obvious.

  1. Ethnographic Field Study

Problem description:

With modern large-scale artificial intelligence projects and API development, it can be difficult to gain a shared understanding at the right level of system description to understand how local decisions, such as the choice of dataset or model architecture, will impact final system behavior. (40)

What's meant with ethnographic here? It's more "ethnography-inspired fieldwork". Main point is that you need to have access to some key people and so semi-structured interviews to explore possible gaps.

On metrics, data and interpretation:

Traditional metrics for artificial intelligence like loss may conceal fairness concerns, social impact risks or abstraction errors [69]. A key challenge is to assess how the numerical metrics specified in the design of an artificial intelligence system reflect or conform with these values. Metrics and measurement are important parts of the auditing process, but should not become aims and ends in themselves when weighing whether an algorithmic system under audit is ethically acceptable for release. Taking metrics measured in isolation risks recapitulating the abstraction error that [69] point out, "To treat fairness and justice as terms that have meaningful application to technology separate from a social context is there- fore to make a category error, or as we posit here, an abstraction error." What we consider data is already an interpretation, highly subjective and contested [23]. (40)

Make explicit which "assumptions and values the metrics apprehend [75]" (40)

Ref 75: Alexander Styhre. 2018. The unfinished business of governance: towards new governance regimes. In The Unfinished Business of Governance. Edward Elgar Publishing.

Q: It's not clear to me how we can study the values captured by a loss functions. This does fascinate me though. I imagine a really cool article on the ethics of the cross-entropy loss function. Obviously, this loss function makes assumptions. But what are the possible ethical implications of these assumptions? In general, I reckon this is really a new field and would also leverage my unique background, as this connects values with very low-level algorithmic details.

Example:

For example, the decision about whether to prioritize the false positive rate over false negative rate (precision/recall) is a question about values and cannot be answered without stating the values of the organization, team or even engineer within the given development context.

Artifact Collection

Often this implies a record of data and model dynamics though application-based systems can include other product development artifacts such as design documents and reviews, in addition to systems architecture diagrams and other implementation planning documents and retrospectives.

  • retro-active documentation requirements may be asked in case of documentation gaps

Key artefacts:

  • Audit checklist that is used to verify that all documentation required for the audit is provided
    • Can include model and data transparency documentation
  • Datasheets and model cards

Model cards, ref on p.48: Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transp arency. ACM, 220–229.

Design checklist

Above-mentioned audit checklist.

  • "This is also a procedural evaluation of the development process for the system, to ensure that appropriate actions were pursued throughout system development ahead of the evaluation of the final system outcome." (40)
    • Pretty generic, see supplementary materials

Datasheets and Model Cards

Ideally should be collected by product stakeholders during system development.

  • A model card includes documentation about how a model was built, what assumptions were made during development, what type of model behavior can be experienced (for different target groups)
    • Also useful internally for communication about the model to stakeholders
  • A datasheet
    • Covers set of questions about data collection "to provide consumers of the dataset with the information they need to make informed decisions about using the dataset"
    • "What mechanisms or procedures were used to collect the data?" (41)
    • "Was any ethical review process conducted?" (41)
    • "Does the dataset relate to people?" (41)

Testing

Test compliance with prioritized ethical values. "Testing should be based on a risk prioritization from t he FMEA" (41)

Testing phase can be highly dependent on the specific usage context and the project. Paper discusses two example artifacts.

Adversial Testing

Pro-active testing before deployment can identify risks before they occur in the wild and have a potentially large impact. Identifying possible failures also allows you to add these to the risk assessment FMEA.

Two examples:

  1. Direct non-statistical testing

direct non-statistical testing uses tailored inputs to the model to see if they result in undesirable outputs. These inputs can be motivated by an intersectional analysis, for example where an ML system might produce unfair outputs based on demographic and phenotypic groups that might combine in non- additive ways to produce harm, or over time recapitulate harmful stereotypes or reinforce unjust social dynamics (for example, in the form of opportunity denial). (41)

  1. Statistical (human-inperceptible) adversial attack

This is distinct from adversarially attacking a model with human-imperceptible pixel manipulations to trick the model into misidentifying previously learned outputs [28], but these approaches can be complementary. (41)

Ethical Risk Analysis Chart

Combines likelihood of failure and severity of failure. Depending on a combination of these, risk is assessed as "low", "middle" or "high".

  • Failure likelihood can be informed by adversial testing.
  • Severity can be informed by previous assessment e.g. of the social impact and the "ethnographic" interviews.

Reflection

  • Discuss results in relation to the original scoping.
  • Based on test results, highlight the ethical principles that may be jeopardized when the AI system is deployed

Key artefacts:

Algorithmic Use-related Risk Analysis and FMEA

  • Format of risk analysis is variable, but good inspiration can be found in field of FMEA

Remediation and Risk Mitigation Plan

  • " The goal is to drive down the risk of ethical concerns or potential negative social impacts to the extent reasonably practicable." (42)
  • What's the threshold for acceptable performance?
    • "If auditors discover, for example, unequal classifier performance across subgroups, how close to parity is necessary to say the classifier is acceptable?" (42)
  • E.g. design choices like opt-out or opt-in functionality

Algorithmic Design History File (ADHF)

collect all the documentation from the activities outlined above related to the development of the algorithm. It should point to the documents necessary to demonstrate that the product or model was developed in accordance with an organization’s ethical values, and that the benefits of the product outweigh any risks identified in the risk analysis process.

Basis for the final algorithmic report, which summarizes the key insights from the ADHF.

Algorithmic Audit Summary Report

The report aggregates all key audit artifacts, technical analyses and documentation, putting this in one accessible location for review. This audit report should be compared qualitatively and quantitatively to the expectations outlined in the given ethical objectives and any corresponding engineering requirements. (42)