Bias in AI

The ethics guidelines for Trustworthy AI provide the following definition of bias in the context of AI:

Bias is an inclination of prejudice towards or against a person, object, or position. Bias can arise in many ways in AI systems. For example, in data-driven AI systems, such as those produced through machine learning, bias in data collection and training can result in an AI system demonstrating bias. In logic-based AI, such as rule-based systems, bias can arise due to how a knowledge engineer might view the rules that apply in a particular setting. Bias can also arise due to online learning and adaptation through interaction. It can also arise through personalisation whereby users are presented with recommendations or information feeds that are tailored to the user’s tastes. It does not necessarily relate to human bias or human-driven data collection. It can arise, for example, through the limited contexts in which a system in used, in which case there is no opportunity to generalise it to other contexts. Bias can be good or bad, intentional or unintentional. In certain cases, bias can result in discriminatory and/or unfair outcomes, indicated in this document as unfair bias. (36)

Sources of bias

We typically say a "model" is biased, but a common culprit in learning systems is data bias. In this case the word "bias" is not the same as the "bias" in the "bias-variance trade-off" which relates to model fitting.

Bias can be caused by human errors, but also reflect systemic bias that is present in the society where the data is collected from.

Examples of human errors are:

  • Sampling errors where the sample is not sufficiently representative of the population
  • Exclusion bias, where a certain group is excluded from data collection
  • Recall bias when people recall events from the past

Societal error:

  • Implicit biases and stereotypes
  • Unjust policies

Where does bias appear in the AI process?

  • Data collection: e.g. a model is trained on data that does match data that will be used in production
  • Data annotation: human annotators may be biased or the annotation task may be difficult or rushed
  • Preprocessing: e.g. if you truncate input texts to fit into memory this affects the model; what if the longer texts correlate with for example patients with a more complex medical history?

STUB: fixing bias in AI