202411102024-EU_AI_Act_definition_of_AI

The AI definition in the EU AI Act is a very broad and flexible definition that tries to fit current AI practices while simultaneously being future proof and very inclusive. This comes at a cost though, because my personal experience is that three people have three different interpretations of this definition when discussing concrete AI cases. Sure, it's quite clear that large language models and computer vision using convolutional networks are covered by this definition, but what about CSP solvers, or a more complex rule-based systems that makes recommendations based on a set of rules? When implementing AI governance in an organization, it's not as simple as just giving this definition and letting dev teams figure it out by themselves. I hope to gain a bit more clarity on which concrete AI cases are (not) covered by this definition by doing a step by step analysis of the AI definition and by contrasting it with definitions used in related notes (such as the OECD memorandum).

Definition

Definition from Article 3, sub 1:

"AI system" mean a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.

Dutch version:

... een op een machine gebaseerd systeem dat is ontworpen om met verschillende niveaus van autonomie te werken en dat na het inzetten ervan aanpassingsvermogen kan vertonen, en dat, voor expliciete of impliciete doelstellingen, uit de ontvangen input afleidt hoe output te genereren zoals voorspellingen, inhoud, aanbevelingen of beslissingen die van invloed kunnen zijn op fysieke of virtuele omgevingen.

The AI definition from the AI Act is in inspired by the definition of the Explanatory memorandum on the updated OECD definition of an AI system, which is as follows:

An AI system is a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. Different AI systems vary in their levels of autonomy and adaptiveness after deployment. (OECD memorandum, p. 4)

This definition, in turn, is inspired by the definition of Russel & Norvig's classic "Artificial Intelligence: A Modern Approach" (2009):

An AI system is a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.

Now step by step.

1. A machine-based system

This is not the most exciting part of the definition, but it's important to note that the EU AI Act definition is not limited to software-based systems. It includes AI systems that are physically embedded, such as robots, drones, and self-driving cars.

The term 'machine-based' refers to the fact that AI systems run on machines. (Recital 12)

2. that is designed to operate with varying levels of autonomy

AI systems are designed to operate with varying levels of autonomy, meaning that they have some degree of independence of actions from human involvement and of capabilities to operate without human intervention. (EU AI Act recital 12)

As far as I can see, the word "autonomy" is used a few times (Article 3, 13, recitals 12, 27, 29, 110, 111), but not defined within the AI Act. Compare with the OECD memorandum, which defines autonomy as:

the degree to which a system can learn or act without human involvement following the delegation of autonomy and process automation by humans. Human supervision can occur at any stage of the AI system lifecycle, such as during AI system design, data collection and processing, development, verification, validation, deployment, or operation and monitoring. Some AI systems can generate outputs without these outputs being explicitly described in the AI system’s objective and without specific instructions from a human. (OECD Memorandum, p. 6)

This provides a lot more support than the AI Act itself. My takeaways from this snippet are:

  • when we talk about the autonomy of AI systems a human has delegated responsibility for a task to an AI system.
  • I take "delegation of autonomy" to mean that the process automation pertains to something that is normally in the sphere of human autonomy, like taking decisions, where there is a degree of freedom.
    • Perhaps this is an important key for distinguishing AI from other software systems that do "automation simple", which are not autonomous in this sense.
    • My reasoning here implies that the type of output is important for determining the autonomy of an AI system. For example, when the output is a decision, there is more autonomy delegated to the AI system than when we use an regression model to predict the price of a house. This reasoning is supported by the OECD memorandum:

AI system outputs generally belong to the broad categories of recommendations, predictions, and decisions. These categories correspond to different levels of human involvement, with "decisions" being the most autonomous type of output (the AI system affects its environment directly or directs another entity to do so) and "predictions" the least autonomous. For example, a driver-assist system might “predict” that a pixel region in its camera input is a pedestrian; it might "recommend" braking, or it might "decide" to apply the brake. (OECD Memorandum, p. 9)

  • the more instructions are needed for an AI system to reach its objective, the less autonomous the system is; conversely, the less instructions are needed and the more freedom an AI system has in determining the best method to reach the specified objective, the more autonomous the system is (increasing autonomy implies increasing degrees of freedom).
    • Foreshadowing the sentence about explicit/implicit objectives: the more explicit the objectives, the less autonomous the system is, because the system is more constrained in its freedom to determine the best method to reach the specified objective. (This may not fit too well with my reasoning that it's mostly the output and the degree of delegated human autonomy that's crucial here.)
  • human supervision is an important, especially when we delegate responsibility to AI systems that have impact, but this is ultimately a matter of risk of AI and not of the definition of AI.

The quote talks about "process automation". This raises the question to which degree "simple automation" is included in this definition. The recital 12 of the EU AI Act states however:

the definition should be based on key characteristics of AI systems that distinguish it from simpler traditional software systems or programming approaches and should not cover systems that are based on the rules defined solely by natural persons to automatically execute operations. (recital 12)

Even though the AI Act is clear in this intention of excluding simple automation, the "autonomy" part of the AI definition strictly speaking does not really help in distinguishing AI from other software systems. As a criterium for AI it's not very helpful on its own, but it is very descriptive of the nature of AI systems and more specifically of the risk involved in deploying AI systems: if you delegate a lot of autonomy to an AI system that has a lot of impact on human lives, it's extra important to have appropriate human supervision without responsibility gaps. Cf. index_MHC_meaningful_human_control.md for more on meaningful human control over AI systems.

Summarizing essay on autonomy (also see "output")

The word “autonomy” is used several times in the AI Act (Article 3 and 13, recitals 12, 27, 29, 110, 111) but nowhere defined. The OECD memorandum offers more support and largely formed the basis of our output-driven explanation of autonomy. When we talk about the autonomy of an AI system a human has (partially) delegated human autonomy and responsibility for a certain task to an AI system. The type of outputs that are mentioned, such as predictions or decisions, pertain to functions that are normally in the sphere of human autonomy because they imply a degree of freedom. When the output of an AI system is a decision to offer a person a mortgage or not, we delegate more human autonomy to the system than when we use an AI system that gives a prediction about house prices.

One important clarification that the OECD adds to the AI Act is that “human supervision can occur at any stage of the AI system lifecycle”. Our reading of this phrase is that the amount of human supervision does not determine whether an AI system is autonomous or not. Rather, the more autonomy we delegate to an AI system, the more risk there is if this system also has real impact and lacks human supervision. The matter of human supervision thus pertains to a subsequent risk assessment, rather than the definition of AI.

3. and that may exhibit adaptiveness after deployment,

The word "adaptiveness" only occurs twice in the AI Act, once in the definition in article 3 and once in recital 12 with a very brief explanation:

The adaptiveness that an AI system could exhibit after deployment, refers to self-learning capabilities, allowing the system to change while in use. (EU AI Act recital 12)

This brief explanation only raises more questions. What is "self-learning"? And is "self-learning" only possible after deployment?

I'm not fond of the term "self-learning" because it's a source of confusion and, again, is not explained in the AI Act. It could mean the capacity of machine learning systems to learn from data without explicit human intervention (cf. autonomy). I say explicit, because humans do specify objectives (cf. explicit vs. implicit objectives) for these machine learning systems.

In this interpretation the training and re-training of machine learning systems are examples of their adaptiveness, which are done typically before deployment or re-deployment. However, the clarification in recital 12 seems to suggest that in the context of the AI Act "adaptiveness" as "self-learning" indicates to the ability to learn independently from human involvement after deployment. This then seems to exclude a typical machine learning flow where a model is trained on a (static) training dataset and then deployed. It also seems to exclude periodic retraining, because this is requires human intervention (either directly or through the orchestration of MLOps pipelines). It includes forms of online learning. For example, a recommender system aims to learn what a certain customer likes to buy, but this is only learned after deployment through tracking the behavior of the customer. Seen this way, this interpretation of adaptiveness as self-learning only captures a small subset of AI/ML applications on the one hand, whereas on the other hand the "may" in the definition makes this part of the definition rather non-discriminative.

Interestingly, the OECD memorandum has a more elaborate explanation. On page 6 of the memorandum we read:

Adaptiveness (contained in the revised definition of an AI system) is usually related to AI systems based on machine learning that can continue to evolve after initial development. (p. 6, my emphasis)

Note that this phrase is still compatible with the definition of the AI Act, but the word "usually" already indicates that there's more to it. What also stands out to me is that this definition takes the initial development as the anchor point, as opposed to the initial deployment. This consequence of this seemingly subtle difference immediately becomes clear in the next sentence

The system modifies its behaviour through direct interaction with input and data before or after deployment. (p. 6, my emphasis)

So, as opposed to the AI Act, this includes learning capabilities before deployment, which brings most typical machine learning approaches into scope. Continuing:

Examples include a speech recognition system that adapts to an individual’s voice or a personalised music recommender system.

Interestingly, this example fits both the definition of the AI Act and the OECD definition. However, the following sentence highlights their difference again:

AI systems can be trained once, periodically, or continually and operate by inferring patterns and relationships in data. Through such training, some AI systems may develop the ability to perform new forms of inference not initially envisioned by their programmers.

This explicitly brings training a model once and period re-training into scope. A model that can be trained once is ipso facto eligible for re-training, but not all AI systems can do continuous online learning after deployment like for example a recommender system could.

I find that the understanding of adaptiveness in the OECD memorandum is more elegant because it captures the various possible relationships between development and deployment phases in AI projects. For example, fine-tuning a model is both post-deployment (from the perspective of the deployer of a base model) but also includes development. The AI Act definition does not sufficiently capture this nuance by exclusively focusing on "self-learning". A machine learning system that you train only once is in my opinion also "adaptive" in the sense that it has the capability to infer patterns from data (see step 5, "infer"); patterns that may be implicit in the training data (see step 4); and do so with degrees of freedom since difference patterns would have been learned if the data would have been different (cf. step 2, autonomy).

My impression is that by narrowing down the definition of adaptiveness to self-learning, the "may" clause became a necessity because at the same time the AI Act does not dare to completely exclude more traditional non-ML AI with symbolic reasoning ("The techniques that enable inference while building an AI system include (...) logic- and knowledge-based approaches that infer from encoded knowledge or symbolic representation of the task to be solved." ). Note that conversely the OECD definition avoids all this vagueness altogether by simply stating that "[d]ifferent AI systems vary in their levels of autonomy and adaptiveness after deployment." (p. 4) This is still inclusive without being so vague!

Another interpretation of the "may" is that they find pre-deployment adaptiveness so obvious that only the optional post-deployment adaptiveness is mentioned in the definition.

Summarizing essay on this point

Depending on one’s interpretation of adaptiveness, the AI Act and the OECD compendium are in conflict. The term “adaptiveness” only occurs only twice in the AI Act and the only explanation is that it refers to “self-learning capabilities, allowing the system to change while in use.” This raises two questions: what is self-learning exactly and is self-learning only possible after deployment?

The clarification in Recital 12 seems to suggest that in the context of the AI Act "adaptiveness" as "self-learning" indicates to the ability to learn independently from human involvement after deployment. An example is a recommender system that learns user preferences on the fly without having to be redeployed. The term “self-learning” is however also often used to describe the capacity of machine learning systems to learn from data without explicit human intervention (cf. Autonomy; although human do specify objectives for these systems). If you use this quite common interpretation of self-learning, then the training and re-training of machine learning systems are examples of how they can adapt to new data, but training is done before rather than after deployment. Is it indeed the intention of the AI Act to target e.g. online learning recommender systems, but exclude for example the vast majority of machine learning projects where a model is trained on a static dataset and then deployed? If indeed the AI Act use of “self-learning” targets a very specific subset of machine learning systems, then this would explain why the word “may” in “may exhibit adaptiveness after deployment” was necessary to avoid accidentally excluding most AI systems from the definition. Another reading of the word “may” could be that the adaptiveness displayed by more typical machine learning systems before deployment is so obvious and implied in other parts of the definition that it’s not worth mentioning it explicitly. This remains unclear.

The OECD compendium largely avoids all this ambiguity by instead stating that adaptiveness is “usually related to AI systems based on machine learning that can continue to evolve after initial development” (our emphasis). Firstly, the word “usually” indicates the scope is broader than the field of machine learning, whereas the scope of the AI Act rather seems to be a smaller subset of machine learning. Secondly, this OECD definition reasons from the moment of development rather than deployment. This is easily overlooked, but has serious implications for our understanding of adaptiveness, which for example becomes clear in the following quote from the OECD memorandum: “The system modifies its behaviour through direct interaction with input and data before or after deployment” (our emphasis). Notably, this does fit with the more common interpretation of “self-learning”. Moreover, this brings training and re-training into scope as forms of adaptiveness, whereas the AI Act explicitly seems to exclude those! Consider the sentence: “AI systems can be trained once, periodically, or continually and operate by inferring patterns and relationships in data. Through such training, some AI systems may develop the ability to perform new forms of inference not initially envisioned by their programmers” (our emphasis).

We decided to take the OECD memorandum as the point of departure with regards to the understanding of adaptiveness. This allows us to avoid the ambiguities of the AI Act use of the word. Moreover, the understanding put forward by the OECD memorandum is more elegant because it captures the various possible relationships between development and deployment phases that actually occur in practice in AI projects. For example, fine-tuning a model is both post-deployment (from the perspective of the deployer of a base model) but also includes development. The AI Act definition does not sufficiently capture this nuance by exclusively focusing on post-deployment self-learning. In our view, a machine learning system that you train only once is also "adaptive" in the sense that it has the capability to infer patterns from data (cf. Section 3 on inference); patterns that may be implicit in the training data (cf. Section 4); and do so with degrees of freedom since difference patterns would have been learned if the data would have been different (cf. Section 6 on autonomy). Finally, we argue that the OECD is in the spirit of the AI Act because of its inclusiveness. It also includes the online-learning examples that the AI Act highlights.

4. and that, for explicit or implicit objectives,

On this point, the AI Act is quite brief, but nevertheless this point is relatively clear and does not contradict the clear explanation from the OECD Memorandum:

The reference to explicit or implicit objectives underscores that AI systems can operate according to explicit defined objectives or to implicit objectives. The objectives of the AI system may be different from the intended purpose of the AI system in a specific context. (Recital 12)

The OECD memorandum distinguish various flavors of objectives:

  • Explicit and human-defined: "the developer encodes the objective directly into the system (e.g., through an objective function)" (p. 7)
    • Examples range from typical ML classifiers to reinforcement learning and planning algorithms.
    • Even though it's "explicit", I find this a bit confusing too, because this "explicit" goal is never a goal in itself but a numerical method to optimize a non-specified human intention.
  • Implicit in specified rules: You may program a rule "If the traffic light is red, stop" , but the true objective (to obey traffic laws and drive safely) remains implicit in the AI system (whereas it's rather common sense for humans).
  • Implicit in training data: in this case the "ultimate" objective is not explicitly programmed in, neither as an objective function or through rules, but is "incorporated through training data and a system architecture that learns to emulate those data (e.g. rewarding large language models for generating a plausible response)" (p. 7)
    • NOTE The difference with "explicit and human-defined" is not so clear to me. For example, the foundational base of LLMs only emulate data through an objective function and the fine-tuning of Q&A pairs is through reinforcement learning from human feedback.
  • Not fully known in advance, for example reinforcement learning from user preferences in recommender systems.
    • The goal is to learn "what the customer likes" but it's not known in advance whether those are children toys or machine guns ;-).

These are essentially different variations of what Stuart Russell calls the "standard model" of AI, where "intelligent" machines are optimized a "purpose put in the machine" (see Human Compatible) which may not in fact be the true underlying objective. Which (as a side note) introduces the problem of alignment the goals that AI optimize with human purpose, since "[i]nevitably, Goodhart’s law will take over, whereby individuals optimize the official measure of outward behavior, just as universities have learned to optimize the “objective” measures of “quality” used by university ranking systems instead of improving their real (but unmeasured) quality." (the book Human Compatible).

W.r.t. this point two endnotes of the OECD memorandum are interesting:

AI systems are supplied with objectives. However, the ultimate human objectives (e.g., respecting traffic rules) may be higher-level or more abstract than the technical objectives supplied to the system (e.g., stopping at a red light), which should serve the higher-level objectives. (p. 11)

From a more technical perspective, the learning process for these imitative systems relies on a formally defined loss function that measures the deviation of the system’s outputs from observed human behaviour. The loss function does not, however, express the human objectives underlying the observed behaviour. This is in contrast with a reinforcement learning approach, where the reward function used for training may explicitly express the humans' ultimate objectives and trade-offs. (p. 11)

5. infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions

On this point the recital is a bit more elaborate:

A key characteristic of AI systems is their capability to infer. This capability to infer refers to the process of obtaining the outputs, such as predictions, content, recommendations, or decisions, which can influence physical and virtual environments, and to a capability of AI systems to derive models or algorithms, or both, from inputs or data. (Recital 12)

For the purposes of this Regulation, environments should be understood to be the contexts in which the AI systems operate, whereas outputs generated by the AI system reflect different functions performed by AI systems and include predictions, content, recommendations or decisions. (Recital 12)

The techniques that enable inference while building an AI system include machine learning approaches that learn from data how to achieve certain objectives, and logic- and knowledge-based approaches that infer from encoded knowledge or symbolic representation of the task to be solved. The capacity of an AI system to infer transcends basic data processing by enabling learning, reasoning or modelling. (Recital 12)

Whereas the "degrees" of "autonomy" and "adaptiveness" where so inclusive that they did not really help distinguishing AI applications from non-AI automation on their own, these section on inference tie everything together.

A basic data dashboard is excluded from the definition because no inference is done on the data (inference "transcends basic data processing").

This explanation is output driven: the outputs generated by an AI system "generally reflect different functions" that in turn "correspond to different levels of human involvement" (OECD p. 9). That is, we should not discuss autonomy (part 2) on its own, because the system output helps determining the relevant degree of delegated autonomy.

This is also one of the few points where symbolic AI (rather than subsymbolic ML) seems to be on top of mind too.

On this point the OECD is quite complementary, adding the nuance that inference is not specifically tied to the building or deployment phase:

The concept of “inference” generally refers to the step in which a system generates an output from its inputs, typically after deployment. When performed during the build phase, inference, in this sense, is often used to evaluate a version of a model, particularly in the machine learning context. In the context of this explanatory memorandum, "infer how to generate outputs" should be understood as also referring to the build phase of the AI system, in which a model is derived from inputs/data. (OECD Memorandum, p. 9)

How a "model is derived from inputs/data" is broader than machine learning alone. This is also highlighted in the following quote

"Symbolic or knowledge-based AI systems typically use logical and/or probabilistic representations, which may be human-generated or machine-generated, to infer outputs such as predictions and decisions. These representations rely on explicit descriptions of variables and of their interrelations. For example, a system that reasons about manufacturing processes might have variables representing factories, goods, workers, vehicles, machines, and so on." (8)

I'm for example thinking of Bayesian networks that are carefully modelled by human domain experts (picking the relevant variables, modelling their (causal) relations), picking their probabilistic representations through prior and conditional distributions) and then updated in a data-driven way.

Finally, AI system != AI model and, according to the following quote, and AI system that infers may use a human constructed model:

AI models can be built manually by human programmers or automatically through, for example, unsupervised, supervised, or reinforcement machine learning techniques. (p. 8)

Might be interesting to work out further and contrast with the AI Act. What does this imply for expert systems or inductive logic programming?

Essay on the word "output"

The term “output” is easily overlooked in the AI Act definition but has a subchapter of its own in the OECD memorandum. In the AI Act the word “output” is mentioned in the context of “inference” as a way to get from inputs to outputs. On closer inspection, the explanation of both inference and autonomy are output driven, which is why we think output deserves a section of its own. We also put emphasis on this term because we believe it’s most helpful to provide examples of AI systems and the type of outputs they infer when trying to distinguish AI from non-AI systems. This informed the approach taken in this document.

The OECD states that outputs generated by an AI system “generally reflect different functions” that in turn “correspond to different levels of human involvement”. This means that considering the type of output of a system is very important for determining the relevant degree of autonomy and which type of inference is being done. For example, Recital 12 of the AI Act explicitly states that the “capacity of an AI system to infer transcends basic data processing by enabling learning, reasoning or modelling.” But if you consider the phrase “varying levels of autonomy” from the AI definition on its own, it’s not clear why basic data processing would be excluded from being an AI system, because indeed any type of programmatic processing has some independence from human involvement. We start a script and then it automatically performs a set of instructions. In the case of starting a programming script that does basic data processing, this independence is typically called automation. To distinguish automation from autonomy, we consider which function is performed. Is this function a type of inference that is normally the responsibility of a human but is now delegated to the system, such as a decision (the output)? Then the system has a relevant degree of autonomy. To summarize, the output of an AI system is a good lens with which to start looking at other parts of the definition.

Essay on the word inference

We have already discussed some relevant aspects of inference in the section on outputs. The OECD compendium is complementary to the AI Act and adds some more nuances. Whereas inference is typically discussed in a post deployment phase, the OECD makes it clear that inference can also occur before deployment. In a machine learning context, inference is also done before deployment to evaluate a model. This is consistent with the AI Act definition of inference as the capability to obtain outputs, because inference during evaluation is a way of getting intermediate outputs to guide the training or modelling process.

Both the AI Act and the OECD memorandum are also in agreement that inference is not limited to subsymbolic reasoning (many machine learning approaches reason using learned representations that are not inherently understandable by humans), but also includes logic- and knowledge-based reasoning approaches that use symbolic or probabilistic representations that encode knowledge. The OECD compendium adds the nuance that these representations may also be human-generated. Consider for example a Bayesian network where domain experts encode their knowledge by picking relevant variables, defining which variables may be related or independent from each other through statistical analysis, and choosing which type of probability distribution is appropriate for representing each variable. This network can consequently be tuned based on evidential data. Another example would be logical inductive programming, where you try to derive hypotheses or rules from data against the background of given examples and background knowledge, which are specified by a human. The insight here is that the model underlying the inference done by the AI system does not have to be fully learned itself.

6. that can influence physical or virtual environments.

Not too exciting.

keywords: (Partially) observable space, observed via data (indirect, supplied via humans or other processes) or sensors (more direct), influenced through actuators (may also be human, for example person taking recommendations from AI and acting on them; or machinal).