|

Following the definition of a provider and an implementer, the agency explains the concept of‘deepfake’. For implementers of AI systems, understanding this definition is important, as transparency obligations apply, amongst other things, specifically to content that constitutes deepfakes.

What does the Artificial Intelligence Act stipulate?

The first transparency obligation under the fourth paragraph of Article 50 of the Artificial Intelligence Act (hereinafter: the AI Act) concerns deepfakes. In Article 3(60), the AI Act defines a deepfake as ‘AI-generated or manipulated image, audio or video content that resembles existing persons, objects, places, entities or events, and which the user is led to believe is genuine or real’.

What do the Guidelines on the Implementation of Transparency Obligations for Certain AI Systems clarify regarding the individual criteria?

Although the Guidelines on the Implementation of Transparency Obligations for Certain AI Systems pursuant to Article 50 of the AI Act (hereinafter: the Guidelines) are not legally binding, they offer practitioners practical guidance for a uniform understanding and definition of deepfakes. For content to constitute a deepfake, all four cumulative criteria derived from the definition must be met:

• Similarity – a high degree of similarity between the deepfake content and the simulated subject.

• Existence – the simulated persons, objects, places, entities or events must resemble someone or something that exists, could plausibly exist or might plausibly exist in reality.

• People, objects, places, subjects or events – for content to constitute a deepfake, it must resemble existing people, objects, places, entities or events.

• The false impression that the content is genuine or real – content that would falsely appear to a person to be genuine or real.

It follows from the definition that the content may be either generated or altered (‘manipulated’) by an AI system. When generating content, an AI system produces new synthetic content; for example, it creates an image. When altering content, the AI system modifies existing content; for example, it changes part of an audio recording. Any content (video, image or audio) created or modified by a AI system is not, in itself, a deepfake. It only becomes one if it meets all four criteria set out in the definition above.

It also follows from the definition that text cannot be deepfakes. Text created or modified using AI, which is published with the aim of informing the public about matters of public interest, is subject to a separate obligation set out in the fourth paragraph of Article 50 (you can read more in the FAQs on transparency already published by the Agency).

1. Similarity: to a significant extent, not necessarily identical

Recital 134 of the AI Act adds that the content must be ‘substantially’ similar to the subject. The guidelines interpret this as a high degree of similarity to the simulated subject or to any of its recognisable features. The content need not be identical to the subject. For example, a deepfake is also an AI-generated video of a person who merely resembles a politician and is speaking to an audience. The assessor evaluates the similarity on a case-by-case basis, by objectively comparing the content with the simulated subject. They take into account the extent to which the subject’s characteristic or distinguishing features are depicted.

2. Existing: it is sufficient that it could exist

The guidelines interpret this criterion in light of the purpose of the fourth paragraph of Article 50, which is to minimise the risks of impersonation, deception, disinformation, manipulation and fraud. It is therefore sufficient for the content to be realistic. A simulated person, object or place does not necessarily have to be real; it is sufficient that it could plausibly exist. A deepfake may therefore also be a realistic avatar or persona created using AI, even if no such person actually exists. Unrealistic scenes that defy the laws of nature or physics and are not capable of deceiving are not covered. The guidelines cite examples of unrealistic scenes such as people flying without mechanical aids, dragons or elephants driving cars, and so on.

3. Persons, objects, places, subjects or events: what each category covers

The guidelines also define in more detail what is to be understood as a person, object, place, subject or event depicted by a deepfake:

• People – human beings, including digital replicas of real people, realistic avatars and AI-generated personas, as well as personal characteristics such as appearance, voice, behaviour and mannerisms

• Objects – inanimate material things, e.g. buildings, machines and consumer goods

• Places – realistic locations

• Entities – non-human living beings, e.g. animals, and other biological forms of life

• Events – scenes or situations that may involve people, objects, places and subjects, e.g. re-enactments of historical events or demonstrations of professional or consumer services

4. False impression of authenticity or reality: the capacity to mislead

This criterion describes the essence of deepfakes: their ability to mislead a person regarding the authenticity or reality of the content. The guidelines distinguish between two concepts. The authenticity of content refers to whether the content is actually what it purports to be, in terms of its source or the process by which it was created (including, for example, the participation of real people or animals and their actual behaviour or actions, the actual appearance or use of objects, or the provision of services, and the precise sequence or development of events). Authenticity refers to whether the content is genuine (e.g. the factual accuracy of depictions in a deepfake).

The assessment is objective, and the creator’s intention to mislead anyone is not a necessary factor. The content must be assessed as a whole, taking into account:

• the degree of similarity,

• any message conveyed by the content,

• the intended and foreseeable contexts of presentation,

• the environment in which the content is presented,

• the composition of the intended and reasonably foreseeable audience and their expectations.

The degree of similarity refers to how similar the content is to the real subject. The guidelines state that, in the case of image and video content, a high degree of photorealism makes it more likely that the content is a deepfake. However, photorealism in itself is not decisive: it is still necessary to assess whether the content could be misleading in the context in which it is presented.

The message conveyed by the content is what the content communicates to the viewer or listener as a fact, e.g. that a certain person said or did something, what a product is like, or that something happened.

• A video clip of a person resembling a politician speaking to an audience conveys to the viewer that the politician did indeed give the speech. The guidelines classify this as a deepfake.

• However, in the case of a real product (e.g. a car) featured in an advert against a background created using AI, the message about the product remains the same. The guidelines do not classify such an advert as deepfake, provided it does not mislead regarding the product and its characteristics. In contrast, a deepfake could be an AI-generated image that depicts the product as different or better than it actually is.

Consequently, the context in which the content is presented – whether created or adapted by the publisher using AI – may rule out a false impression. If, in a given context, the audience does not expect the content to be authentic or true, the criterion is not met (e.g. special effects and background scenes in a film created using AI, or fictional environments in a video game).

At the same time, when defining the intended and reasonably foreseeable audience, it is necessary to take into account contexts in which children, older people and those with lower digital literacy or AI literacy may be present. If the content were to mislead even these groups, this may be sufficient to classify it as deepfake. In this regard, the publisher need not anticipate the further dissemination of the content beyond the audience they can reasonably foresee based on the distribution channels; for example, if a piece of content, shared on an internal website amongst colleagues, later appears on a social network.

Test your understanding with examples from the Guidelines

The Guidelines provide examples that help distinguish deepfakes from other content created or manipulated using AI. You can test your understanding of the definition yourself: read the example in the first column, decide whether it is a deepfake, and only then check the answer in the second column.

! It is important to emphasise that the examples are not exhaustive. They illustrate how the Guidelines apply the criteria, but they do not replace judgement in individual cases. The same content may be assessed differently in a different context, before a different audience, or with a different message. The publisher must therefore assess each piece of content in light of the circumstances in which it is published. !

When does the obligation to disclose that the content is AI-generated or AI-modified, as set out in the fourth paragraph of Article 50 of the AI Act, arise?

Content defined as a deepfake is only one of the conditions under which the transparency obligation set out in the fourth paragraph of Article 50 of the AI Act applies. The content must be labelled when, in addition, the following condition is also met:

• the operator uses the AI system for professional purposes, and

• the use is not permitted by law for the detection, prevention, investigation or prosecution of criminal offences.

We have explained in more detail how AI system operators should label deepfakes in the news article ‘Labelling of content and obligations applicable to AI system operators’, pursuant to paragraph 4 of Article 50 of the AI Act, and the Agency has also published a set of icons in Slovenian.

The guidelines on implementing transparency obligations for certain AI systems, on which this clarification is based, are now also available in Slovenian.