This position paper presents work in progress on the foundation of the not yet finalized AI Act and discusses whether the proposed AI Act has the resources to adequately identify and mitigate the risks posed by generative AI models like LLMs. All the details of AI Act are not yet decided but after the EU Council reached a consensus on a revised version of proposal issued by the Commission, the main principles of this risk-based model of regulation are in place. We argue that the methods for risk identification and mitigation in the act, while adequate to the management of many risks emerging from the use of AI systems, are not suitable for the management of all risks emerging from generative AI systems like LLMs In April 2021 the European Commission issued the draft proposal for the regulation of AI, the so-called AI Act [1]. In the public consultation on the draft proposal more than 300 stakeholders, including several of the tech industries and NGOs, issued position papers expressing diferent types of concerns. The European Council reached a compromise on a revised proposal for the AI Act in November 2022 and the European Parliament will vote on their position on the act in May this year. The process is expected to be completed and the AI Act turned into a law by the end of the year, after negotiations between the Council, the Parliament, and the Commission. Even though the process is not finished the main elements of the risk-based model for how to regulate AI are in place and it is therefore meaningful - and important - to critically discuss the overarching principles in the proposal. In this paper I will argue that the proposed regulation does not adequately handle the risks posed by generative AI like large language models (LLMs). As the basis of the argument, I will use the text of the most recent version of the regulation, issued by the European Council on November 25, 2022 [2].
The regulative model of the AI Act is risk-based and distinguishes between diferent levels of risk. Some types of AI applications are considered to pose unacceptable risks to individuals and are therefore prohibited under the proposed law. These include “AI-enabled manipulative techniques” [2], p. 25 that undermine people’s autonomy by using subliminal techniques to persuade, deceive, or nudge people to act in diferent ways. Also, AI-systems are not allowed to be used for “social scoring”, i.e., the use AI systems to collect and classify people’s social behavior based on data collected across diferent social contexts. The reason is that such an application could lead to discriminatory and unfair treatment of individuals. The proposal then identifies several types of AI-applications that are characterized as “high risk”. These include AI-applications used as part of products and might potentially harm the health and safety of consumers. But other types of applications are also considered to be “high risk”.
As regards high-risk AI systems other than those that are safety components of products, or which are themselves products, it is appropriate to classify them as high-risk if, in light of their intended purpose, they pose a risk of harm to the health and safety or the fundamental rights of persons, taking into account both the severity and the possible harm and its probability of occurrence, and they are used in a number of specifically pre-defined areas specified in the Regulation. [2], p. 35.
The regulation specifies these application areas in Annex III, and includes the use of AI in border control, education, recruitment, law enforcement, and critical infrastructure. The regulation also specifies criteria for the expansion of this list. In all the areas where the use of AI systems is classified as high-risks people are put in a vulnerable position in relation to the AI system with little control over the decisions that are made with the use of the systems. For such systems to be legal their development and application must comply with several requirements that are designed to minimize these risks.
Requirements should apply to high-risk AI systems as regards the quality of data sets used, technical documentation and record-keeping, transparency and the provision of information to users, human oversight, and robustness, accuracy and cybersecurity. Those requirements are necessary to efectively mitigate the risks for health, safety and fundamental rights, as applicable in the light of the intended purpose of the system, and no other less trade restrictive measures are reasonably available, thus avoiding unjustified restrictions to trade. [2], p. 42
We see that AI-applications are classified as high-risk with reference to individual rights in clearly specified situations where these individuals are put in vulnerable positions with little control and that the providers and users of the applications are required to minimize the risk as they arise in these specific situations.
In addition to this, the regulation includes requirements of risk mitigation which apply to all AI systems, including a requirement of transparency of systems that involve interaction with “natural persons”, whether they are classified as high-risk or not. “In particular, natural persons should be notified that they are interacting with an AI system, unless it is obvious from the point of view of a natural person . . . ” [2], p. 55.
It is useful in the ethical evaluation of technological risks [
General -purpose AI models that can be built into diferent types of application and used in diferent situations pose a challenge for this model. Should general models that could, potentially, be used in applications that are high-risk, themselves be classified as high-risk? In the public consultation following the Commission’s original proposal several of the position papers issued by the tech industry, in particular Microsoft and Facebook, argued that would be very dificult for developers of these general models to take measures to mitigate the risks of possibly unknown downstream applications. Instead, they argued, the responsibility for mitigating risks should be put on those that deployed the high-risk applications using the general-purpose models in the situations specified in the regulation. The problem was addressed in the Council’s revised version of the proposal, where a new section(12c) in the preamble on general-purpose AI was added. Here, the problem is recognized but not included in the text of the law.
In particular, it is necessary to clarify that general purpose AI systems are AI systems that are intended by the provider to perform generally applicable functions, such as image/speech recognition, and in a plurality of contexts. They may be used as high-risk AI systems by themselves or be components of other high risk AI systems. Therefore, due to their particular nature and in order to ensure a fair sharing of responsibilities along the AI value chain, such systems should be subject to proportionate and more specific requirements and obligations under this Regulation while ensuring a high level of protection of fundamental rights, health and safety. [2], p. 23
While the Council chose to delegate the regulation to future implementation rules the EU Parliament, which is currently finalizing its treatment of the AI Act, has chosen to include the regulation of general-purpose AI in the act. According to reports, they will require developers of general-purpose AI models to provide the deployers of AI-systems that builds on the general models, with information necessary to enable the deployers to comply with the requirements for the mitigation of risks of high-risk systems.
We now turn to the central question of the paper: does the proposed regulation adequately
manage the risks associated with generative AI models? The risks associated with these models
have been discussed extensively in the months since the public release of ChatGPT and a call for
a moratorium on the development of generative AI models have got the support of prominent
ifgures in the research community [
The challenge from the perspective of the AI Act is that, with its methods for risk identification the regulation would not recognize the generative AI applications as high-risk and the act’s proposed measures for risk mitigation are inadequate for reducing the risks associated with these technologies. I will argue for the following claim:
The model for regulating the use of AI in the AI Act, based on minimizing risks in specified situation, is not adequate for regulating generative AI applications like LLMs, because 1) with such applications, all potential contexts of use and all relevant stakeholders cannot be specified, and the risks are substantial but hard to estimate, 2) the ethical justification for the principles for regulating “high risk application” in the AI Act does not extend to the evaluation of the risks that generative AI pose.
As pointed out above, the risk model of the AI Act identifies high-risk AI applications in relation to specific situations of use where individuals are put in a vulnerable position. Moreover, the risk in questions are risks of harm to individual rights. As an example, consider the justification for classifying systems used in the administration of education as high-risk:
AI systems used in education or vocational training, notably for determining access, admission or assigning persons to educational and vocational training institutions or programmes at all levels or to evaluate learning outcomes of persons should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore afect their ability to secure their livelihood. When improperly designed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination. [2], p. 36)
We see that the high-risk application is this area is identified in relation to specific situations
where individuals are exposed to risks in virtue of their roles in the situations specified.
Importantly for our discussion, the wider societal risks related to the technology are not considered
with this method of risk identification. The risks associated that the generative AI models
cannot not be captured in a similar way. Consider the problem of radicalization. A recent report,
funded by OpenAI, documented how GPT3 could be used to generate highly convincing white
supremacist and other extremist content [
It might be suggested that the generative AI models are general purpose systems and the risk they pose can be handled though risk evaluations of the applications that are built on top of them. While it is true that many generative AI systems are general purpose systems, the risks of the systems that are built on them must, in the end, be identified and characterized with the methods discussed above. The risk assessment is transferred to the embedding applications, but the methods of risk identification will still be tied to specific contexts and the risks evaluated with reference to the way vulnerable individuals are treated with the use of AI application in those situations.
It is also clear that the requirement of transparency that applies to systems interacting with “natural persons” will not mitigate the risks posed by generative AI applications. As mentioned above, the requirement is that individuals that interact with AI systems shall be med aware that they are interacting with an artificial agent, unless it is obvious “from the point of view of a natural person who is reasonably well-informed, observant and circumspect taking into account the circumstances and context of use.” [2], p. 55) It is of course possible that generative AI systems will be embedded in applications that deceive users into believing they are interacting with a human. But the risks we have seen so far emerging from the generative models do not stem from ignorance of the part of users of the fact that the answers are generated by an AI. Those that abuse the system to produce extremist or toxic content are of course fully aware that they are using an AI system and disinformation can spread because the system provides convincing disinformation, not because people think they are interacting with a human.
Furthermore, one could ask if the generative AI models, in virtue of the risks they can give rise
to qualify as posing unacceptable risks and therefore be prohibited under the AI Act? There are
several reasons to think that they do not fall under the category of systems posing unacceptable
risks. First, the treatment of the proposal for the AI Act in the EU parliament, which were not
yet completed at the time of writing, was complicated by the impact of the release of ChatGPT.
However, according to the ‘Draft Compromise Amendments’ [
[F]oundational models should have information obligations and prepare all
necessary technical documentation for downstream providers to be able to comply
with their obligations under this Regulation. Generative foundational models
should ensure transparency about the fact the content is generated by an AI system,
not by humans. These specific requirements and obligations do not amount to
considering foundational models as high risk AI systems, but should guarantee that
the objectives of this Regulation to ensure a high level of protection of fundamental
rights, health and safety, environment, democracy and rule of law is achieved. [
We see that generative models are classified in a lower risk category than high-risk systems, subject only to transparency requirements.
Second, those applications that are classified as posing unacceptable risks and therefore prohibited on the regulation, are either violating individual integrity (systems using subliminal and manipulative techniques), giving rise to discriminatory practices (social scoring), or surveillance. The risks posed by generative AI models, spreading of disinformation and toxic content, are diferent from any of these. Furthermore, generative AI models have a very large number of useful applications and have the potential to contribute significantly to innovations with social benefits, which it is a central goal of the AI Act is to facilitate.
My second argument for the claim that the risk management model in the AI Act does not adequately address the risks posed by generative AI models was that the ethical justification of the principles that underlie the AI Act does not extend to the evaluation of the risks that generative AI models pose. That ethical justification was based on a distinction between three diferent roles or positions in connection with the use of technologies involving risk: those who benefit from the risk, those who decide about the risk, and those that are exposed to the risk. The ethical argument for the model of identification and mitigation in the AI Act took as its starting point the observation that in the situations specified as high-risk in the AI Act, there was a challenging distribution of roles: the same actors that befit from using the risky AI systems is also deciding about the use of the systems, while a diferent actor is exposed to the risks that the systems pose and will sufer harm if they fail. This is an ethically challenging type of situation because of the vulnerability of those that are risk exposed, the power relation between the decision maker and the risk exposed, and the incentive the decision maker could have to exploit the risk exposed. In some cases, the risk exposed could also to some extent benefit from the use of the technology, if the use of the AI system makes an application process in which the risk exposed is involved more eficient, for example. But unless the distribution of roles is unchanged the ethically challenging nature of the situation does not change significantly.
The regulation requires that the party that benefits from the risk and decides about the risk take the cost of putting in place measures to reduce the risk of the risk exposed to an acceptable level. Given the asymmetry of the situation and that the risk exposed often will not have the freedom to choose not to be exposed to the risk, this is a fair distribution of costs and responsibilities. With the generative AI models, however, the same ethical problem remains, with one party both benefiting from the risk and deciding whether to run it, while a diferent party, the risk exposed, bears the risk. But with the generative models the relationship of the risk exposed to the technology is diferent from those targeted by the AI Act. The crucial diference is that the risks do not arise primarily in the situation of use but in the context of communication in the wider social contexts of which the user is a member. As argued above, with applications like LMMs, where the risks are related to such things as the spread of false information, radicalization, and erosion of trust, the risk situations cannot be specified in the same way as with the AI Act because these risks arise across a broad range of diferent situations of use and potentially afect a broad and unspecified range of stakeholders. The measures the act prescribes for the mitigation of the risk will therefore not resolve the ethical challenge in this case.
We have argued for the negative conclusion that, with the resources provided by the risk model
in the AI Act, the ethical challenges of general-purpose cannot be adequately met. This does
not show, of course, that regulations that mitigate the risks associated with these models cannot
be found. But such regulations will have to force the developers of the systems to make sure
that they are mostly truthful and prevent the systems from spreading disinformation. Whether
this level of control is possible to achieve with LLMs seems to be an open question [
1 European Commission. Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS. 2021. 2 Council of the European Union. Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts. 2022.