Under the headline \AI safety", a wide-reaching issue is being discussed, whether in the future some \superhuman arti cial intelligence" / \superintelligence" could pose a threat to humanity. In addition, the late Steven Hawking warned that the rise of robots may be disastrous for mankind. A major concern is that even benevolent superhuman arti cial intelligence (AI) may become seriously harmful if its given goals are not exactly aligned with ours, or if we cannot specify precisely its objective function. Both the de nition and communication of such goals have to be conducted with extreme caution. Metaphorically, this is compared to king Midas in Greek mythology, who expressed the wish that everything he touched should turn to gold, but obviously this wish was not speci ed precisely enough. In our view, this sounds like requirements problems and the challenge of their precise formulation. As usual in requirements engineering (RE), ambiguity or incompleteness may cause problems. That is why we actually propose (and partly work on already) yet another RE problem, guring out the wishes and the needs with regard to a superintelligence, which will in our opinion most likely be a very complex software-intensive system based on AI. To our best knowledge, this view of \AI safety" has not been pointed out yet. Requirements engineering appears to be the right approach to address it rst, preferably before serious attempts of building \superhuman arti cial intelligence" are being conducted.
The idea of technology becoming sentient has been a common theme in the literature and movies for decades. One might think of the famous movie \2001: A Space Odyssey" by Stanley Kubrick, the Matrix, or the Terminator movies. These stories seem to be mostly consistent in the impression that a suddenly arising uncontrolled Arti cial Intelligence (AI) will not mean us well. Furthermore, the AIs described in books and movies are never dumb machines, but rather potent entities with cognitive superpowers far beyond the capacities of human general intelligence | they are superintelligent. In his book \Superintelligence" [Bos14], Oxford philosophy professor Nick Bostrom provides good reasons to believe that we can create such an entity, and he is not the only one (see, e.g., [Yam15]).
In the AI research priorities document published in [RDT15], apart from the desirability of safety, e.g., of self-driving cars, \AI safety" is addressed with regard to \superhuman AI". This is partly based on forecasting work on intelligence explosion and \superintelligence" [Bos14, ELH18], where the importance of given goals to be exactly aligned with those of mankind is emphasized. Even that may be insu cient, however, the system must also somehow be deliberately constructed to pursue them [Bos14].
In terms of requests like \Make paperclips", such a system is envisaged to take everything it can (with high \intelligence") and to make as many paperclips as it can make, whatever it may cost. Much like the example of king Midas or the core theme of the movie with the title \Bedazzled", we view this as a requirement that is not speci ed precisely enough. While other references regarding \AI safety" can be found in [ELH18] and at https://vkrakovna.wordpress.com/ai-safety-resources/, it is mentioned nowhere that it is actually a problem with requirements.
In addition to specifying and communicating requirements to a superintelligence, once available, we see a more pressing problem before it may be built. Since mankind may eventually create a \superintelligence", it should rather sooner than later specify the requirements on such a system. Hence, we propose RE on a superintelligence here.
Having a glass-box view on a system implementing a superintelligence may be helpful for keeping control of it, such as following an approach to making architectural decisions upfront, e.g., for a generic architecture [MLK+00]. Another approach is to model safety frameworks, see, e.g., [EKKL19], where deep neural networks are components of a framework. In contrast, we take a black-box view here and propose to focus on requirements.
Strictly speaking, the notion \AI safety" is misleading in our opinion, since it may also include the safety of certain AI-based systems, e.g., self-driving cars. Hence, we prefer the notion of \superintelligence safety". We also prefer this notion over \AGI safety", since the issue is not so much about an \Arti cial General Intelligence", but a \superintelligence" based on AI, where an intelligence explosion may arise [Bos14].
The remainder of this paper is organized in the following manner. First, we provide some background material on superintelligence safety, in order to explain the challenge involved. Then we view superintelligence safety from the perspective of RE, and envisage specifying and communicating requirements accordingly. Finally, we propose RE on a superintelligence in the rst place. 2
First, let us be more precise on what \superintelligence safety" really means. This concept includes minimizing the existential risk to humanity posed by superintelligent agents as well as mitigating unwanted social consequences that may arise even if the existential risk has been averted, such as social manipulation, advanced warfare, AI-driven unemployment, status quo preservation or unfair redistribution of resources. In a more technical sense, this means solving the "Control Problem" [Bos14] and designing and incorporating safety mechanisms in a seed AI which are still functional after arbitrarily many iterations of self-improvement by the system. "Ideally, every generation of self-improving system should be able to produce a veri able proof of its safety for external examination" [Yam15, p.138].
Superintelligence safety also does not only apply to superintelligent systems to-be-built but also extends to the development process, which in itself poses an existential risk. In this context, Yampolskiy proposes the setup of AI research review boards consisting of a team of experts which are to evaluate each research proposal and decide if it can potentially lead to the development of a full-blown AGI [Yam15, p.139]. RE may be applied to the development process, the system to-be-built, to the goal(s) the system will be given and maybe also to the process by which the goal(s) will be selected.
Assuming the creation of a superintelligence is possible, how can we control it and avoid the doomsday scenarios so eloquently depicted by authors and lm directors? While an emerging superintelligence might not be inherently malevolent, it still poses a great risk. Without any information about the internal workings of such an entity, we have to assume that its way of `thinking' might be very di erent from the way a human thinks. A superintelligence most likely will not share human morals or have a concept of value at all, if the developers did not intentionally design it to have one.
While the most terrifying scenario is a malevolent superintelligence with the intent to eradicate humanity, another potential scenario is a superintelligence with no inherent motivations at all, simply following the instructions of it's creator to the best of its abilities. Bostrom describes some of the abilities a superintelligence could have under the term \Cognitive Superpowers" [Bos14, p. 91]. A superintelligence could excel at strategic planning, forecasting, analysis for optimizing chances of achieving distant goals, social and psychological modeling and manipulation, rhetoric persuasion, hacking into computer systems, design and modeling of advanced technologies, and the list goes on [Bos14, p. 94].
One can see that even equipped with only a subset of these skills, a superintelligence pursuing a goal to the best of its abilities and without being constrained by concepts like morality or value, can cause substantial damage to its environment. Bostrom depicts this with his famous thought experiment where a superintelligence is given the simple task to make as many paper clips as possible. While the superintelligence might start o by acquiring monetary resources by predicting the stock market and building paper clip factories, it may not just stop there. Since its goal is the unconstrained maximization of the production of paperclips, it will soon discover that humans pose a hindrance to its endeavor, as we might try to stop the superintelligence from producing more paperclips or even try to shut it o . Since the superintelligence is multiple magnitudes smarter than humanity combined, humanity fails to stop the superintelligence and ultimately the whole world (the whole observable universe, in fact) will be made into paperclips.
Even if this was just a contrived example, one can see the risks of giving a superintelligence an unconstrained optimization problem. This argument is also extended in [Bos14] to constrained optimization problems, but the details are not necessary for our paper. 3
Under the (usually unrealistic) assumption that we knew the requirements already, `just` telling the superintelligence about them properly is the problem here. It can be decomposed into specifying and communicating.
According to the Standard ISO/IEC 10746-2:2009 Information technology { Open Distributed Processing { Reference Model: Foundations, 7.4, a speci cation is a \concrete representation of a model in some notation". In order to better understand this sub-problem of specifying requirements, let us follow the observation in [KS10] that requirements representations are often confused with requirements per se. This confusion is also widespread in practice, as exempli ed in the very recent Standard ISO SE Vocabulary 24765:2017. In fact, it de nes a requirement both as a \statement that translates or expresses a need and its associated constraints and conditions" and \a condition or capability that must be present in a product . . . " (dots inserted).
As it is well-known in RE, specifying requirements can be done informally, say, using some natural language, or formally, e.g., using some formal logic as the notation. Possibilities for semi-formal representations within this spectrum are many-fold. With regard to specifying requirements to a superintelligence, especially ambiguity and incompleteness appear to be major concerns, cf. the paperclip example and king Midas.
In order to avoid ambiguity in the course of specifying requirements for a superintelligence, formal representation of the requirements should be the choice, of course, possibly in some formal logic. Grounding the logic used in the domain will, however, still leave loopholes for the superintelligence to `misunderstand' the given speci cation of our requirements. For example, for some predicate on paperclips, a grounding of what paperclips are in the real world will be necessary.
Also incompleteness of the speci cation remains an open issue. Unfortunately, no general solution appears to be feasible at the current state of the art in RE.
Communicating a given requirements speci cation to a superintelligence may be investigated according to [JMF08], based on speech-act theory [Sea69], under the premise that stakeholders communicate information in the course of requirements engineering. This view was not taken for the sake of really communicating requirements in [JMF08], but for using the semantics of various speech acts to theoretically (re-)de ne the requirements problem originally de ned by Zave and Jackson [ZJ97] (see also below). Everything is assumed to be communicated by speech acts here. However, we can only communicate representations of requirements, and not requirements per se. It is clear that the text given in the examples in [JMF08] represents something by describing it, the very confusion addressed in [KS10], where speci c examples of this confusion are given.
Anyway, for communicating speci cations of requirements to a superintelligence, speech-act theory could be helpful for annotating the speci c kind of speech act, e.g., Question or Request. For example, the text \Can you produce paperclips?" may be interpreted either way, unless speci ed more precisely.
Still, we have to face that `perfectly' specifying and communicating requirements to a superintelligence may not be possible. Even if we could, this would not help in case of malevolent superintelligence that would not
Motivation Selection Boxing methods Incentive methods
Stunting
Tripwires necessarily satisfy requirements as speci ed and communicated. Hence, let us consider building a superintelligence in such a way that these problems can be mitigated. 4
In our view, building a superintelligence should certainly include RE (much as building any non-trivial system). For speci cally addressing superintelligence safety, we may ignore de ning functional requirements on its superintelligent abilities. Still, some functionality may have to be de ned for operationalizing certain \non-functional" safety requirements, which will in the rst place be cast as constraints on the superintelligence.
Note, that such requirements can and should be de ned, elaborated and implemented for AI-based systems already before these may actually become superintelligent. This may happen all of a sudden, and then it may be too late.
Working on these requirements could be done as usual according to best practice of RE. This will involve major stakeholders (in particular, AI researchers), requirements elicitation may be done including questionnaires in the Web, etc. While the core questions will be about requirements on superintelligence safety, of course, a bigger question should be asked in our opinion, what the goals and requirements are regarding AI in the rst place. As its stands, most AI system are created and deployed without doing any RE at all.
We expect that RE for superintelligence safety will pose even greater challenges than the usual practice of RE. After all, it is not just about conceiving such a superintelligence but about making sure that its creation will not raise uncontrollable safety risks. The currently most widely used approach to functional safety of softwareintensive systems [SS11, VCMG18, LL03, HKR+16] will not be su cient. It is primarily concerned with hazards in the environment of the system and related safety risks, and as dealt with, e.g., in the current automotive standard ISO 26262, it focuses on failures of functions and how to manage them. Superintelligence safety will have to make sure that the superintelligent system will not create hazards even if none of its functions has a failure. This will have to be dealt with in a new kind of safety requirements.
Let our RE endeavor be also informed by some preliminary thought by the author who raised the issue of superintelligence safety. In [Bos14, Chapter 9], controlling a superintelligence is discussed, in order to deal with this speci c safety problem. Two broad classes of potential methods are distinguished | capability control and motivation selection. Within each of them, several speci c techniques are examined, see Figures 1 and 2 for overviews.
Viewed from an RE perspective, it seems as though most of what is discussed here could be elaborated as constraint requirements. Both current theory and practice of RE will most likely be su cient for dealing with capability control as laid out here.
The ideas of motivation selection will be much harder to elaborate according to the current state of the art in RE than the ideas on capability control above. This may even involve extending the current theoretical formulation of the RE problem. 5
In this paper, we address the potentially very important issue of \AI safety", in the sense of superintelligence safety [Bos14], from an RE perspective. To our best knowledge, this is the rst RE approach to this issue, although it may seem obvious that RE is the very discipline of choice here (for someone being aware of RE).
We actually distinguish two di erent approaches:
Specifying and communicating requirements for speci c problems to a concrete superintelligence, and Doing RE in the course of building a superintelligence in the sense of an AI-based software-intensive system.
Based on common wisdom of RE at the current state of the art, we tentatively conclude that `perfectly' specifying and communicating requirements to a superintelligence may not be possible. And even if this became possible in the future, it would not help in case of malevolent superintelligence that would not necessarily satisfy requirements as speci ed and communicated.
Hence, we already pursue RE on a superintelligence to be built. This RE endeavor is informed by some preliminary thought on controlling a superintelligence in [Bos14]. The rst approach through capability control may be dealt with properly through constraint requirements. The second approach for controlling through motivation selection, however, appears to go beyond the current theory of RE. In particular, we raise the challenge of extending Goal Oriented Requirements Engineering (GORE) with dynamic goals of a superintelligence. In [KF19], we motivate how the theory of the requirements problem should be extended to cover goals of a superintelligence.
More speci cally, at the time of this writing we are already preparing for gathering requirements on an AIbased superintelligence through Web-questionnaires from a wide variety of stakeholders from di erent business sectors, age groups and educational backgrounds. These should contribute to a better understanding of the wishes and (perceived) needs of di erent stakeholders with regard to a superintelligence, especially related to safety of humans or even mankind.
Our road map includes
Distributing this document to the ones having lled-in the questionnaires for receiving further feedback; and Integrating this feedback into a second baseline of a requirements speci cation on superintelligence safety.
In the long run, this may lead to a more elaborate approach along the lines of the moral machine experiment [ADK+18]. [Bos14]
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford, 2014. [EKKL19] Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg. Modeling AGI safety frameworks with causal in uence diagrams. In Proceedings of the Workshop on Arti cial Intelligence Safety 2019, volume Vol-2419. CEUR Workshop Proceedings, 2019.
arXiv:1805.01109 [cs.AI], arXiv, 2018.
[JMF08]