Robust alignment is arguably both critical and extremely challenging. Loosely, it is the problem of designing algorithmic systems with strong guarantees of always being beneficial for mankind. In this paper, we propose a preliminary research program to address it in a reinforcement learning framework. This roadmap aims at decomposing the end-toend alignment problem into numerous more tractable subproblems. We hope that each subproblem is sufficiently orthogonal to others to be tackled independently, and that combining the solutions to all such subproblems may yield a solution to alignment.
As they are becoming more and more capable and
ubiquitous, AIs are raising numerous concerns, including
fairness, privacy, filter bubbles, addiction, job displacement or
even existential risks
Unfortunately, it has also been argued that alignment is
an extremely difficult problem. In fact,
In particular, we shall introduce a complete research program to robustly align AIs. Robustness here refers to numerous possible failure modes, including overfitting, hazardous exploration, evasion attacks, poisoning attacks, crash tolerance, Byzantine resilience, reward hacking and wireheading. To guarantee such a robustness, we argue that it is desirable to structure (at least conceptually) our AI systems in their 1https://futureoflife.org/landscape/ entirety. This motivated us to propose a roadmap for robust end-to-end alignment.
While much of our proposal is speculative, we believe that several of the ideas presented here will be critical for AI safety and alignment. More importantly, we hope that this will be a useful roadmap for both AI experts and non-experts to better estimate how they can best contribute to the effort.
Given the complexity of the problem, our roadmap here will likely be full of gaps and false good ideas. It is important to note that our purpose is not to propose a definite perfect solution. Rather, we aim at presenting a sufficiently good starting point for others to build upon.
Our roadmap consists of identifying key steps to alignment. For the sake of exposition, these steps will be personified by 5 characters, called Alice, Bob, Charlie, Dave and Erin. Roughly speaking, Erin will be collecting data from the world, Dave will use these data to infer the likely states of the world, Charlie will compute the desirability of the likely states of the world, Bob will derive incentive-compatible rewards to motivate Alice to take the right decision, and Alice will optimize decision-making. This decomposition is graphically represented in Figure 1.
Evidently, Alice, Bob, Charlie, Dave and Erin need not be 5 different AIs. Typically, it may be much more computationally efficient to merge Charlie and Dave. Nevertheless, at least for pedagogical reasons, it seems useful to first dissociate the different roles that these AIs have.
In the sequel, we shall further detail the challenges posed by each of the 5 AIs. We shall also argue that, for robustness and scalability reasons, these AIs will need to be further divided into many more AIs. We will see that this raises additional challenges. We shall also make a few non-technical remarks, before concluding.
It seems that today’s most promising framework for largescale AIs is that of reinforcement learning. In reinforcement learning, an AI can be regarded as a decision-making process. At time t, the AI observes some state of the world st. Depending on its inner parameters t, it then takes (possibly randomly) some action at.
The decision at then influences the next state and turns it into st+1. The transition from st to st+1 given action at is usually assumed to be nondeterministic. In any case, the AI then receives a reward Rt+1. The internal parameters t of the AI may then be updated into t+1, depending on previous parameters t, action at, state st+1 and reward Rt+1.
Note that this is a very general framework. In fact, we humans are arguably (at least partially) subject to this framework. At any point in time, we observe new data st that informs us about the world. Using an inner model of the world t, we then infer what the world probably is like, which motivates us to take some action at. This may affect what likely next data st+1 will be observed, and may be accompanied with a rewarding (or painful) feeling Rt+1, which will motivate us to update our inner model of the world t into t+1.
Let us call Alice the AI in charge of performing this reinforcement learning reasoning. Alice can thus be viewed as an algorithm, which inputs observed states st and rewards Rt, and undertakes actions at so as to typically maximize some discounted sum of expected future rewards.
Such actions will probably be mostly of the form of messages sent through the Internet. This may sound benign. But it is not. The YouTube recommender system might suggest billions of antivax videos, causing a major decrease of vaccination and an uprise of deadly diseases. Worse, if an AI is in control of 3D-printers, then a message that tells them to construct killer drones to cause a genocide would be catastrophic. On a brighter note, if an AI now promotes convincing eco-friendly messages every day to billions of people, the public opinion on climate change may greatly change.
Note that, as opposed to all other components, in some sense, Alice is the real danger. Indeed, in our framework, she is the only one that really undertakes actions. More precisely, only her actions will be unconstrained (although others highly influence her decision-making and are thus critical as well).
As a result, it is of the utmost importance that Alice be
well-designed. Some of the past work
Indeed, there is already a race between competing companies in competing countries to construct powerful AIs. While it might be possible for some countries to impose some restrictions to some AIs of some companies, it is unlikely that all companies of all countries will accept to be restricted, especially if the restrictions are too constraining. In fact, AI safety will be useful only if the most powerful AIs are all subject to safety measures. As a result, the safety measures that are proposed should not be too constraining. In other words, there are constraints on the safety constraints that can be imposed. This is what makes AI safety so challenging.
As a result, what is perhaps more interesting are the ideas
proposed by
But this is not all. Given that AIs will likely be based on machine learning, and given the lack of verification methods for AIs obtained by machine learning, we should not expect AIs to be correct all the time. Just like humans, AIs will likely be sometimes wrong. But this is extremely worrysome. Indeed, even if an AI is right 99.9999% of the time, it will still be wrong one time out of a million. Yet, AIs like recommender systems or autonomous cars take billions of decisions every day. In such cases, thousands of AI decisions may be unboundedly wrong every day!
This problem can become even more worrysome if we
take into account the fact that hackers may attempt to take
advantage of the AIs’ deficiencies. Such hackers may
typically submit only data that corresponds to cases where the
AIs are wrong. This is known as evasion attacks
Now, even if Alice is well-designed, she will only be an
effective optimization algorithm. Unfortunately, this is no
guarantee of safety or alignment. Typically, because of
humans’ well-known addiction to echo chambers
In fact,
To make sure that Alice will want to behave as we want her to, it seems critical to at least partially control the observed state st+1 or the reward Rt+1. Note that this is similar to the way children are taught to behave. We do so by exposing them to specific observed states, by punishing them when the sequence (st; at; st+1) is undesirable, and by rewarding them when the sequence (st; at; st+1) is desirable.
Whether or not Alice’s observed state st is constrained, her rewards Rt are clearly critical. They are her incentives, and will thus determine her decision-making. Unfortunately, determining the adequate rewards Rt to be given to Alice is an extremely difficult problem. It is, in fact, the key to alignment. Our roadmap to solve it identifies 4 key steps incarnated by Erin, Dave, Charlie and Bob.
In order to do good, it is evidently crucial to be given a lot of reliable data. Indeed, even the most brilliant mind will be unable to know anything about the world if it does not have any data from that world. This is particularly true when the goal is to undertake desirable actions, or to make sure that one’s action will not have potentially catastrophic consequences.
Evidently, much data is already available on the Internet. It is likely that any large-scale AI will have access to the Internet, as is already the case of the Facebook recommender system. However, it is important to take into account the fact that the data on the Internet is not always fully reliable. It may be full of fake news, fraudulent entries, misleading videos, hacked posts and corrupted files.
It may then be relevant to invest in more reliable and
relevant data collection. This would be Erin’s job. Typically,
Erin may want to collect economic metrics to better assess
needs. Recently, it has been shown that satellite images
combined with deep learning allow to compute all sorts of
useful economic indicators
To guarantee the reliability of such data, cryptographic
and distributed computing solutions are likely to be
useful as well, as they already are on the web. In
particular, distributed computing, combined with recent
Byzantineresilient consensus algorithms like Blockchain
Note though that such data collection mechanisms could
pose major privacy issues. It is a major current challenge
to balance the usefulness of collected data and the privacy
violation they inevitably cause. Some possible solutions
include differential privacy
Unfortunately, raw data are usually extremely messy, redundant, incomplete, unreliable, poisoning and even hacked. To tackle these issues, it is necessary to infer the likely actual states of the world, given Erin’s collected data. This will be Dave’s job.
The overarching principle of Dave’s job is probably going
to be some deep representation learning. This corresponds
to determining low-dimensional representations of
highdimensional data. This basic idea has given rise to today’s
most promising unsupervised machine learning alogrithms,
e.g. word vectors
2The possible use of quantum computers may require postquantum cryptography.
Given how crucial it is for Dave to have an unbiased
representation of the world, much care will be needed to make
sure that Dave’s inference will foresee selection biases. For
instance, when asked to provide images of CEOs, Google
Image may return a greater ratio of male CEOs than the
actual ratio. More generally, such biases can be regarded as
instances of Simpson’s paradox
In fact, data can be worse than unintentionally misleading.
Given how influential Alice may be, there will likely be great
incentives for many actors to bias Erin’s data gathering, and
to thus fool Dave. This is known as poisoning attacks
This inference approach is well captured by the Bayesian
paradigm
P[DjT ]P[T ] P[T jD] = :
P[D]
One typical instance of Dave’s job is the problem of inferring global health from a wide variety of collected data. This is what has been done by (Institute for Health Metrics and Evaluation (IHME), University of Washington 2016), using a sophisticated Bayesian model that reconstructed the likely causes of deaths in countries where data were lacking.
Importantly, Bayes rule also tells us that we should not fully believe any single theory. This simply corresponds to saying that data can often be interpreted in many different mutually incompatible manners. It seems important to reason with all possible interpretations rather than isolating a single interpretation that may be flawed.
When the space of possible states of the world is large, which will surely be the case of Dave, it is often computationally intractable to reason with the full posterior distribution P[T jD]. Bayesian methods often rather propose to sample from the posterior distribution to identify a reasonable number of good interpretations of the data. These sampling methods include Monte-Carlo methods, as well as MarkovChain Monte-Carlo (MCMC) ones.
In some sense, Dave’s job can be regarded as writing a compact report of all likely states of the world, given Erin’s collected data. It is an open question as of what language Dave’s report will be in. It might be useful to make it understandable by humans. But it might be too costly as well. Indeed, Dave’s report might be billions of pages long. It could be unreasonable or undesirable to make it humanly readable.
Note also that Erin and Dave are likely to gain cognitive capabilities over time. It is surely worthwhile to anticipate the complexification of Erin’s data and of Dave’s world models. It seems unclear so far how to do so. Some high-level (purely descriptive) language to describe world models is probably needed. In addition, this high-level language may need to be flexible enough to be reshaped and redesigned over time. This may be dubbed the world description problem. It is arguably still a very open and uncharted area of research.
Given any of Dave’s world models, Charlie’s job will then
be to compute how desirable this world model is. This is
the desirability learning problem
Unfortunately, determining what, say, the median human considers desirable is an extremely difficult problem. But again, it should be stressed that we should not aim at deriving an ideal inference of what people desire. This is likely to be a hopeless endeavor. Rather, we should try our best to make sure that Charlie’s desirability scores will be good enough to avoid catastrophic outcomes, e.g. world destruction, global sufferance or major discrimination.
One proposed solution to infer human preferences is
socalled inverse reinforcement learning
First, it is important to keep in mind that, despite Dave’s effort and because of Erin’s limited and possibly biased data collection, Dave’s world model is fundamentally uncertain. In fact, as discussed previously, Dave would probably rather present a distribution of likely world models. Charlie’s job should be regarded as a scoring of all such likely world models. In particular, she should not assign a single number to the current state of the world, but, rather, a distribution of likely scores of the current state of the world. This distribution should convey the uncertainty about the actual state of the world. Besides, as we shall see, this uncertainty is likely to be crucial for Bob to choose incentive-compatible rewards for Alice adequately.
Another challenging aspect of Charlie’s job will be to provide a useful representation of potential human disagreements about the desirability of different states of the world. Humans’ preferences are diverse and may never converge. This should not be swept under the rug. Instead, we need to agree on some way to mitigate disagreement.
3To avoid raising eyebrows, we shall try to steer away from polarizing terminologies like values, moral or ethics.
This is known as a social choice problem. In its general
form, it is the problem of aggregating the preferences of a
group of disagreeing people into a single preference for the
whole group that, in some sense, fairly well represents the
individuals’ preferences. Unfortunately, social choice theory
is plagued with impossibility results, e.g. Arrow’s theorem
One particular proposal, known as majority judgment
Now, to avoid an oppression of a majority over some minority, it might be relevant to choose a small value of q, say q = 0:1. This would mean that Charlie’s scoring of a state of the world will be less than a number score, if more than 10% of the people believe that this state should be given a score less than score. But evidently, this point is very much debatable. It seems unclear so far how to best choose q.
While majority judgment seems to be a promising
approach, it does raise the question of how to compare two
different individuals’ scores. It is not clear that score = 5 given
by John has a meaning comparable to Jane’s score = 5. In
fact, according to a theorem by von Neumann and
Morgenstern
Now, arguably, humans’ current preferences are almost
surely undesirable. Indeed, over the last decades,
psychology has been showing again and again that human
thinking is full of inconsistencies, fallacies and cognitive biases
And arguably, we should prefer what we would prefer to prefer, rather than what we instinctively prefer. Typically, one might prefer to watch a cat video, even though one might prefer to prefer mathematics videos over cat videos. Desirablity scores should arguably encode what we would prefer to prefer, rather than what we instinctively prefer.
To understand, a thought experiment may be useful. Let us imagine better versions of us. Each current me is thereby associated with a me++. A me++ is what current me would desire, if current me were smarter, thought much longer about what he finds desirable, and analyzed all imaginable data of the world. Arguably, me++’s desirability score is “more right” than current me’s.
This can be illustrated by the fact that past standards are often no longer regarded as desirable. Our intuitions about the desirability of slavery, homosexuality and gender discrimination have been completely upset over the last century, if not over the last few decades. It seems unlikely that all of our other intuitions will never change. In particular, it seems unlikely that me++ will fully agree with current me. And it seems reasonable to argue that me++ would be “more right” than current me.
These remarks are the basis of coherent extrapolated
volition
It is noteworthy that we clearly have epistemic uncertainty about me++’s. Determining me++’s desirability scores may be called the coherent extrapoled individual volition problem. Interestingly, this is (mostly) a prediction problem. But it is definitely too ambitious to predict them with absolute uncertainty. Bayes rule tells us that we should rather describe these desirability scores by a probability distributions of likely desirability scores.
Such scores could also be approximated using a large
number of proxies, as is done by boosting methods
Computing the desirability of a given world state is Charlie’s job. In some sense, Charlie’s job would thus be to remove cognitive biases from our intuitive preferences, so that they still basically reflect what we really regard as preferable, but in a more coherent and informed manner. This is an incredibly difficult problem, which will likely take decades to sort out reasonably well. This is why it is of the utmost importance that it be started as soon as possible. Let us try our best to describe, informally and formally, what better versions of ourselves would likely regard as desirable. Let us try to predict the volition of me++’s.
This attempt is likely going to be shocking to us all. Indeed, we should expect that better versions of ourselves will find desirable things that the current versions of ourselves find repelling. Unfortunately though, we humans tend to react poorly to disagreeing jugments. And this is likely to hold even when the oppositions are our better selves. This poses a great scientific and engineering challenge. How can one be best convinced of the judgments that he or she will eventually embrace but does not yet? In other words, how can we quickly agree with better versions of ourselves? What could someone else say to get me closer to my me++? This may be dubbed the individual improvement problem.
To address this issue,
This question is critical for alignment as it will likely be a key challenge to build trust in the systems we design. But evidently, this is a more general question that should be of interest to anyone who desires to do good.
The last piece of the jigsaw is Bob’s job. Bob is in charge of computing the rewards that Alice will receive, based on the work of Erin, Dave and Charlie. Evidently he could simply compute the expectation of Charlie’s scores for the likely states of the world. But this is probably a bad idea, as it opens the door to reward hacking.
Recall that Alice’s goal is to maximize her discounted expected future rewards. But given that Alice knows (or is likely to eventually guess) how her rewards are computed, instead of undertaking the actions that we would want her to, Alice could hack Erin, Dave or Charlie’s computations, so that such hacked computations yield large rewards. This is sometimes called the wireheading problem.
Since all this computation starts with Erin’s data collection, one way for Alice to increase her rewards would be to feed Erin with fake data that will make Dave infer a deeply flawed state of the world, which Charlie may regard as ideal. Worse, Alice may then find out that the best way to do so would be to invest all of Earth’s resources into misleading Erin, Dave and Charlie. This could potentially be extremely bad for mankind. Indeed, especially if Alice cares about discounted future rewards, she might eventually regard mankind as a possible threat to her objective.
This is why it is of the utmost importance that Alice’s incentives be (partially) aligned with Erin, Dave and Charlie performing well and being accurate. This will be Bob’s job. Bob will need to make sure that, while Alice’s rewards do correlate with Charlie’s scores, they also give Alice the incentives to guarantee that Erin, Dave and Charlie perform as reliably as possible the job they were given.
In fact, it even seems desirable that Alice be incentivized
to constantly upgrade Erin, Dave and Charlie for the
better. Ideally, she would even want them to be
computationally more powerful than herself, especially in the long run.
This approach would bear resemblance with the idea of
self-nudge
Unfortunately, it seems unclear how Bob can best make sure that Alice has such incentives. Perhaps a good idea is to penalize Dave’s reported uncertainty about the likely states of the world. Typically, Bob should make sure Alice’s rewards are affected by the reliability of Erin’s data. The more reliable Erin’s data, the larger Alice’s rewards. Similarly, when Dave or Charlie feel that their computations are unreliable, Bob should take note of this and adjust Alice’s rewards accordingly to motivate Alice to provide larger resources for Charlie’s computations.
Now, Bob should also mitigate the desire to retrieve more reliable data and perform more trustworthy computations with the fact that such efforts will necessarily require the exploitation of more resources, probably at the expense of Charlie’s scores. It is this non-trivial trade-off that Bob will need to take care of.
Bob’s work might be simplified by some (partial) control
of Alice’s action or world model. Although it seems unclear
so far how, techniques like interactive proofs (IP)
It may or may not be useful to enable Bob to switch off
Alice. It should be stressed though that (safe) interruptibility
is nontrivial, as discussed by
On another note, it may be computationally more efficient for all if, instead of merely transmitting a reward, Bob also feeds Alice with ”backpropagating signals”, that is, information not about the reward itself, but about its gradient with respect to key variables, e.g. Charlie’s score or Erin’s reliability. Having said this, we leave open the technical question of how to best design this.
4Note though that this may be very relevant assuming that there are several Alices, as will be proposed later on.
We have decomposed alignment into 5 components for the sake of exposition. However, any component will likely have to be decentralized to gain reliability and scalability. In other words, instead of having a single Alice, a single Bob, a single Charlie, a single Dave and a single Erin, it seems crucial to construct multiple Alices, Bobs, Charlies, Daves and Erins.
This is key to crash-tolerance. Indeed, a single
computer doing Bob’s job could crash and leave Alice
without reward nor penalty. But if Alice’s rewards are an
aggregate of rewards given by a large number of Bobs, then
even if some of the Bobs crash, Alice’s rewards will remain
mostly the same. But crash-tolerance is likely to be
insufficient. Instead, we should design Byzantine-resilient
mechanisms, that is, mechanisms that still perform correctly
despite the presence of hacked or malicious Bobs. Estimators
with large statistical breakdowns
Evidently, in this Byzantine environment, cryptography, especially (postquantum?) cryptographical signatures and hashes, are likely to play a critical role. Typically, Bobs’ rewards will likely need to be signed. More generally, the careful design of secure communication channels between the components of the AIs seems key. This may be called the secure messaging problem.
Another difficulty is the addition of more powerful and
precise Bobs, Charlies, Daves and Erins to the pipeline. It
is not yet clear how to best integrate reliable new comers,
especially given that such new comers may be malicious. In
fact, they may want to first act benevolent to gain
admission. But once they are numerous enough, they could take
over the pipeline and, say, feed Alice with infinite rewards.
This is the upgrade problem, which was recently discussed
by
Now, in addition to reliability, decentralization may also enable different Alices, Bobs, Charlies, Daves and Erins to focus on specific tasks. This would allow to separate different problems, which could lead to more optimized solutions at lower costs. To this end, it may be relevant to adapt different Alices’ rewards to their specific tasks. Note though that this could also be a problem, as Alices may enter in competition with one another like in the prisoner’s dilemma. We may call it the specialization problem. Again, there seems to be a lot of new research needed to address this problem.
Another open question is the extent to which AIs should be exposed to Bobs’ rewards. Typically, if a small company creates its own AI, to what extent should this AI be aligned? It should be noted that this may be computationally very costly, as it may be hard to separate the signal of interest to the AI from the noise of Bobs’ rewards. Intuitively, the more influential an AI is, the more it should be influenced by Bobs’ rewards. But even if this AI is small, it may be important to demand that it be influenced by Bobs to avoid any diffusion of responsibility, i.e. many small AIs that disregard safety concerns on the ground that they each hardly have any global impact on the world.
What makes this nontrivial is that any AI may gain capability and influence over time. An unaligned weak AI could eventually become an unaligned human-level AI. To avoid this, even basic, but potentially unboundedly selfimproving5 AIs should be given at least a seed of alignment, which may grow as AIs become more powerful. More generally, AIs should strike a balance between some original (possibly unaligned) objective and the importance they give to alignment. This may be called the alignment burden assignment problem.
Figure 2 recapitulates our complete roadmap.
Given the difficulty of alignment, its resolution will surely require solving a large number of non-technical challenges as well. We briefly mention some of them here.
Perhaps most important is the lack of respectability that is sometimes associated with this line of research. For alignment to be solved, it needs to gain respectability from the scientific community, and perhaps beyond this community as well. This is why it seems to be of the utmost importance that discussions around alignment be carried out carefully to avoid confusions.
Evidently, alignment definitely needs much more manpower, which will require funding and recruiting. It seems particularly important to attract mathematical talents towards this line of work. This evidently also raises the challenge of training as many brilliant minds as possible.
Finally, questions around AI, AI safety and moral philosophy are sadly often poorly debated. There often is a lot of overconfidence, and a lack of well-founded reasoning. For alignment research to gain momentum, it seems crucial to make debating more informative, respectful and stimulating.
This paper discussed the alignment problem, that is, the problem of aligning the goals of AIs with human preferences. It presented a general roadmap to tackle this issue. Interestingly, this roadmap identifies 5 critical steps, as well as many relevant aspects of these 5 steps. In other words, we have presented a large number of hopefully more tractable subproblems that readers are highly encouraged to tackle. We hope that combining the solutions to these subproblems could help to partially address alignment. And we hope that any reader will be able to better determine how he or she may best contribute to the global effort6.
Acknowledgment. The author would like to thank El Mahdi El Mhamdi, Henrik Aslund, Se´bastien Rouault and Alexandre Maurer for fruitful discussions.