Utility functions or their equivalents (value functions, objective functions, loss functions, reward functions, preference orderings) are a central tool in most current machine learning systems. These mechanisms for defining goals and guiding optimization run into practical and conceptual difficulty when there are independent, multi-dimensional objectives that need to be pursued simultaneously and cannot be reduced to each other. Ethicists have proved several impossibility theorems that stem from this origin; those results appear to show that there is no way of formally specifying what it means for an outcome to be good for a population without violating strong human ethical intuitions (in such cases, the objective function is a social welfare function). We argue that this is a practical problem for any machine learning system (such as medical decision support systems or autonomous weapons) or rigidly rule-based bureaucracy that will make high stakes decisions about human lives: such systems should not use objective functions in the strict mathematical sense. We explore the alternative of using uncertain objectives, represented for instance as partially ordered preferences, or as probability distributions over total orders. We show that previously known impossibility theorems can be transformed into uncertainty theorems in both of those settings, and prove lower bounds on how much uncertainty is implied by the impossibility results. We close by proposing two conjectures about the relationship between uncertainty in objectives and severe unintended consequences from AI systems.
Conversations about the safety of self-driving cars often turn
into discussion of “trolley problems”, where the vehicle must
make a decision between several differently disastrous
outcomes
There are, however, other kinds of AI systems for which ethical discussions about the value of lives, about who should exist or keep existing, are more genuinely relevant. The clearest category is systems that are designed to make decisions affecting the welfare or existence of people in the future, rather than systems that only have to make such decisions in highly atypical scenarios. Examples of such systems include: autonomous or semi-autonomous weapons systems (which may have to make trade-offs between threats to and the lives of friendly and enemy combatants, friendly and enemy civilians, and combatants who have surrendered).2 risk assessment algorithms in criminal justice contexts (which at least notionally are making trade-offs between risk to society and the harm that life-shortening incarceration does to individual defendants in the criminal justice system and their families). decision support systems in high-stakes bureucratic settings. For instance a system that helps to allocate scarce resources like doctors’ time, laboratory test access and money in a medical system would implicitly or explicitly have to make trade-offs between the life span, quality of life, and fertility of numerous people. autonomous types of open-ended AI taking important actions without their designers’ input.
Governmental organizations already have to make
decisions that involve weighing benefits to present or future
people against risks to present or future people, or trading off
benefits (such as wealth) to one group against benefits of
another sort (such as freedom) to another. Therefore, we might
1There are related and more fruitful questions like, “how should
self-driving cars make trade-offs between time saved by driving
more aggressively and accident risk?” See for instance
2The creation of autonomous weapons systems may well be
an extremely bad idea (see, eg
Economists and ethicists study individual utility functions
and their aggregations, such as social welfare functions,
social choice functions and formal axiologies, and have
discovered various impossibility theorems about these functions.
Perhaps the most famous of these is Arrow’s Impossibility
Theorem
Unfortunately, ethicists have discovered other situations in
which the problem isn’t learning and computing the tradeoff
between agents’ objectives, but that there simply may not
be such a satisfactory tradeoff at all. The “mere addition
paradox”
The Repugnant Conclusion For any population of very happy people, there exists a much larger population with lives barely worth living that is better than this very happy population (this affects the "maximise total wellbeing" objective).
The Sadistic Conclusion Suppose we start with a population of very happy people. For any proposed addition of a sufficiently large number of almost-as-happy people, there is a small number of horribly tortured people that is a preferable addition (this affects the "maximise average wellbeing" objective).
tion of two or more people which has uniform happiness, there exists another population of the same size which has lower total and average happiness, and is less equal, but is better.
Anti-Dominance Population B can be better than population A even if A is the same size as population B, and every person in A is happier than their equivalent in B. Anti-Addition It is sometimes bad to add a group of people B to a population A (where the people in group B are worse of than those in A), but better to add a group C that is larger than B, and worse off than B.
Extreme Priority There is no n such that create of n lives of very high positive welfare is sufficient benefit to compensate for the reduction from very low positive welfare to slightly negative welfare for a single person (informally, “the needs of the few outweigh the needs of the many”).
The structure of the impossibility theorem is to show that no objective function or social welfare function can simultaneously satisfy these principles, because they imply a cycle of world states, each of which in turn is required (by one of these principles) to be better than the next. The structure of that proof is shown in Figure 1. 2.2
Arrhenius’s unpublished book contains a collection of many more uncertainty theorems. Here is the simplest, which is a more compelling version of Parfitt’s Mere Addition Paradox. It shows the impossibility of an objective function satisfying the following requirements simultaneously:
The Quality Condition: There is at least one perfectly equal population with very high welfare which is at least as good as any population with very low positive welfare, other things being equal.
The Inequality Aversion Condition: For any triplet of welfare levels A, B, and C, A higher than B, and B higher than C, and for any population A with welfare A, there is a larger population C with welfare C such that a perfectly equal population B of the same size as A[C and with welfare B is at least as good as A[C, other things being equal.
is a perfectly equal population of the same size as population B, and every person in A has higher welfare than every person in B, then A is better than B, other things being equal.
lives with positive welfare and an increase in the welfare in the rest of the population doesn’t make a population worse, other things being equal.
The cyclic structure of the proof is shown in Figure 2. 2.3
The particular impossibility theorems summarized above,
and those in the related literature, result from the
incompatibility of several different utilitarian objectives: maximizing
total wellbeing, maximizing average wellbeing, and avoiding
suffering. Similar problems are also likely to arise when
considering almost any kinds of competing objectives, such as
attempting to simultaneously maximize different notions of
wellbeing, freedom, knowledge, fairness, or other widely
accepted or domain-specific ethical goods
Arrhenius’s impossibility results and others like them are quite troubling. They show that we do not presently have a trustworthy framework for making decisions about the welfare or existence of people in the future, and are representative of a broader problem with the inability of the objective functions to reasonably optimise for multiple objectives simultaneously. Next we will consider five possible responses to these impossibility theorems: 3.1
One response may be to claim that the stakes are simply low
for present AI systems that only make small-scale changes
to the world. AGI systems, if they exist, may one day need
to confront impossibility theorems or difficult conflicting
objectives in some cases, but none of that is presently relevant.
Unfortunately, it appears that although the stakes are indeed
much lower at present, the fundamental tensions that drive
these paradoxes are already at play in decisions made by
bureaucracies and decision support systems, and small scale
decisions can easily violate constraints we would want to
see respected. Asking for human feedback helps, but humans
often miss important ethical principles, and can easily make
chains of decisions that have problematic consequences in
proportion to their degree of power or agency. Cases of
particular concern are those where ML or algorithmic heuristics
are deployed in decision support tools that humans are
inadequately inclined to question.
One might argue that impossibility theorems are only a
problem for an AI system if we are attempting to explicitly define
an objective function, rather than letting an AI system
acquire its objective function in a piecemeal way via a method
like co-operative inverse reinforcement learning (CIRL) or
other forms of human guidance
4Although one philosophical concern is that methods like CIRL may already commit us to more ‘person-affecting’ views within ethics by pooling the preferences of existing agents. This could lead to the implementation of principles that satisfy undesirable axioms network is mathematically a utility function or total ordering, these problems will persist at all stages of network training. The theorems do not just reflect a tension between principles that agents are committed to: they are also reflected in the decisions that human supervisors will make when presented with a sequence of pairwise choices.
Therefore, although learning objectives from humans may be a prudent design choice, such learning systems will still need to either violate the ethical intuitions that underpin the various impossibility theorems (some combination of Sections 3.3 and 3.4) or explicitly incorporate uncertainty into their outputs (Section 3.5). 3.3
One natural response to impossibility theorems is to try harder to figure out good ways of making tradeoffs between the competing objectives that underlie them — in Arrhenius’s paradox these are total wellbeing, average wellbeing, and avoidance of suffering. This tradeoff can be simple, such as trying to define a linear exchange rate between those objectives. Unfortunately, linear exchange rates between a total quantity and an average quantity do not make conceptual sense, and it is easy to find cases under which one of them totally outweighs the other.5
One can try more complicated strategies such as trying to
write down convex tradeoffs (see eg
Another type of response – one that is commonly pursued
in the population ethics literature – is to argue that although
each of the axioms above strikes us as undesirable, at least
one of them ought to be accepted in some form. For example,
it has been argued that although the Repugnant Conclusion
might appear undesirable, it is an acceptable consequence of
an objective function over populations
5For instance, if a linear weighting is chosen that appears to make sense for the present population of the world, a change in technology that allowed a much larger population might quickly cause the average wellbeing to cease affecting the preferred outcome in any meaningful way.
6See https://pdollar.github.io/toolbox/classify/softM
softmin is the counterpart to the more widely discussed
It is already the case that in some problems where humans
attempt to provide oversight to AI systems, it is pragmatically
better to allow the human to express uncertainty
We can represent this notion with a partially ordered objective function that sometimes says the comparison between world states cannot be known with certainty. Cyclical impossibility theorems like those surveyed in Sections 2.1 and 2.2 can be viewed as evidence of this uncertainty. Here we prove a lower bound on how much uncertainty is evidenced by each such theorem.7
7Arrhenius considered this direction of interpretation
It turns out that only two of the edges in the cycle need to be
uncertain. Note that Broome’s vague neutral-level theory
Suppose that W is the set of possible states of the world, and that there is a an impossibility theorem TC showing that there is no totally ordered definition of “better” Z over W that satisfies a set of constraints fC1; C2; ::Cng over the comparison function Z : W W ! f<; >; =g. Each of these constraints are motivated by strong human ethical intuitions, and insist that for a set of pairs of inputs x and y, Ci A(x; y) takes some value. We use the notation x Z y to indicate that Ci requires that x Z y for some x and y). Z also has the usual properties of total orderings:
Antisymmetry: (1) (2) (3) (4) (5) (6) (7) wn, w1
C1
Z w2
C2
Z :::
Cn 1
Z
wn wn
Cn
Z w1
Violating (by induction) transitivity, TC shows that no totally ordered Z that satisfies fC1; ::Cng can exist. This is a cyclic impossibility theorem.
Now suppose we attempt to escape TC by looking for a partially ordered notion of “better” Z0 with a comparison function Z0 : W
W ! f<; >; =; =? g. By necessity, the uncertain case is symmetric: a =? b ! b =? a.
Ci
In some instances some of the requirements x Z0 y of some constraint Ci will be failed, but only weakly: they will result in incomparability x =? Z0 y rather than violation, x >Z0 y. We call this uncertain satisfaction of a constraint.
Theorem: A cyclic impossibility theorem TC can be transformed into an uncertainty theorem only if two or more constraints are uncertainly satisfied.
Proof: We might hope that it would be possible to only uncertainly satisfy one of the constraints fC1; C2; ::Cng, while fully satisfying all of the others. But this is impossible, and at minimum two of the constrains will be uncertain. The proof is by contradiction. Assume w.l.o.g. that Cn is the only uncertainly satisfied constraint, such that
a a
Z b ^ b
Z a ! a = b Z b ^ b
Z c ! a
Z c a
Z b _ b
Z a
Suppose further that T is provable by providing a series of example worlds fw1; w2; :::wng such that: w1
C1 Z0 w2
C2 Z0 :::
Cn 1
Z0 wn Cn ? wn =Z0 w1
However by transitivity and (6), we know w1 which by the symmetry of =? contradicts (7). So in order to treat TC as uncertainty theorem, at least two constraints must be uncertainly satisfied. QED.
C1::n 1
Z0
The structure of this transformation from cyclic impossibility to uncertainty is shown in Figure 1 and 2 for the two theorems introduced in Sections 2.1 and 2.2 function or uncertain ordering. Suppose that an uncertain ordering is denoted by:
The largest difficulty with accepting partially ordered, uncertain objectives is knowing how an agent should act when facing ethically uncertain choices. There are a few possible strategies, including: doing nothing, asking for human supervision, choosing randomly, searching harder for a clearlypreferable action, treating all of the uncertain options as equally good, or (for very capable agents) conducting research to try to break the tie in some way. None of these options are ideal. But depending on the application domain, some of them will be workable. An may be necessary for ML systems deployed in high stakes, safety-critical applications. 5
A second way to encode uncertainty about a system’s objectives, is to use what we will call an uncertain objective (8) (9)
Zk(a; b) = P (a > b) the uncertain ordering Zk is a function returning the probability, based on the available prior philosophical and practical evidence k, that one of the states a is better. Conversely, (For simplicity, we assume that if a and b appear exactly equally good, Zk(a; b) = 0:5 rather than allowing a non-zero probability of explicit equality.)
This formulation can for instance be justified by holding that there is some ultimately correct total ordering of states of the world, we just don’t know what it is. The framing is also compatible with the claim that the correct ordering is ultimately unknowable.
Uncertain orderings are a convenient formalization if the comparison function is itself the output of a machine learning model to be trained from human guidance (ideally from a large number of people, over a large number of real or hypothetical scenarios). This formulation is also convenient for systems that do not have a clear architectural separation between abstract objectives and the predictive reasoning about the real world that is necessary to accomplish them.
Uncertain orderings are also a more flexible basis for strategies/decision rules than the partially ordered objective functions discussed in Section 4. If a partial order returns =? , the agent is left torn and unable to act. But with with a distribution Zk decisions rules such as “take an action drawn via weighted probability from those that are sufficiently likely to be best” are available.8 Or for systems where more supervision is available, “take the action that is most likely to be best, provided it beats the second best action by some probability margin ; otherwise, ask for supervision.”
In this framing, whether a given ethical constraint or principle has been violated becomes a probabilistic matter. There will almost always be some probability that Zk violates each constraint, unless all of the corresponding output probabilities have converged to 1 or 0 (as appropriate). Decision rules based on Zk will by mathematical necessity sometimes violate the constraints, at least by inaction. But if the actions are sampled from a space of reasonably supported ones, then at least the violating actions will not systematically prioritize some objectives over others, and will be more similar to the way that wise and cautious humans react to difficult moral dilemmas.
8This has been termed a “satisficing” or “quantilized” decision
rule
What do we know about the values Zk(a; b)? Consider a series of cyclical constraints C0; C1; :::Cn across points x0; x1; ::xn where Ci requires Zk(xi; xi+1 mod n+1) 1. If the uncertain ordering (or the process that generates it) has emitted values for some of the pairwise comparisons Zk(xi; xi+1), this imposes some bounds on the comparison that spans them, Zk(x0; xn). For transitivity, in order for the spanning comparison to be in a given direction, at least one of the pairwise comparisons must also be in that direction. So for the simple two-step case over x0; x1; x2:
Zk(x0; x2)
Zk(x0; x1) + Zk(x1; x2) (10)
Note that the pairwise probabilities are additive because the bound is least tight when those probabilities are disjoint. Or more generally:
Zk(x0; xn) (11) n 1 X Zk(xi; xi+1) i=0
Zk(xn; x0)
Zk(xi+1; xi) 1
Zk(x0; xn)
Zk(xi; xi+1) (12) Zk(x0; xn)
Zk(xi; xi+1)
So we have both upper and lower bounds for Zk(x0; xn) : 1 n 1 X 1 Zk(xi; xi+1)
Zk(x0; xn) i=0 (13)
In the special case where x0; x1; x2; :::xn correspond to points ordered by the constraints C0; C1; C2:::Cn of an n + 1 step cyclic impossibility theorem, the satisfaction of Ci means Zk(xi; xi+1 mod n+1) 1 and the probability of violation Zk(xi+1 mod n+1; xi) 0.
We might ask, what is the lowest probability that we can obtain for any of these chances of violating a constraint? Or in other words, what is the lower bound B that can be achieved on the odds of each such possible violation: n 1 X Zk(xi; xi+1) i=0 n 1 X i=0 n 1 X 1 i=0 1 n 1 X 1 i=0 n 1 B = mZikn mi=a0xf1
n 1 = mZikn mi=a0xfZk(x0; xn); 1 1 1
By symmetry, we know that this bound will be minimized when all the values Zk(xi; xi+1 mod n) are equal to a single value z:
B B min maxf1
z min maxf1 z n(1 z); 1
z; :: ; 1 n + nz; 1 zg z; ::g
Since in the domain z 2 [0; 1] the function 1 n + nz monotonically increasing from 1 n, and the function 1 z is monotonically decreasing from 1, and the functions intersect, the minimax is satisfied at that intersection: (16) (17) (18) At which point we have: 1 z =1 z =
n n + 1
n + nz B B 1
1 n + 1
n n + 1
This bound is a minimum uncertainty theorem for uncertain objective functions: when confronted with a n + 1-step cyclic ethical impossibility theorem, no choice of uncertain ordering can reduce the probabilities of constraint violation so that they are all below n+11 .
In practice, systems that estimate which of two outcomes are better than another could emit values like Zk that violate these constraints, but in such circumstances the outputs could no longer be interpreted as probabilities over a well-defined ordering of states, and the agent would exhibit self-contradictory, non-transitive preferences.
6
We shown the existence of ethical uncertainty theorems for objectives formulated either as partial orders over states, or probability distributions over total orderings. Other formalizations are possible and deserve investigation.
Where constraints are sourced from human intuition, there is a question of how they should be interpreted, prioritized and kept in data structures. For instance, rather than treating all constraints as equally important, and all violations of constraints as equally serious, it might be better to learn or specify weightings for each constraint, to measure the degree to which a given action violates a constraint, and to reconstruct ethical uncertainty theorems in such a framework.
Alternatively, when building AI systems that learn their objectives from the combination of observed world states and human feedback, it may be computationally inconvenient to require the agent to produce a probability distribution over total orderings at each moment, or even to draw samples from such an object. Instead, learned objectives may be viewed as a set of pairwise comparisons that are too sparse and unstable to be taken as a commitment to any global total order (or probability distribution over global total orders). In such a frame, the goal of transitive preferences and avoidance of cycles is an aspiration, and may require both record keeping and “regret” for past actions that now seem sub-optimal due to subsequent learning of objectives. The appropriate framing of ethical uncertainty theorems in such settings would be productive further work. 7.1 7
Many of the concerns in the literature about the difficulty
of aligning hypothetical future AGI systems to human
values are motivated by the risk of “instrumental convergence”
of those systems — the adoption of sub-goals that are
disempowering of humans and other agents
We believe that the emergence of instrumental subgoals
is deeply connected to moral certainty. Agents that are not
completely sure of the right thing to do (which we believe
is an accurate summary of the state of knowledge about
ethics, both because of normative impossibility and
uncertainty theorems, and the practical difficulty of predicting the
consequences of actions) are much more likely to tolerate the
agency of others, than agents that are completely sure that
they know the best way for events to unfold. This appears
to be true not only of AI systems, but of human ideologies
and politics, where totalitarianism has often been built on a
substructure of purported moral certainty
This leads us to propose a conjecture about the relationship between moral certainty and instrumentally convergent subgoals:
9Any open-ended and non-trivial objective appears to be
vulnerable to instrumental convergence. Bostrom argues that objectives
that are bounded rather than open-ended also lead to instrumental
convergence if there is any probability that they will not be achieved
with mathematically certain, monotonically increasing, openended objective functions will adopt sub-goals to disable or dis-empower other agents in all or almost all cases.10 A second conjecture is the converse of the first:
agents with mathematically uncertain objectives will not adopt sub-goals to disable or dis-empower other agents unless those agents constitute a probable threat to a wide range of objectives. 7.2
We have shown that impossibility theorems in ethics have
implications for the design of powerful algorithmic systems,
such as high-stakes AI applications or sufficiently rigid and
rule-based bureaucracies. We showed that such paradoxes
can be avoided by using uncertain objectives, such as
partial orders or probability distributions over total orders; we
proved uncertainty theorems that place a minimum bound
on the amount of uncertainty required. Some previously
proposed ethical theories (such Broome’s vague neutral-level
theory
In the light of these results, we believe that machine learning researchers should avoid using totally ordered objective functions or loss functions as optimization goals in highstakes applications. Systems designed that way appear to be suffering from an ethical “type error” in their goal selection (and action selection) code.
Instead, high-stakes systems should always exhibit uncertainty about the best action in some cases. Further study is warranted about the advantages of various probabilistic decision rules to handle such uncertainty, and about whether other mathematical models of uncertainy are better alternatives to the two models examined in this paper. Further research could also be productive on the relationship between ethical certainty and various observed and predicted pathological behaviour of AI systems. We proposed two conjectures on this topic, the Totalitarian Convergence Conjecture and the Pluralistic Non-Convergence Conjecture.
The author would like to thank the late Aaron Swartz for having originally inspired this work and first attempted to make progress on this question; Toby Ord for many valuable conversations and a detailed framing of the problem; Rohin Shah for suggesting a simple proof strategy for the uncertainly satisfied bound; and Amanda Askell for providing valuable research assistance, and presenting an early poster version of this work at NIPS 2017, and Daniel Ford for spotting issues and making valuable suggestions on the uncertain ordering bound. Thanks also to Dylan Hadfield-Menell, Laurent Orseau, and Anca Dragan for helpful suggestions on earlier versions of this work.
10The qualifier “almost all” requires some further specification. It requires that the objective not be finely tuned in some very intricate way that builds in preservation of all other agents. It is unclear if such fine-tuning is either properly definable or possible.
Economica