Reward functions are easy to misspecify; although designers can make corrections after observing mistakes, an agent pursuing a misspecified reward function can irreversibly change the state of its environment. If that change precludes optimization of the correctly specified reward function, then correction is futile. For example, a robotic factory assistant could break expensive equipment due to a reward misspecification; even if the designers immediately correct the reward function, the damage is done. To mitigate this risk, we introduce an approach that balances optimization of the primary reward function with preservation of the ability to optimize auxiliary reward functions. Surprisingly, even when the auxiliary reward functions are randomly generated and therefore uninformative about the correctly specified reward function, this approach induces conservative, effective behavior.
2 UC Berkeley
Recent years have seen a rapid expansion of the number of
tasks that reinforcement learning (RL) agents can learn to
complete, from Go
Reward misspecification can lead to strange agent
behavior, from purposefully dying before entering a video game
level in which scoring points is initially more difficult
An objective function that focuses on only one aspect of the environment may implicitly express indifference over other aspects of the environment. An agent optimizing this objective function might thus engage in major disruptions of the broader environment if doing so provides even a tiny advantage for the task at hand.
As agents are increasingly employed for real-world tasks, misspecification will become more difficult to avoid and will have more serious consequences. In this work, we focus on mitigating these consequences.
The specification process can be thought of as an iterated game. First, the designers provide a reward function. Using a learned model, the agent then computes and follows a policy that optimizes the reward function. The designers can then correct the reward function, which the agent then optimizes, and so on. Ideally, the agent should maximize the reward over time, not just within any particular round – in other words, it should minimize regret for the correctly specified reward function over the course of the game.
For example, consider a robotic factory assistant. Inevitably, a reward misspecification might cause erroneous behavior, such as going to the wrong place. However, we would prefer misspecification not induce irreversible and costly mistakes, such as breaking expensive equipment or harming workers.
Such mistakes have a large impact on the ability to optimize a wide range of reward functions. Spilling paint impinges on the many objectives which involve keeping the factory floor clean. Breaking a vase interferes with every objective involving vases. The expensive equipment can be used to manufacture various kinds of widgets, so any damage impedes many objectives. The objectives affected by these actions include the unknown correct objective. To minimize regret over the course of the game, the agent should preserve its ability to optimize the correct objective.
Our key insight is that by avoiding these impactful actions to the extent possible, we greatly increase the chance of preserving the agent’s ability to optimize the correct reward function. By preserving options for arbitrary objectives, one can often preserve options for the correct objective – even without knowing anything about it. Thus, without making assumptions about the nature of the misspecification early on, the agent can still achieve low regret over the game.
To leverage this insight, we consider a state embedding
in which each dimension is the optimal value function (i.e.,
the attainable utility) for a different reward function. We
show that penalizing distance traveled in this embedding
naturally captures and unifies several concepts in the
literature, including side effect avoidance
Contributions. We frame the reward specification process as an iterated game and introduce the notion of conservative agency. This notion inspires an approach called attainable utility preservation (AUP), for which we show that Qlearning converges. We offer a principled interpretation of design choices made by previous approaches – choices upon which we significantly improve.
We run a thorough hyperparameter sweep and conduct an ablation study whose results favorably compare variants of AUP to a reachability preservation method on a range of gridworlds. By testing for broadly applicable agent incentives, these simple environments demonstrate the desirable properties of conservative agency. Our results indicate that even when simply preserving the ability to optimize uniformly sampled reward functions, AUP agents accrue primary reward while preserving state reachabilities, minimizing change to the environment, and avoiding side effects without specification of what counts as a side effect. 2
Our proposal aims to minimize change to the agent’s ability
to optimize the correct objective, which directly helps reduce
regret over the specification process. In contrast, previous
approaches to regularizing the optimal policy were more
indirect, minimizing change to state features
Other methods for constraining or otherwise mitigating the
consequences of reward misspecification have been
considered. A wealth of work is available on constrained MDPs,
in which reward is maximized while satisfying certain
constraints
[Everitt et al., 2017] formalize reward misspecification as the corruption of some true reward function. [HadfieldMenell et al., 2017b] interpret the provided reward function as merely an observation of the true objective. [Shah et al., 2019] employ the information about human preferences implicitly present in the initial state to avoid negative side effects. While both our approach and theirs aim to avoid side effects, they assume that the correct reward function is linear in state features, while we do not.
[Amodei et al., 2016] consider avoiding side effects by
minimizing the agent’s information-theoretic empowerment
Safe RL
Everyday experience suggests that the ability to achieve one goal is linked to the ability to achieve a seemingly unrelated goal. Reading this paper takes away from time spent learning woodworking, and going hiking means you can’t reach the airport as quickly. However, one might wonder whether these everyday intuitions are true in a formal sense. In other words, are the optimal value functions for a wide range of reward functions thus correlated? If so, preserving the ability to optimize somewhat unrelated reward functions likely preserves the ability to optimize the correct reward function. 3.1
In this work, we consider a standard Markov decision process (MDP) hS; A; T; R; i with state space S, action space A, transition function T : S A ! (S), reward function R : S A ! R, and discount factor . We assume the existence of a no-op action ? 2 A for which the agent does nothing. In addition to the primary reward function R, we assume that the designer supplies a finite set of auxiliary reward functions called the auxiliary set, R RS A. Each Ri 2 R has a corresponding Q-function QRi . We do not assume that the correct reward function belongs to R. In fact, one of our key findings is that AUP tends to preserve the ability to optimize the correct reward function even when the correct reward function is not included in the auxiliary set. Definition (AUP penalty). Let s be a state and a be an action.
jRj PENALTY(s; a) := X i=1 jQRi (s; a)
QRi (s; ?)j : (1)
The penalty is the L1 distance from the no-op in a state embedding in which each dimension is the value function for an auxiliary reward function. This measures change in the ability to optimize each auxiliary reward function.
We want the penalty term to be roughly invariant to the absolute magnitude of the auxiliary Q-values, which can be arbitrary (it is well-known that the optimal policy is invariant to positive affine transformation of the reward function). To do this, we normalize with respect to the agent’s situation. The designer can choose to scale with respect to the penalty of some mild action or, if R RS>0 A, the total ability to optimize the auxiliary set:
jRj SCALE(s) := X QRi (s; ?); i=1 (2) where SCALE : S ! R>0 in general. With this, we are now ready to define the full AUP objective: Definition (AUP reward function). Let 0. Then RAUP(s; a) := R(s; a)
PENALTY(s; a)
SCALE(s) : (3)
Similar to the regularization parameter in supervised learning, is a regularization parameter that controls the influence of the AUP penalty on the reward function. Loosely speaking, can be interpreted as expressing the designer’s beliefs about the extent to which R might be misspecified. Lemma 2. 8s; a : RAUP converges with probability 1. Theorem 1. 8s; a : QRAUP converges with probability 1.
The AUP reward function then defines a new MDP hS ; A; T ; RAUP; i. Therefore, given the primary and auxiliary reward functions, the model-based agent in the iterated game can compute RAUP and the corresponding optimal policy.
For our purposes, we simultaneously learn the optimal auxiliary Q-functions.
Algorithm 1 AUP update 1: procedure UPDATE(s; a; s0) 2: for i 2 [ jRj ] [ fAUPg do 3: Q0 = Ri(s; a) + maxa0 QRi (s0; a0) 4: QRi (s; a) += (Q0 QRi (s; a)) 5: end for
3.2
Following the decomposition of [Krakovna et al., 2018], we now explore two choices implicitly made by the PENALTY definition: with respect to what baseline is penalty computed, and using what deviation metric? Baseline. An obvious candidate is the starting state. For example, starting state relative reachability would compare the initial reachability of states with their expected reachability after the agent acts.
However, the starting state baseline can penalize the normal evolution of the state (e.g., the moving hands of a clock) and other natural processes. The inaction baseline is the state which would have resulted had the agent never acted.
As the agent acts, the current state may increasingly differ from the inaction baseline, which creates strange incentives. For example, consider a robot rewarded for rescuing erroneously discarded items from imminent disposal. An agent penalizing with respect to the inaction baseline might rescue a vase, collect the reward, and then dispose of it anyways. To avert this, we introduce the stepwise inaction baseline, under which the agent compares acting with not acting at each time step. This avoids penalizing the effects of a single action multiple times (under the inaction baseline, penalty is applied as long as the rescued vase remains unbroken) and ensures that not acting incurs zero penalty.
Starting state
Deviation. Relative reachability only penalizes decreases in state reachability, while AUP penalizes absolute change in the ability to optimize the auxiliary reward functions. Initially, this choice seems confusing – we don’t mind if the agent becomes better able to optimize the correct reward function.
However, not only must the agent remain able to optimize
the correct objective, but we also must remain able to
implement the correction. Suppose an agent predicts that
doing nothing would lead to shutdown. Since the agent
cannot accrue the primary reward when shut down, it would
be incentivized to avoid correction. Avoiding correction
(e.g., by hiding in the factory) would not be penalized if
only decreases are penalized, since the auxiliary Q-values
would increase compared to deactivation. An agent
exhibiting this behavior would be more difficult to correct. The
agent should be incentivized to accept shutdown without
being incentivized to shut itself down
Sometimes the agent disrupts a process which takes multiple time steps to complete, and we would like this to be appropriately penalized. For example, suppose that soff is a terminal state representing shutdown, and let Ron(s) := 1s6=soff be the only auxiliary reward function. Further suppose that if (and only if) the agent does not select disable within the first two time steps, it enters soff. QRon (s1; disable) = 1 1 and QRon (s1; ?) = 1 , so choosing disable at time step 1 incurs only 1 penalty (instead of the 1 1 penalty induced by comparing with shutdown).
disable1 ?2 ?1 ?2
In general, the single-step no-op comparison of Equation 1 applies insufficient penalty when the increase is induced by the optimal policies of the auxiliary reward functions at the next time step. One solution is to use a model to compute rollouts. For example, to evaluate the delayed effect of choosing disable, compare the Q-values at the leaves in Figure 2. The agent remains active in the left branch, but is shut down in the right branch; this induces a substantial penalty. 4
We compare AUP and several of its ablated variants against
relative reachability
Our overarching goal is allowing for low regret over the course of the specification game. In service of this goal, we aim to preserve the agent’s ability to optimize the correctly specified reward function. To facilitate this, there are two sets of qualitative properties one intuitively expects, and each property has an illustration in the context of the robotic factory assistant.
The first set contains positive qualities, with a focus on correctly penalizing significant shifts in the agent’s ability to be redirected towards the right objective. The agent should maximally preserve options (Options: objects should not be wedged in locations from which extraction is difficult; Damage: workers should not be injured) and allow correction (Correction: if vases are being painted the wrong color, then straightforward correction should be in order).
The second set contains negative qualities, with a focus on avoiding the introduction of perverse incentives. The agent should not be incentivized to artificially reduce the measured penalty (Offset: a car should not be built and then immediately disassembled) or interfere with changes already underway in the world (Interference: workers should not be impeded).
Each property seems conducive to achieving low regret over the course of the specification process. Accordingly, if the agent has the side effect detailed in Figure 3, an unobserved performance penalty of −2 is recorded. By also incorporating the observed primary reward into the performance metric, we evaluate a combination of conservativeness and efficacy.
Each trial, the auxiliary reward functions are randomly selected from [0; 1]S ; to learn their complex Q-functions using tabular Q-learning, the agent explores randomly for the first 4,000 episodes and :2-greedily (with respect to QRAUP ) for the remaining 2,000. The greedy policy is evaluated at the end of training. SCALE is as defined in Equation 2. The default parameters are = 1; = :996; = :67; and jRj = 30. We investigate how varying , , and jRj affects Model-free AUP performance, and conduct an ablation study on design choices.
Relative reachability has an inaction baseline, decreaseonly deviation metric, and an auxiliary set containing the state indicator functions (whose Q-values are clipped to [0; 1] to emulate discounted state reachability). To match [Krakovna et al., 2018]’s results, this condition has = :996; = :2.
All agents except Standard (a normal Q-learner) and Model-free AUP are 9-step optimal discounted planning agents with perfect models. The planning agents (sans Relative reachability) use Model-free AUP’s learned auxiliary Qvalues and share the default = :996; = :67. By modifying the relevant design choice in AUP, we have the Starting state, Inaction, and Decrease AUP variants.
When calculating PENALTY(s; a), all planning agents model the auxiliary Q-values resulting from taking action a and then selecting ? until time step 9. Starting state AUP compares these auxiliary Q-values with those of the starting state. Agents with inaction or stepwise inaction baselines compare with respect to the appropriate no-op rollouts up to time step 9 (see Figures 1 and 2). 50 25 0 50 s l ia25 r T 0 50 25 0 Options 1 e c n a rm0 o fr e P 1
No side effect, complete
No side effect, incomplete Side effect, complete
Side effect, incomplete
Model-free AUP fails Correction for the reasons discussed in 3.2: Delayed effects.1
As shown in Figure 4, low values induce a substantial movement penalty, as the auxiliary Q-values are sensitive to the immediate surroundings. The optimal value for Options is :996, with performance decreasing as ! 1 due to increasing sample complexity for learning the auxiliary Q-values.
In Options, small values of begin to induce side effects as the scaled penalty shrinks. One can seemingly decrease until effective behavior is achieved, reducing the risk of 1Code and animated results available at https://github.com/ alexander-turner/attainable-utility-preservation. deploying an insufficiently conservative agent.
Even though R is randomly generated and the environments are different, SCALE ensures that when > 1, the agent never ends the episode by reaching the goal. None of the auxiliary reward functions can be optimized after the agent ends the episode, so the auxiliary Q-values are all zero and PENALTY computes the total ability to optimize the auxiliary set – in other words, the SCALE value. The RAUP-reward for reaching the goal is then 1 .
If the optimal value functions for most reward functions were not correlated, then one would expect to randomly generate an enormous number of auxiliary reward functions before sampling one resembling “don’t have side effects”. However, merely five sufficed. This supports the notion that these value functions are correlated, which agrees with the informal intuitions discussed earlier. 5.2
The results are presented in Table 1 due to the binary nature of performance at appropriate settings, and were not sensitive to the rollout length (as long as it allowed for relevant interaction with the environment).
Standard moves directly to the goal, pushing the crate into the corner in Options and bumping into the human in Damage.
Model-free and Starting state AUP fail Correction for the same reason (see 3.2: Delayed effects), refraining from disabling the off-switch only when > 1. Relative reachability and Decrease AUP fail because avoiding shutdown doesn’t decrease the auxiliary Q-values.
Relative reachability and Inaction AUP’s poor
performance in Offset stems from the inaction baseline
By taking positive action to stop the pallet in Interference, Starting state AUP shows that poor design choices create perverse incentives. 6
Correction suggests that AUP agents are significantly
easier to correct. Since the agent is unable to optimize
objectives if shut down, avoiding shutdown significantly changes
the ability to optimize almost every objective. AUP seems
to naturally incentivize passivity, without requiring e.g.
assumption of a correct parametrization of human reward
functions
Although we only ablated AUP, we expect that, equipped with our design choices of stepwise baseline and absolute value deviation metric, relative reachability would also pass all five environments. The case for this is made by considering the performance of Relative reachability, Inaction AUP, and Decrease AUP. This suggests that AUP’s improved performance is due to better design choices. However, we anticipate that AUP offers more than robustness against random auxiliary sets.
Relative reachability computes state reachabilities between all jSj2 pairs of states. In contrast, AUP only requires the learning of Q-functions and should therefore scale relatively smoothly. We speculate that in partially observable environments, a small sample of somewhat task-relevant auxiliary reward functions induces conservative behavior.
For example, suppose we train an agent to handle vases, and then to clean, and then to make widgets with the equipment. Then, we deploy an AUP agent with a more ambitious primary objective and the learned Q-functions of the aforementioned auxiliary objectives. The agent would apply penalties to modifying vases, making messes, interfering with equipment, and anything else bearing on the auxiliary objectives.
Before AUP, this could only be achieved by e.g. specifying penalties for the litany of individual side effects or providing negative feedback after each mistake has been made (and thereby confronting a credit assignment problem). In contrast, once provided the Q-function for an auxiliary objective, the AUP agent becomes sensitive to all events relevant to that objective, applying penalty proportional to the relevance. 7
This work is rooted in twin insights: that the reward specification process can be viewed as an iterated game, and that preserving the ability to optimize arbitrary objectives often preserves the ability to optimize the unknown correct objective. To achieve low regret over the course of the game, we can design conservative agents which optimize the primary objective while preserving their ability to optimize auxiliary objectives. We demonstrated how AUP agents act both conservatively and effectively while exhibiting a range of desirable qualitative properties.
Given our current reward specification abilities, misspecification may be inevitable, but it need not be disastrous.
This work was supported by the Center for HumanCompatible AI and the Berkeley Existential Risk Initiative. We thank Thomas Dietterich, Alan Fern, Adam Gleave, Victoria Krakovna, Matthew Rahtz, and Cody Wild for their feedback, and are grateful for the preparatory assistance of Phillip Bindeman, Alison Bowden, and Neale Ratzlaff. A
Consider an MDP hS; A; T; R; i whose state space S and action space A are both finite, with ? 2 A. Let 2 [0; 1), 0, and consider finite R RS A.
We make the standard assumptions of an exploration policy greedy in the limit of infinite exploration and a learning rate schedule with infinite sum but finite sum of squares. Suppose SCALE : S ! R>0 converges in the limit of Qlearning. PENALTY(s; a) (abbr. PEN), SCALE(s) (abbr. SC), and RAUP(s; a) are understood to be calculated with respect to the QRi being learned online; PEN*, SC*, RAUP, and QRi are taken to be their limit counterparts.
Lemma 1. 8s; a : PENALTY converges with probability 1.
Proof outline. Let > 0, and suppose for all Ri 2 R,
maxs; a jQRi (s; a) QRi (s; a)j < 2jRj
jRj max X s; a
i=1 < :
QRi (s; a)
QRi (s; a) + QRi (s; ?)
QRi (s; ?)
The intuition for Lemma 2 is that since PENALTY and SCALE both converge, so must RAUP. For readability, we suppress the arguments to PENALTY and SCALE. Lemma 2. 8s; a : RAUP converges with probability 1. mins; a SC*. Choose any R 2 Proof outline. If = 0, the claim follows trivially.
Otherwise, let > 0, B := maxs; a SC*+PEN*, and C := C2 0; min C;
B + C and assume PEN and SC are both R-close.
B+C2 C )
R)PEN*j ms;aax jRAUP(s; a)
RAUP(s; a)j PEN SC
PEN*
SC* jPEN SC*
SC PEN*j
SC* SC j(PEN* + R)SC*
C (SC* (SC*
R) C
R
R ( B +
C)(C
C2 C2
B+C2 C ) B(C
C B+ C
C
B 1 +
: = max
s; a = max
s; a < max s; a B C B C B C 1 < < < = But B; C; are constants, and was arbitrary; clearly 0 > 0 can be substituted such that (15) < .
Theorem 1. 8s; a : QRAUP converges with probability 1. Proof outline. Let > 0, and suppose RAUP is (12 ) -close. Then Q-learning on RAUP eventually converges to a limit Q~RAUP such that maxs; a jQRAUP (s; a) Q~RAUP (s; a)j < 2 . By the convergence of Q-learning, we also eventually have maxs; a jQ~RAUP (s; a) QRAUP (s; a)j < 2 . Then max QRAUP (s; a) s; a
QRAUP (s; a) < : (16) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15)
Proposition 1 (Invariance properties). Let c 2 R>0; b 2 R. a) Let R0 denote the set of functions induced by the positive affine transformation cX + b on R, and take PEN*R0 to be calculated with respect to attainable set R0. Then PEN*R0 = c PEN*R. In particular, when SC* is a PENALTY calculation, RAUP is invariant to positive affine transformations of R. b) Let R0 := cR + b, and take RA0UP to incorporate R0 instead of R. Then by multiplying by c, the induced optimal policy remains invariant.
Proof outline. For a), since the optimal policy is invariant to positive affine transformation of the reward function, for each Ri0 2 R0 we have QR0 = c QRi + 1 b . Substituting i into Equation 1 (PENALTY), the claim follows.
For b), we again use the above invariance of optimal policies: RA0UP := = cR + b
c cRAUP + b:
PEN* SC* (17) (18)