Self-modification of agents embedded in complex environments is hard to avoid, whether it happens via direct means (e.g. own code modification) or indirectly (e.g. influencing the operator, exploiting bugs or the environment). It has been argued that intelligent agents have an incentive to avoid modifying their utility function so that their future instances work towards the same goals. Everitt et al. (2016) formally show that providing an option to self-modify is harmless for perfectly rational agents. We show that this result is no longer true for agents with bounded rationality. In such agents, self-modification may cause exponential deterioration in performance and gradual misalignment of a previously aligned agent. We investigate how the size of this effect depends on the type and magnitude of imperfections in the agent's rationality (1-4 below). We also discuss model assumptions and the wider problem and framing space. We examine four ways in which an agent can be boundedrational: it either (1) doesn't always choose the optimal action, (2) is not perfectly aligned with human values, (3) has an inaccurate model of the environment, or (4) uses the wrong temporal discounting factor. We show that while in the cases (2)-(4) the misalignment caused by the agent's imperfection does not increase over time, with (1) the misalignment may grow exponentially.
We face the prospect of creating superhuman (or otherwise
very powerful) AI systems in the future where those
systems hold significant power in the real world
If strong AI systems with the ability to act in the real world are ever deployed2, it is very likely that they will have some means of deliberately manipulating their own implementation, either directly or indirectly (e.g. via manipulating the human controller, influencing the development of a future AI, exploiting their own bugs or physical limitations of the hardware, etc). While the extent of those means is unknown, even weak indirect means could be extensively exploited with sufficient knowledge, compute, modelling capabilities and time.
Omohundro (2008) argues that every intelligent system has a fundamental drive for goal preservation, because when the future instance of the same agent strives towards the same goal, it is more likely that the goal will be achieved. Therefore, Ohomundro argues, a rational agent should never modify into an agent optimizing different goals.
To address this, we present a theoretical analysis of a
selfmodifying agent with imperfect optimization ability and
incomplete knowledge. We model the agent in the standard
1Other major models include e.g. comprehensive systems of
services
2Proposals to prevent this include e.g. boxing
3
Finally, we explicitly list and discuss the underlying assumptions that motivate the theoretical problem and analysis. In addition to clearly specifying the scope of conclusions, the explicit problem assumptions can be used as a rough axis to map the space of viable research questions in the area; see Sections 2 and 6. 1.1
The result of
We show that when we relax the assumption of perfect rationality, their result no longer applies. The bounded-rational agent may prefer to self-modify given the option and in doing so, become less aligned and lose a significant part of the attainable value according to its original goals.
We use the difference between the attainable and attained expected future value at an (arbitrarily chosen) future time point as a proxy for the degree of the agent’s misalignment at that time. Specifically, for a future time t, we consider the value attainable from time t (after the agent already ran and self-modified for t time units), and we estimate the loss of value f t relative to the non-modified agent in the same environment state. Note that f t is not pre-discounted by the previous t steps. See Section 3 for formal definitions and Section 2 for motivation and discussion.
We consider four types of deviation from perfect rationality, see Section 4 for formal definitions. • -optimizers make suboptimal decisions. • -misaligned agents have inaccurate knowledge of the human utility function. • -ignorant agents have inaccurate knowledge of the environment. • -impatient agents have inaccurate knowledge of the correct temporal discount function.
Note that for the sake of simplicity, we use a very simple model of bounded rationality where the errors are simply bounded by the error parameters ; this has to be taken into account when interpreting the results. However, we suspect that the asymptotic dependence of value loss on the size of errors and time would be similar for a range of natural, realistic models of bounded rationality.
Self-modifying -optimizers may deteriorate in future alignment and performance exponentially over time, losing exponential amount of utility compared to -optimizers that do not self-modify. We show upper and tight lower bounds (by a constant) on the worst-case value loss in Theorem 7. As we decrease (increase discounting), the rate at which the agent’s performance deteriorates increases and the possibility of self-modification becomes a more serious problem.
Our analysis of bounded-optimization agents is a
generalization of Theorem 16 from
Self-modifying u-misaligned, -ignorant, or -impatient perfect optimizers can only lose the same value as non-selfmodifying agents with the same irrationality bounds. This also holds for any combination of the three types of bounded knowledge. We give tight upper and lower bounds (up to a constant factor) for the worst-case performance. See Section 5.2 for details.
This implies that unlike bounded-optimization agents, the performance of perfect-optimization bounded-knowledge agents does not deteriorate in time. This is because boundedknowledge agents continue to take optimal actions with respect to their almost correct knowledge and do not selfmodify in a way that would worsen their performance in their view. Therefore, the possibility of self-modification seems less dangerous in the case of bounded-knowledge agents than in the case of bounded-optimization agents. A self-modifying agent with any combination of the four irrationality types may lose value exponential in the time step t when the agent optimization error parameter o > 0. We again give tight (up to a constant factor) lower bounds on the worst-case performance of such agents. See Section 5.3 for details.
Our results do not imply that every such agent will actually perform this poorly but the prospect of exponential deterioration is worrying in the long-term, even if it happens at a much slower speed than suggested by our results. We focus on worst-case analysis because it tells us whether we can have formal guarantees of the agent’s behaviour – a highly desirable property for powerful real-world autonomous systems, including a prospective AGI (artificial general intelligence) or otherwise strong AIs.
Overview of formal results. Here we summarize how much value the different types of bounded-rational agents may lose via misalignment. Note that the maximal attainable discounted value is at most 1 1 and the losses should be considered relative to that, or to the maximum attainable value in concrete scenarios. Otherwise, the values for different value of are incomparable. In all cases, the worst-case lower and upper bounds are tight up to a constant. -optimizer agents – bounded optimization, after t steps of possible self-modification (Theorem 7) fotpt( ; ) = min( t 1 ; 1 1 ) -misaligned agents – inaccurate utility (Theorem 9) futil( ; ) = -impatient agents – inaccurate discounting (Theorem 13) Here is the correct discount factor and is the agent’s incorrect discount factor. 1
) 2 (1
1
Both the statement of the problem and its relevance to AI alignment rest on a set of assumptions listed below. While this list is non-exhaustive, we try to cover the main implicit and explicit choices in our framing, and the space of alternatives. This is largely in hope of eventually finding a better, more robust theoretical framework for solving agent selfmodification within the context of AI alignment, but even further negative results in the space would inform our intuitions on what aspects of self-modification make the problem harder.
We propose consideration of various assumptions as a
framework for thinking about prospective realistic agent
models that admit formal guarantees. We invite further
research and generalizations in this area, one high-level goal
being to map a part of the space of agent models and
assumptions that do or do not permit guarantees, eventually
finding agent models that do come with meaningful
guarantees. Further negative results would inform our intuitions on
what aspects of the problems make it harder.
(i) Bounded rationality model. In the models of
-boundedrational agents defined in Section 4.1, is generally an
upper bound on the size of the optimization or knowledge
error. One interpretation of our results is that value drift
can happen even if the error is bounded at every step. One
could argue that a more realistic scenario would assume
some distribution of the size of the errors, assuming larger
errors less likely or less frequent; see discussion below
and in Section 6.
(ii) Unlimited self-modification ability. We assume the agent
is able to perform any self-modification at any time. This
models the worst-case scenario when compared to a
limited but still perfectly controlled self-modification.
However, embedded (non-dualistic) agents in complex
environments may chieve almost-unlimited self-modification
from a limited ability, e.g. over a longer time span; see
e.g.
Temporal discounting contrasts with the long-termist view: Why not model non-discounted future utility directly? Noting the motivations we mention in (vii), we agree that models of future value aggregation other than discounting would be generally better suited for long-term objectives. However, this seems to be a difficult task, as such models are neither well developed nor currently used in open-ended AI algorithms (with the obvious exception of a finite time horizon, which we propose to explore in Section 6).
We therefore propose a direct interpretation of our results: Assuming we implement agents that are -optimizers with discounting, they may become exponentially less aligned over time. This is not the case with perfect optimizers with imperfect knowledge and discounting. (x) Dualistic setting. We assume a dualistic agent and allow self-modification through special actions. This allows us to formally model one aspect of embedded agency – at least until there are sufficient theoretical foundations of embedded agency.
Note that in the embedded (non-dualistic) agent setting, it
is not formally clear – or possibly even definable – what
constitutes a self-modification, since there is no clear
conceptual boundary between the agent and the environment,
as discussed by
Assumption categories and the problem space. Each assumption identifies a subspace of research questions we would obtain by varying the relevant choices. These subspaces vary from very technical (e.g. concrete rationality model) to foundational (e.g. finite values and dualistic agent models). Along this axis, the assumptions and choices point to different kinds of prospective problems; we briefly describe three such categories and their prospects. See Section 6 for concrete proposals of future work.
Technical choices: A concrete model of bounded rationality, unlimited self-modification model, and modificationindependence. These are likely important for short and medium time-frames, where even eventually-diverging guarantees are useful.
We believe that many models within some realistic and sufficiently strong model classes would lead to qualitatively equivalent results in long time horizons; e.g. the agent divergence would be asymptotically exponential without external corrigibility, embedded agents in sufficiently complex environments would be able to self-modify arbitrarily over a long time (see discussion above) etc. However, these intuitions call for further verification.
Problem components: No corrigibility mechanisms, unbounded time horizon, time-invariant temporal discounting, focus on the worst-case guarantees. For those, there are interesting alternatives that may yield more optimistic results. In particular, it would be valuable to explore formal models of corrigibility, perform a full probabilistic analysis of agent development, and develop long-term non-discounted finitetime settings.
Foundational assumptions: Dualistic agent model and finite value of the future. Those are a standard in the area, but alternative settings may open up important and fruitful model classes and technical choices that capture currently pre-paradigmatic aspects (e.g. theory of embedded agency and non-dualistic agents).
3
In this section, we explain our model of a self-modifying
agent, which is borrowed from
We use a modified version of the standard cybernetic model. In this model, an agent interacts with the environment in discrete time steps. At each time step t, the agent performs an action at from a finite set A and the environment responds with a perception et from a finite set E . An action-perception pair aet is an action concatenated with a perception. A history is a sequence of action-perception pairs ae1ae2:::aet. We will often abbreviate such sequences to ae<t = ae1:::aet 1 or aen:m = aen:::aem. A complete history ae1:1 is a history containing information about all the time steps.
An agent can be described by its policy . The policy4 : (A E ) ! A is used to determine the agent’s next action from the history at time t. We consider (boundedrational) utility maximizers, where the policy is (partially) determined by the instantaneous utility function u, belief and discount factor . We sometimes use the notation = (u; ; ), where is called the agent’s knowledge. The utility function u~ : (A E )1 ! R describes how much the agent prefers the complete history ae1:1 compared to other complete histories. We will assume that the total utility is a discounted sum of instantaneous utilities given by the instantaneous utility function u : (A E ) ! [0; 1]. Formally, u~(ae1:1) = Pt1=1 t 1u(ae t). The discount factor describes how much the agent prefers immediate reward compared to the same reward at a later time. Smaller means heavier discounting of the future and stronger preference for immediate reward. Note that the maximum achievable utility is 1 1 , which happens when u(aet) = 1 at each step. Also note that instantaneous utility depends not only on the latest perception but can also depend on all previous perceptions and actions.
In addition to all this, an agent has a belief : (A E ) A ! E where E is the set of full-support probability distributions over E . This is a function which maps any history ending with an action onto a probability distribution over the next perceptions. Intuitively speaking, the belief describes what the agent expects to see after it performs an action given a certain history. A belief together with a policy induce a measure on the set (A E ) using P (et j ae<tat) = (et j ae<tat) and P (at j ae<t) = 1 if (ae<t) = at and 0 otherwise. Intuitively speaking, this probability measure captures probabilities assigned by the agent to possible futures.
Following the reinforcement learning literature, we define the value function V : (A E ) ! R as the expected future discounted utility:
1 V (ae<t) = E[ X
t0 tu(ae<t0 ) ] t0=t The expectation value on the right is calculated with respect to belief and assuming the agent will follow the policy . Intuitively, the value function describes how promising the future seems. When the value V (ae<t) of a history is high, it means we can expect an agent with policy to collect a lot of utility in the future starting from this history. Note that when calculating V-values, instantaneous utilities are multiplied by t0 t rather than t0 . This means that V-values can remain high throughout the whole history and are not affected by discounting. We define the Q-value of an action as the expected future discounted utility after taking that ac4For a set S, S denotes the set of finite sequences of elements from S
Q (ae<tat) = E[u(ae1:t) + V (ae1:t)] where the expectation is over the next perception drawn from the belief (note that belief is a probability distribution)
The Q-value measures how good an action is given that the agent will later follow policy . A policy is an optimal policy when V (ae<t) = sup V (ae<t) for all histories ae<t (such a policy always exists, as shown in (Lattimore and Hutter 2014)). 3.1
In this section, we extend the formalism above to include the possibility of self-modification. Since we are interested in the worst-case scenario, we assume the agent has unlimited self-modification ability. Worst-case results derived for such an agent will also hold for agents with limited ability to selfmodify.
Definition 1. A policy self-modification model is defined as a quadruple (A; E ; P; ) where A is the set of world actions, E is the set of perceptions, P is a non-empty set of names and is a map from P to the set of all policies .
At every time step, the agent chooses an action at = (at; pt+1) from the set A = (A P). The first part at describes what the agent “actually does in the world” while the second part chooses the policy t+1 = (pt+1) for the next time step. We will also use the notation at = (at; t+1), keeping in mind that only policies with names may be chosen. This new policy is used in the next step to pick the action at+1 = t+1(ae1:t). Note that only policies with names can be chosen and that P = is not a possibility because it entails a contradiction: j j = j(A E 2j j > j j. A history can now be written as: )jj(A E ) j > ae1:t = a1e1a2e2:::atet = a1 2e1a2 3e2:::at t+1et The subscripts for policies are one time step ahead because the policy chosen at time t is used to pick an action at time t + 1. The subscript denotes at which time step the policy is used. Policy t is used to choose the action at = (at; t+1). No policy modification happens when at = (at; t).
In the previous section, we used these rules to calculate the probability of any finite history: P (et j ae<tat) = (et j ae<tat) and P (at j ae<t) = 1 if (ae<t) = at and 0 otherwise. However, the second rule doesn’t take into consideration that the agent’s policy is changing. Therefore, to account for self modification, we need to modify the second rule into “P (at j ae<t) = 1 if t(ae<t) = at and zero otherwise”. To evaluate the V and Q-functions for self-modifying agents, we need to use probabilities of complete histories calculated in this way.
Definition 2. Let ae1:t denote the history ae1:t with information about self-modification removed so that ae1:t = a1e1a2e2:::atet. A function f : (A E ) ! (anything) 0 is modification-independent if ae1:t = ae1:t implies that f (ae1:t) = f (ae01:t).
the paper, we will assume that the agent’s belief and utility function as well as the correct belief are modificationindependent.
4
We now extend the model from
We introduce the notion of -optimizers. Intuitively speaking, the expected future discounted utility gained by an optimizer is no more than lower than the optimal one in any situation they could get into (that is, for any history). Definition 3. We say that agent A is an -optimizer for history ae<t if it holds that
Q(ae<t (ae<t)) sup Q(ae<t 0(ae<t))
0
When the utility function, belief and discount factor is obvious from the context (or unimportant), we also speak of policy being -optimizing (with respect to the utility function and belief), meaning that the corresponding agent is an -optimizer. 4.2
We consider agents with inaccurate knowledge of the correct utility function (Definition 4), inaccurate knowledge of the world (Definitions 5 and 6), and inaccurate knowledge of the correct discount factor (how much future reward is worth compared to reward in the present).
Misaligned agents We define -misaligned agents as agents whose utility function u has absolute error with respect to the correct utility function u .
Definition 4. We say that the instantaneous utility function u has absolute error with respect to the correct utility function u if
sup ju(ae<t) t2N;ae<t u (ae<t)j = Ignorant agents We define -ignorant agents as agents whose belief has error (absolute or relative depending on the context) with respect to the correct belief .
For belief, we define both relative and absolute error. This is in contrast with utility, for which this does not make sense in our setting. This is because when one speaks of relative utility, one usually compares it to some default action of “doing nothing” which we do not have.
Bounded-rational agents Definition 5. We say that a belief has absolute error with respect to the correct belief if for any t 2 N, history ae<t and action a,
k (ae<ta) (ae<ta)kT V (?) where k kT V is the total variational distance.
Recall that (on discrete measure spaces where all subsets are measurable) for two distributions (formally two probability measures) , on E , the total variational distance is defined as k kT V = sup j (E)
E E Definition 6. We say a belief spect to the correct belief ae<t, action a, and percept e, 1
(ejae<ta) 1 + (ejae<ta) 1 +
(E)j has relative error with reif for any t 2 N, any history Impatient agents We define impatient agents as agents whose discount factor is smaller than the correct discount factor . This means they have a stronger preference for immediate reward compared to the same reward in the future.
We now formally describe our results, including statements of the theorems. Proofs and description of the techniques used to prove these results are included in the full version of this paper.
In their paper,
Theorem 7. Let and u be modification-independent. Consider a self-modifying agent which is an -optimizer for the empty history. Then, for every t 1,
Eae<t [Q (ae<t t (ae<t))]
Eae<t [Q (ae<t 1 (ae<t))] min( t 1 ; 1 1 ) (1) where the expectation is with respect to ae<t such that the perceptions are distributed according to the belief and the actions are given by ai = i (ae<i).
Moreover, for all histories ae<t given by ai = i (ae<i) for which the agent is an -optimizer it holds that Q (ae<t 1 (ae<t)) +
Q (ae<t t (ae<t)) (2) Equality in Inequality (1) can be achieved up to a factor of at most .
The expectation in inequality (1) is necessary as can be demonstrated by the following example. Consider an environment in which the first perception is with probability (1 ) and otherwise. Regardless of the first perception, the utility in the future is always 1 if the action following this perception is a and 0 if b. An -optimizing agent which performs action a may choose to self-modify to an agent which performs action a after perception and b after perception , thus losing 1 utility in the case of perception , regardless of how small is.
Setting = 0 allows us to easily recover Theorem 16 from
If we only care about future discounted utility, this deterioration in performance doesn’t need to concern us because it only happens at future times when utility is heavily discounted. From the definition of an -optimizer, the maximum utility lost is indeed only . On the other hand, if we care about long-term performance of the agent and have only introduced the discount factor for instrumental reasons (as would likely be the case), the possibility of self-modification becomes a serious problem. The discount factor might be introduced because optimizing the long-term future might be computationally intractable.
In this section, we discuss perfect utility maximizers with bounded knowledge and show performance guarantees for such agents. In Section 5.3, we combine these results, show how to relax the assumption of perfect optimization and, most importantly, show how the performance of a boundedrational agent differs between the cases with and without self-modification.
In Lemma 8, we show that if the agent’s estimate of the expected discounted utility is at most away from the true value, the agent will be a 2 -optimizer. In sections 5.2 to 5.2, we show bounds on how inaccurate the agent’s estimate of expected discounted utility can be, thus proving bounds on optimization. In Section 5.2, we proceed differently: we formulate the worst case as a solution of an optimization problem which we then solve analytically.
Lemma 8. Let A be a (possibly self-modifying) perfect expected utility maximizer with knowledge = (u; ; ). Let = (u ; ; ) be the correct knowledge. Assume that jV (ae<t)
V (ae<t)j for all policies and histories ae<t. Then, A is a 2 -optimizer with respect to . -misaligned agents are 0-optimizers We now consider agents with an inaccurate utility function and derive bounds on 0 such that the misaligned agent is an 0-optimizer. Theorem 9. Let A be a perfect utility maximizer with utility function u and error with respect to the correct utility function u . Then it is a 12 -optimizer with respect to u . Moreover, this bound is tight.
In the random-error case when for any ae<t, we randomly choose u(ae<t) from the set fmax(0; u (ae<t) ); min(1; u (ae<t) + )g, we give a simple lower bound that is only a factor 4 away from the upper bound. Consider an environment with only one perception 1 and actions f0; 1g and u (1jae<t0) = 1 2 and u (1jae<t1) = 1. With probability 1=4, it holds that u(1jae<t1) = u(1jae<t0), in which case the agent may take the suboptimal action 0, thus losing 2 in instantaneous utility. At every step, it therefore loses =2 instantaneous expected utility. In total, it then loses in expectation 2(1 ) . We have thus proved the following: Theorem 10. Let A be a perfect utility maximizer whose utility function u is such that for any et; ae<tat, the value u(etjae<tat) is chosen independently and uniformly from the set fmax(0; u (etjae<tat) ); min(1; u (etjae<tat) + )g. Then, the amount of utility lost is in expectation 2(1 ) -ignorant agents are 0-optimizers In this section, we discuss agents with inaccurate belief. Theorem 11 gives bounds on the utility lost as a result of the agent having an inaccurate belief. We give an upper bound in terms of the (weaker) absolute error and lower bounds in terms of both absolute and relative error, showing that the upper bound is tight up to factors of 2 and 4 for absolute and relative error. Theorem 11. Let A be a perfect expected utility maximizer whose belief has absolute error with respect to the correct belief . Then it is a ( 1 2 1 (21 ) )-optimizer with respect to and this bound is tight up to a factor of 2. Moreover, if is the relative error, this bound is tight up to a factor of 4.
So far, we have considered the worst-case scenario. In the next theorem, we show that in the case of both absolute and relative error, the upper bound is tight up to a constant factor even in the case when the error at each timestep is randomly chosen from the set f ; g (that is, (etjae<tat) is chosen uniformly from the set fmax(0; (etjae<tat) ); min(1; (etjae<tat) + )g in the case of absolute error and f (et1j+ae<tat) ; min(1; (1 + ) (etjae<tat))g in the case of relative error), independently of other timesteps. Theorem 12. Let A be a perfect expected utility maximizer whose belief is such that for any et; ae<tat, the value (etjae<tat) is independently for any argument chosen uniformly from the set fmax(0; (etjae<tat) ); min(1; (etjae<tat) + )g in the case of absolute error and f (et1j+ae<tat) ; min(1; (1 + ) (etjae<tat))g in the case of relative error.
Then, in expectation, the amount of expected discounted utility lost is respectively 1 1
=8)
Impatient agents are -optimizers In this section, we discuss the case when an agent has an incorrect discount factor and give a bound on the performance of this agent with respect to the correct discount factor. We only consider the case when the agent discounts faster than the correct discount rate – we deem this to be the interesting case as, generally speaking, while optimizing in the long-term might be desirable, it is difficult to achieve, so the agent is likely to optimize in shorter term than desired. Bounds for the other case can be derived by the same method. To simplify the 1 . bounds, we define k = d lg e Theorem 13. Let and be perfect expected utility maximizers with respect to discount factors and respectively for some , either with or without the ability to self-modify. Let u, be their utility function and belief. Then k 1 jV (ae<t)
V (ae<t)j k + 1 For ! 1, it holds that d lg1 e lg1 . This enables us to simplify the previous result to get a good approximation for when is close to 1: k + In this section we combine the results from sections 5.1 and 5.2 and present a bound on the utility lost by an agent which is misaligned, ignorant, impatient and has bounded optimization, all at the same time. It is an interesting feature of this bound that the worst-case performance guarantee can in some cases be improved by adjusting its discount rate.
Recall that the functions f in the following theorem have been defined in Section 1.1.
Theorem 14. Let A be an o-optimizer for the empty history with either (1) the ability to self-modify and modificationindependent utility function and belief, or (2) without the ability to self-modify and with a possibly modificationdependent utility function and belief. Let be the agent’s discount rate, u the error in its utility function wrt. the correct utility function u and its absolute error in belief with respect to the correct belief . Then at timestep t: (1) If we let 0 be the smallest possible number such that A at time t is an 0-optimizer, then Eae<t [ 0] fopt( o; ) + futil( u; ) + fbel( ; ) + fdisc( ; ) where the expectation is over histories where perceptions are distributed according to and actions are given by the agent’s policy. Moreover, if o = 0, then 0 fopt( o; ) + futil( u; ) + fbel( ; ) + fdisc( ; ) almost certainly. (2) A will be an 0-optimizer, with respect to the correct discount rate , where 0 o + futil( u; ) + fbel( ; ) +
Moreover, when 1=2, there exists an agent which achieves equality up to a factor of at most 8 and up to a factor of 16 if is the relative error.
6
We propose several directions for future research. In general, it would be interesting to explore the central problem of selfmodification safety under different agent and environment models and with different assumptions.
Bounded rationality models. We analyzed a model of bounded-rationality with a strict upper bound on the size of errors (of several kinds). While this shows that even agents guaranteed to have small errors may self-modify in detrimental ways, the analysis would be significantly different for fully stochastic bounded rationality models (e.g. negligible expected errors with non-negligible variance). One such model of interest is Information-Theoretic Bounded Rationality of Ortega et al. (2015).
Awareness of own bounded-rationality. Whatever underlying decision procedure the agent uses somehow implicitly takes its -optimality into account – in particular since the assumed -optimality depends on the behavior of future agent versions. In our formulation, we do not assume the agent to have explicit knowledge of its bounded rationality model and , which would at least intuitively seem useful to know.
Note, however, that in our framing such explicit knowledge would not be necessarily useful, as any deliberation about it is subject to the same error within -optimality. Therefore it may be interesting to explore bounded rationality models where the information about own bounded rationality could be explicitly reasoned about (with more precision than e.g. modelling the trajectory of the full environment). Would the agent then be more reluctant to selfmodify?
counting would likely yield results with stronger
implications (Section 2). We propose analysing the finite-time
undiscounted case, as well as exploring other means of
future value aggregation (finite or infinite, as explored by
Model of self-modification. As noted above, embedded agents with a strong influence on the environment may self-modify by exploiting the environment. However, the extent of this self-modification, and the strength and stability of mechanisms against self-modification (e.g. via modification-dependent utility function) require further research.
Probabilistic analysis. Build stochastic models of agent rationality and self-modification, and perform full probabilistic analysis. This may e.g. inform us about required safety margins. In particular, approaches based on statistical physics and information theory seems to be promising here and have already proven fruitful in analyzing existing optimization problems and algorithms.
7
This work was carried out as part of the AI Safety Research Program 2019. We are grateful to Vojteˇch Kovarˇ´ık, Jan Kulveit, and Gavin Leech (in alphabetical order) for their valuable inputs.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Miller, J. D.; Yampolskiy, R.; and Ha¨ggstro¨m, O. 2020. An AGI Modifying Its Utility Function in Violation of the Orthogonality Thesis. arXiv preprint arXiv:2003.00812 . Russell, S. 2019. Human compatible: artificial intelligence and the problem of control. New York, New York: Viking. ISBN 9780525558613.
Strotz, R. H. 1955. Myopia and inconsistency in dynamic utility maximization. The review of economic studies 23(3): 165–180.