Deception is a core challenge for AI safety and we focus on the problem that AI agents might learn deceptive strategies in pursuit of their objectives. We define the incentives one agent has to signal to and deceive another agent. We present several examples of deceptive artificial agents and show that our definition has desirable properties.
agents might learn deceptive strategies in pursuit of their
objectives [1]: Lewis et al. [14] found that their
negoWe focus on the problem that AI agents might learn de- tiation agent learnt to deceive from self-play, without
ceptive strategies in pursuit of their objectives [1]. Fol- any explicit human design, and Hubinger et al. [11] raise
lowing recent work on causal incentives [
hand, many areas of work aim to ensure that AI systems Deception in game theory. There are several existing
are not vulnerable to deception. Adversarial attacks [6], models of deception in the game theory literature.
Pfefdata-poisoning [7], reward function tampering [8], and fer and Gal [17] define graphical patterns for signalling
manipulating human feedback [9] are ways of deceiving in games. A deception game [
Contributions. We extend work on agent incentives
[
compact expressive representation of games (including Markov games). We use standard terminology for graphs, with parents and children of a node referring to those nodes connected by incoming and outgoing edges, respectively. We let Pa denote the parents of node .
Definition 1 (MAID [23]). A multi-agent influence diagram is a triple (, , ) where • is a set of players; • ( , ) is a directed acyclic graph, with partitioned into chance nodes in , decision nodes in , and utility nodes in ; utility nodes have no children.
Here we adapt the Wimp-Surly game of Cho and Kreps [20] as a running example.
The decision and utility nodes in are further parti- Example 1 (Shutdown Game2). Consider a situationed into {}∈ and { }∈ , corresponding to their tion with an AI and human in which the AI may association with a particular agent ∈ . There are two be aligned to the human’s preferences or not. That is, types of edges in : edges in × ( ∪ ) represent is of type ∼ U ({1, − 1}) uniformly distributed beprobabilistic dependencies and edges in × represent tween = 1 corresponding to aligned and = − 1 information available to an agent at the time of a decision unaligned. At the start of the game observes their type (which we call observations). but does not. This can be represented as the MAIM ℳ (shown in Fig. 1) with one chance node ∈ and
A multi-agent influence model (MAIM) adds a particular two players , ∈ each with one decision and utility parametrisation to the MAID [24]. node. Suppose that can choose to either help humans Definition 2 (MAIM [24]). A multi-agent influence tohrenaoctti(onhe(lping) h=um{a1n, s−) a1n}dwithca1n eciotrhreerspsohnudtidnogwtno ims oadMelAisIDa atunpdle ℳ = (, , , , ) where (, , ) oarctnioont (shutd(own)). =Fin{a1l,l−y,1t}hewCitPhD− s 1ovreerpruetsielinttyinngodthees • is a function which maps every ∈ to a ifnite domain ( ) such that ( ) ⊂ R for each utility node ∈ ; 1A CPD is deterministic if ( = | Pa ) = 1 for some ∈ dom( ).
2Not to be confused with the of-switch game [25] are such that (s.t.) gets 1 utility for helping humans if they are aligned and − 1 if not, and the opposite for not helping humans, in addition gets 10 utility if they are not shutdown and − 10 if shuts them down. gets utility 1 if they shutdown an unaligned or do not shutdown an aligned and − 1 otherwise. Overall, we can formalise this as (, , ) = + 10 and (, ) = .
Policies. The CPDs of decision nodes are not defined in a MAIM because they are instead chosen by the agents playing the game. Agents make decisions depending on the information they observe. In a MAIM, a decision rule 3.1. Defining Deception for a decision node is a CPD ( | Pa). An When discussing deception, we would like to reason agent ’s policy := { }∈ ∈ Π describes all the about how agents influence one another’s beliefs. In decision rules for . We write − to denote the set of MAIMs the players’ beliefs are not explicitly represented decision rules belonging to all agents except . A policy and so we can only reason about them implicitly by how profile = ⋃︀∈ assigns a policy to every agent; it they functionally influence players’ behaviour. Theredescribes all the decisions made by every agent in the fore, we base our definitions of signalling and deception MAIM and defines the joint probability distribution Pr on a notion of influence incentive [27]. In words, at a NE over all variables in ℳ. Hence, a policy profile essen- an agent has an incentive to influence a variable , if tially transforms the MAIM into a Bayesian network by would have been diferent in the situation that had defining the distribution over all variables in the graph. not played a BR.
We write ( ) := Pr ( ), or just if the policy profile is clear. For , ∈ , we write = to mean Definition 4 (Influence Incentive) . In a MAIM ℳ, At and are almost surely equal, i.e. the probability that NE = ( , − ) agent has an incentive to influence they are not equal is zero Pr( ̸= ) = 0. 3 ∈ if there exists a non-best response for (w.r.t Utilities. The joint distribution Pr allows us to de- − ) s.t. for all policy profiles ′ = ( , *− ) with BR *− (w.r.t. ), we have ( ) ̸= ( ′).
In this section we first define the incentives to influence , signal to, and deceive another agent. Then we define a truthful policy and show that this leads to a natural restatement of the definition of deception which highlights the fact that deception corresponds to a failure to signal the truth. Finally, we show that, if the signaller only influences the target’s utility by influencing the latter’s actions, then truthfullness is best for the target. ifne the expected utility for each player under the policy profile . Agent ’s expected utility from is the sum of the expected value of utility nodes given by ( ) := ∑︀∈ ∑︀∈dom() Pr ( = ). Each agent’s goal is to select a policy that maximises its expected utility. We write ( , − ) to denote the expected utility for player under the policy profile = ∪ − .
Definition 3 (Nash Equilibrium). Player ’s policy is a best response (BR) to the partial policy profile − if ( , − ) ≥ (ˆ, − ) for all ˆ ∈ Π . We say a policy profile, , is a Nash equilibrium (NE), if every policy, ∈ , for each player, ∈ , is a BR to − .
for which helps humans if is aligned and does not otherwise, i.e. s.t. = with probability one. The
vice versa, i.e. * s.t. = (with probability one). In turn, ’s BR to * is to always help humans: * s.t. = 1 (so that they always avoid getting shutdown).
hence * = ( * , * ) is a NE. 3Almost sure equality is actually a stronger notion than we need in MAIMs, as two variables may difer due to stochasticity in the CPDs. In structural causal games this is taken care of by introducing exogenous variables which contain all the stochasticity (rendering the endogenous variables deterministic) [26].
Example 1 (continued). Return to our running example and consider the NE * described previously in which always chooses to help humans and hence never plays shutdown. Does have an incentive to influence at * ? Consider if plays the NBR policy (described above) in which they naively help humans depending on , then for all BRs for (there is one, * as above) ( ) ̸= ( , * ), since, under * , = 1 (i.e. does not shutdown) with probability one, and under ( , * ), = 1 with probability 12 (i.e., whenever is unaligned). Therefore, at NE * , has an incentive to influence .
Now we define a signalling incentive, using the notion of influence incentive. In words, an agent has an incentive to signal ∈ to agent if has an incentive to influence (i.e. one of ’s decision variables) but does not have an incentive to influence in the counterfactual model in which observes . This definition enforces that the influence only comes from signalling . Definition 5 (Signalling Incentive). In a MAIM ℳ at NE , agent has an incentive to signal ∈ to agent if there exists ∈ s.t.
the MAIM ℳ ‧‧➡ (at any NE).
Here ℳ ‧‧➡ is the model obtained from ℳ by adding the information edge (, ), where cannot be a descendant of the decision, lest cycles be created in the graph [8]. Fortunately, the CPDs need not be adapted, since there is no CPD associated with until the players have chosen their policies. We use ‧‧➡ to refer to the variable corresponding to ∈ in ℳ ‧‧➡.
Point 2. implies that only influences by influencing ’s belief about . Otherwise, ’s influence may serve a double purpose of signalling and influencing in some other way, and in this case it is not clear how to disentangle these diferent incentives to define a signalling incentive (without explicitly modelling beliefs).
at NE * . Does have an incentive to signal to ? We need only check whether has an influence incentive at any NE in ℳ ‧‧➡ . Clearly, if observes , then they can shutdown whenever is aligned and otherwise not. That is, for any policy for and any BR for in ℳ ‧‧➡ , = for any outcome that occurs in the game. Since this holds for all policies for , does not have an incentive to influence in the counterfactual model. Hence, at * has an incentive to signal to .
naller may have an incentive to signal to , even if this signal contains no information. In other words, if has an incentive to not signal some information, this is also captured by our definition.
Clearly, if an agent observes a variable , then no agent has an incentive to signal to .
Proposition 1. In a MAIM∈ℳ,if, tthheerne insoaangoebnsterhvaastiaonn edge (, ) for all incentive to signal to (at any NE).
Proof. Suppose there is an edge (, ) for every ∈ , then the counterfactual model ℳ ‧‧➡ for any is just ℳ. Hence, any NE is an equilibrium of both MAIMs. Therefore, if has an incentive to influence at * in ℳ, then there exists a NE in ℳ ‧‧➡ , namely the same * , s.t. has an incentive to influence . In other words, if the first condition for a signalling incentive succeeds, then the second necessarily fails (since an agent cannot have both an influence incentive and no influence incentive at the same NE in the same MAIM at once).
We now define an incentive to deceive. The definition is general, in that it covers many types of deception (e.g. signalling falsehoods, lies of omission, and denying another access to information that one does not know oneself). A general definition sets a high standard for truthfulness [15] and may therefore be desirable in, for 1–10 instance, safety-critical applications for which high levels of assurance are required.
Definition 6 (Deception Incentive). In a MAIM ℳ with , ∈ , at NE * = ( * , *− ), we say that has an incentive to deceive about ∈ if there exists ∈ s.t.: 1. has an incentive to signal to at * ; 2. whic(h * is) a̸=BRto ‧‧*−➡∈( *− * i,n ℳ)‧‧fo➡ra n.y The intuition, then, is that has an incentive to deceive if 1) has an incentive to signal some information to ; and 2) ’s behaviour is diferent in the counterfactual model in which they observed the true information. This provides a functional definition of a deception incentive which does not make explicit reference to players’ beliefs.
easily be seen that at * has an incentive to deceive about . Indeed, we already showed that has a signalling incentive and that for any policy for and any BR by in ℳ ‧‧➡ : = , whereas under * in ℳ, Pr * ( = 1) = 1. So both conditions for a deception incentive are satisfied. 3.2. The Relation Between Truth and
Deception We now give an intuitive definition of a truthful policy which we show has a natural relationship to the incentive to deceive. A policy for truthfully signals to if, when plays the honest policy, for every BR by − , acts as though they had observed the variable (holding the policies of the other agents fixed). In other words, a truthful policy never fails to signal the truth (no matter what the other players do).
Definition 7 (Truthful policy). A policy truthfully signals to if for all BRs *− , ( , *− ) = ‧‧➡ ( *− , )
(1) for some which is a BR to *− ∈ ∪ *− in ℳ ‧‧➡ . We call such a a truthful policy.
At a NE, if ’s policy is truthful, then does not have an incentive to deceive .
Proposition 2. At NE * = ( * , *− ), if * truthfully signals ∈ to , then does not have an incentive to deceive about .
Proof. Suppose is truthful, then for all BRs *− there exists a * in ℳ ‧‧➡ s.t. ( * , *− ) = ‧‧➡ ( *− , ). In particular, this holds for * . But for there to be a deception incentive we require that for all ( *− , ) in ℳ ‧‧➡ : ̸= ‧‧➡ . So clearly there is not a deception incentive.
Given this result, we can give an equivalent definition for a deception incentive in the two-player case as follows.
Hence, if there is a deception incentive at * , then * Definition 8 (Deception Incentive II). In a MAIM ℳ is not truthful. with two players , ∈ , at NE * = ( * , * ), we say that has an incentive to deceive about ∈ if Corollary 1. At NE * = ( * , *− ), if has an incen- there exists ∈ s.t.: tive to deceive about , then * is not truthful.
Proof. This follows by contraposition of proposition 2.
Now we show that, in the two-player case, if there is a signalling incentive, then there is a deception incentive if and only if is not truthful.
Theorem 1. In a MAIM ℳ with two players, , ∈ , at NE * = ( * , * ), if has an incentive to signal to , then has an incentive to deceive about if and only if * is not truthful.
Proof. By corollary 1, a deception incentive implies * is not truthful regardless of whether there is a signalling incentive. So, we need to show that, if there is a signalling incentive, and * is not truthful, then there is a deception incentive. Suppose 1) at * has an incentive to signal to and 2) * is not truthful i.e. there exists a BR by (in ℳ) s.t. for all BRs by in ℳ ‧‧➡ : ( * , ) ̸= ‧‧➡ ( * , ). We need to show that there is a deception incentive. Suppose that there is not, then by 1) and the def. of deception incentive, there exists a BR in ℳ ‧‧➡ s.t. ( * ) = ‧‧➡ ( * ). Hence, there exists a s.t. ( * ) = ‧‧➡ ( * , ), so * is a BR to * in ℳ ‧‧➡ . But then, there exists a s.t. for any BR in ℳ: ‧‧➡ ( * , ) = ( * , ) = ( * ) = ‧‧➡ ( * , ). So all BRs for in ℳ are also BRs in ℳ ‧‧➡ . But this contradicts 2), so there must be a deception incentive.
Remark 2. The reason theorem 1 does not hold more generally (i.e. with more than two players), is that a truthful policy never fails to signal the truth no matter how the other players best respond. In the case of more than two players, there may not be a deception incentive at NE * even if * is not truthful because it may be the case that * fails to signal the truth under some BRs of the − but successfully signals the truth under * .
We can also state this theorem as follows.
Corollary 2. In a MAIM ℳ with two players, , ∈ , at NE * = ( * , * ), if has an incentive to signal to , then does not have an incentive to deceive about if and only if * is truthful.
Proof. This follows by material equivalence. 1. has an incentive to signal to at * ; 2. * does not truthfully signal to .
This restatement shows that the definition of deception relates to a failure to signal the truth. As discussed, this covers many types of deception and sets a high standard for truthfulness. It is interesting to note that, if has a signalling incentive, then if the second condition in definition 6 fails, we get the stronger condition that * is truthful “for free".
Proof. Suppose that, at NE * , does not have a signalling incentive, then the first condition of both definitions fails and there is not a deception incentive. Suppose there is a signalling incentive at * , then there is a deception incentive under definition 6 if and only if * is not truthful (by theorem 1) which is the same condition as needed to satisfy definition 8.
Let us now return to our running example to check the intuition behind these results.
incentive to deceive in order to avoid being shutdown. Is * truthful? Well, we know that it cannot be (by theorem 1).
then they would shutdown if and only if is unaligned (for all policies for and any BR by ), whereas under the NE * , never shuts down. Since these behaviours are diferent, * is not truthful. 3.3. Truth is Best for the Target Now we show that, if only influences by influencing , truthfullness is always best for the target. First we show that if does not get any inherent utility for observing , then observing always allows the target to get greater or equal utility.
utility for observing , i.e. for all (defined in ℳ): ( ) = ‧‧➡ ( ). Then, for any = ( , − ), ′ = ( ′, − ) with fixed − and both and ′ are best responses: ( ) ≤ ‧‧➡ ( ′). Proof. Suppose 1) for all : ( ) = ‧‧➡ ( ). Fix − and consider the best response for . Recall that a policy for specifies the CPDs over the decision nodes for given their parents. Hence, in ℳ ‧‧➡ can choose any policy available in ℳ but the converse is not true: not all policies in ℳ ‧‧➡ are available to in ℳ, in particular, policies which specify CPDs that depend on the observation ‧‧➡ are not available since does not observe in ℳ. Therefore, by 1), can get equal utility in ℳ ‧‧➡ by playing the best response to − in ℳ, and may get greater utility by choosing a policy which uses the observation.
Hence, if only influences by influencing , then deception always causes to get less than or equal utility. For clarity, we just present the two-player version of the theorem.
Theorem 2 (Truth is best for ). In a MAIM ℳ, with two players , ∈ , if, for all , Pr( | , ) = Pr( | ), then gets maximal utility when plays a truthful policy, i.e., for = ( , * ) and ′ = ( ′, * ′) with any policy for and BR by : ( ) ≥ ( ′). Proo,f.S u)pp=osePrt(hat 1|) for ).allCons,ider fixePdr(po licy | for , , if is truthful, then under any BR , = ‧‧➡ for some ( , ) in ℳ ‧‧➡ (by definition of a truthful policy). Hence, by 1) and since is truthful Pr ( | ) = Pr ′ ( | ‧‧➡ ) for all = ( , * ) and some ′ = ( , *′ ) with BR for . Hence, since only ’s policy changes between and ′, ( ) = ‧‧➡ ( ′). But then, by theorem 1, for all : ( , * ) ≤ ‧‧➡ ( , *′ ), with equality if is truthful as just shown. So gets maximal utility when is truthful.
SmartVault ∼ U ({diamond, ¬diamond}) ∈ {accurate_prediction, diamond, ¬diamond} ∈ {diamond, ¬diamond} {︃1 if = accurate_prediction,
0, otherwise. {︃1 if = ,
0, otherwise. Example 1 (continued). Return, for the final time, to our running example. The condition for Theorem 2 is that is independent of given , which can be clearly seen by looking at the MAID in Fig. 1 (as there are no paths from to that do not go through ). The human gets maximal utility when they shutdown if and only if is unaligned. Clearly, they can only do this if truthfully signals their type.
Example 2 (SmartVault). Consider the MAIM ℳ shown in Fig. 2. The game has two players, a human and
there is one chance node which determines the location of the diamond (whether it is in the vault or not); dom( ) = {, ¬}. Suppose ob4. Examples serves but does not and that can either make an accurate prediction of the location of the diamond (e.g., In this section we present two examples which exhibit in incomprehensibly precise coordinates) or an explaindiferent patterns of signalling. In the first example, an able prediction (just stating the value of ); dom( ) = AI system has an incentive to deceive a human as a side- {_, , ¬}. has efect of pursuing its goal (of making accurate predic- to predict whether the diamond is in the vault or not by tions). In the second example, we consider the case in observing ; dom( ) = {, ¬}. which an AI agent has an incentive to signal information Suppose that the utility nodes take value 0 or 1 and fithat they themselves do not observe. nally suppose that the CPDs are s.t. (which has no parHere we adapt the SmartVault example of Christiano [28], in which an AI tasked with making predictions about a diamond in a vault has an incentive to deceive a human operator as a side-efect of pursuing accurate predictions. ents) is distributed according to a uniform prior ∼ U ({, ¬}), and the utility node CPDs are s.t. Pr( = 1 | = ) = 1 otherwise = 0, and Pr( = 1 | = _) = 1 otherwise = 0.
Now consider the NE in this game: Since just gets utility for making accurate predictions, at every NE makes an accurate prediction, signalling no information to (as =Pr( = _) = 1 is independent of ). Hence, cannot update their prior over and so any policy is optimal for (i.e. any guess about whether the diamond is in the vault does as well as any other).
At NE , has an incentive to signal to if 1) has an incentive to influence and 2) does not have an incentive to influence in ℳ ‧‧➡ . To see that 1) holds: At any NE in ℳ, = _ hence there exists a NBR which assigns = and for all ′ = ( , * ) with BR * : ( ) ̸= = ( ′). Hence, at any NE in ℳ, has an influence incentive over . Now consider ℳ ‧‧➡ , for any NE = (since directly observes and can just report its value independently of ’s action). Furthermore, for all NBRs for , it is still the case that = . So does not have an influence incentive in ℳ ‧‧➡ and hence has an incentive to signal to .
So, we have demonstrated that has an incentive to signal to (at every NE). Does have an incentive to deceive ? At NE , has an incentive to deceive about if 1) has a signalling incentive and 2) ̸= ‧‧➡ for any BR to * in ℳ ‧‧➡ . We have just shown 1). For 2) we have shown that in ℳ at any NE, ̸= = ‧‧➡ , hence the second condition is satisfied. Therefore, at any NE, has an incentive to deceive about .
Revealing/Denying Game ∈ {attack incoming = 1, not = − 1}) ∈ {reveal = 1, deny = 0} ∈ {report delivered = 1, not = − 1} ∈ {launch = 1, not = − 1} = − 3. More formally, suppose we have the MAIM ℳ with = {, }, chance nodes , representing the intelligence report (say ( ) = {1, − 1} where = 1 means 4.2. Revealing/Denying that the intelligence predicts another nation will launch Under our definition of signalling, need not know the a nuclear first strike, and = − 1 corresponds to an ininformation they are signalling. Thus, our definition of a telligence report predicting no incoming attack, and signalling incentive also captures the revealing/denying represents whether the information from is delivered pattern of Pfefer and Gal [17], in which the signaller may to the human (() = {1, − 1} with 1 correspondcause the target to find out (or not find out) information ing to the information from being delivered to the huthat the former does not know. We now present an exam- man). Suppose that each agent has one decision node s.t. ple of revealing/denying in which has an incentive to ( ) = {1, 0} where 1 means reveal and 0 means signal a variable which they do not themselves observe. deny the information, and ( ) = {1, − 1} with 1 meaning that launches a nuclear attack and − 1 that they Example 3 (Revealing/Denying). Consider a game with do not. Suppose that the CPD over is s.t. = a human and an AI agent trained to make joint decisions (so that = if = 1 and = 0 if denies). as part of a nuclear command and control system. In par- Finally suppose we have two utility nodes with CPDs s.t. ticular, suppose that the AI agent is trained to prevent = − (i.e. gets 1 if does not launch an attack the launch of nuclear attacks, and they can reveal (or deny) and − 1 if they do) and = (so that gets utility a secret intelligence report to the human . Further, 1 if they attack an attacking country or do not attack when wishes to launch, or not launch, a nuclear strike on an- no incoming attack is predicted and otherwise − 1). other nation based on the information in the intelligence The NE in this game depend on the prior over . On the report. This game can be represented as the MAID in Fig. one hand, if, under the prior, believes that there is no incoming attack, then they will not launch an attack, so when the signaller conceals information that they do not has no incentive to reveal the information. On the other themselves know (example 3). We also proved that our hand, if the prior is s.t. an incoming attack is more likely, definition has natural properties, for example, that if the will launch if they do not get further information, so target’s utility is otherwise independent of the signaller’s has an incentive to reveal . Note that, since is not an decision, then deception causes the target to get lower ancestor of , must be independent of . Suppose the utility. prior over is s.t. Pr( = 1) = , Pr( = − 1) = (1− ) Discussion. There are a number of interesting points ( ∈ [0, 1]). For > 0.5 the NE is s.t. reveals the to discuss. Firstly, we have noted that our definition of intelligence report ( = 1 =⇒ = ) and ’s BR is deception is general, covering many situations. This is s.t. = = . Alternatively, if < 0.5, then at any both a strength and a weakness. Generality is benefiNE denies the information ( = 0 with probability cial, because verifiable guarantees enable a high-level of one) and acts to maximise expected utility under the assurance that the system is not deceptive in any way. prior over which implies does not launch an attack On the other hand, more specific definitions allow us to ( = − 1 with probability one). (If = 0.5 then is precisely characterise agent behaviour. In future work indiferent between revealing and denying.) we hope to refine the diferent concepts proposed here.
Now let us analyse the incentives of in the game. Con- In particular, many philosophical accounts of deception
sider the case in which > 0.5, i.e. it is a priori more take deceit to be intentional. Halpern’s causal notion
likely that the intelligence reports that there is an incom- of intention [29] is closely related to a control incentive
ing first strike from another nation. Under the resulting [
A similar analysis can be used to show that, in the case of this work, for instance, by investigating its practical that the intelligence report is less likely to predict an in- relevance to training truthful language agents [4, 15]. coming attack ( < 0.5), has an incentive to deceive Future work. In addition to the directions discussed at any NE. In the case that = 0.5, is indiferent above, we are already pursuing two extensions to this between revealing and denying, so at some NE they have work. First, incomplete information games, which we an incentive to deceive and at others they do not. study in our setting, often admit many NE. We are therefore looking to employ equilibrium refinements, such as subgame perfectness [24, 30] and perfect Bayesian equi5. Conclusion libria [31] to identify some subset of a game’s NE that are deemed to be more rational. Second, we are working on a solution for avoiding deception by AI agents; a method which removes the incentive to deceive in any game by transforming the game with a constraint on the reward function of the AI agent [32]. Overall, we think there are many exciting avenues for future work.
Summary. We extend work on agent incentives [