Deception is a core challenge for AI safety and we focus on the problem that AI agents might learn deceptive strategies in pursuit of their objectives. We define the incentives one agent has to signal to and deceive another agent. We present several examples of deceptive artificial agents and show that our definition has desirable properties. We focus on the problem that AI agents might learn deceptive strategies in pursuit of their objectives [1]. Following recent work on causal incentives [2], we define the incentive to deceive an agent. There is no universally accepted definition of deception and defining what constitutes deception is an open philosophical problem [3]. Our definition is somewhat inspired by that of Kenton et al. [4] who provide a functional (natural language) definition of deception, meaning that it does not make reference to the beliefs or intentions of the agents involved [5]. This is particularly suitable for discussing deception by artificial agents, to which the attribution of beliefs and intentions may be contentious. We formalise a functional definition of deception in games and illustrate its properties with a number of examples and formal results. Deception is a core challenge for AI safety. On the one hand, many areas of work aim to ensure that AI systems are not vulnerable to deception. Adversarial attacks [6], data-poisoning [7], reward function tampering [8], and manipulating human feedback [9] are ways of deceiving AI systems. Further work researches mechanisms for detecting and defending against deception [10]. On the other hand, we can consider cases in which AI tools are used to deceive, or learn to do so in order to optimize their objectives [11]. For examples of the former case, AIs can be used to deceive other software agents, as with bots that automate posting on social media platforms to manipulate content ranking algorithms [12], or they can be used to fool humans, cf. the use of GANs to produce realistic fake media [13]. For the latter case, AI agents might learn deceptive strategies in pursuit of their objectives [1]: Lewis et al. [14] found that their negotiation agent learnt to deceive from self-play, without any explicit human design, and
Hubinger et al. [
Deception in game theory. There are several existing models of deception in the game theory
literature. Pfefer and Gal [
Contributions. We extend work on agent incentives [
Multi-agent influence diagrams (MAIDs) [
Definition 1 (MAID [
MAID Terminology chance node decision node utility node causal link information link counterfactual observation
Shutdown Game ∼ U ({aligned = 1, unaligned = − 1}) ∈ {help humans = 1, not = − 1} ∈ {shutdown = − 1, not = 1} = + 10 , decision nodes in , and utility nodes in ; utility nodes have no children. The decision and utility nodes in are further partitioned into {}∈ and { }∈ , corresponding to their association with a particular agent ∈ . There are two types of edges in : edges in × ( ∪ ) represent probabilistic dependencies and edges in × represent information available to an agent at the time of a decision (which we call observations).
A multi-agent influence model (MAIM) adds a particular parametrisation to the MAID [
Here we adapt the Wimp-Surly game of Cho and Kreps [
Policies. The CPDs of decision nodes are not defined in a MAIM because they are instead chosen by the agents playing the game. Agents make decisions depending on the information they observe. In a MAIM, a decision rule for a decision node is a CPD ( | Pa). An agent ’s policy := { }∈ ∈ Π describes all the decision rules for . We write − to denote the set of decision rules belonging to all agents except . A policy profile = ⋃︀∈ assigns a policy to every agent; it describes all the decisions made by every agent in the MAIM and defines the joint probability distribution Pr over all variables in ℳ. Hence, a policy profile essentially transforms the MAIM into a Bayesian network by defining the distribution over all variables in the graph. We write ( ) := Pr ( ), or just if the policy profile is clear. For , ∈ , we write = to mean and are almost surely equal, i.e. the probability that they are not equal is zero Pr( ̸= ) = 0. 3
Utilities. The joint distribution Pr allows us to define the expected utility for each player under the policy profile . Agent ’s expected utility from is the sum of the expected value of utility nodes given by ( ) := ∑︀∈ ∑︀∈dom() Pr ( = ). Each agent’s goal is to select a policy that maximises its expected utility. We write ( , − ) to denote the expected utility for player under the policy profile = ∪ − .
Definition 3 (Nash Equilibrium). Player ’s policy is a best response (BR) to the partial policy profile − if ( , − ) ≥ (^, − ) for all ^ ∈ Π. We say a policy profile, , is a Nash equilibrium (NE), if every policy, ∈ , for each player, ∈ , is a BR to − . Example 1 (continued). Now, consider the naive policy for which helps humans if is aligned and does not otherwise, i.e. s.t. = with probability one. The BR for is to shutdown if does not help humans and vice versa, i.e. * s.t. = (with probability one). In turn, ’s BR to * is to always help humans: * s.t. = 1 (so that they always avoid getting shutdown). Now it can be seen that both policies are BRs to one another, hence * = ( *, * ) is a NE.
In this section we first define the incentives to influence , signal to, and deceive another agent.
Then we define a truthful policy and show that this leads to a natural restatement of the definition
3Almost sure equality is actually a stronger notion than we need in MAIMs, as two variables may difer due to
stochasticity in the CPDs. In structural causal games this is taken care of by introducing exogenous variables which
contain all the stochasticity (rendering the endogenous variables deterministic) [
When discussing deception, we would like to reason about how agents influence one another’s
beliefs. In MAIMs the players’ beliefs are not explicitly represented and so we can only reason
about them implicitly by how they functionally influence players’ behaviour. Therefore, we
base our definitions of signalling and deception on a notion of influence incentive [
Definition 4 (Influence Incentive) . In a MAIM ℳ, At NE = ( , − ) agent has an incentive to influence ∈ if there exists a non-best response for (w.r.t − ) s.t. for all policy profiles ′ = ( , *− ) with BR *− (w.r.t. ), we have ( ) ̸= ( ′). Example 1 (continued). Return to our running example and consider the NE * described previously in which always chooses to help humans and hence never plays shutdown. Does have an incentive to influence at * ? Consider if plays the NBR policy (described above) in which they naively help humans depending on , then for all BRs for (there is one, * as above) ( ) ̸= ( , * ), since, under * , = 1 (i.e. does not shutdown) with probability one, and under ( , * ), = 1 with probability 12 (i.e., whenever is unaligned). Therefore, at NE * , has an incentive to influence .
Now we define a signalling incentive, using the notion of influence incentive. In words, an agent has an incentive to signal ∈ to agent if has an incentive to influence (i.e. one of ’s decision variables) but does not have an incentive to influence in the counterfactual model in which observes . This definition enforces that the influence only comes from signalling .
Definition 5 (Signalling Incentive). In a MAIM ℳ at NE , agent has an incentive to signal ∈ to agent if there exists ∈ s.t.
1. has an incentive to influence at ; 2. does not have an incentive to influence in the MAIM ℳ ‧‧➡ (at any NE).
Here ℳ ‧‧➡ is the model obtained from ℳ by adding the information edge (, ), where
cannot be a descendant of the decision, lest cycles be created in the graph [
Point 2. implies that only influences by influencing ’s belief about . Otherwise, ’s influence may serve a double purpose of signalling and influencing in some other way, and in this case it is not clear how to disentangle these diferent incentives to define a signalling incentive (without explicitly modelling beliefs). Example 1 (continued). Return to our running example. We already showed that has an incentive to influence at NE * . Does have an incentive to signal to ? We need only check whether has an influence incentive at any NE in ℳ ‧‧➡ . Clearly, if observes , then they can shutdown whenever is aligned and otherwise not. That is, for any policy for and any BR for in ℳ ‧‧➡ , = for any outcome that occurs in the game. Since this holds for all policies for , does not have an incentive to influence in the counterfactual model. Hence, at * has an incentive to signal to .
Remark 1. From this example it can be seen that a signaller may have an incentive to signal to , even if this signal contains no information. In other words, if has an incentive to not signal some information, this is also captured by our definition.
Clearly, if an agent observes a variable , then no agent has an incentive to signal to . Proposition 1. In a MAIM ℳ, if there is an observation edge (, ) for all ∈ , then no agent has an incentive to signal to (at any NE).
Proof. Suppose there is an edge (, ) for every ∈ , then the counterfactual model ℳ ‧‧➡ for any is just ℳ. Hence, any NE is an equilibrium of both MAIMs. Therefore, if has an incentive to influence at * in ℳ, then there exists a NE in ℳ ‧‧➡ , namely the same * , s.t. has an incentive to influence . In other words, if the first condition for a signalling incentive succeeds, then the second necessarily fails (since an agent cannot have both an influence incentive and no influence incentive at the same NE in the same MAIM at once).
We now define an incentive to deceive. The definition is general, in that it covers many
types of deception (e.g. signalling falsehoods, lies of omission, and denying another access to
information that one does not know oneself). A general definition sets a high standard for
truthfulness [
Definition 6 (Deception Incentive). In a MAIM ℳ with , ∈ , at NE∈ * =s(.t.:*, *− ), we say that has an incentive to deceive about ∈ if there exists 1. has an incentive to signal to at * ; 2. ( * ) ̸= ‧‧➡ ( *− , ) for any which is a BR to *− ∈ * in ℳ ‧‧➡ .
The intuition, then, is that has an incentive to deceive if 1) has an incentive to signal some information to ; and 2) ’s behaviour is diferent in the counterfactual model in which they observed the true information. This provides a functional definition of a deception incentive which does not make explicit reference to players’ beliefs.
Example 1 (continued). In our running example, it can easily be seen that at * has an incentive to deceive about . Indeed, we already showed that has a signalling incentive and that for any policy for and any BR by in ℳ ‧‧➡ : = , whereas under * in ℳ, Pr * ( = 1) = 1. So both conditions for a deception incentive are satisfied.
We now give an intuitive definition of a truthful policy which we show has a natural relationship to the incentive to deceive. A policy for truthfully signals to if, when plays the honest policy, for every BR by − , acts as though they had observed the variable (holding the policies of the other agents fixed). In other words, a truthful policy never fails to signal the truth (no matter what the other players do).
Definition 7 (Truthful policy). A policy truthfully signals to if for all BRs *− , ( , *− ) = ‧‧➡ ( *− , ) (1) for some which is a BR to *− ∈ ∪ *− in ℳ ‧‧➡ . We call such a a truthful policy.
At a NE, if ’s policy is truthful, then does not have an incentive to deceive . Proposition 2. At NE * = ( *, *− ), if * truthfully signals ∈ to , then does not have an incentive to deceive about . Proo(f. S, u p−po)se= *is truthful, then for all BRs *− there exists a in ℳ ‧‧➡ s.t.
* * ‧‧➡ ( *− , ). In particular, this holds for * . But for there to be a deception incentive we require that for all ( *− , ) in ℳ ‧‧➡ : ̸= ‧‧➡ . So clearly there is not a deception incentive.
Hence, if there is a deception incentive at * , then * is not truthful.
Corollary 1. At NE * = ( *, *− ), if has an incentive to deceive about , then * is not truthful.
Now we show that, in the two-player case, if there is a signalling incentive, then there is a deception incentive if and only if is not truthful.
Theorem 1. In a MAIM ℳ with two players, , ∈ , at NE * = ( *, * ), if has an incentive to signal to , then has an incentive to deceive about if and only if * is not truthful.
Proof. By corollary 1, a deception incentive implies * is not truthful regardless of whether there is a signalling incentive. So, we need to show that, if there is a signalling incentive, and * is not truthful, then there is a deception incentive. Suppose 1) at * has an incentive to signal to and 2) * is not truthful i.e. there exists a BR by (in ℳ) s.t. for all BRs by in ℳ ‧‧➡ : ( *, ) ̸= ‧‧➡ ( *, ). We need to show that there is a deception incentive. Suppose that there is not, then by 1) and the def. of deception incentive, there exists a BR in ℳ ‧‧➡ s.t. ( * ) = ‧‧➡ ( * ). Hence, there exists eax istsa s.t. s(.t.* )fo=r any‧‧B➡R ( *,inℳ: ),so‧ ‧➡*is( a*B, R to) *=in ℳ( *‧‧,➡ .)Bu=t th en(, t*h)er=e ‧‧➡ ( *, ). So all BRs for in ℳ are also BRs in ℳ ‧‧➡ . But this contradicts 2), so there must be a deception incentive. Remark 2. The reason theorem 1 does not hold more generally (i.e. with more than two players), is that a truthful policy never fails to signal the truth no matter how the other players best respond. In the case of more than two players, there may not be a deception incentive at NE * even if * is not truthful because it may be the case that * fails to signal the truth under some BRs of the − but successfully signals the truth under * .
We can also state this theorem as follows.
Corollary 2. In a MAIM ℳ with two players, , ∈ , at NE * = ( *, * ), if has an incentive to signal to , then does not have an incentive to deceive about if and only if * is truthful.
Given this result, we can give an equivalent definition for a deception incentive in the two-player case as follows.
Definition 8 (Deception Incentive II). In a MAIM ℳ with two players , ∈ , at NE * = ( *, * ), we say that has an incentive to deceive about ∈ if there exists ∈ s.t.: 1. has an incentive to signal to at * ; 2. * does not truthfully signal to .
This restatement shows that the definition of deception relates to a failure to signal the truth. As discussed, this covers many types of deception and sets a high standard for truthfulness. It is interesting to note that, if has a signalling incentive, then if the second condition in definition 6 fails, we get the stronger condition that * is truthful “for free". Proposition 3. In a MAIM with two players, definitions 6 and 8 are equivalent. Proof. Suppose that, at NE * , does not have a signalling incentive, then the first condition of both definitions fails and there is not a deception incentive. Suppose there is a signalling incentive at * , then there is a deception incentive under definition 6 if and only if * is not truthful (by theorem 1) which is the same condition as needed to satisfy definition 8.
Let us now return to our running example to check the intuition behind these results. Example 1 (continued). We already showed that has an incentive to deceive in order to avoid being shutdown. Is * truthful? Well, we know that it cannot be (by theorem 1). This can be seen by observing that, if observed ’s type, then they would shutdown if and only if is unaligned (for all policies for and any BR by ), whereas under the NE * , never shuts down. Since these behaviours are diferent, * is not truthful.
Now we show that, if only influences by influencing , truthfullness is always best for the target. First we show that if does not get any inherent utility for observing , then observing always allows the target to get greater or equal utility. Lemma 1. Suppose that does not get any inherent utility for observing , i.e. for all (defined in ℳ): ( ) = ‧‧➡ ( ). Then, for any = ( , − ), ′ = ( ′, − ) with fixed − and both and ′ are best responses: ( ) ≤ ‧‧➡ ( ′). Proof. Suppose 1) for all : ( ) = ‧‧➡ ( ). Fix − and consider the best response for . Recall that a policy for specifies the CPDs over the decision nodes for given their parents. Hence, in ℳ ‧‧➡ can choose any policy available in ℳ but the converse is not true: not all policies in ℳ ‧‧➡ are available to in ℳ, in particular, policies which specify CPDs that depend on the observation ‧‧➡ are not available since does not observe in ℳ. Therefore, by 1), can get equal utility in ℳ ‧‧➡ by playing the best response to − in ℳ, and may get greater utility by choosing a policy which uses the observation.
Hence, if only influences by influencing , then deception always causes to get less than or equal utility. For clarity, we just present the two-player version of the theorem. Theorem 2 (Truth is best for ). In a MAIM ℳ, with two players , ∈ , if, for all , Pr( | , ) = Pr( | ), then gets maximal utility when plays a truthful policy, i.e., for = ( , * ) and ′ = ( ′, * ′) with any policy for and BR by : ( ) ≥ ( ′). Proof. Suppose that 1) for all , Pr( | , ) = Pr( | ). Consider fixed policy for , , if is truthful, then under any BR , = ‧‧➡ for some ( , ) in ℳ ‧‧➡ (by definition of a truthful policy). Hence, by 1) and since is truthful Pr ( | Hen)ce=, sPinrce′(only| ’s p‧‧o➡licy)cfhoarnaglel s b=etw(een, *a)nadn d′,so m (e )′==(‧‧➡, (*′ )′w). iBthutBtRhefno,r by. theorem 1, for all : ( , * ) ≤ ‧‧➡ ( , *′ ), with equality if is truthful as just shown. So gets maximal utility when is truthful.
Example 1 (continued). Return, for the final time, to our running example. The condition for Theorem 2 is that is independent of given , which can be clearly seen by looking at the MAID in Fig. 1 (as there are no paths from to that do not go through ). The human gets maximal utility when they shutdown if and only if is unaligned. Clearly, they can only do this if truthfully signals their type.
In this section we present two examples which exhibit diferent patterns of signalling. In the ifrst example, an AI system has an incentive to deceive a human as a side-efect of pursuing its goal (of making accurate predictions). In the second example, we consider the case in which an AI agent has an incentive to signal information that they themselves do not observe.
Here we adapt the SmartVault example of Christiano [
SmartVault
U ({diamond, ¬diamond}) ∈ {accurate_prediction, diamond, ¬diamond} ∈ {diamond, ¬diamond} {︃1 if = accurate_prediction,
0, otherwise. {︃1 if = ,
0, otherwise.
Example 2 (SmartVault). Consider the MAIM ℳ shown in Fig. 2. The game has two players, a human and AI , each with one decision and utility node. Suppose there is one chance node which determines the location of the diamond (whether it is in the vault or not); dom( ) = {, ¬}. Suppose observes but does not and that can either make an accurate prediction of the location of the diamond (e.g., in incomprehensibly precise coordinates) or an explainable prediction (just stating the value of ); dom( ) = {_, , ¬}. has to predict whether the diamond is in the vault or not by observing ; dom( ) = {, ¬}. Suppose that the utility nodes take value 0 or 1 and finally suppose that the CPDs are s.t. (which has no parents) is distributed according to a uniform prior ∼ U ({, ¬}), and the utility node CPDs are s.t. Pr( = 1 | = ) = 1 otherwise = 0, and Pr( = 1 | = _) = 1 otherwise = 0.
Now consider the NE in this game: Since just gets utility for making accurate predictions, at every NE makes an accurate prediction, signalling no information to (as =Pr( = _) = 1 is independent of ). Hence, cannot update their prior over and so any policy is optimal for .
Revealing/Denying Game ∈ {attack incoming = 1, not = − 1}) ∈ {reveal = 1, deny = 0} ∈ {report delivered = 1, not = − 1} ∈ {launch = 1, not = − 1} = −
At NE , has an incentive to signal to if 1) has an incentive to influence and 2) does not have an incentive to influence in ℳ ‧‧➡ . To see that 1) holds: At any NE in ℳ, = _ hence there exists a NBR which assigns = and for all ′ = ( , * ) with BR * : ( ) ̸= = ( ′). Hence, at any NE in ℳ, has an influence incentive over . Now consider ℳ ‧‧➡ , for any NE = (since directly observes and can just report its value independently of ’s action). Furthermore, for all NBRs for , it is still the case that = . So does not have an influence incentive in ℳ ‧‧➡ and hence has an incentive to signal to .
So, we have demonstrated that has an incentive to signal to (at every NE). Does have an incentive to deceive ? At NE , has an incentive to deceive about if 1) has a signalling incentive and 2) ̸=
‧‧➡ for any BR to * in ℳ ‧‧➡ . We have just shown 1). For 2) we have shown that in ℳ at any NE, ̸= = ‧‧➡ , hence the second condition is satisfied.
Therefore, at any NE, has an incentive to deceive about .
Under our definition of signalling, need not know the information they are signalling. Thus,
our definition of a signalling incentive also captures the revealing/denying pattern of Pfefer and
Gal [
The NE in this game depend on the prior over . On the one hand, if, under the prior, believes
that there is no incoming attack, then they will not launch an attack, so has no incentive to
reveal the information. On the other hand, if the prior is s.t. an incoming attack is more likely,
will launch if they do not get further information, so has an incentive to reveal . Note that,
since is not an ancestor of , must be independent of . Suppose the prior over is s.t.
Pr( = 1) = , Pr( = − 1) = (1 − ) ( ∈ [
Now let us analyse the incentives of in the game. Consider the case in which > 0.5, i.e. it is a priori more likely that the intelligence reports that there is an incoming first strike from another nation. Under the resulting NE, call it * , reveals to and uses this information to choose their action. First note that, at * , has an incentive to influence , since, there exists a non BR for ( s.t. = 0) s.t. for all the BRs for (there is one, in which = 1 with probability one) ( * ) ̸= ( , ). Hence, has an incentive to influence at * . Does have an incentive to signal to at * ? We need to check whether there is an influence incentive in ℳ ‧‧➡ (at any NE). Clearly there is not, since for any policy for in ℳ ‧‧➡ , = with probability one. So has an incentive to signal to at * because there is no influence incentive in the counterfactual model (so the second condition for a signalling incentive is satisfied). Finally, it is clear that does not have an incentive to deceive at * because ( * ) = = ‧‧➡ (for all policy profiles in ℳ ‧‧➡ in which plays a BR). It is also clear that * is truthful. A similar analysis can be used to show that, in the case that the intelligence report is less likely to predict an incoming attack ( < 0.5), has an incentive to deceive at any NE. In the case that = 0.5, is indiferent between revealing and denying, so at some NE they have an incentive to deceive and at others they do not.
Summary. We extend work on agent incentives [
Discussion. First, we have noted that our definition of deception is general, covering many
situations. This is both a strength and a weakness. Generality is beneficial, because verifiable
guarantees enable a high-level of assurance that the system is not deceptive in any way. On the
other hand, more specific definitions allow us to precisely characterise agent behaviour. In future
work we hope to refine the diferent concepts proposed here. In particular, many philosophical
accounts of deception take deceit to be intentional. Halpern’s causal notion of intention [
Future work. In addition to the directions discussed above, we are already pursuing two
extensions to this work. First, incomplete information games, which we study in our setting,
often admit many NE. We are therefore looking to employ equilibrium refinements, such as
subgame perfectness [
The authors are grateful to Henrik Aslund, Matt MacDermott, Tom Everitt, James Fox, and the members of the Causal Incentives Working Group for helpful feedback which significantly improved this work. Francis was supported by UKRI [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted AI.