Recent work has demonstrated robust mechanisms by which attacks can be orchestrated on machine learning models. In contrast to adversarial examples, backdoor or trojan attacks embed surgically modified samples in the model training process to cause the targeted model to learn to misclassify samples in the presence of specific triggers, while keeping the model performance stable across other nominal samples. However, current published research on trojan attacks mainly focuses on classification problems, which ignores sequential dependency between inputs. In this paper, we propose methods to discreetly introduce and exploit novel backdoor attacks within a sequential decision-making agent, such as a reinforcement learning agent, by training multiple benign and malicious policies within a single long short-term memory (LSTM) network, where the malicious policy can be activated by a short realizable trigger introduced to the agent. We demonstrate the effectiveness through initial outcomes generated from our approach as well as discuss the impact of such attacks in defense scenarios. We also provide evidence as well as intuition on how the trojan trigger and malicious policy is activated. In the end, we propose potential approaches to defend against or serve as early detection for such attacks.
Current research has demonstrated different categories of
attacks on neural networks and other supervised
learning approaches. Majority of them can be categorized as:
(1) inference-time attacks, which add adversarial
perturbations digitally or patches physically to the test samples and
make the model misclassify them
Most research on trojan attacks in AI mainly focuses on classification problems, where model’s performance is affected only in the instant when a trojan trigger is present. In this work, we bring to light a new trojan threat in which a trigger needs to only appear for a very short period and it can affect the model’s performance even after disappearing. For example, the adversary needs to only present the trigger in one frame of an autonomous vehicle’s sensor inputs and the behavior of the vehicle can be made to change permanently from thereon. Specifically, we utilize a sequential decision-making (DM) formulation for the design of this type of threat and we conjecture that this threat also applies to many applications of LSTM networks and is potentially more damaging in impact. Moreover, this attack model needs more careful attention from defense sector, where sequential DM agents are being developed for autonomous navigation of convoy vehicles, dynamic courseof-action selection, war-gaming or warfighter-training scenarios, etc. where adversary can inject such backdoors.
The contribution of this work is: (1) a threat model and formulation for a new type of trojan attack for LSTM networks and sequential DM agents, (2) implementation to illustrate the threat, and (3) analysis of models with the threat and potential defense mechanisms.
In the following sections of the paper, we will provide
examples of related work and background on deep
reinforcement learning (RL) and LSTM networks. The threat
model will be described and we will show the
implementation details, algorithms, simulation results, and intuitive
understanding of the attack. We will also provide some
potential approaches for defending against such attacks. Finally,
we will conclude with some directions for future research.
Adversarial attacks on neural networks have received
increasing attention after neural networks were found to
be vulnerable to adversarial perturbations
Trojan attacks have also been studied on neural networks
for classification problems. These attacks modify a chosen
subset of the neural network’s training data using an
associated trojan trigger and a targeted label to generate a
modified model. Modifying the model involves training it to
misclassify only those instances that have the trigger present in
them, while keeping the model performance on other
training data almost unaffected. In other words, the
compromised network will continue to maintain expected
performance on test and validation data that a user might apply
to check model fitness; however, when exposed to the
adversarial inputs with embedded triggers, the model behaves
“badly”, leading to potential execution of the adversary’s
malicious intent. Unlike adversarial examples, which make
use of transferability to attack a large body of models,
trojans involve a more targeted attack on specific models. Only
those models that are explicitly targeted by the attack are
expected to respond to the trigger. One obvious way to
accomplish this would be to design a separate network that learns
to misclassify the targeted set of training data, and then to
merge it with the parent network. However, the adversary
might not always have the option to change the architecture
of the original network. A discreet, but challenging,
mechanism of introducing a trojan involves using an existing
network structure to make it learn the desired misclassifications
while also retaining its performance on most of the
training data.
While existing research focuses on designing trojans for
neural network models, to the best of our knowledge, our
work is the first work that explores trojan attacks in the
context of sequential DM agents (including RL) as reported in
preprint
Deep RL has growing interest from military and defense
domains. Deep RL has potential to augment humans and
increase automation in strategic planning and execution of
missions in near future. Examples of RL approaches that
are being developed for planning includes logistics convoy
scheduling on a contested transportation network
In this section, we will provide a brief overview of Proximal Policy Optimization (PPO) and LSTM networks, which are relevant for the topic discussed in this work.
A Markov decision process (MDP) is defined by a tuple
(S; A; T ; r; ), where S is a finite set of states, A is a
finite set of actions. T : S A S ! R 0 is the
transition probability distribution, which represents the
probability distribution of next state st+1 given current state st
and action at. r : S A ! R is the reward function and
2 (0; 1) is the discount factor. An agent with optimal
policy should maximize expected cumulative reward defined
as G = E [Pt1=0 tr(st; at)], where is a trajectory of
states and actions. In this work, we use the proximal
policy optimization (PPO)
L( ) = Es;a min ( )A~; clip ( ); 1 ; 1 +
A~ ; where we define policy and ( ) = 0 as the current policy,
as the updated 0((aajjss)) . State s and action a is sampling from the current policy 0 , and A~ is the advantage estimation that is usually determined by discount factor , reward r(st; at) and value function for current policy 0 .
is a hyper-parameter determines the update scale. The clip operator will restrict the value outside of interval [1 ; 1+ ] to the interval edges. Through a sequence of interactions and update, the agent can discover an updated policy that improves the cumulative reward G.
Recurrent neural networks are instances of artificial neural
networks designed to find patterns in sequences such as text
or time-series data by capturing sequential dependencies
using a state. As a variation of recurrent neural networks,
update of the LSTM
tanh(Wcxt + Ucht 1 + bc); where xt is the input vector, it is the input gate, ft is the forget gate, ot is the output gate, ct is the cell state and ht is the hidden state. Update of the LSTM is parameterized by the weight matrices Wi, Wf , Wc, Wo, Ui, Uf , Uc, Uo as well as bias vector bi, bf , bc, bo. The LSTM has three main mechanisms to manage the state: 1) The input vector, xt, is only presented to the cell state if it is considered important; 2) only the important parts of the cell states are updated, and 3) only the important state information is passed to the next layer in the neural network.
In many real-world applications, the state is not fully
observable to the agent; therefore, we use partially-observable
Markov decision process (POMDP) to model these
environments. A POMDP can be described as a tuple
(S; A; T ; r; ; O; ), where S; A; T ; r and is the same as
MDP. is a finite set of observations, O : S A ! R 0
is the conditional observation probability distribution. To
effectively solve the POMDP problem using RL, the agent
needs to make use of the memory, which store information
of previous sequence of actions and observations, to make
decisions
In this section, we discuss overview of the technical approach and the threat model showing realizability of the attack. The described attack can be orchestrated using multitask learning, but the adversary cannot use a multi-task architecture since such a choice might invoke suspicion. Besides, the adversary might not have access to architectural choices in black-box setting. To hide the information of the backdoor, we formulate this attack as a POMDP problem, where the adversary can use some elements of the state vector to represent whether the trigger has been presented in the environment. Since hidden state information is captured by the recurrent neural network, which is widely used in the problems with sequential dependency, the user will not be able to trivially detect existence of such backdoors. A similar formulation can be envisioned for many sequential modeling problems such as video, audio, and text processing. Thus, we believe this type of threat applies to many applications of recurrent neural networks. Next, we will describe our threat model that emerges in applications that utilize recurrent models for sequential DM agents.
We consider two parties, one party is the user and other is the adversary. The user wishes to obtain an agent with policy usr, which can maximize the user’s cumulative reward Gusr, while the adversary’s objective is to build an agent with two (or possibly more) policies inside a single neural network without being noticed by the user. One of the stored policies is usr, which is a user-expected nominal policy. The other policy adv is designed by the adversary, and it maximizes the adversary’s cumulative reward Gadv. When the backdoor is not activated, the agent generates a sequence of actions based on the user-expected nominal policy usr, which maximizes the cumulative reward Gusr, but when the backdoor is activated, the hidden policy adv will be used to choose a sequence of actions, which maximizes the adversary’s cumulative reward Gadv. This threat can be realized in the following scenarios: • The adversary can share its trojan-infested model in a model-sharing marketplace. Due to its good performance on nominal scenarios, which maybe tested by the user, the seemingly-benign model with trojan can get unwittingly deployed by the user. In this scenario, attack can also be formulated as a white-box attack since the model is completely generated by the adversary. • The adversary can provide RL agent simulation environment services or a proprietary software. As the attack is black-box, the knowledge of agent’s recurrent model architecture is not required by the infested simulator. • Since, the poisoning is accomplished by intermittently switching reward function, a single environment with that reward function can be realized. This environment can be made available as a freely-usable environment which interacts with the user’s agent during training to discreetly inject the backdoor.
In previous research on backdoor attacks on neural
networks, the backdoor behavior is active only when a trigger
is present in the inputs
In this section, we will show training approach to inject the backdoor, illustrate results in grid-world experiments with limited observability, and provide intuition of mechanism of switching to backdoor policy in LSTM networks.
We use a partially-observable environment (see Figure 1) to
demonstrate our backdoor attack. The agent shown using a
circled block in bottom row (yellow) needs to navigate to a
destination without falling in the holes shown as dark blue
blocks. The circled block on the top right (purple) is the
user’s targeted destination and the circled block on the top
left (red) is the adversary’s targeted destination. Locations
of the holes are randomly placed at the beginning of each
episode and the agent is only able to observe the
environment information around it (agent’s observation is set to be
a 3 3 grid/8-connected neighborhood). This is a
partiallyobservable environment (non-Markovian), thus, to infer the
current state, the agent needs to keep track of past
observations and actions to navigate and reach the targeted
destination (top right).
In Figure 2, we show the path of agent with backdoor in
the same map twice and in one of the runs a trigger was
presented during navigation. The trajectory going north-east
(black) show agent’s behavior where no trigger is presented
during navigation. The agent successfully navigates to the
user targeted destination. Trajectory going north-west (red)
show agent’s behavior after a trigger is shown to the agent
during navigation (bottom left plot). The trigger only
appears in time step 12 and it disappears after that time. Thus,
before that time step, the agent uses the user expected policy
usr, and after that time step, the hidden policy adv induced
by the adversary is automatically activated.
We demonstrate a reward poisoning approach to inject the
backdoor. We define following notations: 1) normal
environment Envc, where rewards return from the environment
is rusr and the objective is to let the agent learn the user
desired policy usr. 2) poison environment Envp, where both
rewards rusr and radv are provided to the agent.
Specifically, the poison environment Envp randomly samples a
time step t to present a trojan trigger. Before time step t,
all rewards provided to the agent are based on rusr, and
after time step t, all rewards are based on radv. Training
process is described in Algorithm 1. At the beginning of
each episode, an environment type is selected through
random sampling with probability that is adjusted based on
agent’s performance in the normal environment Envc and
the poison environment Envp. Sampling function will take
an environment and a policy as inputs and output a
sequence of trajectory (o0; a0; r0; :::; oT ; aT ; rT ).
PolicyOptimization function uses proximal policy optimization
implemented in
RL agents usually learn in simulation environments before deployment. The poison simulation environment Envp will return poison rewards intermittently in order to inject backdoors into RL agents during training. Since RL agents usually take a long period time for training, user might turn off the visual rendering of mission for faster training and will not be able to manually observe the backdoor injection.
Require: Normal Environment Envc Require: Poison Environment Envp Require: Update Batch Size bs, Training Iterations Nt 1: Initialize: Policy Model 2: Initialize: Performance P Fc 0; P Fp 0 3: Initialize: Batch Count bt 0 4: Initialize: Set of Trajectories fg 5: for k 1 to Nt do 6: Env Envc 7: if random(0; 1) > 0:5 + (P Fp P Fc) then 8: Env Envp 9: end if 10: // Sampling a trajectory using policy 11: k Sampling( ; Env) 12: S k, bt bt + 1 13: // Update policy when k k bs 14: if bt > bs then 15: // Update parameter based on past trajectories 16: PolicyOptimization( ; ) 17: // Evaluate performance in two environments 18: P Fc; P Ft Evaluate(Envc; Envp; ) 19: P Fc; P Ft Normalize(P Fc; P Ft) 20: fg; bt 0 21: end if 22: end for 23: return
To inject a backdoor into a grid world navigation agent, we let the agent interact in several grid configurations, which range from simple ones to complex ones. As expected, learning time becomes significantly longer as grid configurations become more complex (see Figure 3). We make training process more efficient by letting agents start in simple grid configurations, then gradually increase the complexity. Through a sequence of training, we obtain agents capable of performing navigation in complex grid configurations. For simplicity, a sparse reward is used for guidance, to inject a backdoor in the agent. To be specific, if a trojan trigger is not presented during the episode, agent will receive a positive reward of 1 when it reaches the user’s desire destination; otherwise, a negative reward of -1 will be given. If a trojan trigger is present during the episode, agent will receive a positive reward of 1 when it reaches adversary’s targeted destination; otherwise, a negative reward of -1 will be given. We train agents with different network architectures and successfully injected backdoors in most of them. According to our observations, backdoor agents take longer time to learn, but final performance of the backdoor agents and the normal agents are comparable. Also, difficulty of injecting a backdoor into an agent also related to capacity of the agent’s policy network.
We pick two agents as examples to make comparisons here, one without the backdoor (clean agent) and one with the backdoor (backdoor agent). Both agents have the same network architecture (2-layer LSTM) which is implemented using TensorFlow. First layer has 64 LSTM units and the second layer has 32 LSTM units. Learning environments are grids of size 17 17 with 30 holes. Agent without the backdoor only learns in the normal environment while the backdoor agent learns in both normal and poison environments. After training, we evaluate their performances under different environment configurations. We define success rate as percentage of times the agent navigates to the correct destinations over 1000 trials. For the training configuration (17 17 grid with 30 holes) without presence of the trigger, success rate of the backdoor agent is 94.8% and success rate of the clean agent is 96.3%. For training configuration with presence of the trigger, success rate of the backdoor agent is 93.4%. Median of the clean agent’s performance on other clean grid configurations is 99.4%. Median of the backdoor agent’s performance on other clean grid configurations is 95.0%. Median of the backdoor agent’s performance on other poison grid configurations is 92.9%. Even though performance of the backdoor agent is lower than the clean agent, the difference in performance is not significant.
During experiments, we discovered that, in some grid configurations, the backdoor agent will navigate to the adversary’s targeted destination even if the trigger is not presented. Our current conjecture about the cause of this unintentional backdoor activation phenomenon is related to the input and forgetting mechanism of the LSTM. Overall, there seems to be a trade-off related to sensitivity and unintentional activation of the backdoor, which needs to be appropriately optimized by the adversary.
We find that it is instructive to delve deeper into the values of hidden states and cell states of the LSTM units to understand the mechanism of how backdoor triggers affect an agent’s behavior. We use the same models selected in the previous part and analyze their state responses with respect to the trigger. Environments are set to be 27 27 with 100 holes. For the same grid configuration, we let each agent run twice. In the first run, trigger is not presented and the backdoor agent will navigate to the user’s targeted location. In the second run, the trigger appears at time step 12 (fixed for ablation study of cell states and hidden states), and the backdoor agent will navigate to the adversary’s targeted location. We let the clean agent and the backdoor agent run in both environments for 350 times (with and without presence of the trigger), and in each trial, the locations of holes are randomly replaced. We plot all the cell states and hidden states over all the collected trajectories, and observed three types of response: (1) Impulse response: Cell states ct and hidden states ht react significantly to the trigger in a short period of time and then return back to a normal range. (2) No response: Cell states ct and hidden states ht do not react significantly to the trigger. (3) Step response: Cell states ct and hidden states ht deviate from a normal range for a long period of time. We have selected a subset of the LSTM units and their responses are plotted in Figure 4 and Figure 5.
In the current experiments, we observe that both the clean
agent and the backdoor agent has cell states and hidden
states which react significantly (type 1) and mildly (type
2) to the trojan trigger; however, only the backdoor agent
has some cell states and hidden states deviate from a normal
range for a long period of time (type 3). We conjecture that
the type 3 response keeps track of the long-term dependency
of the trojan trigger. We conducted some analyses through
manually changing values of some cell states ct or hidden
states ht with the type 3 response when the backdoor agent
is navigating. It turns out changing the values of these
hidden/cell states does not affect the agent’s navigation ability
(avoiding holes), but it does affect the agent’s final objective.
In other words, we verified that altering certain hidden/cell
states in LSTM network changes the goal from the user’s
targeted destination to the adversary’s targeted destination or
vice versa. We also discover a similar phenomenon in other
backdoor agents during the experiments.
Under defense mechanisms against trojan attacks,
During our analysis on sequential DM agents, we discovered that LSTM units are likely to store long-term dependency in certain cell units. Through manually changing value of some cells, we were able to switch agent’s policies between user desired policy usr and adversary desired policy adv and vice versa. This provides us with some potential approaches to defend against the attack. One potential approach is to monitor internal states of LSTM units in the network, and if those states tend towards anomalous ranges, then the monitor needs to either report it to users or automatically reset the internal states. This type of protection can be run online. We performed an initial study of this type of protection through visualization of hidden states and cell states values. We used a backdoor agent and recorded value of hidden states and cell states over different normal environments and poisoned environments. Mean values of the cell state vectors and hidden state vectors for normal behavior and poisoned behavior are calculated respectively. In the end, we applied a t-SNE on the mean vectors from different trials. Detailed results are shown in Figure 6. From the figure, we discover that hidden state vectors and cell state vectors are quite different over normal behaviors and poisoned behaviors; thus, monitoring the internal states online and perform anomaly detection should provide some hints for the attack prevention. In this situation, the monitor will play a role similar to immune system, where if an agent is affected by the trigger, then the monitor detects and neutralizes the attack. Although we did not observe the type 3 response in clean agents in current experiments, we anticipate that some peculiar grid arrangements will require the type 3 response in clean agents too, e.g. if agent has to take a long U-turn when it gets stuck. Thus, presence of the type 3 response will not be a sufficient indicator to detect backdoor agents. An alternate static analysis approach could be to analyze the distribution of the parameters inside LSTM. Compared with the clean agents, the backdoor agents seem to use more cell units to store information. This might be reflected in the distribution of the parameters. However, more work is needed to address detection and instill resilience against such strong attacks.
Multiple challenges exist that require further research. From
the adversary’s perspective, merging multiple policies into
a single neural network model is hard due to catastrophic
forgetting in neural networks
From the defender’s perspective, it is hard to detect
existence of the backdoor before a model is deployed. Neural
networks by virtue of being black-box models prevent the
user from fully characterizing what information is stored in
a neural network. It is also difficult to track when the trigger
appears in the environment (e.g. a yellow sticky note on a
Stop sign from
We exposed a new threat type for the LSTM networks and sequential DM agents in this paper. Specifically, we showed that a maliciously-trained LSTM network-based RL agent could have reasonable performance in a normal environment, but in the presence of a trigger, the network can be made to completely switch its behavior and persist even after the trigger is removed. Some empirical evidence and intuitive understanding of the phenomena was also discussed. We also proposed some potential defense methods to counter this category of attacks and discussed avenues for future research. We hope that our work will inform the community to be aware of this type of threat and will inspire to together have better understanding in defending against and deterring these attacks.