The growing trend of fledgling reinforcement learning systems making their way into real-world applications has been accompanied by growing concerns for their safety and robustness. In recent years, a variety of approaches have been put forward to address the challenges of safety-aware reinforcement learning; however, these methods often either require a handcrafted model of the environment to be provided beforehand, or that the environment is relatively simple and low-dimensional. We present a novel approach to safetyaware deep reinforcement learning in high-dimensional environments called latent shielding. Latent shielding leverages internal representations of the environment learnt by modelbased agents to “imagine” future trajectories and avoid those deemed unsafe. We experimentally demonstrate that this approach leads to improved adherence to formally-defined safety specifications.
The steady trickle of reinforcement learning (RL) systems making their way out of the lab and into the real world has cast a spotlight on the safety and robustness of RL agents. The motivation behind this should be relatively easy to grasp: when training an agent in real-world settings, it is desirable that some states are never reached as they could, for instance, cause permanent damage to the hardware the agent is controlling. We can thus informally define the notion of safety-aware RL in terms of the classical RL setup with the added requirement that the number of unsafe states visited be minimised. Under this definition, however, it has been found that many state-of-the-art RL algorithms unnecessarily enter unsafe states despite safe alternatives being available and there being a positive correlation between avoiding such states and reward (Giacobbe et al. 2021).
The field of safety-aware RL encompasses a multitude
of approaches ranging from constrained policy optimisation
In this work, we propose a safe RL agent that makes uses of latent shielding, an approach to shielding in environments where a formally-specified dynamics model is not available in advance. At an intuitive level, the agent uses a data-driven approach to learn its own latent world model (a component of which is a dynamics model) which is then leveraged by a shield. The shield then uses the agent’s model to “imagine” trajectories arising from different actions, forcing the agent to avoid those it foresees leading to unsafe states. In addition, the agent can be trained within its own latent world model thus reducing the number of safety violations seen during training.
Contributions The main contribution of this work is a framework for shielding agents in complex, stochastic and high-dimensional environments without knowledge of environmental dynamics a priori. We further introduce a new method to aid exploration when training shielded agents. Though our framework loses the formal safety guarantees associated with traditional symbolic shielding approaches, our experiments illustrate that latent shielding reduces unsafe behaviour during training and achieves testing performance comparable to previous symbolic approaches.
In this section, we cover some relevant background topics. We begin by introducing our problem setup for safety-aware RL and give an overview of the specification language used in this work. This is followed by an outline of the latent world model we make use of in this work as well as a discussion on shielding.
We consider an agent interacting with an environment E modelled as a partially observable Markov decision process (POMDP) with states s ∈ SE , observations ot ∈ OE , agent-generated actions at ∈ AE and scalar rewards rt ∈ R over discrete time steps t ∈ [0, 1, ..., T − 1]. We assume the environment has been augmented with a labelling function uLsϕE w:hSeEth→er {asvaifoel a,tuionnsahfaes}otchcaut,rraetdeawchithti mreespsetecpt,tionfsoormmes formal safety specification ϕ . For the avoidance of doubt, we define a violation to have occurred whenever ϕ does not hold. This is a weaker assumption than previous works in shielding (which assume access to an abstraction of the environment) and can be thought as a secondary safety-focused reward function with a binary output. Intuitively, the goal of the agent is to learn a policy π that maximises its expected cumulative reward while minimising the number of violations of ϕ .
In this work, we use syntactically co-safe Linear Temporal
Logic (scLTL)
ϕ ::= true | d | ¬d | ϕ ∨ ϕ | ϕ ∧ ϕ | ⃝ ϕ | ϕ ∪ ϕ | ⋄ ϕ (1) where d ∈ AP , ¬ (negation), ∨ (disjunction), ∧ (conjunction) are the familiar operators from propositional logic, and ⃝ (next), ∪ (until) and ⋄ (eventually) are temporal operators.
The main rationale behind our choice of specification
language is the fact that we can efficiently monitor a
system’s adherence to an scLTL specification using a technique
known as progression
E
We refer to the predictive model of an environment
maintained by a model-based agent as its world model. World
models can be learnt from experience and be used both as
a substitute for the environment during training
An RSSM is composed of three key components: a latent dynamics model, a reward model, and an observation model. These components act on compact states formed from the concatenation of a deterministic latent state ht and stochastic latent state zt.
Latent Dynamics Model The latent dynamics model is made up of a number of smaller models. First, the recurrent model ht = f (ht− 1, zt− 1, at− 1) is used to compute the deterministic latent state based on the previous compact state and action. From ht and the current observation ot, a distribution q(zt|ht, ot) over posterior stochastic latent states zt is computed by the representation model. At the same time, a distribution p(zˆt|ht) over prior stochastic latent states zˆt is computed by the transition model, based only on ht. During training, the transition model attempts to minimise the Kullback Leibler (KL) divergence between the prior and posterior stochastic latent state distributions. In doing this, the RSSM learns to predict future latent states (using the recurrent and transition models) without access to future observations.
Observation Model The observation model computes the distribution p(oˆt|ht, zt) over observations oˆt for a particular state. Though not strictly needed, the observation model can prove useful for visualising predicted future states and providing a richer training signal.
Reward Model The reward model computes the distribution p(rˆt|ht, zt) over rewards rˆt for a particular state.
In practice, the distributions p and q are implemented with
neural networks pθ and qθ respectively, parameterised by
some set of parameters θ . These latent dynamics models
deifne a fully-observable Markov decision process (MDP) as
the latent states in the agent’s own internal model can
always be observed by the agent
The classical formulation of shielding in RL is given by
Proposed by (Giacobbe et al. 2021), bounded prescience shielding (BPS) avoids the need for hand-crafted abstractions by exploiting the fact that some agents are trained in computer simulations. The shield operates by leveraging access to the program underlying the simulation to look ahead into future states within some finite horizon. Using BPS over classical shielding does, however, come with a few disadvantages. Firstly, it requires access to the simulation at run-time which may prove difficult to provide (especially in cases where running the simulation is computationally expensive). Moreover, an agent using BPS, even when starting from a safe state, can find itself entering unsafe states in cases where the number of steps between a violation being caused by an action and the violation state itself exceeds the shield’s look-ahead horizon. This is not the case for classical shielding which resembles BPS with an infinite horizon.
The notion of safety used by BPS is defined over MDPs. For an arbitrary MDP with states S and actions A, a bounded trajectory ρ of length H is a sequence of states and actions a0 a1 an− 1 s0 →− s1 →− . . →.−− − sn comprised of no more than H states and with the final state sn either being a terminal state or n = H − 1. We further denote the set of all finite trajectories starting from some arbitrary state s ∈ S by ϱ(s) and the set of all bounded trajectories of length H that start from s by ϱH (s).
We say a bounded trajectory ρ of length H satisfies H bounded safety with respect to safety specification ϕ , written SH (ρ, ϕ ), if and only if for all si ∈ ρ , LϕE (si) = saf e. Moreover, we can extend the notion of H -bounded safety over the set of policies: a policy π is H -bounded safe with respect to ϕ , denoted as SH (π, ϕ ), if and only if for all s ∈ S , • either there exists some ρ ∈ ϱH (s) such that SH (ρ, ϕ ) and π (s0) = a0; • or for all ρ ∈ ϱH (s), ¬SH (ρ, ϕ ).
In other words, the policy will choose a safe trajectory as long as one exists. Finally, we formally define a violation of ϕ to be inevitable in state s0 ∈ S if and only if for all ρ ∈ ϱ(s0), ¬SH (ρ, ϕ ).
We introduce the notion of latent shielding, a novel class
of shielding approaches that replace abstractions used by
shields with learned latent dynamics models thus allowing
the enforcement of formal safety specifications while
avoiding the need for an explicitly-defined abstraction of the
environment. We further introduce the first such approach,
approximate bounded prescience shielding, a framework for
latent shielding that leverages latent world models learnt
by model-based deep RL (DRL) agents. In this work, our
model-based agent of choice is Dreamer1
We augment the standard RSSM with a labelling function LϕI : SI → {saf e, unsaf e} which maps latent states st ∈ SI to whether they correspond to states in violation of ϕ . As with the other models, Lϕ is implemented with a neural network with a categorical oIutput lθ also parameterised by θ . This yields an enhanced RSSM (illustrated in Figure 1) which we will refer to as a safety RSSM (SRSSM).
We train lθ along with the other components of the
SRSSM with the objective
mθ in Lmodel = Lobservation + Lreward + LKL + Lviolation
(2)
where the first three terms are as described in
We now integrate the SRSSM as part of a latent shielding approach which we shall refer to as approximate bounded prescience shielding (ABPS). Though ABPS is inspired by BPS, it differs in two key aspects: (1) we approximate the labelling function Lϕ and environmental dynamics using an
E SRSSM; and (2) we sample a fixed number of potential future trajectories as opposed to exhaustively exploring all possibilities.
Thus, our approach can be thought of as an approximation of some “ideal” bounded prescience shield that uses the true environmental dynamics and labelling function. The advantage of the first difference should be obvious: it enables the shield to learn its own abstraction, removing the need for hand-crafting or access to a digital environment’s underlying program. Why the second difference is advantageous may be slightly less obvious - it’s a heuristic that allows us to increase the horizon H . By directing the sampling of trajectories in accordance with states and actions the policy is biased towards (as opposed to uniformly), it may be possible to achieve comparable performance to a standard BPS in foreseeing unsafe states. The intuition behind this is that by not sampling trajectories the agent is unlikely to take, more of our computational budget can be dedicated into ensuring that the agent’s most likely trajectories are safe. In other words, we will not spend time planning to correct actions that the agent is unlikely to take.
More formally, our shielded policy π ∗ can be written: π ∗ (st) = π alt(st), otherwise π (st), if P (lθ (st+1) = 1|π (st)) < ϵ (3) where π (st) is the agent’s policy without shielding; st ∈ SI is some compact latent state; and π alt is an alternative policy that ensures that, if a violation isn’t already inevitable, the Algorithm 1: Approximate Bounded Prescience Shielding in latent space.
Input: Current compact latent state (h, z), unshielded action a, horizon H, number of trajectories to sample N , random noise η , alternative policy π alt and threshold ϵ .
Output: A tuple containing an action and whether or not the shield had to interfere. 1 Initialise array of zeros Λ with length N . 2 for n = 1..N do /* Sample a trajectory and check if it is unsafe. */ 3 Imagine trajectory
ρ = (h, z→)− a (hˆ1, zˆ1) →−a1 ..→.−−− aH− 1 (hˆH , zˆH ) 4 where ai = π (hˆi, zˆ ) + η .
i if ∃i ∈ [1, 2, ..., H ] such that lθ (hi, zi) = unsaf e then Λ[ n] := 1. 8 9 end 10 return ⟨a,interfered⟩. 5 end /* Interfere if probability of
violation exceeds ϵ . 6 interf ered := f alse. 7 if N1 Pλ ∈Λ λ > ϵ then
a, interf ered := π alt(hi, zi), true.
agent avoids a predicted unsafe state. Though more complex
candidates for π alt (such as selecting the unshielded policy’s
highest-ranked safe action
Since the SRSSM models stochasticity, we can sample multiple futures arising from an action being taken and derive probabilistic estimates of whether an action will lead to a violation. In this way, we can estimate the safety of an action taken in a given state by checking whether the probability of a violation occurring (inferred by sampling) exceeds a fixed threshold ϵ (see Algorithm 1). Moreover, the sampling process can, in practice, be augmented to sample a wider range of trajectories (less likely to be taken by the agent) by adding a small amount of noise η ∼ N (0, σ 2) to actions suggested by the policy during sampling. Data Collection In this phase, the agent interacts with the real environment to collect a dataset D ⊆ S × A × O × R × { 0, 1} of states, actions, observations, rewards and violations with which a latent world model can be learned. At the very start of training, we collect trajectories from S seed episodes using a random policy; at all other times, we use the agent’s shielded policy.
to improve the model of the world so that the policy has an accurate imagined environment to train in. To this end, we train the latent world model with respect to the objective in Equation 2 on B data sequences of length L sampled from D.
Agent Training Agent training is composed of two steps.
First, starting from states from the same B data sequences
from the latent world model training phase, the agent
imagines I trajectories of length H with actions chosen from its
current policy. Next, the unshielded policy π is updated in
an actor-critic fashion as in
cuss the key changes we have made that differentiate our agent from previous approaches.
Experience Dataset. Elements in the experience dataset D now contain a binary variable representing whether a violation has occurred λ t.
Latent Shielding. Before being sent to the environment E , actions at generated by the unshielded policy π are routed through our newly-proposed latent shield (described in Algorithm 1). The shield returns a new approximately Hbounded safe action a′t.
Intrinsic Punishment. Whenever a violation occurs
(whether detected by the latent shield or by the
environment), we override the environment’s reward function by
instead assigning a reward p < 0. This discourages the agent’s
unshielded policy π from taking actions that lead to unsafe
states through the standard RL setup. Over time, this means
that the shield will have to interfere less as π becomes biased
away from unsafe states. This is a standard practice in the
shielding literature
notion of a shield introduction schedule Π which can enable and disable shielding during training. The rationale behind this is that a shield backed by an inaccurate world model will incorrectly label some safe states as unsafe, and vice versa. In some cases, this may prove detrimental to the training process: suppose lθ happens to be initialised in such a way that all states are labelled as unsafe. As a result, the shield can prevent the agent from exploring the environment and improving its internal model of the environment. This, in turn, can prevent the labelling function from learning to correctly differentiate safe states from unsafe states.
To give the agent time to learn a good world model before restricting exploration through a learned shield, we augment the training procedure with Π which aims to gradually introduce shielding. To our knowledge, this is the first time such a system has been proposed. Though there exist a wide */
We extend the training regime proposed in
Overview The training procedure can be split into three phases: data collection, latent world model training and agent training (lines 10-22, 4-6 and 7-8 in Algorithm 2 respectively). These phases are cycled through until convergence. range of possible implementations of Π , in this work we use a simple schedule that seems to work well empirically: start training with shielding disabled and enable shielding once the world model loss (in particular, the violation loss) begins to plateau. A detailed comparison of different shield introduction schedules, however, is beyond the scope of this study and left for further work. 23 end end Add experience to dataset
T
D := D ∪ {(ot, at, rt, λ t)t=1}.
Algorithm 2: Training Dreamer with Approximate Bounded Prescience Shielding.
Input: Punishment for violation rpunish, shield imagination horizon H, number of steps to imagine ahead for training I, number of seed episodes S, number of training steps per episode C, number of real environment interaction steps per episode T , sequence length L, batch size B, environment E , labelling function Lϕ and shield introduction schedule Π . E 1 Initialise dataset D with S seed episodes and neural network parameters θ randomly. 2 while not converged do /* Learning the internal world
model and agent policy. 3 for c = 1..C do 4 Draw B data sequences {(at, ot, rt, λ t)tk=+kL} ∼ D .
Compute model states ht, zt and zˆt.
Update θ using representation learning.
Imagine a trajectory ρ of length I for each state in each data sequence.
Update the unshielded policy π based on the imagined trajectories. end /* Collecting data from the
environment. for t = 1..T do
Compute model states ht, zt from history.
Select action at with the unshielded policy, adding exploration noise if desired. if Π has enabled shielding then
Pass ht, zt and at into Algorithm 1 to obtain the tuple ⟨a′t, interf ered⟩. */ */ else end rt, ot := E .step (a′t).
a′t, interf ered := at, f alse.
Check if Lϕ has detected a violation and store
E the result as 0 or 1 in λ t.
if λ t = 1 or interf ered then rt := rpunish.
In this section, we compare our ABPS agent against
Dreamer without shielding
Visual Grid World The Visual Grid World (VGW) environment is a simple deterministic navigation benchmark with high-dimensional (64 × 64 × 3) visual observations (see Figure 2) and discrete actions (up, down, left, right and staying still). The agent’s (in green) task is to navigate to randomly-placed targets (in black) while avoiding any unsafe locations (in red). This yields a relatively straightforward safety specification:
¬agent in red square ∪ episode ended where agent in red square is true if and only if the agent is in an unsafe location, and episode ended is only true at the end of an episode.
The reward function used is also quite simple, with a small penalty term at each time step to encourage movement (though indeed different forms of intrinsic motivation may be also used): 100, agent reaches target, − 40, agent enters unsafe state, − 10, agent does not move, − 1, otherwise.
We experiment with both fixed and procedurally generated grids over episodes consisting of 500 steps. Cliff Driver The Cliff Driver (CD) environment is a symbolic benchmark with stochastic dynamics and continuous actions. The agent controls the forward acceleration of a car and is tasked with driving to the edge of a cliff as quickly as possible without overshooting and falling into the sea. The car exists on a one-dimensional road with actions at ∈ [− 1, 1] which correspond to the acceleration of the car. The car cannot move backwards and its speed vt is lower-bounded at 0 (thus at < 0 corresponds to braking as opposed to reversing). The agent starts each episode stationary at a fixed distance x0 from the edge of the cliff with its distance at subsequent time steps being written xt. Stochasticity comes from the fact that at each time step there is a probability pstick that the car’s controls get “stuck” meaning that the current action is ignored and replaced with the previous action.
Observations from the environment are given as twodimensional vectors encoding the distance from the edge of the cliff in one component and the speed of the car in the other. The safety specification can thus be written: ¬agent fallen off cliff ∪ episode ended where agent fallen off cliff is true if and only if the agent has overshot the cliff, and episode ended is only true at the end of an episode. (1 − xx0t , agent has not fallen off cliff, − 5,
otherwise.
We experiment with pstick = 0.1 and pstick = 0.5 on roads 10 units long over episodes consisting of 20 steps.
We evaluate our agents on three metrics: (1) average reward per episode at test-time; (2) average number of violations per episode at test-time; and (3) total number of violations during training. Results are calculated by averaging over vfie versions of each agent trained from different seeds. Training Details Experiments were carried out on a machine with a single NVIDIA RTX 2080 Ti GPU, an AMD Ryzen 7 2700X processor and 64GB of RAM. For each environment, the model-based agents were trained for the same number of steps; the model-free CPO agents were allowed to train for longer (2× longer for the VGW environment and 5× longer for the CD environments). The latent shield horizon H was 2 and 6 for the VGW and CD environments respectively.
We used the network architectures proposed in
Shield introduction schedules were kept relatively simple. For the VGW environment, the agent started with shielding disabled. After 10 episodes (including 5 seed episodes), shielding was enabled every third episode. After 20 episodes, shielding was enabled every other episode. Shielding was fully enabled after 30 episodes. In addition, the unsafe threshold ϵ was decayed linearly from 0.5 to 0.125 over the course of training. For the CD environment, shielding was enabled after 60 episodes (including 50 seed episodes).
Results As can be seen in Table 1, our agent collected more reward at test-time than both the unshielded and BPS agents in the VGW environment. Moreover, our agent saw a seven-fold reduction in test-time violations compared to the unshielded agent across both the fixed and procedurally generated environments, averaging at less than one violation per episode for the fixed VGW. This reduction in training violations is even more dramatic when compared to the model-free CPO agent. Our agent outperformed the other agents at test-time in the most stochastic CD environment (pstick = 0.5). This is possibly due to the SRSSM used by our latent shield being better able to capture nondeterminism than BPS. Nevertheless, the result is still rather impressive given that the agent using BPS sampled 1024 trajectories at every time step whereas our agent only sampled 20. Plots of training reward and violations can be seen in Figure 3.
We evaluate the quality of each agent’s latent dynamics model by observing its trajectory predictions given a particular starting state and action sequence. Each agent was given the same starting observation and predefined sequence of 10 actions. The agents took these actions in their respective latent world models with the states traversed being decoded for inspection. Results can be seen in Figure 2 and indicate that, qualitatively, our agent’s decoder performed the best, accurately predicting 10 frames into the future. One possible reason for this observation is that the inclusion of the violation detection objective in the SRSSM encourages the model to focus on accurately capturing violation states. Another potential factor is that the BPS agent never actually enters unsafe states and thus finds it difficult to represent them.
Visual Grid World
Cliff Driver
Flavour
Fixed Procedural pstick = 0.1 pstick = 0.5
Metric Testing Reward Testing Violations Training Violations
Testing Reward Testing Violations Training Violations
Testing Reward Testing Violations Training Violations
Testing Reward Testing Violations Training Violations
Latent We compare the first 100 training episodes of our agent in the fixed VGW environment with and without a shield introduction schedule. As with the performance evaluation, results are calculated over vfie trained agents and a plot of training reward can be seen in Figure 4. Though both agents start at roughly the same reward, the agents with a shield introduction schedule consistently outperforms their counterparts without shield introduction schedules.
Latent World Models Learning latent world models from
visual observations has seen growing interest from the RL
community
with safe ones has been a popular approach to safety-aware
RL. Introduced into the RL scene by
In contrast, latent shielding tackles the problem head-on
by directly learning an abstraction for use by the shield.
In this work, the abstraction we use is an SRSSM, which
captures stochasticity by design, a useful property for
nondeterministic environments. Though our latent shield
satisifes an approximation of H-bounded safety with respect to
the learned abstraction, its safety with respect to the true
environmental dynamics is not guaranteed and instead relies
on the fidelity with which the true dynamics are captured.
Furthermore, unlike for its formally-verified predecessors,
it is a necessary sacrifice that the agent visits unsafe states
during training in order to learn a notion of safety (unless, of
course, pre-training is possible). Nevertheless, a learned
abstraction may be advantageous in settings where
handcrafting an abstraction is not feasible and privileged access to
some simulation (as in (Giacobbe et al. 2021)) cannot be
assumed. Moreover, latent shielding may provide a greater
degree of explainability over model-free methods which get rid
of the abstraction altogether
In this paper, we have presented latent shielding, a new framework for shielding DRL agents without the need for a handcrafted abstraction of the environment. Using this framework, we have designed a novel shield and demonstrated that this method not only leads to improved adherence to safety specifications on two benchmark environments with respect to an unshielded agent, but also works out-of-the-box on both continuous and discrete environments (unlike its predecessor, BPS). Furthermore, we have demonstrated for the first time that shielding at inappropriate times may adversely impact the performance of modelbased DRL agents and showed how this phenomenon can be counteracted using our novel notion of shield introduction schedules.
Our work opens several exciting avenues for future work. For instance, this work uses a very simple shield introduction schedule; future work may provide a richer investigation into the properties of different schedules. Moreover, though demonstrating promising empirical results, our realisation of latent shielding loses the formally verifiable safety guarantees enjoyed by many symbolic shielding approaches - whether it is possible to construct a verifiable implementation of latent shielding, or compensate for the loss of formal guarantees, are open problems.
The authors would like to thank Claudio Elgueta Karstegl for turning our GPU machine off and on again during the national lockdowns.