Safe Reinforcement Learning (Safe RL) aims to produce constrained policies with constraints typically motivated by issues of physical safety. This paper considers the issues that arise from regulatory constraints or issues of legal safety. Without guarantees of safety, autonomous systems or agents (A-bots) trained through RL are expensive or dangerous to train and deploy. Many potential applications for RL involve acting in regulated environments and here existing research is thin. Regulations impose behavioural restrictions which can be more complex than those engendered by considerations of physical safety. They are often inter-temporal, require planning on behalf of the learner and involve concepts of causality and intent. By examining the typical types of laws present in a regulated arena, this paper identifies design features that the RL learning process should possess in order to ensure that it is able to generate legally safe or compliant policies.
In this position paper I will consider the problem of learning a solution to a sequential decision making problem in an environment governed by some laws via Reinforcement learning (RL). I will assume that the learned policy should not break these laws because doing so would impose sanctions by the environment’s regulator or law enforcer. By presenting a taxonomy of laws which exist in real life, whose features are relevant to RL, I am able to make some inferences about the general design of a RL process that can produce legal policies.
RL can produce novel policies to solve sequential
decision problems. Its potential has been demonstrated in the
super-human mastery of Go
RL requires an environment which allows ample exploration and feedback. In game applications such as Go and
Supported by the EPSRC Copyright c 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). computer games the training environment is the same as the deployment environment and training is costless (excepting the ecological impact of the vast computational power that is often used). Potential real world applications of RL are often more complex, almost certainly regulated, and the cost of mistakes made in training or deployment could be catastrophic. In such applications where safety, cost or legality are issue, one approach is to conduct learning in a simulator of the environment where the cost of bad policies is negligible. The use of any simulator raises the risk of misspecification and poor generalisation on deployment. An alternative to using a simulator is placing the RL agent very carefully in the target environment with a human overseer ready to take over in tricky spots. This approach has limitations according to the complexity of the task (Saunders et al. 2018). It might not be feasible to use this approach in applications like trading because the speed of decision making is beyond the ability of a human overseer to monitor.
Whether learning takes place in a simulator or carefully in the target arena, the ability to generate legal policies with high probability is highly desirable if the policy is to be deployed in a regulated setting. Laws can present different challenges to other types of constraint. A legally transgressive policy might not be obvious in the way a physically transgressive one might be. The nature of laws will dictate the methods of RL used to generate optimal, legal policies.
Markov Decision Processes (MDPs) are a common framework underpinning RL. In this formulation time is discretised and labelled t = 1; 2; 3; : : : . A MDP is described by a tuple (S; S0; A; T ; R; ) where: 1. S is the set of states in the environment. 2. s0 is a distribution over the initial states of the environment p(s) for s 2 S . 3. A is the set of all actions available. 4. T (s; a; s0) = P(s0js; a) is the transition probability distribution; the probability of transitioning to state s0 when in state s 2 S and choosing action a 2 A. 5. R : S A ! R is the reward function, the feedback mechanism through which learning is possible.
2 (0; 1] the discount factor to differentiate the value of rewards now vs those received in the future. In finite horizon cases = 1 and can be ignored.
The learner then has the objective of funding a policy function from the set of all policy functions : S ! A which solves the maximisation of the expected discounted sum of rewards: = arg max E X tR(st; at)j
2 t=0
The policy function is often a probability distribution over actions (ajs) = P(ajs) 8a 2 A, s 2 S
The Markovian property of this process comes from the transition function. It is satisfied if the probability of transition to a new state is determined only by the current state and chosen action.
An extension of the MDP is the Partially Observable MDP (POMDP). This covers the very probable contingency where the full state of the world is not visible to the decision maker. It is described by the tuple (S; S0; A; T ; R; ; ; O). The two additions to the tuple are as follows:
is the set of all observations that the learner can receive. For convenience we assume that it includes the reward rt received in any time period. 8. O = P(!js0; a) is the probability distribution of receiving observation ! 2 after transition to state s0 and action a was chosen.
The domain of the policy function then becomes the
history of all observations and actions which we write (ajht)
where ht is short hand for (o1; a1; o2; a2; : : : at 1; ot).
Consequently the complexity of solving a POMDP is much
higher than that of a MDP
This paper will show that the legality of behaviour can
depend on establishing the causal effects of an action. The
definition of causality is a complicated topic and there is a
distinction between predictive causality which refers to
predicting the effect of actions and actual causality which refers
to evidential analysis after actions have been taken.
Structural Causal Models (SCMS)
RL has been used infer the intent of others
In this section I present a number of law-types which are likely to exist in a regulated environment. I differentiate between states and actions. Actions are assumed to be originated by the learner only and their commission is voluntary. States refer to some measurable property of the environment, stable for the duration of the time period. Actions cause a measurable change in the environment but I assume that their duration is instantaneous so there is no state that records an action in progress.
This the simplest type of law and the one which Safe RL research has concentrated on as many physical safety constraints are of this type. Examples might include ’drive below 30 miles per hour in urban environments’ or ’do not fly a drone in the vicinity of the airport’. For a state restriction to be a law, its realisation should avoidable by the learner through its actions. This is the case for the speed restriction example. Even though I have described these laws as simple, they might be context dependent. Roads with a solid central line generally do not allow overtaking, but the presence of a stationery vehicle blocking progress might allow it, providing it is safe to do so.
Some states exist which could be both caused through the learner’s actions and through an external mechanism. This marks the first departure from conventional safe-RL research because the safety constraints traditionally considered do not differentiate between those states caused by the learner and those that are not. It does not matter whether the drone caused itself to fly over the volcano or whether it was blown by the wind, the state of being located over the volcano is the one to be avoided. For legal restrictions, certain states might only be restricted if they were caused by the learner1. A concrete example of such a causal dependent state restriction can be found within financial markets where the UK’s financial regulator prohibits trading algorithms from creating or contributing to disorderly market conditions. Such conditions could arise independently of the behaviour of the learner, if the learner has no mechanism of determining whether this the case, learning efficiency will be compromised. For caused state restriction laws to be broken two conditions should be satisfied. Firstly a restricted state s^ should occur and secondly that the actions of the learner foreseeably caused the state to occur.
Laws exist in a variety of settings which are restrictions on conduct with no necessary requirement for a lasting change in the state of the world. Examples in the UK include the offence of Careless Driving and more seriously Dangerous driving. There is no causation requirement since I assume that the A-bot has the freedom to choose its own actions at any time2 Action sequence laws could be transformed into a simple state restriction law by adding a state variable that indicates whether a restricted action sequence has occurred. Such an approach might not be efficient if a large number of conduct laws exist in the environment as it would cause the dimension of the state space to grow.
Laws exist which combine a sequence of actions that cause restricted state(s). Continuing the driving examples from the previous section, in the UK there exist statutory offences of causing death by careless or reckless driving. These laws constitute a restriction on how certain states are arrived at.
Inchoate offences are restrictions on action sequences and
states which may lead to restricted states which are not
necessarily realised. Examples in the UK include attempt crimes
such as attempted murder or possession of drugs with intent
1Death is not generally prohibited, but causing it generally is!
2Situations where there is no action which won’t break a law are
analogous to the concept of deadlock in model checking
Common law as practised in UK, USA, India and Canada amongst others requires that the accused had mens rea (the mental element of intent to commit a crime) for certain criminal offences to have been committed by them. Different levels of intent exist, ranging from direct intent where the accused deliberately caused and wanted to cause a prohibited outcome in the extreme to oblique intent where prohibited outcomes were caused as a side-effect of their behaviour, to recklessness and negligence where the prohibited outcomes were foreseeable outcomes of their behaviour to various degrees. Certain offences will specify what level of mens-rea is required so murder requires direct or oblique intent.
Aside from the crimes of specific intent, certain laws exist
which require establishing for what purpose the accused did
something. This is called basis intent by
There exist a number of challenges to developing an RL method which produces legal policies under a rich set of laws. I classify them into three areas: 1. Encoding The environment’s laws need to be described in such a way as to be machine interpretable. 2. Determination A mechanism needs to exist which can determine the legality of any behaviour either in advance or in retrospect.
method to constrain the policy of the A-bot to be law abiding when it is acting or learning.
Generally these problems should be solved in the order displayed. The determination of legality requires an encoding of law to reference (planned) behaviour against and constrained policy learning relies on knowing what policies are acceptable and what are not. By looking at the taxonomy of laws, some inferences can be made about all three elements of this process. I will run through each of the three tasks and make comments on how laws affect them. In practice the three tasks might be heavily intertwined.
Safe RL research is only beginning to pay attention to how laws should be described. This is because the types of restrictions considered have largely been of the simple state type which can be encoded using simple algebraic expressions. This approach becomes untenable when considering more complicated laws like the ones identified in the previous example. Furthermore the quantity of applicable rules in regulated environments is larger than most rulesets hitherto considered in research. I identify four desirable features of an encoding which should be used to convey laws. 1. Temporal A number of laws restrict sequences of states or actions. Moreover there is no requirement that these sequences are contiguous. An encoding needs to be rich enough to express multiple states and actions with temporal relations like always, until, next etc. 2. Probabilistic As in the case of Inchoate offences, some laws refer to future states not realised. Since the space of all possible future events is a large one, law reasonably concerns itself with restricting foreseeable consequences of behaviours. An encoding of laws should be rich enough to express this. 3. Causal As we saw in the previous section, laws will often prohibit the causation of a state, not the state itself per se. Our death is not usually prohibited, but causing it normally is. Causation when considered in prospect will also normally require some sort of probabilistic reasoning. 4. Intent Certain laws require establishing levels of intent on the part of the transgressor. Different levels of intent exist and are applicable to different offences.
In Table 1 I summarise what level of encoding
expressiveness is required for each of the law types described in
the previous section. Nearly all of the laws require a
temporal expressiveness. Temporal Logic systems allows us to
express when conditions should be true. A wide variety
exist such as Linear Temporal Logic (LTL)
The analysis is not exhaustive. For example in any given regulated environment there are likely to be a large number of rules that the learner should obey simultaneously. This is likely to mean deadlock situations arise where not breaking one law may result in the breaking of another. A meta ordering of laws may be required to deal with this situation. Law Type
Given that we have a codified set of rules, we require a device to determine when a sequence of behaviour has or is contravening a law. I assume that the states referred to in the encoding are either the same as those perceived by the learner or there is an available mapping function between the learning and encoding statespaces. This is of course not a given since the learner may be perceiving continuous states and the encoding is likely to refer to high level states. In the case of simple state restrictions this is not a difficult task but it becomes increasingly complex with richer laws. I have separated the requirements into four main features. 1. Domain Consider an arbiter function which determines legality to the binary set - denoting legal or illegal. The domain of this function is dependent on the type of law it is considering. Laws which reference more than one state or action for example require a domain which includes the history of states and actions ht = (s1; a1; s2; a2; : : : st). Laws which reference future paths will also require the policy function of the learner ( ). Laws which require intent may also require information about the reward function of the learner or its estimation of state values. 2. Future Projection A model of the environment is required for almost all laws. In the case where strong safety is required in training, the legality of any action needs to be assessed before choosing it and this means assessing the likely state transitions which occur as a result. Projection is also a requirement in causal reasoning. 3. Causal Reasoning As certain laws are defined by causation, a method is required to determine whether restricted states have or are likely to be caused by the learner’s action. This requires a causal model of the environment. 4. Intentional Reasoning In environments where laws are defined by intent, a learner must be aware of what they are intending to do (ie their likely policy trajectory) by choosing a particular action at any moment in time.
In Table 2 I show the necessary features of a legality determination process according to the law type present. Reasoning about intent requires an algorithmic definition of intent. This is an open area of research since the concept of intent has been deliberately left as a primitive by legal practitioners. Care must be taken to ensure that the definition of intent used in safe RL corresponds to what a court would find sufficient.
Future trajectory prediction Law Type State Maybe Caused State Yes Action Seq Action State Maybe sequence Inchoate Yes Intent Yes
Arbiter function domain
Reasoning State path Action path Policy Causal Intent Yes
Yes Yes Yes
Yes Maybe
Yes Yes Yes
Probably
Yes
The taxonomy of laws informs us how those laws should be
described and the process which determines the legality of
behaviour. Finally, it can also inform us about the
properties of RL methods which will generate legally constrained
policies.
1. Memory Reinforcement Learning approaches typically
use a MDP formulation to model their task. Whilst a
record of the current state might still be valid for the
transition model, most of the law types that I have identified
rely to some extent on sequences of states and actions.
Thus the device which chooses actions at any state (most
likely the policy function) must include histories in its
domain. Otherwise the standard MDP learner would not be
able to understand whether its current action is legal or
not. Including the history of actions and states is
something that POMDPs do in order to make inference on the
hidden states of this model.
2. Model for planning Determining the legality of any
action requires a predicting the likelihood for future states.
How far prediction is expected to go into the future
depends on the laws present - avoiding inchoate offences
presumably requires greater foresight. Much of RL is
’model free’ and successfully so, but they seem
unavoidable here. Established methods like Dyna-Q
simultaneously learn to act and create a world model
The task of learning a legally constrained policy through
RL has seldom been mentioned in isolation but instead cited
as a possible use case in more general Safe RL work.
Surprisingly the learning of ethical policies has loomed larger
in published research. For a RL approach see Abel,
MacGlashan, and Littman (2016) or
Safe RL methods which constrain exploration include
approaches where a policy is learnt from observing a safe
policy in a process known as Inverse RL or
Apprenticeship RL
A developing area of Safe RL are those methods which
combine formal methods based on symbolic logic into the
learning machinery of RL. Many of these techniques
originate from the research area of formal verification methods
and model checking
Seldonian Reinforcement learning
The paper is motivated by an aim to design Safe RL processes which are capable of producing policies constrained under a general rule set. By creating a brief taxonomy of laws in the language of states and actions specifically for the application I have been able to draw some conclusions about the requirements of legally-safe RL. Laws are commonly defined in inter-temporal ways over actions and state. This means that a learning process must include a memory of past states and actions. Thus the domain of a legal policy function will include history just as it does in RL under a POMDP. Causality and Intent can be key concepts in determining whether and which laws have been broken. Whilst RL is beginning to tackle causality, it has not done in the context of constrained learning. Intent is barely defined quantitatively but it will have to be if generally legal RL systems are to be produced. Causality, Intent and the existence of inchoate offences mean that a legally-safe RL algorithm will require prediction about likely future trajectories. This will require some type of environment model to be learned or supplied to the learner and planning take place.
This work could be viewed as an application of legal
requirements engineering. Care should be taken since it
originates singularly from a computer scientist and not a legal
practitioner