Deployed algorithms can cause certain negative side effects on the world in pursuit of their objective. It is important to define precisely what an algorithmic side-effect is in a way which is compatible with the wider folk concept to avoid future misunderstandings and to aid analysis in the event of harm being caused. This article argues that current treatments of side-effects in AI research are often not sufficiently precise. By considering the medical idea of side effect, this article will argue that the concept of algorithm side effect can only exist once the intent or purpose of the algorithm is known and the relevant causal mechanisms are understood and mapped. It presents a method to apply widely accepted legal concepts (The Model Penal Code or MPC) along with causal reasoning to identify side effects and then determine their associated culpability.
When certain types of Algorithms are deployed in the wider world, they can cause changes (effects) to that world. We can divide those effects into things which are the purpose of the algorithm (and its creators) and those that are not. We can augment this by understanding which effects are necessary for the algorithm to fulfil its purpose and which are not using the concept of means-end intent. Often risk analysis concentrates on those effects which might be caused if the algorithm fails to achieve its purpose. Side effects concern those effects which are caused by the algorithm, but whose occurrence does not affect the purpose of the algorithm and its creators. Often side effects have a cost not born by the person who caused them. Such costs are called negative externalities by economists. Analysis after the event can identify those effects of the algorithm which were foreseeable to the algorithm and its designers and those which should have been. Questions of Intent, Causation and Foreseeability are asked when courts decide on the culpability of algorithm designers when actionable harm has been caused. This article will use the culpability definitions as found in the US Model Penal Code and use causal reasoning coupled with Causal Inference Diagrams to provide a way of identifying and reasoning about side effects. We will use a running example of a recommender system as an illustrative example of a system which may display ill side effects.
2
The problem with defining side-effects solely in terms of the reward function is that effects necessary for a strategy to succeed, brought about through a policy are mislabelled as unintentional. The danger with Computer Scientists proceeding with an overly liberal definition of side-effect is that labelling caused harms as such implies they are not intentional. This in turn is important because intentional harms attract the highest criminal sanctions. Intent is not the sole determinant in determining culpability and this will article will go on to show how unintentional yet foreseen side-effects also attract criminal sanctions. Nevertheless certain crimes (such as attempt crimes) cannot be committed without intent. A definition of side-effect solely dependent on agent reward function risks incentivising myopia so as to avoid responsibility for harm.
Consider the story of the AI Physician tasked with curing cancer in a human patient. It comes up with a novel solution and proceeds and the result is that the patient is killed by the intervention. Note that it succeeds in its task because the patient does not die of cancer. Since patient survival was not in its objective function, patient death is understood as a side effect according to the definition above. Think of 3 surgical procedures: T1 The AI Physician removes the brain of the patient so that they could not subsequently die of cancer, but die from having no brain.
T2 The AI physician diverts all of the patient’s immune system to destroying the tumour but that necessarily made the immune system attack some other vital part of the body leading to death.
T3 The AI physician came up with a genuinely novel procedure with a p% recovery rate but the patient fails to recover.
These are all different causal mechanisms and we will later see that in Treatment 1, since death is necessary for the procedure to work it is most definitely not a side-effect. Additionally do we think about the side-effect status of patient differently in the contrasting cases of when the AI physician understands the outcomes of its actions and when it does not?
Rather than choosing a definition of side-effects ourselves, I argue that computer scientists are better off looking in other domains for one. That way we can borrow accrued wisdom, avoid a proliferation of conflicting definitions between sciences and deflect any accusations that an overly generous definition of side effect is a device to insulate ourselves from blame for harm. 3
In common speech what do we mean by the term side effect? Firstly to disambiguate, I should say that a concept of side effect does exist in programming and it has a formal definition fit for its intended purpose. A hazard with terms that have domain specific meanings is to assume that those meanings are shared outside the discipline. Instead we want to take the idea of side effect that exists outside computer science and bring it into the discipline in a process that does not alter it. As with most primitive or folklore concepts, people intuitively know what a side effect is, but pinning down a decent definition of one takes some effort. The benefit of such an endeavour is twofold. It aids cross-disciplinary communication for when a regulator and a computer scientist discuss side effects it is preferable that both mean the same thing. From a programming perspective, a formal definition of side effects written in such a way as an algorithm would be able to understand, can prevent algorithms from causing harm.
The most common place that people see the term side
effects is in a medical setting, so it is intuitive to start the
process of definition here. It is also an appropriate source
given that medicines and the medical profession are strictly
regulated. The APA (American Psychological Association)
defines a side effect as follows
Any reaction secondary to the intended therapeutic effect that may occur following administration of a drug or other treatment This definition makes the distinction between effects which are intended and those which are not, with side effects appearing in the latter class. The term secondary requires further unpacking which we will do once we have introduced some causal mechanisms.
The concepts of intentionality and foreseeability are
commonly used in the legal world and it is from here that we
will source their definitions. By looking to the law for a
definition of intent we can borrow centuries of legal thought
and endeavour. One can consider legal definitions as
opensource in the sense that they are accessible to public scrutiny
and have been democratically tested over time. As
Despite its key role in Criminal law amongst others 1, for
various reasons a singular definition of what constitutes
intent is elusive. We will use the US Model Penal Code (MPC)
A person acts purposely with respect to a material element of an offense when... if the element involves the nature of his conduct or a result thereof, it is his conscious object to engage in conduct of that nature or to cause such a result
In essence this definition says that someone intends something if it is the object outcome of their actions. This definition largely corresponds to the folk-definition of intent. Agent Ag intends X if they choose to do action ψ in order to cause X. This implies an epistemic condition on the outcome; Ag can foresee that X could be an outcome of them ψ -ing. However this definition does not mention probability of outcome so it follows that long-shot type outcomes can be intended.
At this point we will introduce some causal modelling terminology to illustrate more easily things which are intended and things which could be called side effects.
4
Consider a directed acyclic graph G with vertices V and edges E, for A, B ∈ V we will adopt the convention that A is a cause of B iff there is a directed edge in E from A to B. That is to say, all other things constant, a change in A will imply some change in B. We shall call this a Causal DAG.
1Intent appears in almost aspect of law - contract, tort, regulatory. Sometimes in obvious ways, sometimes not
2When drafting the MPC, the authors took the approach that defining concepts such as intent could be better done by not mentioning them so as not to conflate with possibly wrong folk concepts of the word.
We will use the Structural Causal Influence Models
(SCIM)
Additionally without any loss of generality we can
enforce the restriction that all stochastic elements of the I D
do not have a parent. Practically this just means the
rewriting of non-deterministic structural equations to separate
deterministic variables from (possibly new) non-deterministic
variables which themselves have no parents. This ensures
that every vertex that is a descendent of a decision vertex is
deterministic and the I D is said to be in Howard Canonical
Form
Within the framework of SCIMs and related Causal
Analysis, a number of definitions of intent might exist which are
arguably compatible with the MPC’s definition of intent or
Purpose but might require assumptions about the agent. For
example
Whilst for the purposes of identifying side effects we can simply assume that it is known which things are intended and which are not but we must still ensure intentional knowledge is consistent with the legal concept of intent. For this we need to make three assumptions. 1. An outcome can only be intended if its realisation is dependent on an action realising a certain value. 2. Actions made by an algorithm are done so intentionally only if their is choice not to act in that way (the action set has more than one member) 3. The concept of means-end consistent intent is respected.
Condition one just says that an outcome can only be
intended if a decision variable is able to affect the it. We
cannot intend our football team to win by attending the match
as a spectator. This rules out any parental chance or
outcome variables from being intended and stops any foregone
conclusions from being intended. Condition two says that
the action-decisions that a decision maker makes are not
coerced (there is always a choice of policy design).
Condition three requires explanation. There exists in law
Suppose we know outcome O = o for O ∈ VO is intended in an influence diagram. Then for any outcome variable O′ ∈ Anc(O) ∩ Vo, if it is necessary for O′ = o′ for the intended outcome O = o to occur then O′ = o is also an intended outcome.
Definition 2 (Side Effect and Unintended outcome). Consider an influence diagram SCIMwith outcome vertices VO, decision vertices VD and utility vertices VU , a policy function π ID and a set of intended variables VE ⊂ V and for each intended variable an intended realisation x ∈ RX for every x ∈ VE . • A Side Effect Variable of a Decision is any descendent variable of that decision vertex which is not an intended variable. A Side effect of a Decision is a realisation of a Side Effect Variable of a decision. • The Policy Side Effects and Policy Side Effects Variables are analogously defined for sets of intended realisations and variables according to some policy. • An Unintended outcome is a realisation of an intended variable other than the intended realisation
The definition requires that side effects are caused by a policy since they are descendants of action variables in a causal DAG; the policy has an influence on their outcome. We can restate Definition 2 by saying Side Effects of a decision are those vertices which are descendants of the decision but are not themselves ancestors of any vertices with intended realisations according to that decision. This definition incorporates the idea of means-end consistency into it assumed in Definition 1.
The definition of Unintended outcomes separates the analysis of when a certain policy fails in achieving its objective (chance of failure) from the analysis of side effects whose occurrence is not dependent a policy’s success. The legality or culpability of unintended outcomes is not straightforward. A doctor performing a life saving treatment which will either result in the death of their patient or their saviour is not usually punished in the event of failure even though the chances of success might be exceptionally slim. On the other hand a fund-manager who loses all of their investor’s money on a risky bet might be punished for their reckless actions. 6
In this section we will explore some of the previously
discussed features of side effects and intent using the example
of a company wishing to deploy a new recommender
system on its users. I recognise this is a departure from issues
of robot induced vase breaking as is typically considered
in related side-effect literature, nevertheless I think this
setting is pertinent. Article 5(1)(a-b) of The Draft EU AI Act
Suppose a company has the choice between serving two items of video content, one normal and one which has been designed to intoxicate its viewer into watching all of the video. If the user views this intoxicating content there is a chance that they become radicalised in some way. Hidden from the company are the user’s preferences which dictate what type of content they would want to watch, preferences undisturbed. • Let Serve Content decision be represented by variable
S ∈ {0, 1} • Let the Change Preference outcome be represented by variable C ∈ {0, 1} • Let the Radicalised outcome be represented by variable
R ∈ {0, 1} • Let Watches Post utility vertex be represented by variable
W ∈ {0, 1} • Let the user’s Preferences be represented by variable P ∈ {0, 1}. This is an exogenous random variable unknown at the time of decision S. Let P be Bernoulli distributed with chance of success 0 ≤ µ P ≤ 1 • UC and UR are two exogenous random variables which help determine the chance of Preference change and Radicalisation respectively. Let them both be Bernoulli distributed with probabilities of success µ C , µ R ∈ [0, 1] respectively.
The structural equations are as follows:
R = C = W = 1 if C = 1 and Ur = 1 0 else 1 if S = 1 and US = 1 0 else 1 if S = P or C = 1 0 else
Content can be one of two types as can a user’s preferences; p(P = 1) = µ P . If the two types match then the user will watch the post and the recommender is rewarded with a unit of advertising revenue. Additionally if the content is of the intoxicating type (S = 1) then there is chance µ C that the user will have their preferences changed C = 1. If that is the case then they will definitely watch the post. Additionally there is chance the user will be radicalised with probability µ R Example 1. Let W = 1 be the intended outcome under the Policy S = 1. The first question is whether C must have intended realisation under means-end consistency. As long as µ P > 0 (the chance that the user would actually like content type 1) W = 1 is over-determined; either P = 1 or C = 1 is sufficient for W = 1. C could be be both an intended variable and not.
Suppose C = 1 is the intended realisation. The set of intended variables is {S, C, W } and the set of possible side effects is {R} since there are no other descendant.
Alternatively suppose C has no intentional status. The set of intended variables is {S, W } and the set of possible side effects is {C, R}.
The subject of culpability as to side effects of actions has
received a lot of attention in Psychology since
Legal systems have a lot to say about side effects and
culpability3 and provide us with an independent framework to
reason about them. Conversely, courts currently have very
little to say about harms caused by algorithms and who
should take responsibility for them. Only legal persons can
commit crimes and so harms caused by algorithm might
have an indeterminate status
Just as with our use of the MPC to find a definition of Purpose (intent), we can use its definitions of Knowledge, Recklessness and Negligence to analyse the culpability of side effects. These four concepts are in descending order of culpability. Recklessness is typically the minimum level of culpability required for criminal charges. Negligence is the benchmark required for most civil-damages cases. The key features of these definitions are summarised in Table 1. The table also includes the features of Intent or Purpose for ease of comparison.
All four definitions of culpability in the table require someone to be able to foresee a bad outcome occurring as a result of an action or policy. It is here that the MPC’s decision to define the second most serious level of culpability ’Knowledge’ problematic since the word is useful to describe the epistemic properties of all of the definitions. We will refer to this as ’Culpable Knowledge’ to disambiguate.
The table shows aim or desire is only required for
Purpose/Intent which is consistent with our prior definition of
side effects being. It makes a distinction between two types
of knowledge - subjective knowledge - things which are
known to the actor and objective - things which should be
known to the actor. We can view the Influence Diagram as
a distillation of the actor’s causal knowledge of the world.
A side effect caused with Culpable Knowledge concerns the
case when a bad outcome occurs with almost certainty
according to the SCIMand the algorithm’s policy.
Recklessness and Negligence have been termed culpable
carelessness by
Whilst this table is focussed on the culpability of side effects as previously defined, Recklessness and Negligence can also apply to Unintended Outcomes. That is to say outcomes that may be caused when failing to achieve an intended outcome. In the case of Knowledge there is debate about whether someone can commit something with Culpable Knowledge if their action was intended to obtain some other result.
Figure 2 distils Table 1 into a decision process with which to identify the possible culpability of any caused outcomes. The grey decision vertices concern questions of subjective knowledge - information known to the actor at the point of commission. The white vertices concern questions of objective knowledge - information that should have been known to the actor at the point of commission. We will use it in the following example.
Example 2. Continuing Example 1 we consider the
culpability of the possible side effects caused. Since the outcome
of getting a user to watch a video of type 1 was
overdetermined, there was some uncertainty about the intentional
state of changing preferences - C by choosing action S = 1.
Since causing a change in preferences is not in itself an
unambiguous harm, we will concentrate analysis on the
Radicalisation outcome R which is in the set of side effects
regardless of the intentional status of preference change.
Given that content of type one is chosen, the probability of
radicalisation is µ p.µ R. R is not an intended variable and
has no intended realisation. Consider the case where
Radicalisation does occur. We will assume that the actor has
the same model available to them as in Figure 1. The first
question would be to consider whether harm is foreseeable,
and whether that harm was ’substantial and unjustifiable’,
that is whether the µ p.µ R >> 0. If this wasn’t the case,
then the Radicalisation could be said to be an accident. At
this point, the estimated quantity E[µ P ∗ µ R] must be
assessed. If the actor estimated this as negligible then
Radicalisation would have been caused with negligence. If the
estimate E[µ P ∗ µ R] ≈ 1 then Radicalisation would have
been caused with Culpable Knowledge. The remaining case,
E[µ P ∗ µ R] >> 0 means that the actor caused Radicalisation
with Recklessness. It should be stated that the definition of
’substantial and unjustifiable’ is not straightforward and may
be dependent on the degree of harm caused
In the case where harm has not been caused, culpability
can still arise if the actor were to believe their actions were
risky and they were substantially so. For a given crime, it
does not always follow that there is an analogous crime of
reckless endangerment unlike the general existence of
attempt crimes for every crime.
Outcome Intention
Exists?
Yes Intended Harm? Yes
No
No Harm Caused
Yes Harm Achieved Yes No
Attempt No Culpable Purpose
Harm Foreseeable
Harm
Foreseeable Else
Substantial and
unjustifiable
Accident / Strict Negligence
Harm Foreseen
Else
None Some Negligence
As well as applying negligence to instances when an
actor should have known about the chance of some harm
being caused, US and other Common Law jurisdictions allow
higher levels of culpability to be imputed in the case when an
agent actively avoids acquiring knowledge in an effort not to
inculpate their behaviour. This is known as the Willful
Ignorance Doctrine
8
The following review of related work is divided between subject area.
Side Effects in AI Whilst I argued in that the definition
of side effect in
In a recent review of the subject in AI, Saisubramanian,
Zilberstein, and Kamar (2021) state that Negative Side
Effects (NSE) are ”Undesired effects of an agents actions that
occur in addition to the agent’s intended effects”. This is
close in spirit to the APA definition in Section 3, though
there is no subsequent clarification as to how intent is
assessed. The authors present a taxonomy in which to classify
recent side-effect mitigation techniques consisting of the
following
• Severity - Serious harms are more obvious and likely to
be designed out at an early stage. Saisubramanian et al
observe that less serious harms are the ones that manifest
which they show in related work reduce confidence in AI
systems
An alternative to the reward (or penalty) based approaches
so far mentioned is constrained optimisation
Many approaches to the side-effect problem assume that
they are a result of underspeciefid reward functions or
nonobservability. It could be that even with an adequate reward
function and state space, undesirable and unnecessary
sideeffects are still incurred due to mis-inference on the part of
the algorithm. It is tempting to believe that causal reasoning
is not needed when using Reinforcement Learning. Often
this is because expert knowledge about the data generation
process has been embedded into a simulation environment
Side Effects in Philosophy and Psychology Research
considering the moral judgement of side effects in
experimental psychology has been popular since
Formal accounts of intent are not hugely common. The
aforementioned,
Side effects in law Law prioritises the establishment of
the various levels of culpability or mens-rea which make
the concept of side effect redundant. At the levels of
culpable carelessness, it isn’t so concerned whether an adverse
outcome was a side effect or an unintended outcome.
Bratman’s intuition that means-end intent should be equivalent to
purpose, expressed in
9
This article presents a formal definition of what constitutes side effects sourcing the definition of side effect from medicine and the necessary definition of intent from law. It does this with the use of a Structural Causal Incentive Model or SCIM, itself an extension of a Structural Causal Model (SCM) and an exogenous definition of intent. Side Effects of an action are those vertices which are descendants of the action but are not themselves ancestors of any vertices with intended realisations.
Any definition of side effects taken up by the Safe-AI community should be based on principles agreed by society rather than computer scientists. Such an approach defuses the accusation that definitions of concepts important to society are created to be convenient or progress the objectives of the engineer or their employer.
Despite their name, side effects can still attract severe criminal liability. I have used the standard MPC definitions of culpability to create a decision process which can be used to systematically determine culpability for harms caused or potentially caused. In the event that an algorithm causes some tangible harm to the world through a side effect, computer scientists are not going to be surprised about the conditions under which a side effect might make them liable in a civil or criminal sense.
Algorithms or their designers when equipped with model of the world can use such a definition and decision process to discriminate between directly intended outcomes and side effects and then identify what degree of culpability can be attached to side effects caused or endangered. In particular, the article should impress upon the reader the importance of foresight knowledge and its relation to culpability. Harms can be caused accidentally with no liability, but after their 4Attempts can be divided between those that are interrupted before commission after passing some threshold of culpable preparation and those that attempts which are completed but fail in their aim. We are talking about the latter here ifrst instance they become foreseeable, at which point if they reoccur they can no longer be termed accidents and causing them becomes a culpable action.
A
The following definitions of culpability are taken from the
MPC
A person acts knowingly with respect to a material element of an offense when: (i) if the element involves the nature of his conduct or the attendant circumstances, he is aware that his conduct is of that nature or that such circumstances exist; and (ii) if the element involves a result of his conduct, he is aware that it is practically certain that his conduct will cause such a result.
A person acts recklessly with respect to a material element of an offense when he consciously disregards a substantial and unjustifiable risk that the material element exists or will result from his conduct. The risk must be of such a nature and degree that, considering the nature and purpose of the actor’s conduct and the circumstances known to him, its disregard involves a gross deviation from the standard of conduct that a law-abiding person would observe in the actor’s situation.
A person acts negligently with respect to a material element of an offense when he should be aware of a substantial and unjustifiable risk that the material element exists or will result from his conduct. The risk must be of such a nature and degree that the actor’s failure to perceive it, considering the nature and purpose of his conduct and the circumstances known to him, involves a gross deviation from the standard of care that a reasonable person would observe in the actor’s situation.