=Paper=
{{Paper
|id=Vol-2419/paper_2
|storemode=property
|title=Towards a Framework for Safety Assurance of Autonomous Systems
|pdfUrl=https://ceur-ws.org/Vol-2419/paper_2.pdf
|volume=Vol-2419
|authors=John McDermid,Yan Jia,Ibrahim Habli
|dblpUrl=https://dblp.org/rec/conf/ijcai/McDermidJH19
}}
==Towards a Framework for Safety Assurance of Autonomous Systems==
https://ceur-ws.org/Vol-2419/paper_2.pdf
Towards a Framework for Safety Assurance of Autonomous Systems
John McDermid, Yan Jia, Ibrahim Habli
Department of Computer Science, University of York, UK
john.mcdermid, yan.jia, ibrahim.habli @york.ac.uk
Abstract • Some of the challenges of analysing shared autonomy.
These issues underpin the proposed framework; they are il-
Autonomous systems have the potential to provide lustrated in section 2 which considers some AS accidents.
great benefit to society. However, they also pose
AS used in safety-related situations should be subject to
problems for safety assurance, whether fully auton- safety processes. Previous analysis of quantitative risk anal-
omous or remotely operated (semi-autonomous). ysis (QRA) [Rae et al 2014] identified limitations of classical
This paper discusses the challenges of safety assur-
safety processes, many of which are relevant to AS. Section
ance of autonomous systems and proposes a novel
3 draws out key issues, considering the impact on both assur-
framework for safety assurance that, inter alia, uses ance and regulation. It also reviews the literature on AI safety
machine learning to provide evidence for a system
seeking to show a correlation with the QRA limitations.
safety case and thus enables the safety case to be up- Section 4 outlines the top-level of the framework and iden-
dated dynamically as system behaviour evolves. tifies some of the work that will underpin (flesh out) the
framework. The framework seeks to address some of the lim-
1 Introduction itations in safety processes identified in the analysis of QRA.
This paper addresses the safety of autonomous systems (AS). The major novelty of the framework is that it includes the use
It is intended to cover the spectrum from those that are semi- of ML in safety assessment as well as its role in the AS itself.
autonomous (remotely controlled or operated) through to As the framework is new, it has not yet been applied to an
those that operate largely free of human oversight. Often AS AS. However, a related framework, which has informed our
use artificial intelligence (AI), including machine learning ideas, has been applied in complex healthcare settings, see
(ML), as a key implementation mechanism. The paper relates Section 5. Using such an example also shows how the frame-
to the System Assurance and Certification and the AI Safety work might be used for AI safety more generally, not just for
Foundations focus areas of the AI Safety Landscape. It pro- safety of AS.
poses a novel safety assurance framework. Sections 6 discusses related work and future plans; Section
7 presents conclusions.
1.1 Spectrum of Autonomy The paper considers some apparently disparate concepts.
In the authors’ view AS are best thought of as systems where The ambition is to draw these concepts together to propose a
decisions that would otherwise have been taken by humans framework that can address the wide range of factors that in-
are allocated to machines. In practice, many systems have fluence safety and assurance of AS, not just AI. As the ideas
‘shared autonomy’ where operators can monitor and poten- are evolving the paper shows a ‘direction of travel’, not a fin-
tially over-ride the system’s decisions and actions. Often the ished product; it is hoped that this will help stimulate debate.
function (decision) allocation between humans and AS can
vary over time. In some sectors, e.g. automotive, the spec- 2 Motivating Examples
trum is codified into ‘levels’ of autonomy. However, diverse Examples are presented which illustrate the problems of the
sectors use different definitions of levels, so we choose here use of AI/ML, and shared autonomy, respectively. The intent
just to refer to a spectrum; ultimately our proposed frame- is primarily to identify the issues that need to be addressed in
work is intended to address the whole spectrum of autonomy. producing a safety assurance framework for AS.
1.2 Scope and Ambition 2.1 Autonomous Road Vehicles
Producing a complete framework for assuring AS, or even the There have been several fatal accidents with autonomous
AI elements of such systems, is a major endeavour. The scope road vehicles (AVs). We consider two accidents which can
in this paper is more limited and we address: be viewed as illustrating some of the issues with AI and ML
• Some of the challenges of use of AI and ML in AS, (as image analysis necessarily uses ML), although this is not
particularly focusing on training and testing; fully covered in the accident reports.
A fatal accident occurred in May 2016 when a Tesla on
autopilot impacted a truck (semi-trailer) near Williston Flor-
ida. The NTSB’s analysis of probable cause refers to the
driver of the truck failing to yield, and to the Tesla driver’s
‘inattention due to over-reliance on vehicle automation’
[NTSB 2017]. However, the report also says: ‘the Tesla’s au-
tomated vehicle control system was not designed to, and did
not, identify the truck crossing the car’s path or recognize the
impending crash’; if it had recognised the obstacle in its path
the Tesla could have applied emergency braking. This shows
the challenge of training and testing image analysis systems
for operation in an open environment but doesn’t give any
specific information on the issues for the ML processes. Figure 1. Abstract Model of WK Accident Causation
The NTSB preliminary report on the Uber accident in
Tempe Arizona in March 2018 [NTSB 2018] gives a more The design model used, including for safety analysis, was
detailed account of the events leading to the accident than is not an accurate predictor of the actual behaviour of the sys-
present in the Tesla analysis. The pedestrian was first ob- tem (in its operational environment). The ground crew train-
served about 6 seconds prior to the accident. The report says: ing and documentation did not help them to understand the
‘the self-driving system software classified the pedestrian as actual system behaviour, including WCAs. Finally, workload
an unknown object, as a vehicle, and then as a bicycle with had a significant impact on the operators’ ability to observe
varying expectations of future travel path. At 1.3 seconds be- the state of the system and to control it. There is a dissonance
fore impact, the self-driving system determined that an emer- between the three sub-models in Figure 1 which is redolent
gency braking maneuver [sic] was needed’. The Uber system of the distinctions made between ‘work as imagined’ and
relied on the driver for emergency braking and was not in- ‘work as done’ [Hollnagel et al 2006]. This dissonance is only
tended to initiate an emergency stop. likely to get worse for systems that learn in operation.
This example illustrates more fully the problems for ML:
how to reliably detect and classify objects in the environment 3 Safety and Assurance Challenges for AS
whilst avoiding ‘false alerts’ that would prevent expeditious By assurance we mean justified confidence or certainty in a
progress? It also illustrates the shared autonomy problem, but system’s capabilities, including its safety. The motiving ex-
this is shown more fully below. amples have given some practical illustrations of why assur-
ing safety of AS is challenging. This section considers these
2.2 Unmanned Air Systems challenges more systematically drawing together three differ-
The United Kingdom (UK) Army uses an Unmanned Air ent perspectives. First, it considers ‘classical’ safety pro-
System (UAS) known as Watchkeeper (WK) for reconnais- cesses, but focusing on their limitations. Second, it considers
sance. WK is operated and supported by a ground crew at a the ‘AI perspective’ on safety. Third, it presents a perspective
Ground Control Station (GCS) but some functions are auton- on assurance challenges from a programme addressing assur-
omous, e.g. the ability to execute a go-around (rather than ance and regulation of AS, including their use of AI/ML.
landing). WK does not use AI or ML but it is a good example
of the shared autonomy problem. 3.1 Safety Processes and their Limitations
WK has suffered five accidents to date; a far higher loss The principles of classical safety processes apply to AS. It is
rate than the safety analysis predicted [Wilby 2019]. The UK necessary to identify hazards, to assess (safety) risks associ-
Defence Safety Authority (DSA) report on the fourth WK ated with the AS, and to change the design (or operational
crash [DSA 2019] draws out three ‘themes’ from all five ac- profile) if the risks are too high, in order to make the system
cidents (the fifth is still being investigated but the DSA has acceptably safe. In this paper we assume that the safety pro-
visibility of the investigation). The themes highlighted are: cess results in the production of a safety case – a justification
1. The incomplete understanding of the full system, and (or argument), supported by evidence, that the system is safe
how sub-systems integrate; to operate in its intended context of use.
2. The need to improve collection and analysis of data; A systematic review of Quantitative Risk Analysis (QRA)
3. Ground crew and engineer workload. [Rae et al 2014] showed that the quantitative results are nor-
For example, in the third area, the report cites the high rate of mally not accurate. Safety analysis is not just quantitative, but
warnings, cautions and advisory (WCA) notifications creat- the problems identified affect the validity of all aspects of
ing a high workload. Further, the ground crew rely on their safety analysis, whether or not the results are quantified [Rae
collective knowledge to understand the situation and how to et al 2014]. The paper proposed a five-level hierarchy for im-
respond, e.g. to interpret WCAs rather than referring to doc- proving QRA, which is also relevant to AS:
umentation (paper or electronic). Thus, based on our reading 1. Unrepeatable, e.g. analysis relies on undocumented
of the reports and from discussions with the manufacturers assumptions;
[Wilby 2019], we propose the abstract model of accident cau- 2. Invalid, e.g. for WK, design models were not accurate
sation set out in Figure 1. predictors of operational behaviour;
3. Valid but inaccurate, e.g. inadequate treatment of un- 3.3 Assurance and Regulation
certainty such as in object classification; The Lloyd’s Register Foundation review of robotics and AS
4. Accurate but challengeable, e.g. insufficient scientific (RAS) identified challenges (they used the term ‘white
knowledge for parts of the analysis (this is likely to be
spaces’) in assurance and regulation of RAS [LRF 2016]. The
a particular issue for AS using ML); Assuring Autonomy International Programme that was set up
5. Ideal (which the paper views as unattainable).
in response to the above review has amplified on these issues
The intention is that flaws at level 1 need to be corrected and identified Critical Barriers to Assurance and Regulation
before a safety analysis can move to level 2, and so on. Flaws (CBARs) [AAIP 2018]. CBARs are characterized as prob-
identified in [Rae et al 2014] are referred to here as §x.y lems that must be solved otherwise there is a risk that:
meaning flaw y at level x, e.g. §2.3 is ‘mismatch between the 1. Unsafe systems are deployed, or
risk assessment and reality’ – a key point drawn out in our 2. Safe systems cannot be deployed.
motivating WK example. The level 4 flaws are of particular
The distinction between these two cases rests on the regu-
relevance to AS – in general we do not have a good scientific latory regime; a permissive regulatory regime which allows
basis for safety assessment of ML. [Rae et al 2014] identify systems to be deployed in the absence of contrary evidence is
other flaws that motivate our framework, and we discuss prone to the former risk (arguably this is what happened in
some of these in Section 4. the case of the AV accidents outlined above); a restrictive re-
3.2 Challenges of AI Safety gime is prone to the latter which might inhibit the beneficial
use of RAS (hereinafter AS for brevity).
The AI community have recently produced a number of pa- CBARs are intended to highlight key issues in the assur-
pers on the problems of ‘AI safety’ e.g. [Domingos 2012], ance and regulation of AS, with the aim of focusing research
[Feige 2019]. It should be noted here that there is a disso- on these issues to expedite safe deployment of AS. The
nance between the use of the term ‘safety’ in the AI and safety CBARs are intended to apply across different application do-
engineering communities. The latter focus on injury or death mains, e.g. UAS or medical devices. Two CBARs relevant to
to humans; the former takes a wider definition that encom- this paper are (simplified and merged from [AAIP 2018]):
passes robustness, privacy and security, and they tend to fo- 1. Monitoring and handover – how can it be ensured and
cus more on mechanisms for producing ‘unsafe’ results assured that operators retain sufficient levels of atten-
whereas safety engineers are more concerned about effects. tion and concentration to handle problems that arise?
Fortunately, these different perspectives are complementary 2. Training and Testing – how can it be shown that the
and safety engineers can usefully consider ‘AI safety’ con- training and test sets used in ML give enough cover-
cerns as potential causes of a hazard. age of the environment to provide sufficient evidence
One of the more influential papers [Amodei et al 2016] to allow controlled use of the AS?
identifies ‘concrete problems in AI’. The paper is largely fo- All three accidents discussed in Section 2 illustrate the
cused on reinforcement learning; the problems they identify monitoring and handover CBAR; the AV accidents illustrate
can be rephrased as follows, to make them more general: the second CBAR.
1. Avoiding negative side effects – ensuring that the be-
haviour meets safety constraints;
2. Avoiding reward hacking – ensuring that the behav-
4 The Proposed Framework
iour doesn’t get a high reward by ‘cheating’ and thus The framework proposed here is intended to provide a basis
avoiding the benefit that was sought; for assurance and regulation of AS, taking into account the
3. Scalable oversight – this concerns the ability of the use of AI/ML in their development. Although safety princi-
human to interact with the system both to monitor it, ples apply to AS the analysis can be difficult, especially
and to respond to requests for confirmation of deci- where the behaviour of the system can evolve in operation
sions prior to actions being taken; (e.g. due to ML). The core difficulty is that traditional safety
4. Safe exploration – if the system can try out new things processes assume that safety can be assessed prior to deploy-
(new strategies) how can negative outcomes be ment and the assessment remains largely valid through life.
avoided if these are circumstances that have not been The framework therefore includes continued and proactive
considered hitherto; assessment in operation – in contrast to current safety man-
5. Robustness to distributional shift – how the system agement that tends only to update safety assessments in re-
adapts to changes in the operational environment. sponse to problems or accidents. It draws on Hollnagel’s
Problem 3 arises for WK even though it does not use AI. ‘work as imagined’ and work as done’, but re-setting these as
The two AV accidents illustrate both problems 1 and 3 (alt- the ‘world as imagined’ (as we have to consider the system
hough there are issues of psychology here too). The Tempe and its environment) and the ‘world as observed’ (or the ‘data
Uber accident seems to reflect problem 4. There are real- world’) reflecting the need to analyse operational data to un-
world examples of both problems 2 and 5, e.g., ship classifi- derstand and control the dissonances identified above.
cation algorithms that work in the Baltic but not near the
equator due to differences in the angle of elevation of the sun, 4.1 Framework Overview
typical wave patterns, etc. The framework has four major elements which are conceptu-
ally distinct, although they physically overlap to some extent:
Figure 2. Top-level of the Proposed Safety Assurance Framework
• The ‘real world’ contains the AS in its operational • Uncertainty – the inability to model all the likely
environment; variations in the environment, or the distribution of
• The ‘world as imagined’ contains design and simu- possibilities, due to ‘real world’ complexity, [§3.3
lation models, and the results of safety analysis ‘insufficient characterization of uncertainty’].
based on those models; These factors (and more) contribute to the mismatch be-
• The ‘world as observed’ contains the operational tween the real and imagined worlds, and thus limit the fidelity
data, e.g. images produced by sensors, and the re- (predictive accuracy) of the safety analyses. The training data
sults of ML analysis of the operational data; ‘gap’ clearly links to the ‘training and testing’ CBAR.
• A safety case, initially reflecting just the ‘world as The ‘world as observed’ is also affected by ‘gaps’ as
imagined’ later updated to respond to the ‘world as shown in Figure 2, which represent different sorts of mis-
observed’, reducing the gaps between the safety match with the ‘real world’. These include:
case and reality. • Sensing – sensor sets, e.g. lidars and cameras, will
The ‘real world’ environment includes both the physical have limited capabilities and are affected by the en-
infrastructure, e.g. roads, and people, including operators, vironment, e.g. rain, dust clouds, etc.;
passengers, etc., as well as the AS. This is the ‘ground truth’. • Algorithmic limitations – characteristics of algo-
Design produces models of the system and environment, rithms, e.g. balancing false positives against false
including simulations both to define what is to be produced, negatives, or sensor data fusion not always selecting
and to analyse it. In the framework, safety analysis including the best combination of data sources to use;
hazard analysis and the identification of risk mitigations are • Failure conditions – hardware can fail, and this can
key elements of the ‘world as imagined’, but this is where affect behaviour (but it should be addressed by con-
model mismatches (dissonances) can start to arise. ventional safety processes);
There are ‘gaps’ between the ‘real world’ and the ‘world • Human/team performance – limitations in human
as imagined’, see Figure 2, that are challenges for safety anal- cognition and capability, allowing for factors that
ysis. The gaps identified in Figure 2 (with example mapping shape human performance, e.g. the high level of
to flaws in [Rae et al 2014]) include: WCAs that affected the WK accidents;
• Assumptions – what is taken as a ‘given’, about the • Resource limitations – many AI/ML algorithms are
system or environment, e.g. driver will be able to non-deterministic and, for example, it is possible for
intervene and apply emergency braking [§2.3 ‘mis- object classification algorithms to ‘lose’ objects if
match between the risk assessment and reality’]; they run out of time between video frames (this is
• Framing (scoping) – deciding what aspects of the viewed as distinct from algorithmic limitations if the
‘real world’ should be considered, and which do algorithm would not ‘lose’ the objects given suffi-
not, e.g. in determining what objects might need to cient processing resources).
be classified by an AV vision system, such as pe- The human/team performance ‘gap’ relates to the monitor-
destrians pushing bicycles [§2.2 ‘Major omissions ing and handover CBAR.
in the analysis’]; The sensor and algorithmic issues are not always disjoint
• Model fidelity – precision and accuracy of the mod- and, for example, the failure of the Tesla to identify the semi-
els used, including simulations, e.g. point mass ap- trailer in the Florida accident, is likely to have been due to a
proximations to the dynamics of AVs [§2.4a ‘Mod- combination of such limitations.
els are used outside their valid scope’]; As with the mismatches between the ‘real world’ and
• Training data – limitations of the data used to train ‘world as imagined’, these gaps are illustrative, not exhaus-
the ML in AS, e.g. biases in data sets used for pe- tive. Again, these reflect problems identified by [Rae et al
destrian detection, so that some ethnic groups are 2014], viz: Sensing maps to §3.1 ‘insufficient rigour in se-
misclassified [§2.8 ‘failure to report limitations’]; lecting source data’ and Algorithmic limitations maps to §3.2
‘incorrect processing of data’, albeit extending these catego- the potential problem of distributional shift between
ries to the operational system, not just the safety analysis. the simulator and AV;
Note that gaps can also arise in the ‘world as observed’ from • The rationale for choosing factors that shape the
the ‘work as imagined’ shown as ‘inherited gaps’ in Figure 2. learning process including selection of training and
The most novel part of the framework is the ML analysis test data sets, and ‘hyperparameters’.
in the ‘world as observed’. Safety analysis on the ‘world as The ‘hyperparameters’ in model learning are important as
imagined’ is hypothetical – it is based on design models and the models learnt are dependent on these values. For exam-
on an imperfect understanding of the environment. The WK ple, Bayesian Network (BN) structure learning can be used to
example shows clearly that such mismatches are a serious identify correlations between elements in the training data.
problem and form significant contributions to the accidents. The relevant hyperparameter is Equivalent Sample Size
Systems employing AI and ML are data rich, and this (ESS) which is used to guide the learning process when BDeu
opens up the possibility of using ML on the operational data (Bayesian Dirichlet equivalent uniform) score is used for the
both to understand the factors that actually influence safe be- structure learning. In general, increasing ESS leads to learn-
haviour, and to validate or to inform refinement of, the safety ing more links in the structure, but the number of links does
analyses. The rate at which operational data is produced will not necessarily increase monotonically. The arguments will
exceed human capability for analysis in real-time, but ML of- be domain specific, e.g. for a data set with a skewed distribu-
fers the opportunity to identify the critical information in the tion, which is very likely with AVs, for example, a smaller
data. In this way the initial safety case would reflect analysis ESS value is preferred [Steck 2012].
of the ‘world as imagined’, but then would be updated with The Assuring Autonomy International Programme is de-
ML analysis of the ‘world as observed’. Such an approach veloping a Body of Knowledge (BoK) which includes tem-
would allow the safety case to stay in (closer) alignment with plate assurance arguments to address these issues [AAIP
the system behaviour, which has been referred to as a dy- 2019]. The BoK contains guidance on the structure of assur-
namic safety case [Denney et al 2015], [Calinescu et al 2018]. ance arguments and the criteria for evidence and will evolve
The framework also supports feedback, enabling the safety to provide more details. e.g. using different search algorithms
analysis to be improved based on the ‘world as observed’, for learning BN structures to improve confidence.
and to improve the data collection and ML analysis.
4.3 Example using ML for Safety Analysis
4.2 Assurance of ML in the Framework
The development of the framework presented here has been
The safety case needs to address explicitly each of the ‘gaps’ influenced by parallel work on safety of medication manage-
identified in Figure 2 and demonstrate that the impact on the
ment which has had to address very similar issues of mis-
behaviour (safety) of the AS is small enough to permit initial match between the ‘world as imagined’ and ‘world as ob-
operations (see the discussion of regulatory issues below).
served’. The following illustrative example focuses on the
Figure 2 is not rich enough to show all the issues relating to
‘ML analysis’ in the ‘world as observed, as this is the most
ML in AS; these will require a more detailed model. The As- novel part of the framework.
suring Autonomy International Programme is working on The example is of a Health IT (HIT) system rather than an
such models [Ashmore et al 2019]. For simplicity we draw
AS [Jia 2019]; as mentioned above, this example is chosen as
out only a small number of issues highlighted in their ML
it best illustrates the framework, in the absence of an applica-
process model relevant to the ‘training and testing’ CBAR. tion to AS. In the HIT work, the mapping to the framework
The safety case needs to provide arguments and evidence
is as follows:
for the coverage of the training and testing data, noting that • Real world – hospital environment, including HIT
the coverage criteria should be informed by the safety pro- systems;
cess. Thus, for an AV, the focus should be on coverage of
• World as imagined – healthcare pathway model re-
those driving situations which are hazardous, e.g. junctions,
lated to post-operative care of patients following oe-
driving into low sun, and not ‘undemanding’ situations such sophagectomy, and safety analysis using Software
as quiet dual carriageways (divided highways). However, it Hazard Analysis and Resolution in Design (SHARD)
is unlikely (undesirable) that good coverage of near-accident [Pumfrey 1999] based on the healthcare pathway;
data will be achieved using real world data – it is too rare, and • World as observed – real data from the HIT system,
too dangerous to collect – thus collection must be augmented
and BN structure learning to explore the correlations
by simulation data to get good coverage of the operational
between the different factors identified in the safety
design domain (ODD), from a safety perspective. Thus, the analysis, representing hazards, causes and their ef-
argument should address a number of issues: fects; this enables the safety analysis to be validated in
• The coverage of the ODD in the data used for train- the real world.
ing and testing, justifying the ‘skew’ in the data to The structure learnt showed new patterns in the ways that
get coverage of ‘demanding’ situations based on the
nurses carry out their work during post-operative care follow-
assessment from the safety analysis, e.g. vehicle
ing oesophagectomy which was not expected as it was neither
cut-in and emergency braking to avoid pedestrians; apparent in the pathway model nor in the safety analysis. This
• The choice between real-world and simulation data, can be characterized as ‘flaw’ §2.3 [Rae et al 2014] ‘mis-
again based on risk assessment, and dealing with match between the risk assessment and reality’.
BN parameter learning was used to quantify the effects of In some ways, this is what AS developers are doing now, e.g.
different causes and controls on the associated hazard in or- progressively expanding the ODDs of AVs. Doing this ‘for-
der to understand their significance, enabling the introduction mally’ through a regulatory process would be a substantial
of the potential hazard controls to be prioritized. culture shift, in many domains, not least for the regulators.
4.4 Regulation based on the Framework 5 Discussion and Future Work
As noted above, regulatory processes assume that system ap- The ideas presented here are evolving and have their roots in
proval is essentially static, only updated for major events, e.g. the earlier analysis of QRA and considerations of how safety
significant design changes or accidents. Where AS evolve engineering can ‘catch up’ with design engineering [McDer-
their capability over time, e.g. using ML, then a more evolu- mid 2017]. They were crystalized by discussions of the WK
tionary approach is needed, with two primary phases. accidents. The aspiration is to use ML to understand and re-
First, initial operation of the AS should be based on safety duce the ‘gaps’ identified in Figure 2. Our initial work on HIT
analysis of the ‘world as imagined’. The decision to approve is encouraging, but the ideas need to be applied to AS. There
the system must be based on an assessment of risk that will is work using ML on operational data, e.g. for assessing
reflect the arguments that the ‘gaps’ identified in Figure 2 drowsiness of car drivers [Schwarz et al 2019]. However, we
have been adequately controlled. For AVs this amounts to de- believe that use of ML on operational data to update safety
ciding whether or not the use of the AV in an initial ODD cases is a unique perspective, but one that we hope might help
carries an acceptable risk. This will, for example, include to build bridges between the safety and AI communities.
consideration of sufficiency of the training and augmentation Also, there is a need to revise safety analysis processes to
data in the specified ODD, as outlined above. reflect the characteristics of AS and ML, for example chang-
Second, operation of the system will provide data, which ing the notion of controllability used in risk assessment for
is then analysed using ML, to either confirm the safety anal- road vehicles as they become more autonomous [Habli et al
ysis in the ‘world as imagined’ or to identify areas in which 2016]. There is other work in this area, and several authors
the safety analysis is not consistent with the ‘world as ob- have made use of Leveson’s STAMP/STPA or referred to its
served’ (or the ‘real world’) in a way that is safety significant. utility [NHTSA 2018], [NASA 2018]. STAMP/ STPA re-
The operational data would enable identification of structural flects a control systems perspective which is highly appropri-
weaknesses or ‘flaws’ in the safety analysis, such as identi- ate for AS, and some aspects of the models might help ad-
fied in [Rae et al 2014]. For example, if using BN structure dress the ‘flaws’ in QRA [Rae et al 2014], however it is not
learning, the learnt structures may show correlations of causal clear that the approach helps particularly with the ‘gaps’ iden-
factors of hazards not reflected in the safety analysis, which tified in Figure 2. Thus, we believe that more will need to be
is an example of §2.2 ‘Major omissions in the analysis’. Iden- done to produce an effective safety analysis process for the
tifying such a problem should prompt review and, if appro- ‘world as imagined’, and thus improve the safety analysis of
priate, revision to the system design, its operation, e.g. limi- AS. There is relevant work for AVs, including on Safety of
tations to ODDs for AVs, and update of the safety analysis to the Intended Function and on a ‘standard for safety for the
reflect a better understanding of the ‘real world’, etc. evaluation of autonomous products’ by the Underwriters La-
In the WK example this might involve an analysis of boratory (UL), known as UL 4600 [Koopman 2019]. UL
ground crew workload (using ML) and a redesign of WK (or 4600 will explicitly address some of the requirements on
more likely its support systems, e.g. the GCS) to reduce safety cases for AVs employing AI and ML discussed above;
workload, with an associated update to the safety analysis and a public draft of UL 4600 should be available during 2019.
the safety case. Put another way, the ML analysis in the We think that the notion of ‘gaps’ is helpful in considering
‘world as observed’ enables organisations to learn from ex- the wider issues of AS safety. Moving decision-making to an
perience, and to update their approach to safety, rather than AS can create a ‘responsibility gap’ where it is unclear who
the safety analysis being ‘open loop’ [McDermid 2017]. is responsible for an action (in an ethical sense) [Porter et al
Of course, the analysis of the ‘world as observed’ may 2019], and there may be no-one responsible in a legal sense
show that the risk is lower than predicted and that there is a (e.g. Arizona determined that Uber did not have a case to an-
‘safety margin’. In principle, this can be used to justify ex- swer for the Tempe fatality). We see the possibility of devel-
tending the use of the AS – perhaps expanding the ODD for oping a new and rich socio-technical ‘theory of safety’ that
an AV, or expanding the fleet size for a UAS, or allowing draws together multiple disciplines, including AI, system
UAS to fly in more congested airspace. safety engineering, law and ethics. A key part of this will be
ML can be applied to data from the AS in (near) real-time reconciling system safety with AI safety – combining system
so, in principle, the safety case can be updated continuously. safety’s view of the harm to humans from AS with the under-
However, unless regulatory approval can be automated so it standing from the AI safety community of what might go
can track the evolution of the evidence base, as might be done wrong in AI and ML.
using run-time certification [Rushby 2008], it is likely that
human regulators would approve changes to usage in ‘incre- 6 Conclusions
ments’ from time to time. This could be done where the evi-
dence produced by ML analysis in the ‘world as observed’ The introduction of AS has the possibility of providing sig-
enables the gaps affecting confidence in the AS to be reduced. nificant benefits to society, for example in social care and in
reducing fatalities on the road. However, there are also fun- [Habli et al 2016] Monkhouse, H., Habli, I. and McDermid,
damental challenges to safety and regulatory processes as the J., 2015, September. The Notion of Controllability in an
WK example, and accidents and incidents with AVs, show. autonmous vehicle context. In CARS 2015-Critical Auto-
The framework presented here reflects both empirical un- motive applications: Robustness & Safety.
derstanding of problems with particular AS, and a much more [Hollnagel et al 2006] Hollnagel, E., Woods, D.D. and
thorough analysis of weaknesses of safety processes in gen- Leveson, N., 2006. Resilience engineering: Concepts and
eral, and QRA in particular. However, it is abstract, and more precepts. Ashgate Publishing, Ltd.
detail is needed; the Assuring Autonomy International Pro-
gramme BoK [AAIP 2019] and standards such as UL 4600 [Jia 2019] Jia, Y., 2019, Improving medication safety using
should provide some of the necessary underpinning. machine learning. AIME 2019, Poland, Springer, In press.
There has been an initial validation of the framework on a [Koopman 2019] Koopman, P., 2019. Private communica-
healthcare example. It is hoped that the proposed framework tion.
will help to unify perspectives on AI safety and contribute to [LRF 2016] https://www.lrfoundation.org.uk/en/news/fore-
the development of the AI Safety Landscape. sight-review-of-robotics-and-autonomous-systems/ (ac-
cessed May 2019)
Acknowledgements
[McDermid 2017] McDermid, J., Playing Catch–Up: The
This work was supported by the Assuring Autonomy Interna- Fate of Safety Engineering? Developments in Systems
tional Programme. Safety Engineering, Safety-Critical Systems Club, M Par-
son T P Kelly (Eds), 2017.
References [NASA 2018] Alves, E.E., Bhatt, D., Hall, B., Driscoll, K.,
[AAIP 2018] Assuring Autonomy International Programme, Murugesan, A. and Rushby, J., 2018. Considerations in
Critical Barriers to Assurance and Regulation. Assuring Safety of Increasingly Autonomous Systems.
https://www.york.ac.uk/assuring-autonomy/projects/bar- [NHTSA 2018] National Highway Traffic Safety Admin-
riers-to-assurance/ (accessed May 2019) istration, Functional Safety of an Automated Lane Cen-
[AAIP 2019] Assuring Autonomy International Programme, tering System, DOT HS 812 573, 2018.
Body of Knowledge. https://www.york.ac.uk/assuring- [NTSB 2017] National Transportation Safety Board, 2017.
autonomy/body-of-knowledge/ (accessed May 2019) Collision Between a Car Operating With Automated Ve-
[Amodei et al 2016] Amodei, D., Olah, C., Steinhardt, J., hicle Control Systems and a Tractor-Semitrailer Truck
Christiano, P.,, Schulman, P., and Mané, D., Concrete Near Williston, Florida May 7, 2016, NTSB/HAR-17/02
Problems in AI Safety, arXiv.org, 2016. [NTSB 2018] National Transportation Safety Board, 2018,
[Ashmore et al 2019] Ashmore R., Calinescu, R., Paterson, Preliminary Highway Report, HWY18MH010.
C., Assuring the Machine Learning Lifecycle: Desiderata, [Porter et al 2018] Porter, Z., Habli, I., Monkhouse, H., &
Methods, and Challenges, arXiv preprint, arXiv: Bragg, J., 2018. The moral responsibility gap and the in-
1905.04223. creasing autonomy of systems. In International Confer-
[Calinescu et al 2018] Calinescu, R., Weyns, D., Gerasimou, ence on Computer Safety, Reliability, and Security (pp.
S., Iftikhar, M.U., Habli, I. and Kelly, T., 2018. Engineer- 487-493). Springer.
ing trustworthy self-adaptive software with dynamic as- [Pumfrey 1999] Pumfrey, D.J., 1999. The principled design
surance cases. IEEE Transactions on Software Engineer- of computer system safety analyses (Doctoral disserta-
ing, 44(11), pp.1039-1069. tion, University of York).
[Denney et al 2015] Denney, E., Pai, G. and Habli, I., 2015, [Rae et al 2014] Rae, A., Alexander, R. and McDermid, J.,
May. Dynamic safety cases for through-life safety assur- 2014. Fixing the cracks in the crystal ball: a maturity
ance. In 2015 IEEE/ACM 37th IEEE International Con- model for quantitative risk assessment. Reliability Engi-
ference on Software Engineering (Vol. 2, pp. 587-590). neering & System Safety, 125, pp.67-81.
IEEE.
[Rushby 2008] Rushby, J., 2008, March. Runtime certifica-
[Domingos 2012] Domingos, P., 2012. A Few Useful Things tion. In International Workshop on Runtime Verifica-
to Know about Machine Learning, Comm. ACM, Vol. 5, tion (pp. 21-35). Springer, Berlin.
Iss. 10, pp 78-87, October 2012.
[Schwarz et al 2019] Schwarz, C., Gaspar, J., Miller, T.,
[DSA 2019] Defence Safety Authority, Service Inquiry, Loss Yousefian, R., 2019. The Detection of Drowsiness Using
of Watchkeeper (WK043) Unmanned Air Vehicle over a Driver Monitoring System, ESV 2019.
Cardigan Bay in West Wales 24 Mar 17,
DSA/DAIB/17/006, 2019. [Steck 2012] Steck, H., 2012. Learning the Bayesian network
structure: Dirichlet prior versus data. arXiv preprint
[Feige 2019] Feige, I., What is AI safety? https://fac- arXiv:1206.3287.
ulty.ai/blog/what-is-ai-safety/ (accessed May 2019)
[Wilby 2019] Wilby, A., 2019. Private communication.