Towards an Ontological Framework for Environmental Survey Hazard Analysis of Autonomous Systems Dr Christopher Harper, Prof Praminda Caleb-Solly1 * 1 Bristol Robotics Laboratory, University of the West of England T-Block, Frenchay Campus, UWE Bristol, Bristol BS16 1QY, United Kingdom chris.harper@brl.ac.uk, praminda.caleb-solly@uwe.ac.uk Abstract Analysis (ESHA) (Harper et al. 2014), which is a relatively new technique of preliminary hazard identification aimed at This paper presents current progress in the development of autonomous system design problems. Environmental Survey Hazard Analysis (ESHA), a method of preliminary hazard identification aimed at autonomous sys- tem application problems. In addition to performing their de- 1.1 The Problem of Autonomy from a Safety sign mission, autonomous systems must be capable of reli- Perspective able and predictable behaviour in their environments, partic- Autonomous systems have unique characteristics and re- ularly when facing potential hazards that are not explicitly quirements that present a considerable challenge for safety included in their design specifications (’non-mission’ tasks). ESHA differs from conventional hazard identification meth- assurance and certification. By definition, an AS may be re- ods in that its scope explicitly covers the identification of quired to operate for extended periods (or even indefinitely) non-mission interactions between a system and its environ- without any human intervention or supervision. This means ment and any associated hazards. Although of general use that they must be capable of interacting safely with any fea- as a safety analysis technique, ESHA has been designed ture of the environment necessary to ensure their ongoing primarily to support a ”so far as is reasonably practicable” survival, and performance of their required mission. Hence, (SFAIRP) style of safety argument. However, early versions there is a set of ”non-mission” interactions that an AS is re- of the method were based on informal models, and therefore quired to perform, which relate to general existence and sur- provided only weak support. This paper reviews the develop- vival, as well as all those ”mission tasks” that are required ment of a formal ontological framework for ESHA, intended for the AS to fulfil its intended purpose. to provide much stronger basis for arguing the completeness and consistency of analyses. A simple example of the concept of mission and non- mission interactions of an autonomous robot waiter is shown in Figure 1. 1 Introduction Autonomous systems (AS) are now emerging from the re- search laboratory into full industrial and social application, for example applications which require collaborative inter- action with humans in shared spaces. Yet one of the key challenges in the development of these systems, and one of the principal barriers to their full deployment, remains unsolved; we still lack the tools and methods for adequate safety assurance and certification. This paper reviews cur- rent progress at Bristol Robotics Laboratory in the devel- opment of a method called Environmental Survey Hazard * Supported by the Assistive Robotics in Healthcare project of the Assuring Autonomy International Programme (www.york.ac.uk/assuring-autonomy/). We thank our colleagues Figure 1: Mission vs. Non-mission Interactions Daniel Delgado Bellamy, Sanja Dogramadzi, Alex Sleat and Jason Welsby for their support. Copyright © 2021 for this paper by its authors. Use permitted un- The robot will be expected to (and designed to) perform der Creative Commons License Attribution 4.0 International (CC tasks related to interacting with customers, or setting down BY 4.0). or picking up drinks from furniture such as tables. However, there are numerous other unexpected features of the envi- more validation scenarios for each identified hazard, to eval- ronment that might be found within a location where such a uate the capability of the AS to avoid them. ESHA is a com- robot might operate(e.g. a cafe), for example unusual terrain plementary technique to a test coverage metric called Situa- (such as holes in the floor), humans such as children who are tion Coverage which has been developed recently (Alexan- not otherwise engaged in the business of ordering drinks, or der, Hawkins, and Rae 2015) as a validation metric for other creatures such as pets or service animals (e.g. guide scenario-based testing, and in Section 5.2 we discuss cur- dogs for visually impaired people). The robot will need to be rent activities using ESHA to identify test scenarios for a capable of performing these non-mission interactions safely connected autonomous vehicle (CAV) using situation cover- and reliably if it is not to present an unacceptable risk while age to evaluate progress and completion of the process. in service. The environment-driven requirements analysis produced Non-mission interactions are often overlooked by typical by ESHA can also support the development of machine design engineering practices. However, this gap is usually learning solutions for autonomous system problems. Since closed either by the supervision of human operators or by all ML processes or devices are developed inductively by constraining the system’s environment, which reduces the training from a set of examples, the correctness of the sys- number of required interactions between a system and its tem behaviour produced from a machine learning process environment to a point where it becomes tractable to con- is highly dependent on the diversity of the samples provided trol them with the resources available to the system. Since for its training. If there is any systematic error of omission in such interactions are an essential feature of AS problems, the training set, then the product of the ML process is likely we need new or revised safety analysis methods that seek to to exhibit similar omissions in its operational behaviour. By identify non-mission interactions, especially in unbounded providing a systematic coverage of potentially hazardous en- environments. Such a goal is challenging - the unbounded- vironmental interactions, ESHA can assist in specifying the ness criterion creates problems of combinatorial expansion scope of the training data set that is needed for adequate in the number of situated states that must be considered. We coverage of potential hazards, and can support the safety encountered these problems when we first started to look at argument that an ML/AI system has been sufficiently well hazard identification of AS, which led to the development of trained that the risk of any incompleteness affecting safety our original version of ESHA, as presented in (Harper et al. has been reduced as far as reasonably practicable. 2014). 1.3 Safety arguments for autonomous systems 1.2 Safety Validation of Autonomous Systems and The unboundedness of the general problem of autonomy ML/AI (Pfeifer and Scheier 1999) causes significant problems for In safety critical systems, hazard analysis is one of the prin- the safety argument(s) of an AS - for example as recom- cipal sources of information for safety validation require- mended by the new UL 4600 standard for autonomous road ments, and autonomous systems are no exception in this re- vehicles (ANSI/UL 2020). The demonstration that the resid- gard. Once a hazard has been identified, it follows that one ual risk of harmful events (a function of severity and prob- or more safety measures need to be introduced (typically ability) of a system is ’acceptable’ becomes questionable. design changes, safeguard mechanisms, or operational pro- Significant underlying causes of this problem include: cedures) to reduce or eliminate the risk of its occurrence, 1. In unbounded environments, the number of features with and the effectiveness of those safety measures must be vali- which an autonomous agent may interact is uncertain. dated. Since ESHA is a process of identifying environmen- This affects the validity of quantitative risk analyses; cal- tal interactions, it lends itself naturally to the specification culations of requirements for or verification of a system’s of test scenarios that can serve to evaluate the effective- probability of failure depend on the set of contributing ness of safety measures. For example, it could be used in hazards to be complete. If any are omitted then quantita- conjunction with scenario definition languages such as M- tive safety targets will be erroneous and the system less SDL (Foretellix 2020), OpenSCENARIO (ASAM 2020) or reliable or fail-safe than it needs to be to fulfil its ’true’ SCENIC (Fremont et al. 2019), where ESHA captures the requirements (i.e. those specified by legislation). high-level specification for scenarios that can be translated into scenario description languages for execution on a simu- 2. It has been a long standing problem in the risk assessment lator. of safety critical systems in many industry sectors, that Scenario-based validation is seen as one of the princi- regulatory requirements demand levels of reliability that pal modes of safety validation for autonomous systems (see are not feasible to demonstrate by practical testing (But- (Fremont et al. 2019) for example). As discussed earlier, one ler and Finelli 1991). Most practical test programmes can of the distinguishing concepts of autonomy relates to the re- only deliver high-confidence estimates of probability of quirement to interact reliably with features of the environ- failure that are usually several orders of magnitude lower ment. So, it follows that one must test those interactions in than the values required. This applies in particular to rare order to validate the correctness of an autonomous system’s events, which are unlikely to occur during testing but may function. Since ESHA is a process of systematic search for nevertheless still occur at a higher rate than is considered (potentially hazardous) environmental interactions, irrespec- acceptable by legislative requirements. tive of their inclusion in the system’s specified mission, it is 3. Where simulation models are used to accelerate the test- a straightforward extension of the method to identify one or ing of a system, and existing simulation data are used for probability estimation based on extrapolation of (bounds harm was not foreseeable, any safety measures not deployed of) probability from the measured results, any such analy- were not reasonably practicable, or the harm was outside sis rests on the assumption that the simulation is a faithful the scope of the undertaking. For these reasons, a safety reproduction of the system and its environment. But any case/argument will need to demonstrate objectively (Eliot simulator will have limits to its fidelity (Koopman and 2007) the following characteristics: Wagner 2018), which are likely to be impossible or too • that exhaustive coverage of potential hazards has been expensive to correct, and this makes the task of identify- achieved ing rare events solely from testing impossible (the event does not occur during the test programme, and there is no • that the analysis of a system’s scope of operation is sys- way to determine whether the lack of occurrence is due to tematic and complete the genuine rarity of the event or due to being masked by • that the identification of safety measures to resolve any errors in simulator fidelity). potential hazard has been exhaustive In all these cases a systematic analysis of environmental in- • that the costs vs. benefits of each identified safety measure teractions such as ESHA will help to improve confidence in have been analysed systematically the quality of the results, but our conclusion is that it will However, this may not be easy for AS - if the environment be difficult to produce strong supporting evidence for an AS is unconstrained or unbounded, then it may not be possible safety case with these techniques. to show that any potential cause of harm is outside the scope Therefore, we are investigating an alternative safety ar- of the undertaking, or that it was not practicable to do more gument strategy (Harper 2020), where the claim is that the than was in fact done to identify potential hazards. risk has been reduced ”so far as is reasonably practicable” ESHA was developed (Harper et al. 2014) specifically to (SFAIRP). This argument concept is explicitly written into support this type of argument by attempting to construct at UK law (HMSO 1974) and is often an acceptable legal in- least part of the evidence necessary to support such claims, terpretation in other countries. We use the UK legal basis of in terms of providing an objectively demonstrable process the concept as the working model for our studies, as well as framework that can assist analysts to show that all categories the work in (Eliot 2006, 2007), which provides supporting of environmental feature have been considered, even if it is interpretive material. not feasible to demonstrate that all instances have been iden- In a SFAIRP-based safety case, the general obligation for tified. This is the motivation for our consideration of onto- a system developer is to demonstrate that any of the follow- logical frameworks; ontology is the disciplined (and, ide- ing have been achieved: ally, formal) practice of trying to build such classification • it was not practicable or not reasonably practicable to do schemes. more than was in fact done to satisfy the duty or require- While the SFAIRP approach takes a different approach by ment; avoiding conventional quantitative risk-driven arguments, it • there was no better practicable means than was in fact is nevertheless the case that reducing risk to as low as can used to satisfy the duty or requirement; practicably be achieved may still mean that the residual risk • the potential cause of harm lies outside the scope of the is too great to be acceptable. This is a residual problem, [system developer’s] undertaking (Eliot 2007). but not one that can be resolved by engineering analysis methods alone; it is a legislative matter that lies within the Compliance with these requirements must be demonstrated purview of legislators and politicians to decide where any by means of objective evidence, namely that no allowance be such boundary may lie. Nevertheless, we argue that ESHA made for personal qualities of any legal defendant, i.e. it will could contribute to such decision making by supporting the not be a valid legal defence if the system developer appeals establishment of risk models that have some formal back- to the actual conduct of the safety assurance process, or the ing, and some confidence that the significant risks have been nature of the people who performed it. Evidence must be found. objective, i.e. epistemically independent of the people who created it. The legal concept of ”reasonableness” is based on the 2 Overview of ESHA concept of a ”reasonable person”, which in the case of engi- ESHA was developed to help identify environmental fea- neering problems may be interpreted as ”competent techni- tures with which an AS might interact (irrespective of cal specialist”. Hence whatever may be considered ’reason- whether it is a mission or non-mission interaction) and ably practicable’ in a safety assurance context would mean whether there is any potential for hazard. whatever might be expected of a suitably qualified practicing safety assurance engineer. If in the design of any system, any 2.1 ESHA Procedure and Guide-words hazard was overlooked or any safety feature of a system de- The ESHA procedure (Harper et al. 2014) is similar in many sign not considered for its practicability, which could plau- respects to conventional system hazard analyses: a set of sibly have been identified or specified by such an engineer, guide-words is considered, which classify the environmental then it could not be said that all foreseeable hazards were features with which an AS may need to interact. The general identified, nor all practicable safety measures deployed. nature of any possible hazardous interactions is identified. A plausible defence against breach of safety regulations The results are compiled in tabular a format similar to tra- can be made if it can be demonstrated objectively that the ditional variants of hazard analysis. A set of procedures and volved were not safety assurance practitioners (academic researchers and students in robotics were the most typical groups). And even though these versions of ESHA have been logically informal, the method still serves as a useful tech- nique for getting human designers to consider in a structured way how hazards might be identified for autonomous sys- tems. But it has been noted that complexity and combina- torial issues may limit the practicability of the method, and ontology has been seen as one of the best approaches to over- coming these problems. 3 Requirements for an ESHA Ontology The guide-words presented in the previous section are an attempt to categorize environmental features in such a way as to be logically complete and therefore exhaustive of any environmental domain. The set was developed on the con- ceptual basis that in an unbounded environment, the only boundary available from which one could extract a logi- cally complete description was the boundary of the sys- tem (robot) itself. The intent of the original ESHA guide- words was to identify environmental features by the char- acteristics of their ’images’ as they impinged upon the sys- tem sensors. Visual perception was the principal modality considered, hence the classification of objects in particular Figure 2: Original ESHA Guide-words as ’point-like’ or ’line-like’; the ’dimensionality’ of images was considered to be a concept that could demarcate all pos- sible environmental features in a logically exhaustive way. checklists were developed to guide analysts in compilation However, the guide-word model was never formally de- of results tables correctly, and the guide-words shown 2 in rived from first principles, and hence there is no rigorous ba- were specified. sis for asserting its completeness. Since we wish to demon- While this guide-word set is not unreasonable in an intu- strate objectively that the ESHA procedure is exhaustive, we itive sense, and was intended to be grounded in a concept are investigating existing formal ontologies to see if they based on how the characteristics of how environmental fea- provide the necessary logical framework we need to under- tures might be perceived by the sensors of an AS, it never- pin ESHA. Based on the previous discussions, the following theless is an informal model, and lacks any objective proof requirements are seen as necessary for the foundational on- of completeness. This compromises the aim that the ESHA tology: method can demonstrate objectively an exhaustive coverage • Support for essential ontological concepts of interactions with environmental features. We are therefore Several core ontological topics are required a a minimum looking to provide such a formal basis, which is the reason of any ontology that could be used to support ESHA: for our investigation of ontological frameworks as a basis for doing so. – Mereotopology: relationships between parts and wholes, that allow the identification of structures and 2.2 Current Experience with ESHA complex objects Our applications of ESHA have so far been restricted to – Boundaries: formal treatment of boundaries within the small or partial application problems in robotics and au- mereotopology, especially fiat as well as bona fide tonomous systems. In the original development phase of boundaries (Smith and Varzi 2000) ESHA (Harper et al. 2014), we investigated some problems – Situations and Events: formal classification scheme of in urban search and rescue and domestic assistance (guide situations and events, which will provide support for robot for elderly persons). More recently, we have trialled hazard identification the method in conference and project workshops, including – Agency, Causality, Autonomy: formal classification the SOCRATES project1 and the European Robotics Forum scheme for the behaviour of environmental features, to 2020 (Caleb-Solly 2020). allow capture of causal relationships and identification Anecdotal and verbal responses from workshop dele- of interactions. gates and participants have generally been positive, although it should be noted that in most cases the participants in- • Particulars and Universals Since we are interested in establishing abstract concepts 1 such as safety properties, we seek a foundational ontology This event is mentioned in passing at URL: http://www. socrates-project.eu/blog/2019/12/02/meeting-in-bristol/ that incorporates Universals as well as Particulars. • Realism or Conceptualism preferred over Nominalism some sub-types, which are derived both from physical and Since we require the existence of universals, we reject the abstract entities simultaneously and therefore inherit con- purely Nominalist stance that Universals do not exist. We flicting definitions of spatial and temporal existence. Once require ontological frameworks that reflect at least a Con- the type hierarchy is corrected for this inconsistency, it be- ceptualist if not a fully Platonic perspective. However, this gins to look similar to other ontologies (such as BFO, dis- does not mean that we cannot incorporate nominalist on- cussed below) that are far more explicitly formal and hence tological models as partial frameworks, applying solely to are preferred for that reason. Particulars, and then ’complete the model’ by adding cor- Basic formal ontology (Smith 2015) has the advantage of responding Universals, which can be done by means of being a consistent formal specification, but is insufficient for model-patterns such as the ontological square. ESHA purposes since (as a matter of pragmatic policy) it • Logical completeness and disjointness only include types and sub-types of Particulars. Addition- We are looking for formal ontologies whose type hierar- ally, the theory of mereology that underpins the BFO model chy is defined as far as possible in a logically complete is less extensive than other frameworks (UFO). and disjoint manner. Completeness ensures that the model Zemach’s ”Four Ontologies” model (Zemach 1970), al- is exhaustive in its scope. Disjointness of types ensures though having a nominalist perspective, does offer a disjoint that our analysis remains tractable, since multiple combi- and complete decomposition of Particulars into four types of nations of parent types will be excluded. continuant and occurrent entity. The model is complete and disjoint, and we believe can be adapted to resolve issues in Our general model of ontological frameworks for ESHA other models to improve the properties of the model eventu- takes the view that they will follow a three-layer organiza- ally used to support ESHA (as discussed later). tion/ structure: While no previously developed formal ontological model • Foundational ontologies are the most abstract layer, defin- had all the attributes we were seeking, our preferred on- ing the most basic entities that underpin other layers. tological framework is the Unified Foundational Ontology (UFO), as it has the greatest number of useful features that • Domain ontologies define the most general concepts that we have seen, and we believe that the few deficiencies of are specific to a particular domain but general to numer- logical completeness that exist in some parts of its model ous applications (e.g. system safety, geospatial domains, (especially in the decomposition into continuants and oc- HMI, etc.); currents) can be modified to correct the problems. While • Application ontologies define entity types intended for the original version of UFO (UFO-A) was developed only specific application categories, such as assistive robots or as an ontology of endurant (continuant) entities (Guizzardi driverless vehicles 2005), it has been extended with additional subsets UFO- B (covering perdurant/occurrent entities) and UFO-C (pro- 4 Review of Candidate Ontologies viding models of causality, interaction and agency). UFO-A It was not the original aim of this work to develop any fun- and UFO-B have recently been integrated into a combined damentally new ontologies to support ESHA, rather to ex- ontology of endurants and perdurants UFO-AB (Benevides, ploit existing work (Harper 2020). However, in respect of the Almeida, and Guizzardi 2019). These extensions develop basic foundational ontology layer, no existing frameworks ontologies for processes, events and their relationships, for have been found to possess all the properties desired. So we example temporal sequences of events, or how processes must adapt existing foundational ontology to introduce the might be composed from (or otherwise related to) set(s) of modifications or new elements as needed. individual events that may be ascribed to them. While numerous proposed ontological frameworks have been reviewed, the following are the major candidates that 5 Conclusion and Current Progress were considered for adoption as the ESHA foundational on- We have selected UFO as the basis for a design of a foun- tology. dational ontology for ESHA, although it will require some • Sowa’s Knowledge Representation Ontology (Sowa modification and extension to ensure that it has the relevant 2000) properties of completeness necessary to fulfil the require- ments of a method that can support a SFAIRP safety argu- • Basic Formal Ontology (Smith 2015) ment. • Zemach’s ”Four Ontologies” (Zemach 1970) 5.1 Modifying UFO to Support ESHA • Unified Foundational Ontology (UFO) (Guizzardi 2005) Requirements Sowa’s ontology (Sowa 2000) was the first we reviewed. It Where the UFO ontological model in its most recent incar- blends together several ontological concepts developed by nation UFO-AB (Benevides, Almeida, and Guizzardi 2019) Peirce and Whitehead, and has some interesting features, but has all the properties we require for ESHA, we propose to two significant flaws. First, as noted by (Degen et al. 2001), use it unchanged. Where it does not, we propose to incorpo- it does not draw clear distinctions between sets, universals rate ideas from other models where they appear to be com- and individuals, nor does it clarify the ontological meaning patible, a partial example of which is shown in the model of modal operators used in their definitions. Second, we have fragment in Figure 3. noticed that the type hierarchy contains inheritance errors in ganized material entities as an extension of BFO. This work could be a bridge between the foundational ontology and domain ontology layers, as it could form an abstract catalogue of patterns for combining foundational material entities in any given domain, which can in principle be logically complete. • It may be worthwhile to incorporate some of the work of Bittner on granular partitions, collections, and tempo- ral mereological relations (Bittner, Donnelly, and Smith 2006). Bittner’s work on partitions may also serve as a meta-model for the development of the ontology itself, as it can serve to establish the consistency and completeness of the type hierarchy. Figure 3: Enhancement of UFO model (where necessary) by incorporation of elements from other frameworks The extent to which it is logically permissible to incorpo- rate all these elements in a consistent manner remains to be seen, and will be a challenge for this research programme. Figure 3 shows a partial representation of one particular Where it proves impossible to complete the UFO ontology, modification that we have already identified, using Zemach’s ’gaps’ will remain in the underlying framework, which then model mentioned above. In the original type model devel- affect the capability of environmental survey analyses to oped for UFO, the major sub-types of Individual consisted conclude that a systematic and complete survey has been only of ’Endurant’ (also known as ’Continuant’) and ’Per- achieved. We anticipate (purely as an informal judgement) durant’ (also known as ’Occurrent’), and this was not a log- that it will be possible to develop a complete Foundational ically complete and disjoint type decomposition. By adding Ontology for ESHA, but that incompleteness may begin to two new sub-types ’Pure Continuant’ and ’Pure Occurrent’ ’creep in’ to the model from the Domain Ontology level on- (known as ’Event’ in Zemach’s model) one can develop this ward (as discussed in Section 3), where the relatively ab- into a complete and disjoint decomposition, albeit with some stract types of the foundational layer begin to be applied to underlying logic regarding spatial and temporal bounds, as real-world domains. mentioned in the list below. For example, the formal ontological ’catalogue’ of consti- By this and other modifications, we aim to create a new tutive material entities developed by (Vogt et al. 2011) would ESHA Foundational Ontology as an evolutionary develop- seem to be an extremely powerful development of profound ment of UFO. There are several ’grafts’ to be done to UFO significance to this research, but early indications are that to incorporate all the elements that we anticipate might be for any given domain the catalogue may be extremely large necessary or useful to support an environmental survey haz- due to the permutations of potential combinations of enti- ard analysis, including: ties, and may well be open-ended. (Consider the set of all • The incorporation of Zemach’s model as the major sub- possible roads that can be composed from road environment types of Individuals, and the extension of this model entites such as junctions, straight sections, road bends, etc.) (which was originally conceived as a nominalist model Hence the domain-level catalogue may become impractica- applicable only to Individuals, also known as ’Particu- ble to complete, even though a complete set of individual lars’) into the corresponding Universal types according entities might be identified. to the ontological square, which UFO applies as a meta- But the advantage of attempting to ground the analysis in model governing the form of its ontology. an ontological model means that at least it will be known where the incompleteness exists, and this can be taken into • The incorporation of the concept(s) of topoids, chronoids, account when identifying hazards and specifying safety re- and situoids from GOL (Degen et al. 2001) as the under- quirements for a system. (Effectively, this is a strategy of lying formal framework for the elements from Zemach’s attempting to transform any ”unknown unknowns” of a haz- model. ard analysis into ”known unknowns”, which can then at least • The incorporation of Fiat Boundaries (Smith and Varzi be managed if not resolved.) 2000) into the mereotopology framework of UFO; fiat boundaries are boundaries that are established by an 5.2 Current Experiments and Further Work observer or social convention rather than existing in a We are currently applying the improved ESHA methodology strictly physical sense. This concept may be useful be- to application problems in the fields of assistive robotics for cause changes in agent behaviour can be driven by such healthcare and social care, and also to driver-less road vehi- boundaries (for example a doorway between two rooms, cles (CAVs). or the centre line of a road), and interactions between In the CAV domain, we are using the ESHA technique to agents and the environment may be influenced or even analyse the operating design domain (ODD) of a connected governed by such boundaries. autonomous vehicle (CAV) in order to develop a specifica- • The incorporation of the work of (Vogt et al. 2011), who tion of test scenarios based on a systematic review of the ve- have developed a complete taxonomy of constitutively or- hicle route. We will generate CAV simulator test scenarios and validate coverage of the domain by use of a validation Eliot, C. 2006. System safety and the law. In Proceedings 1st metric called situation coverage, which is a new validation IET International Conference on System Safety, 344–351. metric concept developed recently by (Alexander, Hawkins, London, UK. ISBN 0-86341-646-2. and Rae 2015). Eliot, C. 2007. What is a reasonable argument in law? In Our experiments in assistive robotics applications are Proceedings of 8th GSN User Club Meeting, 344–351. York, aimed at gaining broader experience in application of the UK. ISBN 0-86341-646-2. method, especially since many applications in this field are concerned with human-robot interaction with vulnerable Foretellix. 2020. Open Measureable Scenario Descrip- users with complex care needs that vary over time. This in- tion Language Manual, v20.10 edition. URL https://www. creases the complexity of the required interactive behaviour foretellix.com/open-language/. both for mission and non-mission tasks, and we are investi- Fremont, D. J.; Dreossi, T.; Ghosh, S.; Yue, X.; Sangiovanni- gating the impact of this on the practicability of the method. Vincentelli, A. L.; and Seshia, S. A. 2019. Scenic: A We will then proceed to develop domain-level and Language for Scenario Specification and Scene Generation. application-level ontologies that support two particular do- In Proceedings of the 40th ACM SIGPLAN Conference on mains (assistive medical/social care robots, and driverless PLDI ’19. Phoenix, AZ, USA. doi:https://doi.org/10.1145/ road vehicles), which are active research topics at Bristol 3314221.3314633. Robotics Lab. As each stage of the ontological framework Guizzardi, G. 2005. Ontological Foundations For Structural is completed it will be published as a journal paper, and we Conceptual Models. Ph.D. thesis, Centre for Telematics and plan in the long term to amalgamate all the work into an Information Technology, University of Twente, The Nether- ESHA Handbook, to be made publicly available as a text- lands. book. Harper, C. 2020. Environmental Survey Hazard Analysis: Current Developments. UK Safety Critical Systems Club References seminar, New Safety Analysis Techniques. URL https://scsc. Alexander, R.; Hawkins, R.; and Rae, A. J. 2015. Situa- uk/e654. tion coverage – a coverage criterion for testing autonomous Harper, C.; Dogramadzi, S.; Giannaccini, M. E.; Sobhani, robots. Technical Report Technical Report YCS-2015-496, M.; Woodman, R.; and Choung, J. 2014. Environmental Department of Computer Science, University of York. hazard analysis - a variant of preliminary hazard analysis for autonomous mobile robots. Journal of Intelligent and ANSI/UL. 2020. Standard for Safety for the Evaluation of Robotic Systems 76(1): 73–117. Autonomous Products., ansi/ul standard 4600, 1st edition. HMSO. 1974. The Health and Safety at Work etc Act. URL ASAM. 2020. OpenSCENARIO Manual, v1.0.0 edition. https://www.legislation.gov.uk/ukpga/1974/37/contents/. URL https://www.asam.net/standards/detail/openscenario/. Koopman, P.; and Wagner, M. 2018. Toward a Framework Benevides, A. B.; Almeida, J. P. A.; and Guizzardi, G. for Highly Automated Vehicle Safety Validation. In Pro- 2019. Towards a Unified Theory of Endurants and Per- ceedings of SAE World Congress 2018, 1–13. durants: UFO-AB. In Proceedings FOUST III: Workshop Pfeifer, R.; and Scheier, C. 1999. Understanding Intelli- on Foundational Ontology, The Joint Ontology Workshops gence. MIT Press. ISBN 0-262-16181-8. (JOWO 2019), volume CEUR Workshop Proceedings Vol. 2518. Graz, Austria. URL http://ceur-ws.org/Vol-2518/. Smith, B. 2015. Basic Formal Ontology 2.0: Specification and User’s Guide. URL https://basic-formal-ontology.org/. Bittner, T.; Donnelly, M.; and Smith, B. 2006. A Spatio- Smith, B.; and Varzi, A. C. 2000. Fiat and Bona fide bound- Temporal Ontology for Geographic Information Integration. aries. Philosophy and Phenomenological Research LX(2): International Journal of Geographical Information Science 401–420. 23(6): 1–29. Sowa, J. F. 2000. Knowledge Representation: Logical, Butler, R. W.; and Finelli, G. B. 1991. The Infeasibility of Philosophical, and Computational Foundations. Pacific Experimental Quantification of Life-Critical Software Relia- Grove, CA: Brooks Cole. ISBN 0-534-94965-7. bility. In ACM Software Engineering Notes (Proc. SIGSOFT ‘91 Conf. on Software for Critical Systems), volume 16(5), Vogt, L.; Grobe, P.; Quast, B.; and Bartolomaeus, T. 2011. 66–76. New Orleans. Top-Level Categories of Constitutively Organized Material Entities - Suggestions for a Formal Top-Level Ontology. Caleb-Solly, P. 2020. ERF 2020 – Workshop Report: As- PLoS ONE 6(4): 1–14. suring Safety for Assistive Robotics in Health and So- Zemach, E. 1970. Four Ontologies. Journal of Philosophy cial Care. URL https://www.eu-robotics.net/robotics forum/ 67(8): 231–247. upload/erf2020/presentations/Workshops 04.03.2020.rar. Degen, W.; Heller, B.; Herre, H.; and Smith, B. 2001. GOL: Towards an Axiomatized Upper-Level Ontology. In Pro- ceedings FOIS’01, volume ACM 1-58113-377-4/01/0010. Ogunquit, Maine, USA.