The safety demonstration of Increasingly-Autonomous Cyber-Physical Systems is posing new challenges for the safety community: standards and practices must be adjusted to account for the system's new capabilities and operations. In this position paper, we advocate for consideration of the CPS architecture in both its functional and non-functional dimensions as a cornerstone for safety assessment. We discuss challenges to support our claim. The IJCAI-ECAI-22 Workshop on Artificial Intelligence Safety (AISafety Current safety standards (such as MIL-STD882 for mili*20C2o2r)r,eJsuployn2d4i-n2g5,a2u0t2h2o, rV.ienna, Austria tary systems, ARP4761 for avionics, or ISO26262 for auto† These authors contributed equally. motive) and more generally the existing body of practice $ jhugues@andrew.cmu.edu (J. Hugues); daniela.cancila@cea.fr usually consider faults as a conjunction of basic events, (D. Cancila) attached to their probability of occurrence. This led to 0000-0003-0148-7175 (J. Hugues); 0000-0002-3483-7947 successful applications in the safety-critical industry. Yet, (D. Cancila) © 2022 Copyright 2022 Carnegie Mellon University and Daniela Cancila. This material is based these standards do not apply to IA-CPS, as the following 1u5p-oDn-0w0o0r2kwfuitnhdCedarannedgiseupMpeolrlotendUinnipvaerrtsibtyytfhoer tDheepoaprtemraetniotnofoDftehfeenSsoeftuwnadreerECnognitnreaecrtiNngo.InFAst8it7u0t2e-, premises are invalid or insuficient for complex AI-based aUNfeIdVeEraRllSyITfuYnAdeNdDreSsOeaFrTcWhaAnRdEdEevNeGloIpNmEeEnRtINceGntIeNr.SNTIOTWUTAERRMAANTTEYR.IATHLIISSCFAURRNNEISGHIEEDMOELNLAONN systems: "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, WEIATRHREARNETXYPROEFSFSIETDNEOSRSIFMOPRLIPEUDR,PAOSSTEOOARNMYEMRCAHTATNERTAINBCILLIUTYD,IENXGC, LBUUSTIVNIOTYT,LOIRMRITEESDULTTOS, • A Human operator can act as the final judge to AONBTYAWINAERDRAFRNOTMY UOSFEAONFYTKHIENMDAWTIETRHIARLE.SCPAERCNTETGOIEFMREEELDLOONMUFNRIOVMERPSAITTEYNDTO,ETSRNAODTEMMAARKKE, control the system: the system can reach a safe CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g 4OC.0REICnUOtePrRnYaRtWIiGonHoaTlr(ICkNCFsRBhIYNo4Gp.0E)M.PErNoT.cUeseepdeirnmigttesd(uCndEerUCrRea-tiWve CSo.momrogn)s License Attribution state (space, nuclear, train) or the operator can
ate emerging faulty behaviors with hardware, AI-enabled
functionality, human operators, and the system
architecIncreasingly-Autonomous Cyber-Physical Systems (IA- ture itself as fault sources. Recent incidents involving
CPS for short) are emerging as the natural evolution of CPS at large, e.g. autonomous vehicles, are posing new
embedded real-time systems[
Generally speaking, IA-CPS depend on fault mitigation mechanisms correctly integrated into a functional architecture to fulfill their mission. If not so integrated, safety mechanisms can play an adversarial role and cre
take over close control of the system (aircraft, car). The Three Mile Island accident, known as TMI-2,
ocThis assumption is no longer true for IA-CPS: the curred in a nuclear reactor in 1979 [
• First failure: one safety valve was expected to be opened, it was closed after a maintenance procedure. This error indirectly contributed to a reactor overheating. • Second failure: another safety valve (PORV pilot-operated relief valve) received the “close command” however it remained in the open position. • Software design error: data reported on the operational control monitor were referred to send command status “close the valve” and not on its execution on the system (i.e. actual state of the valve, still opened).
One path to improve the state of practice is to increase
trust in AI functions [
This approach is incomplete. First, the functions of an IA-CPS cannot be fully characterized; second, they operate in an environment that is sampled by a network of sensors, processors, and software functions. Thus, we advocate for an alternative approach: we consider faults in IA-CPS as emergent behavior that a system must resist or eventually control.
In this situation, the emergent behavior arose from a set of events, never even imagined, which have resulted in inconsistent data that have been presented to the operator. And on this information, the operators made choices, which turned out to be incorrect. The design of the TMI-2 Nuclear Power Plant was robust enough and resilient to mitigate the consequences of this unexpected emergent behavior.
Although TMI-2 demonstrates far less autonomy that
3. Emergent Behaviors and modern AI-based systems, it allows us to propose a
charSystem’s Architecture acterisation for the root cause of some emergent
behaviors. Indeed, TMI-2 illustrates that emergent behaviors
Sifakis and al. define the emergent properties of a system arise from the conjunctions of multiple minor events
such as those properties that were not in the original whose confluence creates a major safety hazard.
test specifications [
Emergent properties and emergent behavior are well- ization of an autonomous system from an architectural known phenomena in systems engineering. Designers perspective. An autonomous system is organized around must pay careful attention to emergent behavior to en- five blocks: Perception, Refection, Goal management, sure the correctness of a system, especially in the critical Planning and Self-adaptation. These blocks are orgasystems domain. A capital example of an emergent unde- nized to fulfill a particular mission and may be subject to sired and not specified behavior happened in a Nuclear emergent behaviors.
Power Plant. Reviewing this example is indicative of the
impact of a system architecture to mitigate these emer- 1See Chapter 9 of [
The attention to emergent behaviors requires an even
more exacerbated analysis when we expand the study
from traditional cyber-physical systems, such as a
Nuclear Power Plant [
Let us focus on the origin of emergent behaviors in IACPS. We mentioned we focus on the confluence of minor events that may have a significant safety impact. An IACPS covers multiple domains: control, energy, real-time, vision. First, we segregate them by the two high-level ones: CPS and AI. This leads to three potential sources of emergent behaviors, two of which are well-studied: • emergence in CPS: elements of a CPS architecture may present emergent behavior due to the inherent nature of the interactions between computational (cyber) and physical part, which is surveyed in [15]. • emergence in AI: Similarly, emergence in AI systems is a large topic, under heavy research investigation, e.g. in [16]. • emergence at the AI/CPS boundary: to the best of the authors’ knowledge, this situation is less investigated. However, it is also the source of many emergent behaviors. In particular, typical CPS components may present a threat to AI ones, and vice versa. We review such scenarios in the next section.
We note that each domain developed specific engineering methodologies and tailored safety assessment processes. Several groups analyze emergent behaviors within CPS or AI systems. We claim this line of research must be complemented by research work on the AI/CPS boundary.
and engineering communities, yet their collaboration. We mentioned in the previous section that the coupling between AI and CPS topics be the source of emergent behaviors. We motivate the existence of these emergent behaviors by introducing one support case study (in figure 1) and multiple scenarios.
The robot in figure 1 has a LEN (Lifelong Exploratory Navigation) architecture [17]. LEN is based on a crosslayer architecture from the robot’s sensors to the occupation grid map, to the generalized Voronoi graph (GVG) until to the navigation and exploration algorithms and rising up to the high-level AI, and from here down across the architecture until the actuators. LEN allows robot navigation and space exploration in dynamic and unknown environments.
LEN is developed by CEA. We chose this example, because for some years now, we have seen an industrial trend to the development of mobile robots, with diferent levels of autonomy with/without AI, in diferent civil applications, such as household robots or toys. Compared to the past, where robots were mainly used in industry with little direct contact with the operator, the current type of applications target the masses and have a closer interaction to humans. This trend is expected to grow over the next decades.
To increase the presence of robots near humans, a few guarantees should be met. First, robots have to operate safely and not hurt users. Second, the user should not be required to program, reset or maintain the robot. Third, unlike a factory, the environments in which the robot operates are not controlled in any way by the constructor or the developer. Thus, the robot must adapt itself to its environment, manage its own resources (memory, computing power and battery), and perform its missions.
the context of robotic IA-CPS, these resources are (but not limited to): (1) the use of energy (battery), (2) the extensive use of the processor, and (3) the use of the network to exchange messages with other systems.
Let us be consider the following mission for the LEN robot: to bring an object from location A to location B, without having a predefined cartography of the environment in memory. Moreover, the robot has to recognize a given person to whom to deliver the transported object.
To move from A to B, the robot requires a navigation A system that has been compromised exhibit an inconand exploration system. Such a subsystem may require sistent behavior, akin to an emergent behavior. an excessive use of the processor and then an important In [22], we showed that from a system perspective, amount of energy for its execution. a faulty sensor and an attacked sensor have a similar
Two emergent behaviors may arise from a bad cou- behavior. We propose a review of attacks and faults and pling between AI functions and the supporting CPS plat- how to detect them. In particular, we show that the form. In this case, we show that the CPS architecture architecture of an IA-CPS can be extended with specific may have an adversarial efect on the AI functions. fault or attack detectors to improve the system resilience.
First, the aforementioned subsystem may use so many Furthermore, adding a fault or attack detector may in resources that it interferes with the nominal mission of turn generate new emergent behaviors in the case of false the robot. For example, the subsystem requires and con- positive: the system may overreact to non-existent atsumes the entire battery, making impossible to fulfill tacks. This is a well-known paradox due to fault detectors the mission. This potential conflict situation increases that are not absolute, but rely on some state evaluation. with the deployment of autonomous subsystems in the robot (e.g., the perception of the environment, the recog- 4.3. Conclusive Remarks nition of the person, ensuring the security level of the document). An on-line optimization (i.e., after the de- In this section, we listed some scenarios for emergent velopment of subsystems from diferent teams and their behaviors at the boundary of AI and CPS. We focused deployment in the robot) could involve emergent behav- on non-functional properties and discussed security and iors. Second, the AI or robotics function may induce performance (energy and CPU). This could be extended a significant computation overload. It is well-known to all timing aspects (latency, jitter, scheduling, . . . ), but that the performance evaluation of robotics platform is also safety. a challenge, e.g. [19]. Uncontrolled CPU workload may We note that the increased complexity of those systems trigger timing violations, leading ultimately to errors in calls for an impossible holistic engineering approach: one computations that could impact multiple subsystems. needs to “tame" emergent behaviors that stem from AI,
An approach to overcome the emergent behaviour by and CPS subdomains, but also evaluate the cross-domain guarantying performance of IA-CPS could be twofold. impact of CPS non-functional properties to AI and viceFrom one hand, the design and development of versa. lightweight solutions for autonomous subsystems. For ex- We deem this approach as impossible not from a sciample, LEN uses resources eficiently [ 20]. From the other entific perspective, but from an economic one: we used hand, to understand which solutions can be developed to TMI-2 as an example of a system that mitigated a signifidetect, control and mitigate emergent behaviors, by guar- cant emergent behavior. However, the nuclear industry anteeing the performance of the AI embedded in the sys- can spend more time and efort to improve its design. tem. LEN is built on a cross-layer architecture. This latter This is not possible for general CPS: the time to deliver a includes two macro blocks. Each block has more layers. new product should be reduced.
The low-level macro-block (the one from the sensors to In this context, we consider one should instead focus GVG) implements the safety-related control-command on assuring the system is suficiently resilient rather than and is devoid of AI functionality. The high-level macro safe. block contains Machine Learning-based making-decision and interfaces with other AI-driven functionality of the 5. From Safety assessment to robot. The management of errors and emergent behaviors between layers uses a contract-based approach [17] Resilience assurance (See section 5.2).
4.2. Cybersecurity and Cyber-Physical ses for safety-related properties, we here discuss existing work that is closer to our approach. This literature
Security ranges from traditional analysis techniques, such as FTA We mentioned that IA-CPS are ultimately networked and FMEA, to pattern-based, contract-based, GSN (Goal software-intensive systems. As such, they can be subject Structuring Notation). to cyber-security attacks, i.e. the malicious tampering of An interesting and mature approach is the one based some CPU or software functions. In addition, sensors and on the use of design patters for achieving security or actuators can be subject to additional attacks, leading to safety objectives [23] and for which today we have encyber-physical attacks [21]. In these situations, the sys- couraging and positive feedback. In this context, design tem may no longer fulfill this mission: data are corrupted, patterns provide an architectural description that imthe system reprogrammed, or resource mismanaged, etc. prove the resilience of the system to specific scenarios, combined with fragments of an assurance case. As an an autonomous mobile robot should be resilient to extension, the authors in [24] promote a methodology internal software errors, manage battery properly, that combines patterns with contract-based and GSN - execute a command, detect physical damage of thus obtaining a modular and structural result for safety sensors and control it, etc. argument.
The main benefit of patterns and, more generally, of Exogenous Resilience deals with the system’s extertechniques such as GSN and contracts, is the ability to nal environment in which it is operated., e.g., ensure both safety- and cybersecurity-related proper- avoiding an obstacle or malicious attacks ties [25, 26]. In [27], the authors propose a safety and To ensure Safety-I, traditional techniques (such as redunsecurity co-engineering framework, based on patters of dancy) are dificult to deploy on IA-CPS systems where process and argumentation. the physical space of the product and its final cost are
However, despite the essential help of the above ap- much more limited than traditional systems. Techniques proaches, safety assurance and safety analysis remain on resilience appear more appropriated for this new genvery expensive. They require redundancy, diversity and eration of systems and could be based on the design and independence at software, architectural and physical lev- the control of a modular architecture, having diferent els. Such an option is not always possible in IA-CPS , levels of trust. where the limitations of the physical space and cost constitute stronger constraints for IA-CPS applications than traditional ones. In many cases, heterogeneous architec- 5.1. Conclusive Remarks tures cannot be applied.
To overcome these dificulties, an interesting approach,
albeit not without live debates in the community (see
e.g. [28]), is to widen the safety definition to Safety I and
Safety II [29]. In [30], the author states that “a system
cannot be resilient, but a system can have a potential for
resilient performance". Hollnagel proposes to change the
classical safety analysis process that focuses mainly on
reducing the number of adverse outcomes by taking into
account the success stories that tend to become invisible
and insignificant, because they are considered as
normal, i.e. as planned. Hollnagel introduces the following
definitions [29]:
Unlike traditional critical systems, in IA-CPS systems,
the emergent behavior is expected to increase. It could
arise from how IA-CPS are used and from the
interaction of IA-CPS with its environment. In [
the success of outputs will be as high as possible.
In [28], the author introduces the notion of Safety III we are interested in Trustworthy Artificially Intelligent as follows Adaptive Autonomous CPS. As discussed in the previous sections, these systems can be afected by emergent Safety-III freedom from unacceptable losses as identi- behaviors, particularly if we consider them in dynamic ifed by the system stakeholders. The goal is to and unpredictable environments. In this research coneliminate, mitigate, or control hazards, which are text, we focus on the navigation and exploration system the states that can lead to these losses. and we use LEN [17] as an excellent case study to experiment the achieved scientific results. More precisely, we
Moreover, Lavenson argues the importance of improv- would like firstly to understand what methods and teching and extending Safety-III. We introduce resilience as niques are needed to guarantee and ensure a sustainable a subclass of Safety III. Resilience could allow us to re- and trustable low-level architecture, i.e from sensors to lease some hard constraints related to safety (e.g. redun- the occupancy grid and the Generalized Voronoi Graph dancy, sensor quality) and provide a given level of no (GVG) and then back down to the actuators. In this first longer safety but system resilience. In this regard, we architectural block, the main safety control-commands should distinguish between Exogenous and Endogenous are implemented (including the control of the admissiresilience. ble incertitude). As argued in section 4.1, the design of Endogenous Resilience is the ability of the system to lightweight solutions has a paramount importance to redetect and manage internal faults. For example, duce the risk of uncontrolled emergent behaviours. In
At the SEI, the MBE team is working on the definition of languages and tool-supported processes to engineer safety-critical systems. This encompasses model-based techniques such as the AADL architecture description language along with code generation techniques, safety analysis capabilities. We have developed a collection of techniques to ensure the correctness of code generated from models, along with model checking or Digital Twins capabilities [32]. These provide the foundations to fully analyze a system either analytically or through simulations, with a close link to the engineering models. The SEI is currently engaged in a project to further strengthen the link between MBSE, resilience in the context of IACPS. Most notable, we plan to address solutions to all challenges highlighted in the previous section.
have shown through the TMI-2 nuclear accident, how undesirable emergent behavior can occur in traditional systems. In Increasingly-Autonomous Cyber-Physical Systems, emergent properties are becoming more critical. Unlike traditional application domains, the potential consequences of undesired emergent behavior may be more dificult to mitigate, given the limitation of the physical space and cost. We discussed the emergent properties from three perspective, performance of AI, cyber-security and resilience, and we have briefly illustrated some of the solutions available in the literature. From our analysis of the state of the art and practice, we advocate that the resilience assurance of Increasingly-Autonomous CyberPhysical Systems requires a careful review of the coupling between AI functions and the architecture of the CPS that hosts it.
[15] S. Tyszberowicz, D. Faitelson, Emergence in cyber- [27] H. Martin, R. Bramberger, C. Schmittner, Z. Ma, physical systems: potential and risk, Frontiers of In- T. Gruber, A. Ruiz, G. Macher, Safety and Security formation Technology & Electronic Engineering 21 Co-engineering and Argumentation Framework, in: (2020) 1554–1566. doi:10.1631/FITEE.2000279. Computer Safety, Reliability, and Security, Springer [16] Z. Li, C. Sim, M. Hean Low, A Survey of Emergent International Publishing, 2017, pp. 286–297.
Behavior and Its Impacts in Agent-based Systems, [28] N. Leveson, Safety III: A Systems Approach to in: 2006 IEEE International Conference on Indus- Safety and Resilience, http://sunnyday.mit.edu/ trial Informatics, IEEE, Singapore, 2006, pp. 1295– safety-3.pdf, 2020. 1300. URL: http://ieeexplore.ieee.org/document/ [29] E. Hollnagel, R. Wears, J. Braithwaite, From Safety-I 4053581/. doi:10.1109/INDIN.2006.275846. to Safety-II: A White Paper, 2015. [17] F. Mayran de Chamisso, D. Cancila, L. Soulier, [30] E. Hollnagel, Rag - the resilience analysis grid, R. Passerone, M. Aupetit, Lifelong Exploratory Nav- Resilience engineering in practice: a guidebook. igation: an Architecture for Safer Mobile Robots, Ashgate Publishing Limited, Farnham, Surrey (2011) IEEE Design and Test (2019). 275–296. [18] Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, J. Zhang, [31] R. Bloomfield, G. Fletcher, H. Khlaaf, L. Hinde, Edge intelligence: Paving the last mile of artificial P. Ryan, Safety Case Templates for Autonomous intelligence with edge computing, Proceedings of Systems, CoRR (2021). the IEEE 107 (2019). doi:10.1109/JPROC.2019. [32] J. Hugues, A. Hristosov, J. J. Hudak, J. Yankel, 2918951. Twinops - devops meets model-based engineer[19] T. Kronauer, J. Pohlmann, M. Matthé, T. Sme- ing and digital twins for the engineering of cps, jkal, G. P. Fettweis, Latency overhead of in: Proceedings of the 23rd ACM/IEEE InternaROS2 for modular time-critical systems, CoRR tional Conference on Model Driven Engineering abs/2101.02074 (2021). URL: https://arxiv.org/abs/ Languages and Systems: Companion Proceedings, 2101.02074. arXiv:2101.02074. MODELS ’20, Association for Computing Machin[20] F. Mayran de Chamisso, L. Soulier, M. Aupetit, ery, New York, NY, USA, 2020. doi:10.1145/ Robust topological skeleton extraction from oc- 3417990.3421446. cupancy grids for mobile robot navigation, in: Proceedings of the twentieth national congress on Shape Recognition and Artificial Intelligence (RFIA’16), 2016. [21] A. Humayed, J. Lin, F. Li, B. Luo, Cyber-physical systems security—a survey, IEEE Internet of Things Journal 4 (2017) 1802–1831. doi:10.1109/JIOT.
2017.2703172. [22] L. Zhai, A. Kanellopoulos, F. Fotiadis, K. G.
Vamvoudakis, J. Hugues, Towards intelligent security for unmanned aerial vehicles: A taxonomy of attacks, faults, and detection mechanisms, in: AIAA SCITECH 2022 Forum, 2022. doi:10.2514/ 6.2022-0969. [23] C. Preschern, N. Kajtazovic, A. Höller, C. Kreiner,
Pattern-based safety development methods: overview and comparison, in: EuroPLoP ’14, 2014. [24] I. Sljivo, G. J. Uriagereka, S. Puri, B. Gallina, Guiding assurance of architectural design patterns for critical applications, J. Syst. Archit. 110 (2020) 101765. [25] J. Dobaj, D. Ekert, J. Stolfa, S. Stolfa, G. Macher,
R. Messnarz, Cybersecurity threat analysis, risk assessment and design patterns for automotive networked embedded systems: A case study, JUCS Journal of Universal Computer Science 27 (2021) 830–849. [26] S. Mouelhi, E. Laarouchi, D. Cancila, H. Chaouchi,
Predictive Formal Analysis of Resilience in CyberPhysical Systems, IEEE Access 7 (2019).