This paper introduces the concept of transfer assurance for Machine Learning (ML) components used as part of an autonomous system (AS). In previous work we developed the first approach for assuring the safety of ML components such that a compelling safety case can be created for their safe deployment. During operation it may be necessary to update an ML component by re-training the model using new or updated development data. If model re-training is required post-deployment, the safety case that was created for the ML component may no longer be valid, since a new model has been created that can no longer be assured to meet its safety requirements. In particular, the nature of machine learnt components means that one may not be able to predict how even small changes in the development data may afect the model and its performance. As a result, current practice would require that a full assurance assessment is undertaken for the re-learned model, and that a new safety case is created. Given the desirability of updating ML components during operation, we see it as imperative that the assurance process become more proportionate to the size of the change that is made to the model, whilst ensuring that assurance can still be demonstrated. Retraining ML components is known to be a costly and complex process and as such techniques such as transfer learning have been developed which aim to reduce this burden through incremental development. Approaches such as transfer learning provide an inspiration for how the challenge of eficiently assuring updated models could be addressed through understanding which aspects of a model may have been afected by changes to the development data. We refer to this as transfer assurance, where parts of the assurance case for an ML component can remain fixed whilst other parts are re-assessed.
nent is based upon predictions and assumptions made during development about the system and the environThe use of ML models promises to revolutionise a number ment in which the ML component will be used. These of societally-significant, safety-critical domains includ- predictions and assumptions may turn out during operaing healthcare, transport and defence [1, 2, 3]. Whilst tion of the system not to be true, or may become untrue such systems may be reported to exceed human perfor- as things change in unexpected ways. When this hapmance [4, 5], their adoption is dependent on establishing pens the safety case created during development may justified confidence in the safety of systems not only at no longer be valid. It is necessary therefore to identify development time but also once deployed in complex when such changes occur and what the impact of those open-world environments. changes may be on the ML component and therefore on
In our previous work we developed a methodology, the safety of the system. called AMLAS (Assurance of Machine Learning for use During operation it may be necessary to update an in Autonomous Systems) that systematically integrates ML component by re-training the model using new or safety assurance into the development of ML components updated development data. There are a number of reain order to generate a safety case to demonstrate the ML sons why it may be desirable to update a learned model component is safe to use in a particular autonomous after deployment. For example, as more data becomes system (AS) application [6]. Through evaluative stud- available during operation of the AS, it may be possible ies [7, 8, 9, 10] it has been shown that through following to use that data to improve the performance of the model. AMLAS it is possible to produce an assurance case which It may also be the case that during operation the operis valid for deployment. ational inputs to the ML model are observed to diverge The assurance case that is created for the ML compo- from the development data used in the model learning process. *SaCfoeArrIe,sFpeobnrudainryg a13u–th1o4,r.2023, Washington, DC Such shifts in the distribution of the operational data $ chiara.picardi@york.ac.uk (C. Picardi); with respect to the development data may require that richard.hawkins@york.ac.uk (R. Hawkins); the model is re-trained with data sets that more closely colin.paterson@york.ac.uk (C. Paterson); ibrahim.habli@york.ac.uk reflect the current operational data in order to ensure (I. Habli) that the model continues to perform as required in the operational context [11]. A further consideration is that the © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) 2. Related Work safety case for the ML component is based upon predictions and assumptions about the environment in which the ML component will be used. Due to the complex- 2.1. Model Updating ity and dynamicity of the operating environment of the autonomous system, there is the possibility that the pre- The need to update models at run-time is well underdictions and assumptions may be incorrect, or become stood, however model updating presents a number of invalid as the environment evolves in unexpected ways challenges [13, 14] including: [12]. Again this may require model re-training to ensure 1. deciding when a model is no longer valid the assurance case remains valid. 2. suitable mechanisms for retraining
In each of these cases, if the model is re-trained post- Model retraining can be scheduled to occur periodideployment, the safety case that was created for the ML cally when the environment in which the system opercomponent may no longer be valid, since a new model ates can be assumed to be stable for a fixed period of has been created that can no longer be assured to meet its time. However when the environment is less predictable, safety requirements. In particular, the nature of machine mechanisms need to be employed which are able to detect learnt components means that one may not be able to the potential for a deterioration in system performance predict how even small changes in the development data through the monitoring of data to identify distribution may afect the model and its performance. As a result, shifts [15]. In order to detect shifts in data distributions current practice would require that a full assurance as- we can monitor the performance of the model, when the sessment is undertaken for the re-learned model, and that ground truth is known, or the input/output distributions a new safety case is created. This would involve applying in order to detect any significant changes [ 16] which may the whole of the AMLAS process again following any invalidate assumptions made at design time. change to the ML component. Given the desirability of Once the need to retrain has been identified we need updating ML components during operation, we see it as to establish how to undertake such retraining. Two apimperative that the assurance process become more pro- proaches are proposed in the literature: portionate to the size of the change that is made, whilst ensuring that assurance can still be demonstrated. 1. ofline using batches of data [13], or
Retraining ML models is known to be a costly and com- 2. online, continuously updating the model in opplex process and as such techniques have been developed eration [17]. which aim to reduce this burden through incremental de- In both cases the dataset used for retraining contains both velopment. Transfer learning is one such technique that new and historical data in order to avoid catastrophic enables large parts of a convolutional neural network forgetting [18]. In this paper we consider only ofline (CNN) to remain fixed during re-learning, whilst some retraining methods. of the layers are learned using the updated development data. Approaches such as transfer learning provide an inspiration for how the challenge of eficiently assuring 2.2. AMLAS updated models could be addressed through understand- AMLAS is a methodology for the assurance of machine ing which aspects of a model may have been afected learning in autonomous systems [6]. AMLAS provides by changes to the development data. We refer to this as a set of defined processes that describe activities that transfer assurance, where parts of the assurance case for should be undertaken as well as the artefacts that are an ML component can remain fixed whilst other parts are used by, or generated from those activities. AMLAS also re-assessed. This paper conceptualises transfer assurance provide a set of safety argument patterns that describe and looks at how a transfer assurance approach can be how these artefacts are used to create a compelling safety used to re-assure the ML model and update the safety case for the ML component. AMLAS is split into six lifecase for the ML model without having to apply the whole cycle stages as shown in Figure 1. The ML assurance AMLAS process. activities run in parallel to the ML development activi
The rest of the paper is structured as follows. In Sec- ties; the assurance activities defined in AMLAS should be tion 2 we discuss related research. Section 3 introduces an integral part of developing an ML component. Feedthe transfer assurance process that we propose. Sec- back and iteration between the stages of AMLAS is also tion 4 discusses some of the key open challenges related important. It is expected that it might be necessary to to transfer assurance that we aim to address. Section 5 revisit stages of AMLAS multiple times. For example, the presents our conclusions. verification stage might identify a need to revisit data management and then learn a new model, and so on.
Figure 1 also shows the input to the AMLAS process from the system safety requirements. Safety considerations for the ML component are only meaningful when
M LM LCoCmompopnoennenttDDeevveellooppmmeenntt Act ivit ies scoped within the wider system and operational context.
The system safety requirements provide that crucial link between AMLAS and the system safety process. Having followed the AMLAS process the outputs from each stage can be integrated together to form an overall safety case for the ML component. [9] provides an example how AMLAS has been used to create a safety case for a neural network used to detect wildfires. AMLAS can be used as mean for compliance with standards as part of emerging regulatory frameworks [19] [20].
There has been some previous work that has investigated run-time assurance of ML systems ([21], [22]), however these typically focus on particular applications such as reinforcement learning for self-driving cars [23] or ML for robotic systems [24]. The approach proposed in [25] uses monitors to determine if safety critical requirements are violated by ML components at run-time and provides feedback to improve the models.
None of this work considers how an assurance case is impacted by changes at run-time or how the validity of the assurance case is maintained. Other work [12] has highlighted the need to dynamically update safety cases in operation, but has not considered how this could be done for ML components.
transfer assurance process shown in Figure 2.
The first stage concerns the initial development of the ML component for deployment into the operational system. The two activities result in the creation of an ML component along with its assurance case and a set of appropriate monitors to be deployed to the autonomous system.
The second and the third phases deal with analysing and responding to changes during operation of the autonomous system. These phases will be executed multiple times during the lifetime of the system as and when the monitor issues a response trigger. In the following subsections we explain in detail each of the proposed stages.
generating two artefacts as well as the ML component and a set of monitors.
The first of the activities creates the ML component and the associated assurance case using the AMLAS process [6]. The resulting assurance case report is used in the second activity in order to create a component monitor. The monitor is created by considering the evidence, assumption, context and justification elements present in the assurance case to identify those environmental and system conditions that, if they were to change, would invalidate the safety argument. The data to be monitored may be concerned with environmental changes outside the autonomous system, changes in physical or software components which constitute the autonomous system platform or in the behaviour of one or more ML components enacting autonomous behaviours.
Thus a monitoring strategy is created which considers how the nature of potential changes in operational data will impact the assurance case to undermine safety assurances. The strategy then defines what data is to be monitored, the nature of the sensors to be used, and the analytical basis upon which a response trigger should be generated.
As an example let us consider an object classification component which is deemed acceptable in the assurance safety case under the assumption that the risk to be controlled is a function of the impact caused by misclassiifcation and likelihood of encounter. Verification of the 1. Initial Development 1 component shows that an accuracy of 0.87 and assump- sensor is correlated since it refers to the same physical tions of the operating domain determine a likelihood of objects in space. From development activities we will encounter rate of one per 10 hours of operation. A moni- have extracted expectations for the distributions of comtor for likelihood may then be created which counts the ponent outputs for each of the 6 channels in operation. number of class predictions such that a trigger is raised For example when Channels 1, 3 and 4 predict outcome A should we identify that there has been a distribution shift then with probability 0.98 Channel 2 will agree. Through in the probability of class encounter. monitoring the combinatorial behaviour characteristics
We can also consider the example of an ML model for of sensors we may be able to identify distribution shifts speech recognition trained on adult voices. The assur- in the data which compromise safety at run-time. ance argument will have a context element defining the Whilst components are typically quoted with point valtraining data as adult voices. This can then be used to ues for expected performance (e.g. accuracy, precision) identify the requirement for a monitor to recognise if the perturbations encountered at run-time can significantly input to the model is not an adult voice, as it cannot be impact ML performance [27]. Understanding the link assured that the model will meet its safety requirements between real world phenomena and component perforwhen exposed to non-adult voices as input. mance allow for the creation of sensors which monitor
There are two main problems with creating monitors such phenomena and allow for the validation of operatfor ML systems during operation. Firstly it is dificult ing assumptions at run-time. Consider for example an to detect distribution shifts in complex inputs such as image classifier utilised to identify road signs in an auimages. Secondly it is often dificult to monitor the cor- tonomous vehicle. Experimental results at development rectness of the output of the model due to unavailability time may provide a performance profile with respect to of ground truth. We can overcome these problems using the amount of fog, or mist, in the air [28]. A simple moissensor fusion, contextual information and established ture monitor deployed at run-time may be employed to beliefs. In the rest of the subsection we illustrates some update the anticipated accuracy of the sensor and to say methods using examples. when the sensor falls outside it’s safe operating mode,
Typically autonomous systems make use of multiple hence triggering a redevelopment of the component and sensors during normal operation and we may be able to the associated assurance case. use these to either directly or indirectly infer properties The output of an ML component can also be monitored of interest. Specifically sensor agreement and compari- using contextual information combined with knowledge son can be used to check the validity of a sensor readings about the physical world in order to estimate ground at run-time. For example we may consider a drone which truth. For example, in a self-driving car an object detecutilises a 6 channel multi-spectral camera for agricultural tion component identifies pedestrians with an associated monitoring [26]. Whilst each channel may have inde- bounding box which is used to estimate the position and pendent analysis pipelines, using ML components with size of the pedestrian. This estimate is updated as each diferent functional requirements, the output from each frame of the video is presented to the component and the movement of the box reflects behaviours in the real- an understanding of the reason that the monitor was world which must be consistent with physical laws. For created (informed by the monitoring strategy justificaexample the pedestrian may not ’disappear’ or suddenly tion), as well as of the safety case itself. The assessment ’relocate’ to impractical areas of the scene. When such therefore relies upon human expertise and judgement. changes are detected this is indicative of a failure of the The change impact report should therefore be used to sensing pipeline. In this way data consistency may be document how the impact of the change was determined. used to estimate false positives, area of union or mean Where the change has an impact on the safety case it average precision at run-time. is necessary to determine an update strategy that will
Another way to estimate the ground truth is combining ensure a valid safety case can be maintained. It may sensor predictions with contextual meaningful informa- be that the impact on the safety case can be mitigated tion. As an example let’s consider a self-driving car with without requiring an update to the model itself. For exan assumption in the assurance case asserting that the ample, system level mitigation may be possible, such as frequency of false positive on speed limit signals shall by introducing additional operating restrictions on the be less than a certain threshold. On a motorway in the autonomous system. These restrictions would serve to UK a vehicle is unlikely to encounter a 30 mph speed ensure that, for example the system was not exposed sign. Using a GPS unit on the car confirms the position to the situations that triggered the change response. In and ,combining this with the contextual information, we such cases the scope of the safety case can be updated to can determine that if a 30 miles per hour speed limit is reflect the change in operation. detected this represents a false positive. Where it is determined that model retraining is re
In order to assess safety during operations usually the quired in order to mitigate the impact of the change on precursor events or leading indicators (i.e. events likely the safety case, an appropriate update strategy must be to happen before an accident) are monitored [29]. In determined. It is desirable to create an update strategy other words monitoring the output of the system detect- that limits the amount of costly and time consuming ing precursor event can be a good indication that the ML assurance rework that is required (such as a need to recomponent is not safe as required. As an example we apply the whole of the AMLAS process). In the transfer could monitor the frequency of emergency stops contex- assurance approach we propose that the model retraintually knowing that these should not happen frequently ing strategy and the assurance strategy should be closely or in self-driving cars we could easily monitor the dis- linked, such that the model retraining strategy enables a engagement rate knowing that if the car hand-over the reduction in the assurance efort required to re-establish control to the human driver too often then something is a valid safety case. not working as it should. When a precursor event happens frequently, the development team should perform a 3.3. Redevelopment casual analysis to understand where the problem could be in the pipeline.
its associated safety case in line with the recommenda3.2. Change Analysis tions from the previous stage.
While the majority of ML learning is stateless [15], The change analysis stage seeks to evaluate the impact i.e. the model is re-learnt from scratch each time, such of any change that activates a response trigger. The re- approaches result in “catastrophic forgetting" where all sponse trigger may be flagged by a monitor, or be initiated previously learnt knowledge is discarded. Given the high by an ML developer who has identified a desire to change cost of building models this is undesirable and where the model (for example to improve model performance knowledge remains valid across model iterations it is during operation). Once the change is triggered, the desirable to retain knowledge encoded in the model. In impact of the change is analysed to determine whether an attempt to achieve this techniques, such as transfer updates to the model, or to the safety case, or to both learning [30], have been developed which allow for apare required. Where an update is required, a strategy for propriate features of a model to be retained whilst also doing so in an efective manner is determined. allowing for new features to be learnt. One benefit of
Changes will generate a response trigger if they have transfer learning is to reduce the amount of data required been determined to be potentially important with respect for model development as well as development costs. to the validity of the safety case. When a response trig- Undertaking the AMLAS process from scratch each ger is activated during the operation of the autonomous cycle is also to be avoided if possible and, where we can system, it is necessary to assess the nature and extent of control the scope of retraining, we may also reduce the the impact the change has on the safety case. It should be assurance efort. For example, the identification of a new noted that the impact assessment activity is not expected subclass may lead us to gather a small set of data and relato be an automated process. The assessment will require bel some existing data. These activities will require us to justify the accuracy of data labelling and the collection of world applications and that research in this area is well appropriate data, however undertaking new assurance ac- underway. We do not, at present however, have a strategy tivities associated with historic data or “frozen” portions for partial assurance of incrementally learnt models. We of the model may be unnecessary. intend to address this by building on the work of AMLAS
A transfer assurance approach would allows us to re- to construct processes and safety patterns to guide the consider only a subset of the AMLAS activities originally assurance of such models. undertaken in stage 1 of the proposed process. The up- In future work we intend to address these challenges dated safety case resulting from the partial reapplication with application to a range of real world problems where of AMLAS will, however, introduce new evidence and we will consider: the needs of diferent operating conmay modify assumptions, contexts and justification ele- texts including autonomous driving and healthcare; a ments. These changes may necessitate the updating of range of model forms including neural networks and existing monitors as well as the creation of new monitors reinforcement learning policies; and diferent modes of for deployment. In addition each modified monitor will change including evolutionary drift and step changes in have an associated monitoring strategy justification doc- the environment. ument which will inform subsequent impact assessment activities.
Once the new model and associated monitors have 5. Conclusions been deployed we return to operation and await a new response trigger to undertake stages 2 and 3 once more.
update, simultaneously, an ML model integrated into an autonomous system and its assurance case as defined 4. Addressing Open Challenges prior deployment. Such an approach is necessary due to the evolutionary nature of the open contexts into which In order to deliver transfer assurance we must address autonomous systems are being deployed and a desire to a number of open challenges which will be the focus of ensure safety through life. future work. We have presented a number of examples of how the
In stage 1 of the proposed approach we suggest a num- approach may be practically achieved and highlighted ber of techniques which may be employed to monitor open challenges and opportunities for future work. We environments for distributional shifts which invalidate believe that transfer assurance promises to allow for reassurance claims. In open environments however such duced development and assurance costs, more responsive changes are combinatorial in nature and identifying im- deployment cycles and ultimately safer autonomous syspact factors in high dimensional space remains challeng- tems in real-world contexts. ing. Being able to identify small regions of the input space will however limit the need for new data to be generated for training and potentially allow for larger regions of 6. Acknowledgements the machine learnt component to be retained between redevelopment cycles. In addition an understanding of This work was supported by the Engineering and Physithe factor interactions in decision space will improve the cal Sciences Research Council through the RAILS project specification of monitoring strategies,reduce the num- (EP/W011344/1) and the Assuring Autonomy Internaber of response triggers and potentially reduce the time tional Programme, a partnership between Lloyd’s Registaken for developing assurance cases. ter Foundation and the University of York.
In stage 2 of the process we proposed the development of update strategies through an analysis of the trigger References response and monitoring strategy justification document.
There exists no guidance at present concerning how trade [1] J. De Fauw, et al., Clinically applicable deep learning ofs and design decisions should be derived. For example for diagnosis and referral in retinal disease, Nature we may become aware that the performance of a com- medicine 24 (2018) 1342–1350. ponent with respect to a subclass is insuficient but it is [2] S. Grigorescu, et al., A survey of deep learning unclear at present as to whether one is better to a) gather techniques for autonomous driving, Journal of Field new data b) label sub classes or c) restrict the scope of Robotics 37 (2020) 362–386. assurance. Each option has costs and benefits and guid- [3] S. A. Azizan, I. A. Aziz, Terrorism detection based ance on how to decide the correct course of action in a on sentiment analysis using machine learning, Jourcontextually relevant manner is needed. nal of Engineering and Applied Sciences 12 (2017)
In stage 3 we recognise that continual learning and 691–698. transfer learning strategies are in sight for many real [4] M. Nagendran, et al., Artificial intelligence versus cessed: 2022-12-12.
clinicians: systematic review of design, reporting [21] D. Cofer, et al., Run-time assurance for learningstandards, and claims of deep learning studies, bmj enabled systems, in: NASA Formal Methods Sym368 (2020). posium, Springer, 2020, pp. 361–368. [5] C.-J. Farrell, Identifying mislabelled samples: ma- [22] S. Chen, et al., Runtime safety assurance for chine learning models exceed human performance, learning-enabled control of autonomous driving Annals of Clinical Biochemistry 58 (2021) 650–652. vehicles, in: 2022 Int. Conf. on Robotics and Au[6] R. Hawkins, et al., Guidance on the assurance of tomation (ICRA), IEEE, 2022, pp. 8978–8984. machine learning in autonomous systems (AMLAS), [23] P. Musau, et al., On using real-time reachability arXiv preprint arXiv:2102.01564 (2021). for the safety assurance of machine learning con[7] S. Laher, et al., Review of the AMLAS methodol- trollers, in: 2022 IEEE Int. Conf. on Assured Autonogy for application in healthcare, arXiv preprint omy (ICAA), IEEE, 2022, pp. 1–10. arXiv:2209.00421 (2022). [24] A. Desai, et al., Soter: a runtime assurance frame[8] J. Hamnert, D. Hägglund, How safe is machine vi- work for programming safe robotics systems, in: sion?: An evaluation of the AMLAS process in a 2019 49th Annual IEEE/IFIP Int. Conf. on Dependmachine vision environment, 2022. able Systems and Networks (DSN), IEEE, 2019, pp. [9] R. Hawkins, et al., Creating a safety assurance 138–150.
case for an ML satellite-based wildfire detection [25] P. Mallozzi, Combining machine-learning with inand alert system, arXiv preprint arXiv:2211.04530 variants assurance techniques for autonomous sys(2022). tems, in: 2017 IEEE/ACM 39th Int. Conf. on Soft[10] P. Festor, et al., Assuring the safety of ai-based ware Engineering Companion (ICSE-C), IEEE, 2017, clinical decision support systems: a case study of pp. 485–486. the ai clinician for sepsis treatment, BMJ Health & [26] J. Franke, G. Menz, Multi-temporal wheat disease Care Informatics 29 (2022) e100549. detection by multi-spectral remote sensing, Preci[11] S. G. Finlayson, et al., The clinician and dataset sion Agriculture 8 (2007) 161–172. shift in artificial intelligence, The New England [27] J. Mohapatra, et al., Towards verifying robustness journal of medicine 385 (2021) 283. of neural networks against a family of semantic [12] E. Denney, G. Pai, I. Habli, Dynamic safety cases for perturbations, in: Proceedings of the IEEE/CVF through-life safety assurance, in: 2015 IEEE/ACM Conf. on Computer Vision and Pattern Recognition, 37th IEEE Int. Conf. on Software Engineering, vol- 2020, pp. 244–252.
ume 2, IEEE, 2015, pp. 587–590. [28] C. Paterson, et al., Deepcert: Verification of con[13] D. Klabjan, X. Zhu, Neural network retraining for textually relevant robustness for neural network model serving, arXiv preprint arXiv:2004.14203 image classifiers, in: Int. Conf. on Computer Safety, (2020). Reliability, and Security, Springer, 2021, pp. 3–17. [14] A. Paleyes, R.-G. Urma, N. D. Lawrence, Challenges [29] N. Leveson, A systems approach to risk managein deploying machine learning: a survey of case ment through leading safety indicators, Reliability studies, ACM Computing Surveys (CSUR) (2020). engineering & system safety 136 (2015) 17–34. [15] C. Huyen, Designing Machine Learning Systems, " [30] C. Tan, et al., A survey on deep transfer learning,
O’Reilly Media, Inc.", 2022. in: Int. Conf. on artificial neural networks, Springer, [16] J. Klaise, et al., Monitoring and explainabil- 2018, pp. 270–279.
ity of models in production, arXiv preprint arXiv:2007.06299 (2020). [17] I. Prapas, et al., Continuous training and deployment of deep learning models, Datenbank
Spektrum 21 (2021) 203–212. [18] R. M. French, Catastrophic forgetting in connectionist networks, Trends in cognitive sciences 3 (1999) 128–135. [19] Artificial intelligence/machine learning (ai/ml)based software as a medical device (samd) action plan, https://www.fda.gov/media/145022/ download, 2021. Accessed: 2022-12-12. [20] EASA concept paper: First usable guidance for level 1 machine learning applications, https://www. easa.europa.eu/en/downloads/134357/en, 2021. Ac