Cyber-physical systems are usually subject to assurance and certi cation processes, including thorough requirements engineering tasks, to ensure that they are acceptably dependable. The underlying activities can be complex and labour-intensive, thus practitioners need tools that facilitate them. We present the AMASS Tool Platform as an example of these tools. This Platform is an open source solution that supports the main activities for assurance and certi cation. It also provides advanced features such as argument fragment composition and automated assurance evidence generation and collection. In addition, we present the main insights gained from tool usage. Among them, practitioners expect improvement in relation to usability, performance, and ease of con guration. Videos showing tool usage are available online, including general usage scenarios1.
Cyber-physical systems (CPS), e.g. aircrafts, cars, and trains, are usually subject to rigorous assurance and certi cation processes to provide adequate con dence and evidence that the systems satisfy given requirements and thus are dependable [Nai14], i.e. acceptably safe, secure, reliable, etc. This is typically performed in compliance with standards. For complex systems, the activities are challenging and labour-intensive because of the large set of compliance criteria to ful l, the amount of evidence to manage, and the need for providing valid dependability justi cations, among other issues [dlV16][Nai15]. Therefore, practitioners need adequate tool support for assurance and certi cation. The activities are however usually executed with di erent tools that are not integrated, and most often with basic tools such as Excel that provide very limited speci c support.
The AMASS project (Architecture-driven, Multi-concern and Seamless Assurance and Certi cation of CPS; [Ama20]) provided innovative tool support for assurance and certi cation. AMASS was an industry-academia research project in which 29 partners from eight countries worked on the creation and consolidation of the de-facto European-wide open tool platform, ecosystem, and self-sustainable community for assurance and certi cation.
The ultimate goal of AMASS was to lower certi cation costs for CPS. To this end, a novel holistic approach was de ned for architecture-driven assurance (compatible with standards such as SysML), multi-concern assurance (for co-analysis and co-assurance of e.g. security and safety aspects), seamless interoperability between assurance and engineering activities (including their tool support), and cross- and intra-domain reuse of assurance assets (e.g. of assurance evidence between projects).
The tool that supports the approach is referred to as AMASS Tool Platform. It is an open source solution that has integrated and further developed several existing technologies and toolsets for compliance management, system modelling, process engineering, traceability, variability management, and tool interoperability. The Platform includes support for requirements engineering (RE) tasks such as elicitation, speci cation, and analysis of dependability requirements. It is managed as an Eclipse project [OpC20].
The following sections introduce the requirements, architecture, and implementation of the AMASS Tool Platform and the experience in using it. These aspects have not been introduced in conjunction in any prior publication. Nonetheless, AMASS deliverables [Ama20] include details about them separately. Prior publications also provide more information about the motivation [Rui16] and objectives [dlV19a] of the AMASS project, the process that underlies Platform usage [dlV19b], and the Eclipse open source project [Esp18][Gal19]. Publications on speci c research topics are available, e.g. on artefact quality analysis [Par19]. 2
The AMASS Tool Platform provides a collaborative tool environment that supports the main activities for CPS assurance and certi cation, including activities dealing with product requirements and with process requirements. The general assurance project stages supported, as high-level features, are Standards Compliance De nition, Process Reusability De nition, Assurance Project De nition, System Design Analysis and V&V, Assurance Case Management, and Evidence Management [Ama18b]. Not all stages need to be performed for each project; e.g. the rst two stages are project independent and the outcome could be re-used for multiple projects.
Three categories of user roles exist [Ama18a]: Manager, such as Project, Assurance, and IT Managers; Engineer, such as Development Engineer (including Process Engineer) and Assurance Engineer (including Safety and Security Engineers), and; Assessor, such as Assurance Assessor (including Independent and Internal Assessors).
For Standards Compliance De nition, the Assurance Manager and the Process Engineer capture and retrieve compliance knowledge from standards about requirements and other compliance criteria. The Assurance Manager and the Process and Assurance Engineers participate in Process Reusability De nition to manage reusable compliant process elements. During Assurance Project De nition, the Assurance Manager and the Process Engineer de ne the compliance needs, reuse possibilities, and compliance means for a project. For System Design Analysis and V&V, the Development and the Assurance Engineers elicit and specify system requirements, de ne the system architecture, de ne and validate component contracts, and execute dependability analyses. Assurance Case Management addresses argumentation using compliance and product arguments, resolution of safety-security trade-o s, and the link to system architecture, involving the Assurance Manager and the Process, Assurance, and Development Engineers. The Assurance Manager and the Assurance and Process Engineers perform Evidence Management by collecting and specifying data about project artefacts, traceability, process execution, and compliance.
The analysis and speci cation of these high-level features, users, and work ow was conducted in parallel to the elicitation and documentation of 151 high-level requirements [Ama17] (e.g. \The AMASS Tool Platform shall be able to validate formal requirements" and \The AMASS Tool Platform shall allow an assurance engineer to specify the characteristics of assurance evidence") and 73 use cases [Ama18a] (e.g. \Analyse requirements" and \Characterise evidence artefact") for the main overall functional areas of the Platform: platform infrastructure, architecture-driven assurance, multi-concern assurance, seamless interoperability, and cross- and intra-domain reuse. The requirements and use cases were also grouped by ner-grain architectural building blocks (Section 3).
The logical architecture of the AMASS Tool Platform is referred to as AMASS Reference Tool Architecture (Figure 1; [Ama18a]). The Architecture provides a conceptual framework for architecture-driven assurance, multi-concern assurance, seamless interoperability, and cross- and intra-domain reuse of assurance assets. It contains both technological building blocks, e.g. System architecture modelling for assurance, and the Common Assurance & Certi cation Metamodel. The metamodel provides an information model for CPS assurance and certi cation, e.g. for Compliance management and for Assurance case speci cation. Basic application services such as Access and Data management are also considered. The main stakeholder groups of the architecture are Manufacturer, Supplier, Assessor & Authorities, and Tool Vendors.
All the building blocks relate to RE, as CPS assurance and certi cation deals with the determination of dependability requirements for the systems, both for products and for processes and both from standards and from system analyses, as well as with the justi cation of requirement satisfaction. The most RE-focused building block is arguably Requirements support, which addresses requirements derivation from dependability analyses, formalisation, quality analysis, and veri cation.
The AMASS Tool Platform is a concrete implementation of the AMASS Reference Tool Architecture with capability for evolution and adaptation (Figure 2). Eclipse is the main environment for Platform usage, but web-based support also exists for some functionality. The Platform has integrated and further developed: OpenCert for compliance management, evidence management, assurance case speci cation, and dependability assurance modelling.
Papyrus and CHESS for system modelling, system analysis, contract modelling, contract-based multiconcern assurance, contract-based trade-o analysis, and design veri cation.
An integrated usage work ow could be as follows. OpenCert would be used to specify and analyse the compliance criteria for a project such as the requirements to satisfy from applicable standards, and EPF-Composer to de ne a project-speci c assurance process according to process requirements. BVR would help a user study their variants, e.g. for compliance. Papyrus and CHESS would then support system modelling and automated analysis, including requirements needs. The results could be traced with Capra and used in evidence management. If data had to be imported from or exported to an external tool, OSLC would enable it. Finally, assurance cases could be managed throughout the work ow to justify requirement satisfaction, updating them according to design decisions and the available evidence, and generating fragments.
In addition, the AMASS Tool Platform interacts with over a dozen tools that provide additional features, typically commercial ones, e.g.:
The resulting open source ecosystem and community are managed as an Eclipse project [OpC20]. The community deals with the maintenance, evolution, and industrialization of the AMASS Tool Platform and is supported by a governance board and by rules, policies, and quality models. Last but not least, the documentation of the AMASS Tool Platform includes a detailed developers' guide [Ama18b] for those interested in implementing new functionality for or on top of the Platform.
The main experience and lessons learned from using the AMASS Tool Platform are a result of three di erent activities during the AMASS project.
Validation [Ama18c][Ama19c]. Three versions of the Platform were released during the AMASS project. Each version underwent validation tasks. The execution of 141 test cases con rmed the implementation of 93% of the high-level requirements speci ed for the Platform. The usability analysis results suggest that EPF-Composer and Papyrus provide a good user experience and that the rest of tools have larger potential for improvement. The documentation of the Platform could also be enhanced, as inconsistency was found because of insu cient homogeneity in the documentation style by all the contributors and for all the features. Finally, it appeared that the main data storage technology could impact performance under certain con gurations.
Application [Ama19a]. The AMASS Tool Platform was used in 11 industrial case studies from air tra c management, automotive, avionics, industrial automation, railway, and space. Each case study selected a subset of the functionality of the AMASS Tool Platform for its application, and each piece of functionality was applied in at least one case study. For example, requirements were modelled, analysed, and veri ed and their satisfaction was justi ed, among other activities, in the scope of safety assessment of multi-modal interactions in cockpits for the avionics domain. Practitioners in the AMASS consortium reported achievements, bene ts, improvement opportunities, and recommendations from the application of the Platform. Suggestions were made on user interaction (e.g. to further guide the users), the value of the new features was stated (e.g. modelling of standards), and easier con guration was expected for the Platform. It is a large tool with many sub-tools, but an organisation would typically be interested only in a subset of the functionality. The selection, con guration, and tailoring of the subset could be better supported, e.g. with a dashboard for feature selection.
Benchmarking [Ama19b]. The industrial case studies were used to compare how assurance and certi cation could be executed with the Platform and how they were executed before. Quantitative evaluations were performed to study the reduction of e ort in assurance and certi cation (initial target: 50%), reduction of risks (35%), reduction of costs in (re)certi cation (40%), and increase in technology harmonisation and interoperability (60%). These goals were achieved in general, but to a varying extent among the case studies. The use of di erent features is one of the reasons.
The above-mentioned activities also allowed the AMASS consortium to estimate the Technology Readiness Level of its components [Ama19c]. Such a level di ers among them. EPF-Composer and Papyrus can be regarded as the most mature technologies. Tool quali cation considerations of the AMASS Tool Platform have also been analysed [Ama19c]. Although the Platform itself cannot directly introduce errors in a system, the speci c aspects to consider will depend on how the Platform is used, e.g. regarding the veri cation of its automatic actions and the toolchain deployed. 5
The AMASS project developed innovative tool support for CPS assurance and certi cation, and thus for certain RE needs of CPS, focusing on architecture-driven assurance, multi-concern assurance, seamless interoperability, and cross- and intra-domain reuse of assurance assets. The resulting AMASS Tool Platform is an open source solution that facilitates system modelling and analysis, compliance management, argumentation, process engineering, variability management, and traceability. The Platform is also integrated with external tools for additional features, e.g. for requirements quality analysis. Although the Platform needs to further mature, we argue that its nalisation and its public release are initial, major milestones. It is the rst integrated environment for assurance and certi cation and bene ts from its use have been demonstrated.
The development of the AMASS Tool Platform focused on supporting assurance and certi cation for CPS, but the Platform can be used for any system or project having to deal with e.g. system modelling or compliance. Nonetheless, Platform usage would in principle have to be tailored.
We plan to continue working on the development of tool support that improves assurance and certi cation. This includes the development of novel solutions for traceability, assurance case management, and privacy assurance.
Acknowledgements The research leading to this paper has received funding from the AMASS (H2020-ECSEL grant agreement no 692474), iRel4.0 (H2020-ECSEL grant agreement no 876659), VALU3S (H2020-ECSEL grant agreement no [Ama17] AMASS Project: Deliverable 2.1 - Business cases and high-level requirements. 2017. [Ama18a] AMASS Project: Deliverable 2.4 - AMASS reference architecture (c). 2018. [Ama18b] AMASS Project: Deliverable 2.5 - AMASS user guidance and methodological framework. 2018. [Ama18c] AMASS Project: Deliverable 2.8 - Integrated AMASS platform (c). 2018. [Ama18d] AMASS Project: Deliverable 4.6 - Prototype for multi-concern assurance (c). 2018. [Ama19a] AMASS Project. Deliverable 1.6 - AMASS demonstrators (c). 2019. [Ama19b] AMASS Project: Deliverable 1.7 - AMASS solution benchmarking. 2019. [Ama19c] AMASS Project: Deliverable 2.9 - AMASS platform validation. 2019. [Ama20] AMASS Project (online) https://www.amass-ecsel.eu/ (Accessed Feb 17, 2020)
[OCR20] OCRA (online) https://ocra.fbk.eu/ (Accessed Feb 17, 2020) [OpC20] OpenCert (online) https://www.polarsys.org/opencert/ (Accessed Feb 17, 2020) [Par19] E. Parra, et al. Advances in Artefact Quality Analysis for Critical Systems. ISSRE 2019. [RiO20] RiskOversee (online) https://www.riskoversee.com/en/home/ (Accessed Feb 17, 2020)