With the use of AI systems growing, particularly in areas that can significantly efect people, trustworthy AI is needed. For this, transparent and accountable trustworthiness risk assessment of AI systems is required. The work here details an initial approach to trustworthiness risk assessment of AI systems that builds on a semi-automated risk assessment tool called Spyderisk and uses information from frameworks and other sources to guide it. AI systems are becoming increasingly prevalent, with increased adoption and use across industries informing or making decisions that afect more and more people in impactful ways. Diferent stakeholders of AI systems, as well as the people they afect, need to be able to trust them to behave in a correct, fair, and appropriate manner. That is, AI systems need to be trustworthy, where risks in the AI system, and in the wider system it is deployed, must be understood so that unacceptable risks can be managed and mitigated, leaving acceptable residual risks. This process must be transparent so that diferent stakeholders across the AI lifecycle can understand the risks and mitigations in place. For this, risk assessment is needed. The aim of the work here is to introduce an approach being developed for the semi-automated trustworthiness risk assessment of AI systems. This approach enhances a preexisting knowledge-based semi-automatic risk assessment tool developed by the University of Southampton, Spyderisk [1, 2], to expand its knowledge base from the cyber-physical domain to also include AI systems. The approach taken here is to use sources of information such as AI risk management and best practices frameworks. This then improves the adoptability and enhancability of this approach whilst also giving it a rigorous foundation. The ENISA Multilayer Framework for Good Cybersecurity Practices (FAICP) [3] and the NIST Artificial Intelligence Risk Management Framework (AI RMF) [ 4] are considered and discussed in the following subsection. Then Spyderisk is introduced. After, a mapping between these frameworks and Spyderisk is established. Finally, initial AI domain Spyderisk knowledge extensions are detailed.
1.1. The ENISA FAICP and NIST AI RMF
The FAICP [
Within it, AI systems are defined to have desired characteristics that contribute to the trustworthiness of the AI system. This AI trustworthiness is “the confidence that AI systems will behave within specified norms, as a function of some characteristics [. . . ]”. The diferent characteristics of AI trustworthiness considered are: Accountability, Accuracy, Explainability, Fairness, Privacy, Reliability, Resiliency, Robustness, Safety, Security, and Transparency. These AI system characteristics are classified as technical, socio-technical, and guiding principles.
The FAICP recommends the ISO 2700x standards [
The NIST AI RMF [
The AI RMF identifies a set of trustworthiness characteristics that address technical, socio-technical and ethical / legal aspects of the AI system. These are Safe, Secure and Resilient, Explainable and Interpretable, Privacy-Enhanced, Fair - With Harmful Bias Managed, Valid and Reliable, and Accountable and Transparent. Valid and Reliable is a necessary condition of trustworthiness, and Accountable and Transparent relate to all the other characteristics. A trustworthy AI system requires balancing these characteristics within its specific context of use, which may entail trade-ofs.
The AI RMF argues that AI risk management should be incorporated into the broader risk management surrounding an AI system, considering also its environment and relevant actors. To support the risk management process, an RMF core is proposed that includes four functions – Govern, Map, Measure, and Manage – which are described at a process level. As such, the AI RMF gives a comprehensive non-prescriptive framework for organisations working with AI systems.
The FAICP and AI RMF have similar definitions and considerations of the socio-technical context of the system. The layered approach of the FAICP addresses the ICT infrastructure, the AI system itself, and the system within a given sector or socio-technical environment. The four-function approach of the AI RMF maps out, measures, and manages the risks of AI, whilst maintaining governance throughout. The FAICP categorises AI threats into types of attacks whilst the AI RMF categorises AI harms into broad categories based on harms at the person-, organisation-, or ecosystem-level. The FAICP identifies 11 trustworthy characteristics of an AI system whilst the AI RMF identifies seven AI trustworthy characteristics. These seven are broader though fit within the FAICP’s 11. Both frameworks complement each other and provide guidelines and methodology. As such, an implemented approach to AI risk assessment can be formed from both of these.
Spyderisk is a semi-automated asset-based risk assessment and management tool following the ISO
27005 [
Within a risk calculation, Spyderisk automatically identifies the threats and risks present in a system, along with their likelihoods. The threat and risk likelihoods are calculated using an automated iterative algorithm that considers how entry point threats propagate to further threats in the system, and how all these threats can cause further threats, and so on. Initial input values are given to represent how much diferent components can be trusted or expected to behave correctly, and these are then used to inform the initial likelihoods of threats in the system. As threats propagate through the system, the likelihoods of other threats increase based on these, if a new cause is more likely than the current cause. This iterative algorithm converges when threat and risk likelihoods stop increasing. This automated approach is advantageous to manual risk calculations as there can be many threats and risks in a system, and manual risk calculations may only consider a subset of them, and may not fully consider their propagation to further threats and risks.
The Spyderisk schema is shown in Figure 2, adapted from [
The Spyderisk knowledge base encodes threats, assets, consequences, and controls for a given domain. It is a generalised abstraction such that it is not so specific to one system that it cannot be used for others within the given domain. Prior to the work here, the knowledge base contained knowledge primarily on the cybersecurity of cyber-physical systems. This knowledge already has relevance to AI systems, e.g. with it containing assets like software processes, ICT hardware, data, computer networks, people, places and jurisdictions. The contribution of the work here is to extend this knowledge base to include assets, threats, consequences, controls, and trustworthiness characteristics inherent to AI systems.
This knowledge extension follows a three-step process of knowledge acquisition, translation, and encoding. Knowledge acquisition includes locating sources of knowledge, accessing their relevance, and collating their knowledge; with key considerations of what are important assets, threats, consequences and harms to model, among others. Knowledge translation involves translating the acquired knowledge into the format of the Spyderisk knowledge schema, through determining what can be modelled as assets, threats, consequences, trustworthiness characteristics, controls, etc. Knowledge encoding involves the practical steps of encoding the new knowledge into the Spyderisk knowledge base. Through this knowledge extension, Spyderisk will be able to perform automated risk assessment of AI systems and its wider ICT system.
Spyderisk already has extensive encoded knowledge regarding computer processing, data storage
and usage, ICT hardware, networking, plus a significant amount in human and social aspects such
as human roles, rights (focusing on privacy), institutions, physical spaces and regulation. It contains
cybersecurity-related threats and controls, the consequences of these threats on diferent asset types,
and how these threats can propagate through systems made of these asset types. Spyderisk is discussed
in detail in [
• AI models reading data as input, writing data as output, running on ICT hardware, communicating over computer networks, and interacting with people. • Cybersecurity threats afecting AI quality / accuracy / robustness / fairness. • Inaccurate AI models afecting the data output from said AI models. • AI models unknowingly exposing personal information, compromising the privacy of citizens. • Cybersecurity controls, e.g., reducing risks of unauthentic training data for AI models.
These are initial examples that illustrate diferent cause and efect relationships between the domains. Patterns such as these have been considered in the knowledge extension and a key observation made so far is that data is a key link across domains, due to the multiple forms it can take. E.g., it is used for AI training, so compromises in training data afect the model results, which is also data.
The FAICP and AI RMF are mapped to the Spyderisk modelling process next. This enables the Spyderisk approach to AI risk assessment to follow, and be compatible with, the FAICP and AI RMF, giving it a rigorous foundation and easier adoptability.
The FAICP and AI RMF identify sets of AI trustworthiness characteristics, harms, threat types, and controls. In terms of AI trustworthiness characteristics, in the work here, an initial focus is given on accuracy, and fairness. Additionally, the FAICP identifies a general set of AI assets that may be present, grouped into categories including data, models, stakeholders, processes, and environment. Assets within these categories already exist in the Spyderisk knowledge base, making the integration and cross-domain interrelationships easy to implement as there is already good coverage of cyber-physical threats and risks for these AI asset categories.
The FAICP and AI RMF have diferent but compatible approaches. The FAICP considers risk management in a three-layer manner. Each layer builds up from the prior layer, where the first covers the cybersecurity foundations, the second covers AI-specific cybersecurity practices, and the third considers best practices for the sectoral context of where the AI system will be deployed. The FAICP three-layer approach is compatible with Spyderisk, since, for a user modelling a system in Spyderisk to produce a correctly functioning system model, they must specify the cyber-physical system components. This means the user will inherently model the ICT infrastructure of the first layer. The second layer of the FAICP is sector-agnostic AI risk management and provides requirements for threats, risks and controls specific to AI, which will be covered by the knowledge extensions. For the sector-specific third layer, some sector-specific AI threats may be encoded, and the modeller of a system will adjust consequence impact levels related to key parts of the sector-specific AI system. These impacts afect the risk levels, so sector-critical risks are highlighted and prioritised.
The four functions of the AI RMF are compatible with Spyderisk: Governance is addressed through the risk assessment of Spyderisk, helping towards the accountability, transparency, and reporting structures, as well as making risk assessment more accessible to non-experts. Map is addressed through the Spyderisk system model creation and risk assessment, with this giving the context of risks concerning the AI system. Measure is addressed through the calculated likelihood and risk levels for threats and consequences identified through the automated risk assessment calculation, aided through the setting of impact levels to highlight important consequences and the setting of controls to mitigate threats and risks. Additionally, Spyderisk evaluates risks stemming from compromises of trustworthy characteristics of the AI system. Through repeating the risk assessment over the lifecycle of the AI system, identified AI risks can be tracked over time, furthering the Measure function. Finally, for Manage, the risk assessment provides and recommends controls that address identified threats and risks.
The FAICP and AI RMF have a large focus on the compromise of AI trustworthiness characteristics, actions to reduce the likelihoods of these compromises occurring, transparent reporting of AI risks, and accountability for the actions and decisions made regarding them. Both also focus on the lifecycle of the AI system and surrounding ICT system, from design to deployment, and its evolution over time. The user of Spyderisk being required to explicitly model the cyber-physical system and AI-system components means the full cyber-physical socio-technical context of the AI system and associated ICT system is included within the Spyderisk system model. As such, the ICT context the AI system is in and the AI system as a socio-technical system are both inherently considered within Spyderisk.
Spyderisk can also be used throughout the lifecycle of an AI system. At the design stage it can assess potential risks to pre-emptively mitigate them and during the lifetime operation of the system it can assess risks using external information to adjust metrics related to vulnerabilities as the system operation evolves. Additionally, diferent hypothetical scenarios can be tested. All these lead to the optimisation of AI system trustworthiness characteristics through testing diferent optimisation scenarios and exploring their risks so that trade-ofs to these optimisations are known, understood, and planned for before any changes are made to the real-life system, allowing for informed decisions to be made and the most appropriate set of optimisations for a given system and context to be chosen. The importance of this is highlighted by the FAICP, which points out that optimisation of some trustworthiness characteristics can negatively afect others, resulting in trade-ofs. With links between the FAICP, AI RMF, and Spyderisk mapped and understood, knowledge can now be encoded within the Spyderisk knowledge base.
The initial work towards knowledge modelling of AI-related assets, threats, harms, consequences for
Spyderisk contains essentials needed to model ML-based harms resulting from compromises of integrity
or fairness of training data, or the actions a malicious user of an ML model. Knowledge has been
elicited primarily from the FAICP [
"ML Model" and "Training Algorithm" are subtypes of software process and the training, testing, validation, hyperparameters, model parameters, inputs, and predictions data are all subtypes of data. Here, software processes run on hardware, can consume data as input, and produce data as output. Data can be related to a software process, stored within databases and filesystems on ICT hardware, and transmitted over computer networks. Data can also link to humans, e.g. humans interact with data via software processes or the data being personal data relating to a person as a data subject. The Process and Data parent types already exist in the Spyderisk knowledge base, meaning there are already cybersecurity-related threats applicable to these AI system components, providing a point of integration between the domains of cybersecurity and AI. The links between the diferent assets in this ifgure represent relationships between them, showing how they interact with or use each other. These assets and relationships enable a user of Spyderisk to construct models of an AI SUT that encompasses diferent phases of the ML lifecycle, since the creation of the ML model as well as its usage is included.
As mentioned, Figure 3 is a high-level schematic of an ML system that encompasses both the training and deployment phases of the system. The top half represents the training phase(s), where training data is used to determine the model parameters, testing data is used to test the model accuracy, hyperparameters are used to define the training process, and validation data is used to guide the hyperparameter values. The bottom half represents the usage phase, where input data is received by the ML model, which, in conjunction with the model parameters, outputs predictions based on this input. These then enable diferent system models to be constructed, which can include ML model creation via training, ML model operation, or the two combined, which can be in a continuous training and usage cycle. This means that in these models, issues in one phase can propagate to afect the other.
With the schematic defined, it can be used to determine matching patterns for threats that could
occur in an ML system. Here, a matching pattern is a specification of assets and relationships required
Equality and equity are represented within the data, with
issues such as harmful bias and discrimination managed [
Fair Processing
Loss of Fair Pro- Process Inputs are processed such that resulting outputs have equality cessing and equity, are not unduly detrimental, and do not contain harmful bias and discrimination.
Accuracy
Loss of Accuracy
Data
“Correctness of output compared with reality.” [
An example matching pattern is shown in Figure 4. The pattern is denoted by the Asset types ("Training Algorithm" and "Training Data") and the relation between them (“usesToTrain”), is caused by the Training Data losing integrity, and results in the Training Algorithm being unreliable.
For ML-related threats, the asset types and relationships in Figure 3 determine the prototype elements from which the matching patterns are constructed. In this initial work, the focus is on the accuracy and fairness trustworthiness characteristics of the ML model predictions and how compromises of other trustworthiness characteristics in the system can lead to these, where these compromises may propagate through the system and directly or indirectly afect other assets of all kinds (e.g. people, institutions, downstream processes). Trustworthiness characteristics and consequences new to the Spyderisk knowledge base have been identified and encoded within it to allow new threats to be defined. These new trustworthiness characteristics and consequences are given in Table 1.
Using these and the asset types and relationships in Figure 3, threats have been encoded to model how training data that become incorrect (unintentionally or maliciously) propagate through the system to eventually afect the accuracy and fairness of model predictions, along with training data being unfair or biased as well. This considers threat paths from the training data to the training algorithm, model parameters, ML model, and ML predictions. Malicious users have also been modelled that maliciously alter the inputs to the ML model, afecting the accuracy and fairness of the model predictions.
Each ML-related threat is encoded independently inside Spyderisk and together, along with other cyber-physical threats, they automatically form chains of threats in the semi-automated risk calculation of Spyderisk. This is due to the consequence of one threat being the cause of another, leading to chains forming as threats and risks propagate through a system. One such chain of threats is given in Figure 5.
Here, there are four threats connected together in a chain. The block boxes represent assets that must be present for the threat to occur, and the block arrows between them represent the relationships that must be present between them. The red boxes represent trustworthiness characteristics that trigger the threat when they become compromised, the red ovals represent consequences of threats occurring, and the green oval represents a control that would block the threat. The yellow arrows show how one threat chains to another, with the consequence of one becoming the cause of another. This chain of threats shows how training data losing integrity causes the training algorithm to lose reliability, since its outputs now will not be correct due to incorrect inputs. This then causes the model parameters to lose integrity since the training algorithm is creating them based on incorrect training data. This leads to the ML model being unreliable as its model parameters are incorrect. Finally, this leads to the ML model predictions being inaccurate as its model parameters are incorrect. This chain may be broken and the final threat not reached if the model is validated and the incorrect parameters detected.
Multiple ML-related threats have successfully been encoded in the Spyderisk knowledge base, covering aspects of fairness and accuracy of the ML system, with threat chains like this formed.
An approach and initial work towards trustworthiness risk assessment of AI systems has been given. This approach builds on and enhances the existing Spyderisk risk assessment tool by extending its knowledge base to AI trustworthiness, threats, risks, assets, and controls.
Key source material has been examined and compared to Spyderisk, primarily the FAICP and AI RMF. It has been concluded that these are compatible with Spyderisk and so the knowledge extensions to it can use both these as sources of information and also be guided by them. This source material has provided an initial set of assets, threats, consequences, and trustworthiness characteristics that model ML threats and risks that has been encoded in the Spyderisk knowledge base. This has then integrated this new knowledge with existing knowledge in the realm of ICT hardware, software, data, networks, human interaction, physical spaces, cybersecurity and human rights such as privacy. This gives an initial crossing of the domains of cyber-physical systems and AI systems, allowing threats and risks from one to afect the other.
The knowledge extension is ongoing work and additional information in the sources already considered will be included in the knowledge base, along with information and considerations from additional sources to be examined, such as ISO/IEC 23894:2023, as well as relevant regulation such as the EU AI Act. The aim of this further work is to include more assets, threats, controls, consequences, and trustworthiness characteristics in the Spyderisk knowledge base so that it has greater fidelity regarding AI trustworthiness risk assessment and encompasses more of the AI trustworthiness characteristics defined in these diferent sources. Additionally, a further aim is to include other types of AI system in the Spyderisk knowledge base, rather than just ML systems. This paper has been adapted from the D2.1 THEMIS 5.0 Methodological Framework and Requirements Analysis deliverable report of the “Human-centered Trustworthiness Optimization in Hybrid Decision Support” (THEMIS 5.0) project (https://www.themis-trust.eu/_files/ugd/a245c2_ 9b7d6e7394b246f5921375833483b3a7.pdf), funded by the European Union’s Horizon Europe research and innovation programme, Grant Agreement No. 101121042. Views expressed are those of the authors.
The author(s) have not employed any Generative AI tools.