Introduction

There is much public discussion on how Artificial Intelligence (AI) differs from human intelligence. We trust the latter, we are wary of the former. Industry practitioners share these concerns and put effort into measuring safety, privacy, etc. Their goal is ensuring AI-based Self-Adaptive Systems (SAS) can at least reach human performance in the tasks they were designed for [1]. However, these efforts are often insufficient for humans to trust SASs, especially with the introduction of foundation models, such as OpenAI's ChatGPT, rapidly permeating society.

Foundation models are based on large-scale self-supervised deep learning algorithms [2], whose inner workings are not transparent, making them difficult to explain to and interpret by users. Moreover, foundation models often use large amounts of unlabelled data, often gathered disregarding ethical concerns, e.g., diversity. The more complex and accurate the models become, the more data is needed to train them, and the harder it is to explain their decision making process. Thus, the conflict between these powerful AI "blackboxes" and user trust [3].

Requirements Engineering (RE) guidelines for ethical AI were reviewed with the aim of building a framework for Trustworthy SASs (T-SASs). The outlined T-SAS framework is motivated by the emergence of semi-autonomous heavy vehicles for mining, as running example, which raise concerns addressed here. Nevertheless, the T-SAS framework could address human values in other fields. The focus will be on human oversight, still needed to promote trust in SASs [4,5,6]. The insights on human-on-the-loop (HOTL) expectations for T-SAS monitoring and human intervention aim to foster discussions among the RE practitioners about creating T-SASs that adhere to ethical principles and laws [4,7].

Background and Mining Context

Aristotle defined credibility in terms of wisdom, virtue, and goodwill. Centuries later, EU guidelines state that AI should be trustworthy, that is robust, lawful, and ethical [4]. Fig. 1 shows requirements related to human autonomy and shared responsibility in EU guidelines. Evaluating whether adaptive systems meet stakeholders' needs often focuses on robustness verification, but this may not capture ethical values [1,8,9]. Nevertheless, embedding ethical values in SASs is challenging, partly due to the recent AI developments such as foundation models, e.g., text-to-image generators for non-expert users [10,2]. Designing comprehensive evaluation strategies for these complex and industrial systems is difficult due to the lack of auditability and sustainability analysis, and the emergence of unforeseen skills during training [2]. Moreover, the lack of open APIs and benchmarks hinders research on foundation models' transparency, robustness, fairness, etc. Moreover, the resources needed to train and test such systems hinder academics' access to evaluating their benefits and harms [2]. Nevertheless, high-risk SASs like Autonomous Vehicles (AV), potentially using foundation models, must nevertheless show transparency to allow for human oversight and intervention [3,4]. SASs must inform diverse end-users, e.g., end-users or third-party audits, about their capacities and limitations and trace them back to input data to enable responsibility reasoning [11,12]. Responsibility sharing and mitigation of foreseeable misuse are challenging and raise ethical questions that need to be answered during the RE process [3,13].

Mining AVs in safety-critical situations are high-risk AI products, therefore a HOTL to monitor the AVs and intervene when prompted is needed [3,4]. Human drivers and AVs primarily rely on vision, or Computer Vision (CV), to avoid danger and their responsibilities must be balanced [14,15]. AI algorithms can help mining vehicles remote operators in critical situations: by measuring user attention, either driver or remote operator, to reduce reaction times or by facilitating fallback to human control in case of low AI confidence [16,7]. Even HOTL AVs can be involved in incidents, potentially fatal with heavy mining machinery, so risks arising from faulty interactions must be mitigated. Human-AI interaction is receiving increasing academic attention together with limitations of AV, including benefits, harms, and development practices [11,17,18]. The AI paradigm is shifting to blackbox models, hindering HOTL-SAS interaction and raising the question of how to split the responsibility of decision-making.

Deep Learning algorithms are increasingly popular to detect edge cases where human intervention might be needed, but they rely on large amounts of annotated data, which is difficult or impossible to gather, expensive and time-consuming to curate [2]. However, sensor difficulties, e.g., extreme weather affecting visibility, or cognitive limits, e.g., insufficient training data, might cause malfunctions [19]. The RE process therefore needs to set standards for data quality, security, and privacy [10]. Based on the data, robustness needs to be periodically evaluated by stakeholders, using performance metrics and criteria that reflect their values and goals, e.g., ore throughput rate [1,10]. Limitations of mining AVs should be clearly explained to the HOTL at all times, e.g., to prevent incidents, improve throughput rates, or audit accidents [12,7]. Transparency, though, is not always possible when using these algorithms, especially in opaque blackbox algorithms or foundation models.

Framework for Trustworthy Self-Adaptive Systems

This section outlines a framework to guide the RE process for T-SAS focusing on requirements for HOTL-mechanisms (see Figure 1) in light of the trend to incorporate foundation models such as GPT-3, DALL-E, or BERT [20]. The relationship between the concepts is also discussed:

Robustness. Classic AIs use annotated data, whilst foundation models use large volumes of unlabeled data, removing the difficult and time-consuming task of curating data sets. This paradigm can particularly benefit AVs for mining, which inherently need to deal with previously unseen scenarios. Nevertheless foundation models, especially learning online, can be affected by incorrect, redundant, or unstable data, which could lead to safety-critical situations. Therefore, the T-SAS framework promotes the usage of high-quality, diverse, self-updating, and self-augmenting data sets [21,4]. Appropriate requirements for data availability, usability, consistency, and integrity, must be discussed [2,1].

Human oversight. Whilst foundation models can accomplish complex tasks, e.g., image synthesis, they still show limitations, e.g., generalizing to new scenes, mainly due to selfsupervised training [2]. Even if totally reliable, SASs incorporating such models would still need to be transparent to facilitate human oversight, foster human autonomy, and, ultimately, be trustworthy. HOTL-SAS interaction is an open and important problem for humans, who should be able to supervise and override SAS decisions at all times. Therefore, T-SASs must integrate HOTL strategies and monitoring interfaces adequate to the end-users, designed to address the transparency and accountability needs of T-SASs [17,7,10].

Transparency. T-SASs should provide concise, complete, correct, and clear explanations that are relevant, accessible and comprehensible to users in a context (use or foreseeable misuse), to avoid risks to health, safety, or fundamental rights [4,3]. These requirements intend to ensure human autonomy and responsibility sharing but integrating these needs into SAS is challenging. Previous work has focused on highly trained operators, e.g., aircraft pilots, but there is still the need to investigate how to design interactions with non-expert users [11]. Training end-users while using SASs could be considered. For that, appropriate metrics and criteria, adapted to the user and the operation context, would be needed to ensure clarity and avoid ambiguity about the state of the T-SAS.

Accountability. As discussed above, many SASs, including AV, cannot ensure safety on their own and need to be monitored by humans during operation. Even when SASs are not entirely robust, might be able to produce priors and convey information that greatly helps the HOTL in critical situations. This has long been a focus of Human-Computer Interaction research [3,17,7]. Moreover, T-SASs must also be accountable to justify their goals, motivations and rationale in post hoc analysis by third parties. This topic is strongly related to detecting, leveraging, and mitigating risks by public authorities. Therefore, the framework should explicitly connect these needs to open communication requirements, critical for T-SASs that closely interact with humans, e.g., AV drivers [4].

Conclusion

Humans often mistrust SASs or show automation bias [3,11]. Both are concerning as SASs increasingly integrate foundation models, far from being transparent or auditable [20,6,22,2]. Much effort has been devoted to support practitioners in addressing human values in the RE process but the absence of clear guidelines, benchmarks, metrics, and evaluation criteria, makes this task challenging. As a result, there is still a need for human oversight, e.g., fallback procedures [11,17,16]. Academics from different backgrounds should examine the models' biases and limitations, and inform society about their trustworthiness [2]. These recommendations are based on existing international laws, domestic legislation, and AI development frameworks and aim to increase awareness among RE practitioners and inspire the development of a generic framework for creating T-SASs. Efforts to homogenise mining processes are already being made but further research is needed to adequately address human values in HOTL mining SASs. For instance, it is necessary to consider the implications that foundation models will entail with respect to other ethical considerations. Agreeing on appropriate recommendations with practitioners to address human values in the RE process for T-SAS would be a necessary next step. Frameworks from other disciplines and the ad-hoc practices of RE practitioners could be studied to propose adaptations to existing frameworks to better address human values in T-SAS development. Data governance should in turn be aligned with stakeholders' values, e.g., non-discrimination, and requirements such as privacy or fairness. These considerations are left out for future work.

This work is based on European Union guidelines but different values might prevail in non-EU countries. Even within the EU, revisions to the AI legislation, which is still in draft form, might have a significant impact on the SAS now in development. As such, it is important for the framework to adapt to new, unforeseeable trust elements introduced by public authorities that might, directly and indirectly, impact the expectations for T-SASs. As a final note, future research must also address the question of how to allow for diverse legislation and context-dependent interpretation of T-SAS requirements.