Symbiotic AI: What is the Role of Trustworthiness?

Abstract

The design, development, and use of Artificial Intelligence (AI) is crucial in modern society. The traditional design of AI systems focuses on models with very high performances without highlighting how relevant the role of humans is in this context. To create AI systems that suit end users' needs and preferences, it is important to involve them in each phase of the system lifetime cycle. AI systems must present interfaces and interaction paradigms that enhance users' cognitive models, ensuring usability and a positive User Experience (UX). In this new scenario, Human-Computer Interaction (HCI) and AI contaminate each other leading to reach the human-AI symbiosis. Researchers should shift the focus toward Symbiotic AI (SAI) systems, which aims to enhance humans' abilities without replacing them. This manuscript presents preliminary considerations for the creation of a framework to design high-quality SAI systems and metrics that can be employed to appropriately evaluate them. Being a novel field, it focuses on the current investigation regarding the definition of the properties of SAI systems, stressing the importance of Trustworthiness, and whether new design principles for SAI systems can be extracted from the AI act.

Keywords

• eol>Symbiotic AI
• Trustworthiness
• Design and Evaluation
• Human-Centered Approach
• AI Act (AIA)

1. Introduction

lationship between humans and AI systems in which "the human understands and intuitively reacts to the machine, and the machine understands and intuitively reacts to the human" [ 9 ].

To reach the human-AI symbiosis, users should trust the system’s decisions and properly comprehend them, making Trustworthiness one of the main properties to consider when dealing with such systems. However, due to the novelty of the field, limited work is available in the literature. Our research aims to propose a comprehensive framework and evaluation metrics to support designers, developers, and AI specialists in creating and evaluating Symbiotic AI (SAI) systems that inspire trust, ensure fairness, and are responsible and compliant with the various domains in which they operate [ 10 ].

This manuscript is structured as follows: Section 3 describes the approach that will be undertaken to design and evaluate SAI systems; Section 2 presents how trustworthiness can be defined in the SAI field, exploring the perspectives of the European Commission and academia; Section 4 concludes and explores the future work of the project.

2. Trustworthiness for SAI Systems

For people and society, trustworthiness is undoubtedly one of the prerequisites that AI systems should have to be used without hesitations [ 11 ]. It, therefore, becomes the starting point of our research because of its breadth and multifaceted nature. In this section, the concept of trustworthiness is explored by analysing the perspectives of European policymakers and academics to determine how to consider it in the context of SAI.

2.1. The European Commission

Perspective

This section focuses on two documents drafted by the European Commission: the Ethics Guidelines for Trustworthy AI and the AIA. The goal is to delineate a clear image of the standpoints of policymakers to create AI products that fully comply with laws, regulations, and norms and track the eforts of the EU concerning human rights, ethics, and philosophical issues.

2.1.1. Ethics Guidelines for Trustworthy AI)

The role of the AI HLEG is to define the approach of the European Commission with respect to AI by indicating the key principles and policies. In 2019, they drafted the "Ethics Guidelines for Trustworthy AI" report, which identifies seven requirements of Trustworthiness, identiifed as the umbrella property to ensure a human-centric approach to AI [ 11, 12 ], illustrated in Figure 1. Such requirements are briefly described in the following: • Human agency and Oversight: incorporating mechanisms for human intervention in critical decision-making processes ensures human control and supervision over AI systems to prevent unintended consequences. • Technical Robustness and Safety: developing AI systems necessitates a risk-preventive approach that ensures reliable behavior, minimizing and preventing unintentional and unexpected harm. • Privacy and Data Governance: ensuring privacy protection requires robust data governance, encompassing both the quality and integrity of the data used in processing to guarantee privacy. • Transparency: encompassing the transparency of elements requires to comprehend the reason that lies behind the decision taken by the system. • Diversity, Non-Discrimination and Fairness: involving all stakeholders throughout the entire system lifecycle ensures equal access through inclusive design processes and equitable treatment. • Societal and Environmental Well-being: maximizing sustainability, social impact, and ecological responsibility of AI systems to positively contribute to society while minimizing negative consequences. • Accountability: creating mechanisms to ensure accountability of AI systems, both before and after their development, deployment and use guarantees fairness [ 11 ].

2.1.2. The Artificial Intelligence Act (AIA)

Starting from the requirements of Trustworthy AI, listed in Section 2.1.1, in 2021, the EU has defined the AIA to regulate the adoption of harmonised and standardized rules for AI systems. Specifically, it merges trustworthiness with the risk-based approach to determine the acceptability of the types of systems through norms and regulations [ 12 ]. The risk-based approach outlines four categories of AI systems in relation to the risks they might cause: 1. Unacceptable Risk: it encompasses systems that might include prohibited AI practices that must be banned to guarantee a well-functioning society, such as those that might threaten minorities or those used by public authorities. 2. High Risk: it regards systems used in fields such as education and vocational training, access to private and public services, law enforcement, etc. (e.g., usable, observable, explainable, resilient, agile, etc.) [ 13 ].

The investigation of our research work consists in understanding what principles are applicable to SAI and identifying the potential new properties.

2.3. The Impact of Trustworthiness in SAI

3. Conceptual Framework for SAI

Systems

2.2. The Academic Perspective

The starting point is understanding the gaps in the traBen Shneiderman, one of the pioneers of HCI, proposes ditional approach to the development of AI systems to trustworthiness as one of three principles, along with determine the changes to propose and the integration of safety and reliability, of human-centered AI (HCAI) sys- new processes into the software lifecycle. This conceptems, which guarantee an appropriate balance of automa- tual framework aims to support designers and developers tion and human control. Specifically: in creating and evaluating SAI systems. The objective • Trustworthiness concerns the property that makes is to provide a standardized methodology to those who systems deserving of being trusted by humans. create AI-powered services that reduce the gap between technology and humans and decrease cognitive demand • Reliability comes from the application of techni- when interpreting and understanding the outputs that cal practices of software engineering that build systems produce. systems that produce appropriate and/or ex- The objective of this work lies in defining a framepected responses. work that considers and merges the two perspectives (i.e.

Ethics Guidelines for Trustworthy AI and AIA), while • Safety is a strategy to guide the refinement of the identifying principles, guidelines, and techniques that model performance to prevent potential failure belong to diferent disciplines by finding the appropriate and improper use [ 13 ]. links. Figure 2 presents an initial version of the conceptual framework that consists in two layers, Design and

Assessment, explained below.

The three above mentioned properties are the most recurrent in the literature since they are the main areas of research and can encompass the other properties; nevertheless, the state of the art concerning the human-AI interaction, considers other 22 properties that can influence the design and development of any kind of system

3.1. Design

This layer embraces four main research areas that contribute equally: Human-Computer Interaction (HCI), Law & Ethics, Software Engineering (SE), and AI. The following sections describe each component of the framework, illustrating its role in the SAI scenario.

Human-Computer Interaction (HCI) HCI is one of the pivotal components of this framework because the symbiotic relationship can be achieved if such systems allow users to reach their goals with efectiveness, eficiency, and satisfaction, thus, by being usable and providing a positive user experience. Other key elements that HCI is responsible for are feedback and afordance, enabling humans to understand how the system should be used, making them feel at ease with proper communication [ 6 ]. Involving humans iteratively during each phase of the system’s lifecycle implies performing interviews, questionnaires, field studies, and focus groups to perform quantitative and qualitative evaluations of the systems and to obtain rich insights concerning the users’ needs, preferences and cognitive models [ 6, 7 ]. Ethical & Legal Factors This dimension considers the regulatory, philosophical, and ethical standpoint since designers and developers must create products that preserve users’ social, working, and personal well-being. One of the main issues concerning AI, which becomes particularly valid for the branch of SAI, consists of avoiding biases and ensuring fairness. This element must be always considered because the root of biases is found in how data is treated by AI models, for example, in the learning phase. This determines the unfair behavior of systems that can influence humans’ decisions when employing AI as an instrument. The legal standpoint must be considered for designing and developing AI systems to create products that comply with regulations and can be released to the public. Currently, the main elements to consider are the AIA and the General Data Protection Regulation (GDPR); the first regulates the design, development, and use of AI systems in the EU, while the GDPR is a law that defines how data is handled, stored, and processed [ 15 ].

These regulations define the ethical principles that any kind of system should possess to be available to society.

Artificial Intelligence (AI) This dimension refers to

AI from a technical and algorithmic standpoint because the framework aims to suggest the appropriate techniques and practices to adopt depending on the requirements of the systems to create. AI models, along with high computational power, can be employed in multiple domains, such as business, finance, healthcare, agriculture, smart cities, and cybersecurity; however, they cannot be used as a one-size-fits-all solution because, depending on the activities, diferent tasks are needed - e.g., classification, prediction, description -, raising the need for context-specific models, parameters, and variables [ 16 ]. The efectiveness of SAI systems is not guaranteed by simply obtaining high-performing models but rather by systems that properly integrate Transparency, Explainability, and Interpretability. This provides users with the right instruments to comprehend the processes behind outputs, influencing their decisions, and what data is responsible for the system’s responses.

Software Engineering (SE) This framework aims to guide design and developers in creating SAI systems, ensuring that they operate by following a human-centered approach while complying with legal requirements and implementing high-performing AI systems. Thus, the objective is to integrate the Agile principles and the processes of the Agile Development Lifecycle with those belonging to the SAI design, creating a mapping that does not exclude any discipline [ 17 ].

3.2. Assessment

In this new scenario, where a strict correlation and contamination exists between human and AI performance, it becomes essential to define novel metrics to assess the human-AI symbiotic relationship.

Traditionally, human beings and AI have been viewed as distinct and unrelated entities, causing UX and AI metrics to be defined independently to evaluate both human behavior and system performance. Considering them in unison, it is possible to draft a preliminary set of metrics that can be employed to assess the symbiosis. By integrating both the dataset and user information and considering the user’s characteristics from the training phase of the AI model, it is possible to foster symbiosis, making the system’s behaviour as much as possible adaptable to the user’s needs.

Since Trustworthiness allows users to trust systems that operate safely and exhibit reliable behavior, it is contemplated as one of the starting points of this research work [ 4 ]. Assessing this aspect is dificult since it varies across many application contexts [ 4 ]; therefore, it is necessary to understand whether its evaluation should consider it as a stand-alone property or as an ensemble of other dimensions, such as safety, fairness, robustness, etc1.

Two potential metrics are proposed to assess how Trustworthy an AI system is: Preventing Undesired System Behaviors, which refers to how efectively the system avoids actions that could potentially harm the user or deviate from expected behavior; Correctness of Decisions, which measures the extent to which system’s decisions align with user expectations and desired outcomes.

4. Conclusions

This paper presents preliminary considerations concerning the novel field of Symbiotic AI with respect Trustworthiness. It presents the main challenges of identifying the principles of this field while stressing the need for a human-centered approach when dealing with AI systems of any kind. This research work is the starting ground for the definition of a comprehensive framework, presented in Section 3, that encompasses multiple disciplines and aims to guide designers and developers in creating SAI systems. This framework is still in its early stages and at a conceptual state. Delineating a standardized approach 1https://ec.europa.eu/futurium/en/ai-alliance-consultation.1.html to assess the behavior and performance of such systems is crucial to ensure the proper deployment of AI, which is part of the daily lives of countless individuals. As Trustworthiness plays a pivotal role in an efective human-AI interaction, the future of this research will focus on determining its complementary principles and its impact on symbiosis by carrying out verticalized case studies and performing in-depth investigations in the literature.

Acknowledgments

The research of Miriana Calvano and Antonio Curci is supported by the co-funding of the European Union Next Generation EU: NRRP Initiative, Mission 4, Component 2, Investment 1.3 – Partnerships extended to universities, research centers, companies, and research D.D. MUR n. 341 del 15.03.2022 – Next Generation EU (PE0000013 – “Future Artificial Intelligence Research – FAIR” - CUP: H97G22000210007).

References

[1] T. E. Commission , Proposal for a regulation of the european parliament and of the council laying down harmonised rules on asrtificial intelligence (artificial intelligence act) and amending certain union legislative acts , 2024 . URL: http://thomas.loc .gov/cgi-bin/query/z?c102: H. CON . RES.1 .IH.

[2] C. Sanderson , D. Douglas , Q. Lu , E. Schleiger , J. Whittle , J. Lacey , G. Newnham, S. Hajkowicz , C. Robinson , D. Hansen , Ai ethics principles in practice: Perspectives of designers and developers , IEEE Transactions on Technology and Society 4 ( 2023 ) 171 - 187 . URL: http://dx.doi.org/10.1109/TTS. 2023 . 3257303 . doi: 10 .1109/tts. 2023 . 3257303 .

[3] R. Guidotti , A. Monreale , S. Ruggieri , F. Turini , F. Giannotti , D. Pedreschi , A Survey of Methods for Explaining Black Box Models , ACM Computing Surveys 51 ( 2019 ) 1 - 42 . URL: https://dl.acm.org/doi /10.1145/3236009. doi: 10 .1145/3236009.

[4] B. Shneiderman , C. Plaisant , M. Cohen , S. Jacobs , N. Elmqvist , N. Diakopoulos , Designing the User Interface: Strategies for Efective Human-Computer Interaction , 6 ed., Pearson Education , 2016 . URL: ht tps://books.google.it/books?id=PpItDAAAQBAJ.

[5] I. O. for Standardization , Iso 9241 : 210 - ergonomics of human-system interaction , 2019 . URL: https:// www.iso.org/standard/77520.html.

[6] H. Sharp , J. Preece , Y. Rogers , Interaction Design: beyond human-computer interaction , 5 ed., John Wiley & Sons, Inc., 2019 .

[7] I. O. for Standardization , Iso 9241 : 210 - ergonomics of human-system interaction: Human-centred design for interactive systems , 2019 . URL: https: //www.iso.org/standard/77520.html.

[8] P. Bourque , R. E. Fairley (Eds.), SWEBOK: guide to the software engineering body of knowledge, version 3 .0 ed., IEEE Computer Society , Los Alamitos, CA, 2014 .

[9] S. S. Grigsby , Artificial intelligence for advanced human-machine symbiosis , in: D. D. Schmorrow , C. M. Fidopiastis (Eds.), Augmented Cognition: Intelligent Technologies , Springer International Publishing, Cham, 2018 , pp. 255 - 266 .

[10] M. Vahabava , The risks associated with generative AI apps in the European Artificial Intelligence Act (AIA) , in: Workshops at the Second International Conference on Hybrid Human-Artificial Intelligence (HHAI) , CEUR Workshop Proceedings , Munich, Germany, 2023 , pp. 1 - 12 .

[11] E. Commission , European commission - ethics guidelines for trustworthy ai , 2021 . URL: https: //ec.europa.eu/f uturium/en/ai-alliance- consu ltation.1 .html.

[12] J. Laux , S. Wachter , B. Mittelstadt , Trustworthy artificial intelligence and the European Union <span style="font-variant:small-caps;">AI</span> act: On the conflation of trustworthiness and acceptability of risk , Regulation & Governance 18 ( 2024 ) 3 - 32 . URL: https://onlinelibrary.wiley.com/doi/10.1111/ rego.12512. doi: 10 .1111/rego.12512.

[13] B. Shneiderman , Human-Centered AI , 1 ed ., Oxford University PressOxford, 2022 . URL: https://academ ic.oup.com/book/41126. doi: 10 .1093/oso/9780 192845290.001.0001.

[14] B. Kitchenham , Procedures for Performing Systematic Reviews , Technical Report , Keele University, 2004 .

[15] Gazzetta Uficiale dell'Unione Europea , General Data Protection Regulation (GDPR): Regulation (EU) 2016 /679, 2018 .

[16] I. Sarker , Ai-based modeling: Techniques, applications and research issues towards automation , intelligent and smart systems , 2022 . doi: 10 .20944 /preprints202202. 0001 . v1 .

[17] D. Salah , R. F. Paige , P. Cairns , A systematic literature review for agile development processes and user centred design integration , in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering , ACM, London England United Kingdom, 2014 , pp. 1 - 10 . URL: https://dl.acm.org/doi/10.1145/2601248.26012 76. doi: 10 .1145/2601248.2601276.