Symbiotic AI: What is the Role of Trustworthiness? Miriana Calvano1 , Antonio Curci1,2 , Rosa Lanzilotti1 and Antonio Piccinno1 1 University of Bari "Aldo Moro", Via Edoardo Orabona 4, 70125, Bari, Italy 2 University of Pisa, Largo B. Pontecorvo 3, 56127, Pisa, Italy Abstract The design, development, and use of Artificial Intelligence (AI) is crucial in modern society. The traditional design of AI systems focuses on models with very high performances without highlighting how relevant the role of humans is in this context. To create AI systems that suit end users’ needs and preferences, it is important to involve them in each phase of the system lifetime cycle. AI systems must present interfaces and interaction paradigms that enhance users’ cognitive models, ensuring usability and a positive User Experience (UX). In this new scenario, Human-Computer Interaction (HCI) and AI contaminate each other leading to reach the human-AI symbiosis. Researchers should shift the focus toward Symbiotic AI (SAI) systems, which aims to enhance humans’ abilities without replacing them. This manuscript presents preliminary considerations for the creation of a framework to design high-quality SAI systems and metrics that can be employed to appropriately evaluate them. Being a novel field, it focuses on the current investigation regarding the definition of the properties of SAI systems, stressing the importance of Trustworthiness, and whether new design principles for SAI systems can be extracted from the AI act. Keywords Symbiotic AI, Trustworthiness, Design and Evaluation, Human-Centered Approach, AI Act (AIA) 1. Introduction prehend the processes that lead to the outputs of such systems, causing low transparency [3]. This can be ad- The fast and broad spread of artificial intelligence (AI) dressed by adopting a human-centered approach when over the past few years has allowed individuals to use designing and developing AI systems to foster a symbi- new services, products, and systems to perform various otic relationship with humans and let technology support tasks and activities. AI has been introduced in various humans’ daily activities without replacing them, adapting fields, such as medicine, law, and education, raising sev- to their mental and physical models [4]. Human-Centred eral concerns because the results of the systems can influ- Design (HCD), which belongs to the Human-Computer ence humans to make decisions that are often irreversible Interaction (HCI) discipline, stresses that end-users must and can impact other individuals. Consequently, legal always be involved in the creation of any kind of product, bodies and governments are working to regulate AI to in order to create clear, appropriate and effective inter- preserve humans with new laws, such as the Artificial faces that allow end-users to interact correctly with the Intelligence Act (AIA), which undertakes a risk-based software they are using [5, 6, 7, 4]. On the other hand, approach regarding the design, development, and de- software engineering (SE) is another pillar in the devel- ployment of AI for EU citizens, identifying its best and opment of quality software systems, as it is the discipline forbidden practices while delineating guiding principles that studies how software should be developed, main- [1]. This implies that the future direction of AI is under- tained and used through specific standards and processes going substantial changes that should be addressed with [8]. It is, therefore, crucial to integrate practices and prin- a multidisciplinary approach [2]. ciples from the two disciplines to support designers and The main issue with AI systems is that the traditional developers in creating artificial intelligence systems that approach to their development heavily focuses on achiev- enable a symbiotic relationship with their end users. ing high-performing models and obtaining excellent met- This research is part of the Future Artificial Intelligence rics (e.g., accuracy, precision, recall). Such models are Research (FAIR) project, which aims to bring innovation also called black boxes: users cannot analyze and com- to the European Union in the context of AI. FAIR fol- lows a holistic and multidisciplinary approach to rethink Ital-IA 2024, 29-30th May 2024, Naples, Italy the foundations of AI and investigate its social impact. * Corresponding author. † Its goal is to build systems capable of interacting and These authors contributed equally. $ miriana.calvano@uniba.it (M. Calvano); antonio.curci@uniba.it collaborating with humans and foster trustworthiness. (A. Curci); rosa.lanzilotti@uniba.it (R. Lanzilotti); Specifically, the research presented in this article is per- antonio.piccinno@uniba.it (A. Piccinno) formed within the Spoke 6, named Symbiotic AI (SAI),  0000-0002-9507-9940 (M. Calvano); 0000-0001-6863-872X which investigates the scientific, social, economic, legal (A. Curci); 0000-0002-2039-8162 (R. Lanzilotti); and ethical challenges related to the growing symbiosis 0000-0003-1561-7073 (A. Piccinno) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License between humans and AI. SAI refers to a collaborative re- Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings lationship between humans and AI systems in which "the • Human agency and Oversight: incorporating human understands and intuitively reacts to the machine, mechanisms for human intervention in critical and the machine understands and intuitively reacts to decision-making processes ensures human con- the human" [9]. trol and supervision over AI systems to prevent To reach the human-AI symbiosis, users should trust unintended consequences. the system’s decisions and properly comprehend them, making Trustworthiness one of the main properties to • Technical Robustness and Safety: develop- consider when dealing with such systems. However, due ing AI systems necessitates a risk-preventive ap- to the novelty of the field, limited work is available in proach that ensures reliable behavior, minimiz- the literature. Our research aims to propose a compre- ing and preventing unintentional and unexpected hensive framework and evaluation metrics to support harm. designers, developers, and AI specialists in creating and • Privacy and Data Governance: ensuring pri- evaluating Symbiotic AI (SAI) systems that inspire trust, vacy protection requires robust data governance, ensure fairness, and are responsible and compliant with encompassing both the quality and integrity of the various domains in which they operate [10]. the data used in processing to guarantee privacy. This manuscript is structured as follows: Section 3 describes the approach that will be undertaken to design • Transparency: encompassing the transparency and evaluate SAI systems; Section 2 presents how trust- of elements requires to comprehend the reason worthiness can be defined in the SAI field, exploring the that lies behind the decision taken by the system. perspectives of the European Commission and academia; • Diversity, Non-Discrimination and Fairness: Section 4 concludes and explores the future work of the involving all stakeholders throughout the entire project. system lifecycle ensures equal access through in- clusive design processes and equitable treatment. 2. Trustworthiness for SAI Systems • Societal and Environmental Well-being: max- imizing sustainability, social impact, and ecologi- For people and society, trustworthiness is undoubtedly cal responsibility of AI systems to positively con- one of the prerequisites that AI systems should have to tribute to society while minimizing negative con- be used without hesitations [11]. It, therefore, becomes sequences. the starting point of our research because of its breadth and multifaceted nature. In this section, the concept of • Accountability: creating mechanisms to ensure trustworthiness is explored by analysing the perspectives accountability of AI systems, both before and af- of European policymakers and academics to determine ter their development, deployment and use guar- how to consider it in the context of SAI. antees fairness [11]. 2.1. The European Commission 2.1.2. The Artificial Intelligence Act (AIA) Perspective Starting from the requirements of Trustworthy AI, listed This section focuses on two documents drafted by the in Section 2.1.1, in 2021, the EU has defined the AIA to European Commission: the Ethics Guidelines for Trust- regulate the adoption of harmonised and standardized worthy AI and the AIA. The goal is to delineate a clear rules for AI systems. Specifically, it merges trustwor- image of the standpoints of policymakers to create AI thiness with the risk-based approach to determine the products that fully comply with laws, regulations, and acceptability of the types of systems through norms and norms and track the efforts of the EU concerning human regulations [12]. The risk-based approach outlines four rights, ethics, and philosophical issues. categories of AI systems in relation to the risks they might cause: 2.1.1. Ethics Guidelines for Trustworthy AI) 1. Unacceptable Risk: it encompasses systems that The role of the AI HLEG is to define the approach of the might include prohibited AI practices that must European Commission with respect to AI by indicating be banned to guarantee a well-functioning soci- the key principles and policies. In 2019, they drafted ety, such as those that might threaten minorities the "Ethics Guidelines for Trustworthy AI" report, which or those used by public authorities. identifies seven requirements of Trustworthiness, identi- fied as the umbrella property to ensure a human-centric 2. High Risk: it regards systems used in fields such approach to AI [11, 12], illustrated in Figure 1. Such as education and vocational training, access to requirements are briefly described in the following: private and public services, law enforcement, etc. (e.g., usable, observable, explainable, resilient, agile, etc.) [13]. The investigation of our research work consists in understanding what principles are applicable to SAI and identifying the potential new properties. 2.3. The Impact of Trustworthiness in SAI Our objective is to define a framework that encompasses both standpoints; in this regard, the authors are perform- ing a Systematic Literature Review (SLR), following the Kitchenham protocol, to identify the guidelines and prin- ciples that can be drawn from the AIA that could be applied to the lifecycle of SAI systems [14]. This SLR has the objective to determine how the research community is investigating and employing the AIA with respect to the design and development AI. From the preliminary Figure 1: The seven key requirements of Trustworthy AI: all results, it emerged that trustworthiness is intrinsic in are of equal importance and support each other [11] SAI because humans must fully trust systems in order to symbiotically with them. Belonging to the domain of AI built following a human- 3. Limited Risk: it encompasses AI systems that centered approach, SAI can include Trustworthiness, must comply with specific transparency obliga- Safety, and Reliability as principles; however, the estab- tions because they interact with humans (e.g., bio- lishment of a symbiotic relationship might require their metric recognition systems, and emotion recog- refinement or to the definition of new ones. The ongoing nition systems). SLR will also serve to establish the new principles and identify new guidelines suitable for the field of SAI. 4. Low or Minimal Risk: it refers to systems that feature AI but do not require specific conformity checks [1]. 3. Conceptual Framework for SAI Systems 2.2. The Academic Perspective The starting point is understanding the gaps in the tra- Ben Shneiderman, one of the pioneers of HCI, proposes ditional approach to the development of AI systems to trustworthiness as one of three principles, along with determine the changes to propose and the integration of safety and reliability, of human-centered AI (HCAI) sys- new processes into the software lifecycle. This concep- tems, which guarantee an appropriate balance of automa- tual framework aims to support designers and developers tion and human control. Specifically: in creating and evaluating SAI systems. The objective is to provide a standardized methodology to those who • Trustworthiness concerns the property that makes create AI-powered services that reduce the gap between systems deserving of being trusted by humans. technology and humans and decrease cognitive demand • Reliability comes from the application of techni- when interpreting and understanding the outputs that cal practices of software engineering that build systems produce. systems that produce appropriate and/or ex- The objective of this work lies in defining a frame- pected responses. work that considers and merges the two perspectives (i.e. Ethics Guidelines for Trustworthy AI and AIA), while • Safety is a strategy to guide the refinement of the identifying principles, guidelines, and techniques that model performance to prevent potential failure belong to different disciplines by finding the appropriate and improper use [13]. links. Figure 2 presents an initial version of the concep- The three above mentioned properties are the most tual framework that consists in two layers, Design and recurrent in the literature since they are the main areas Assessment, explained below. of research and can encompass the other properties; nev- ertheless, the state of the art concerning the human-AI interaction, considers other 22 properties that can influ- ence the design and development of any kind of system Figure 2: Conceptual Comprehensive Framework for the design and the evaluation of Symbiotic AI 3.1. Design ploying AI as an instrument. The legal standpoint must be considered for designing and developing AI systems This layer embraces four main research areas that con- to create products that comply with regulations and can tribute equally: Human-Computer Interaction (HCI), Law be released to the public. Currently, the main elements & Ethics, Software Engineering (SE), and AI. The follow- to consider are the AIA and the General Data Protec- ing sections describe each component of the framework, tion Regulation (GDPR); the first regulates the design, illustrating its role in the SAI scenario. development, and use of AI systems in the EU, while the GDPR is a law that defines how data is handled, stored, Human-Computer Interaction (HCI) HCI is one of and processed [15]. the pivotal components of this framework because the These regulations define the ethical principles that any symbiotic relationship can be achieved if such systems kind of system should possess to be available to society. allow users to reach their goals with effectiveness, effi- ciency, and satisfaction, thus, by being usable and pro- Artificial Intelligence (AI) This dimension refers to viding a positive user experience. Other key elements AI from a technical and algorithmic standpoint because that HCI is responsible for are feedback and affordance, the framework aims to suggest the appropriate tech- enabling humans to understand how the system should niques and practices to adopt depending on the require- be used, making them feel at ease with proper commu- ments of the systems to create. AI models, along with nication [6]. Involving humans iteratively during each high computational power, can be employed in multi- phase of the system’s lifecycle implies performing inter- ple domains, such as business, finance, healthcare, agri- views, questionnaires, field studies, and focus groups to culture, smart cities, and cybersecurity; however, they perform quantitative and qualitative evaluations of the cannot be used as a one-size-fits-all solution because, de- systems and to obtain rich insights concerning the users’ pending on the activities, different tasks are needed - e.g., needs, preferences and cognitive models [6, 7]. classification, prediction, description -, raising the need for context-specific models, parameters, and variables Ethical & Legal Factors This dimension considers the [16]. The effectiveness of SAI systems is not guaranteed regulatory, philosophical, and ethical standpoint since by simply obtaining high-performing models but rather designers and developers must create products that pre- by systems that properly integrate Transparency, Explain- serve users’ social, working, and personal well-being. ability, and Interpretability. This provides users with the One of the main issues concerning AI, which becomes right instruments to comprehend the processes behind particularly valid for the branch of SAI, consists of avoid- outputs, influencing their decisions, and what data is ing biases and ensuring fairness. This element must be responsible for the system’s responses. always considered because the root of biases is found in how data is treated by AI models, for example, in the Software Engineering (SE) This framework aims to learning phase. This determines the unfair behavior of guide design and developers in creating SAI systems, en- systems that can influence humans’ decisions when em- suring that they operate by following a human-centered approach while complying with legal requirements and to assess the behavior and performance of such systems implementing high-performing AI systems. Thus, the is crucial to ensure the proper deployment of AI, which is objective is to integrate the Agile principles and the pro- part of the daily lives of countless individuals. As Trust- cesses of the Agile Development Lifecycle with those worthiness plays a pivotal role in an effective human-AI belonging to the SAI design, creating a mapping that interaction, the future of this research will focus on de- does not exclude any discipline [17]. termining its complementary principles and its impact on symbiosis by carrying out verticalized case studies 3.2. Assessment and performing in-depth investigations in the literature. In this new scenario, where a strict correlation and con- tamination exists between human and AI performance, Acknowledgments it becomes essential to define novel metrics to assess the human-AI symbiotic relationship. The research of Miriana Calvano and Antonio Curci is Traditionally, human beings and AI have been viewed supported by the co-funding of the European Union - as distinct and unrelated entities, causing UX and AI met- Next Generation EU: NRRP Initiative, Mission 4, Com- rics to be defined independently to evaluate both human ponent 2, Investment 1.3 – Partnerships extended to behavior and system performance. Considering them in universities, research centers, companies, and research unison, it is possible to draft a preliminary set of met- D.D. MUR n. 341 del 15.03.2022 – Next Generation EU rics that can be employed to assess the symbiosis. By (PE0000013 – “Future Artificial Intelligence Research – integrating both the dataset and user information and FAIR” - CUP: H97G22000210007). considering the user’s characteristics from the training phase of the AI model, it is possible to foster symbio- References sis, making the system’s behaviour as much as possible adaptable to the user’s needs. [1] T. E. Commission, Proposal for a regulation of Since Trustworthiness allows users to trust systems that the european parliament and of the council laying operate safely and exhibit reliable behavior, it is contem- down harmonised rules on asrtificial intelligence plated as one of the starting points of this research work (artificial intelligence act) and amending certain [4]. Assessing this aspect is difficult since it varies across union legislative acts, 2024. URL: http://thomas.loc many application contexts [4]; therefore, it is necessary .gov/cgi-bin/query/z?c102:H.CON.RES.1.IH. to understand whether its evaluation should consider [2] C. Sanderson, D. Douglas, Q. Lu, E. Schleiger, it as a stand-alone property or as an ensemble of other J. Whittle, J. Lacey, G. Newnham, S. Hajkowicz, dimensions, such as safety, fairness, robustness, etc1 . C. Robinson, D. Hansen, Ai ethics principles in prac- Two potential metrics are proposed to assess how tice: Perspectives of designers and developers, IEEE Trustworthy an AI system is: Preventing Undesired Sys- Transactions on Technology and Society 4 (2023) tem Behaviors, which refers to how effectively the system 171–187. URL: http://dx.doi.org/10.1109/TTS.2023. avoids actions that could potentially harm the user or 3257303. doi:10.1109/tts.2023.3257303. deviate from expected behavior; Correctness of Decisions, [3] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Gi- which measures the extent to which system’s decisions annotti, D. Pedreschi, A Survey of Methods for align with user expectations and desired outcomes. Explaining Black Box Models, ACM Computing Surveys 51 (2019) 1–42. URL: https://dl.acm.org/doi /10.1145/3236009. doi:10.1145/3236009. 4. Conclusions [4] B. Shneiderman, C. Plaisant, M. Cohen, S. Jacobs, This paper presents preliminary considerations concern- N. Elmqvist, N. Diakopoulos, Designing the User ing the novel field of Symbiotic AI with respect Trust- Interface: Strategies for Effective Human-Computer worthiness. It presents the main challenges of identifying Interaction, 6 ed., Pearson Education, 2016. URL: ht the principles of this field while stressing the need for a tps://books.google.it/books?id=PpItDAAAQBAJ. human-centered approach when dealing with AI systems [5] I. O. for Standardization, Iso 9241:210 - ergonomics of any kind. This research work is the starting ground for of human-system interaction, 2019. URL: https:// the definition of a comprehensive framework, presented www.iso.org/standard/77520.html. in Section 3, that encompasses multiple disciplines and [6] H. Sharp, J. Preece, Y. Rogers, Interaction Design: aims to guide designers and developers in creating SAI beyond human-computer interaction, 5 ed., John systems. This framework is still in its early stages and at Wiley & Sons, Inc., 2019. a conceptual state. Delineating a standardized approach [7] I. O. for Standardization, Iso 9241:210 - ergonomics of human-system interaction: Human-centred de- 1 https://ec.europa.eu/futurium/en/ai-alliance-consultation.1.html sign for interactive systems, 2019. URL: https: //www.iso.org/standard/77520.html. [8] P. Bourque, R. E. Fairley (Eds.), SWEBOK: guide to the software engineering body of knowledge, ver- sion 3.0 ed., IEEE Computer Society, Los Alamitos, CA, 2014. [9] S. S. Grigsby, Artificial intelligence for advanced human-machine symbiosis, in: D. D. Schmorrow, C. M. Fidopiastis (Eds.), Augmented Cognition: In- telligent Technologies, Springer International Pub- lishing, Cham, 2018, pp. 255–266. [10] M. Vahabava, The risks associated with genera- tive AI apps in the European Artificial Intelligence Act (AIA), in: Workshops at the Second Interna- tional Conference on Hybrid Human-Artificial In- telligence (HHAI), CEUR Workshop Proceedings, Munich, Germany, 2023, pp. 1–12. [11] E. Commission, European commission - ethics guidelines for trustworthy ai, 2021. URL: https: //ec.europa.eu/f uturium/en/ai-alliance-consu ltation.1.html. [12] J. Laux, S. Wachter, B. Mittelstadt, Trustworthy ar- tificial intelligence and the European Union AI act: On the conflation of trustworthiness and acceptability of risk, Regulation & Governance 18 (2024) 3–32. URL: https://onlinelibrary.wiley.com/doi/10.1111/ rego.12512. doi:10.1111/rego.12512. [13] B. Shneiderman, Human-Centered AI, 1 ed., Oxford University PressOxford, 2022. URL: https://academ ic.oup.com/book/41126. doi:10.1093/oso/9780 192845290.001.0001. [14] B. Kitchenham, Procedures for Performing System- atic Reviews, Technical Report, Keele University, 2004. [15] Gazzetta Ufficiale dell’Unione Europea, General Data Protection Regulation (GDPR): Regulation (EU) 2016/679, 2018. [16] I. Sarker, Ai-based modeling: Techniques, appli- cations and research issues towards automation, intelligent and smart systems, 2022. doi:10.20944 /preprints202202.0001.v1. [17] D. Salah, R. F. Paige, P. Cairns, A systematic liter- ature review for agile development processes and user centred design integration, in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, ACM, London England United Kingdom, 2014, pp. 1–10. URL: https://dl.acm.org/doi/10.1145/2601248.26012 76. doi:10.1145/2601248.2601276.