=Paper=
{{Paper
|id=Vol-3762/528
|storemode=property
|title=Symbiotic AI: What is the Role of Trustworthiness?
|pdfUrl=https://ceur-ws.org/Vol-3762/528.pdf
|volume=Vol-3762
|authors=Miriana Calvano,Antonio Curci,Rosa Lanzilotti,Antonio Piccinno
|dblpUrl=https://dblp.org/rec/conf/ital-ia/CalvanoCLP24
}}
==Symbiotic AI: What is the Role of Trustworthiness?==
https://ceur-ws.org/Vol-3762/528.pdf
Symbiotic AI: What is the Role of Trustworthiness?
Miriana Calvano1 , Antonio Curci1,2 , Rosa Lanzilotti1 and Antonio Piccinno1
1
University of Bari "Aldo Moro", Via Edoardo Orabona 4, 70125, Bari, Italy
2
University of Pisa, Largo B. Pontecorvo 3, 56127, Pisa, Italy
Abstract
The design, development, and use of Artificial Intelligence (AI) is crucial in modern society. The traditional design of AI
systems focuses on models with very high performances without highlighting how relevant the role of humans is in this
context. To create AI systems that suit end users’ needs and preferences, it is important to involve them in each phase of the
system lifetime cycle. AI systems must present interfaces and interaction paradigms that enhance users’ cognitive models,
ensuring usability and a positive User Experience (UX). In this new scenario, Human-Computer Interaction (HCI) and AI
contaminate each other leading to reach the human-AI symbiosis. Researchers should shift the focus toward Symbiotic
AI (SAI) systems, which aims to enhance humans’ abilities without replacing them. This manuscript presents preliminary
considerations for the creation of a framework to design high-quality SAI systems and metrics that can be employed to
appropriately evaluate them. Being a novel field, it focuses on the current investigation regarding the definition of the
properties of SAI systems, stressing the importance of Trustworthiness, and whether new design principles for SAI systems
can be extracted from the AI act.
Keywords
Symbiotic AI, Trustworthiness, Design and Evaluation, Human-Centered Approach, AI Act (AIA)
1. Introduction prehend the processes that lead to the outputs of such
systems, causing low transparency [3]. This can be ad-
The fast and broad spread of artificial intelligence (AI) dressed by adopting a human-centered approach when
over the past few years has allowed individuals to use designing and developing AI systems to foster a symbi-
new services, products, and systems to perform various otic relationship with humans and let technology support
tasks and activities. AI has been introduced in various humans’ daily activities without replacing them, adapting
fields, such as medicine, law, and education, raising sev- to their mental and physical models [4]. Human-Centred
eral concerns because the results of the systems can influ- Design (HCD), which belongs to the Human-Computer
ence humans to make decisions that are often irreversible Interaction (HCI) discipline, stresses that end-users must
and can impact other individuals. Consequently, legal always be involved in the creation of any kind of product,
bodies and governments are working to regulate AI to in order to create clear, appropriate and effective inter-
preserve humans with new laws, such as the Artificial faces that allow end-users to interact correctly with the
Intelligence Act (AIA), which undertakes a risk-based software they are using [5, 6, 7, 4]. On the other hand,
approach regarding the design, development, and de- software engineering (SE) is another pillar in the devel-
ployment of AI for EU citizens, identifying its best and opment of quality software systems, as it is the discipline
forbidden practices while delineating guiding principles that studies how software should be developed, main-
[1]. This implies that the future direction of AI is under- tained and used through specific standards and processes
going substantial changes that should be addressed with [8]. It is, therefore, crucial to integrate practices and prin-
a multidisciplinary approach [2]. ciples from the two disciplines to support designers and
The main issue with AI systems is that the traditional developers in creating artificial intelligence systems that
approach to their development heavily focuses on achiev- enable a symbiotic relationship with their end users.
ing high-performing models and obtaining excellent met- This research is part of the Future Artificial Intelligence
rics (e.g., accuracy, precision, recall). Such models are Research (FAIR) project, which aims to bring innovation
also called black boxes: users cannot analyze and com- to the European Union in the context of AI. FAIR fol-
lows a holistic and multidisciplinary approach to rethink
Ital-IA 2024, 29-30th May 2024, Naples, Italy
the foundations of AI and investigate its social impact.
*
Corresponding author.
† Its goal is to build systems capable of interacting and
These authors contributed equally.
$ miriana.calvano@uniba.it (M. Calvano); antonio.curci@uniba.it collaborating with humans and foster trustworthiness.
(A. Curci); rosa.lanzilotti@uniba.it (R. Lanzilotti); Specifically, the research presented in this article is per-
antonio.piccinno@uniba.it (A. Piccinno) formed within the Spoke 6, named Symbiotic AI (SAI),
0000-0002-9507-9940 (M. Calvano); 0000-0001-6863-872X which investigates the scientific, social, economic, legal
(A. Curci); 0000-0002-2039-8162 (R. Lanzilotti);
and ethical challenges related to the growing symbiosis
0000-0003-1561-7073 (A. Piccinno)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License between humans and AI. SAI refers to a collaborative re-
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
lationship between humans and AI systems in which "the • Human agency and Oversight: incorporating
human understands and intuitively reacts to the machine, mechanisms for human intervention in critical
and the machine understands and intuitively reacts to decision-making processes ensures human con-
the human" [9]. trol and supervision over AI systems to prevent
To reach the human-AI symbiosis, users should trust unintended consequences.
the system’s decisions and properly comprehend them,
making Trustworthiness one of the main properties to • Technical Robustness and Safety: develop-
consider when dealing with such systems. However, due ing AI systems necessitates a risk-preventive ap-
to the novelty of the field, limited work is available in proach that ensures reliable behavior, minimiz-
the literature. Our research aims to propose a compre- ing and preventing unintentional and unexpected
hensive framework and evaluation metrics to support harm.
designers, developers, and AI specialists in creating and • Privacy and Data Governance: ensuring pri-
evaluating Symbiotic AI (SAI) systems that inspire trust, vacy protection requires robust data governance,
ensure fairness, and are responsible and compliant with encompassing both the quality and integrity of
the various domains in which they operate [10]. the data used in processing to guarantee privacy.
This manuscript is structured as follows: Section 3
describes the approach that will be undertaken to design • Transparency: encompassing the transparency
and evaluate SAI systems; Section 2 presents how trust- of elements requires to comprehend the reason
worthiness can be defined in the SAI field, exploring the that lies behind the decision taken by the system.
perspectives of the European Commission and academia; • Diversity, Non-Discrimination and Fairness:
Section 4 concludes and explores the future work of the involving all stakeholders throughout the entire
project. system lifecycle ensures equal access through in-
clusive design processes and equitable treatment.
2. Trustworthiness for SAI Systems • Societal and Environmental Well-being: max-
imizing sustainability, social impact, and ecologi-
For people and society, trustworthiness is undoubtedly
cal responsibility of AI systems to positively con-
one of the prerequisites that AI systems should have to
tribute to society while minimizing negative con-
be used without hesitations [11]. It, therefore, becomes
sequences.
the starting point of our research because of its breadth
and multifaceted nature. In this section, the concept of • Accountability: creating mechanisms to ensure
trustworthiness is explored by analysing the perspectives accountability of AI systems, both before and af-
of European policymakers and academics to determine ter their development, deployment and use guar-
how to consider it in the context of SAI. antees fairness [11].
2.1. The European Commission
2.1.2. The Artificial Intelligence Act (AIA)
Perspective
Starting from the requirements of Trustworthy AI, listed
This section focuses on two documents drafted by the
in Section 2.1.1, in 2021, the EU has defined the AIA to
European Commission: the Ethics Guidelines for Trust-
regulate the adoption of harmonised and standardized
worthy AI and the AIA. The goal is to delineate a clear
rules for AI systems. Specifically, it merges trustwor-
image of the standpoints of policymakers to create AI
thiness with the risk-based approach to determine the
products that fully comply with laws, regulations, and
acceptability of the types of systems through norms and
norms and track the efforts of the EU concerning human
regulations [12]. The risk-based approach outlines four
rights, ethics, and philosophical issues.
categories of AI systems in relation to the risks they
might cause:
2.1.1. Ethics Guidelines for Trustworthy AI)
1. Unacceptable Risk: it encompasses systems that
The role of the AI HLEG is to define the approach of the might include prohibited AI practices that must
European Commission with respect to AI by indicating be banned to guarantee a well-functioning soci-
the key principles and policies. In 2019, they drafted ety, such as those that might threaten minorities
the "Ethics Guidelines for Trustworthy AI" report, which or those used by public authorities.
identifies seven requirements of Trustworthiness, identi-
fied as the umbrella property to ensure a human-centric 2. High Risk: it regards systems used in fields such
approach to AI [11, 12], illustrated in Figure 1. Such as education and vocational training, access to
requirements are briefly described in the following: private and public services, law enforcement, etc.
(e.g., usable, observable, explainable, resilient, agile, etc.)
[13].
The investigation of our research work consists in
understanding what principles are applicable to SAI and
identifying the potential new properties.
2.3. The Impact of Trustworthiness in SAI
Our objective is to define a framework that encompasses
both standpoints; in this regard, the authors are perform-
ing a Systematic Literature Review (SLR), following the
Kitchenham protocol, to identify the guidelines and prin-
ciples that can be drawn from the AIA that could be
applied to the lifecycle of SAI systems [14]. This SLR has
the objective to determine how the research community
is investigating and employing the AIA with respect to
the design and development AI. From the preliminary
Figure 1: The seven key requirements of Trustworthy AI: all results, it emerged that trustworthiness is intrinsic in
are of equal importance and support each other [11] SAI because humans must fully trust systems in order to
symbiotically with them.
Belonging to the domain of AI built following a human-
3. Limited Risk: it encompasses AI systems that centered approach, SAI can include Trustworthiness,
must comply with specific transparency obliga- Safety, and Reliability as principles; however, the estab-
tions because they interact with humans (e.g., bio- lishment of a symbiotic relationship might require their
metric recognition systems, and emotion recog- refinement or to the definition of new ones. The ongoing
nition systems). SLR will also serve to establish the new principles and
identify new guidelines suitable for the field of SAI.
4. Low or Minimal Risk: it refers to systems that
feature AI but do not require specific conformity
checks [1]. 3. Conceptual Framework for SAI
Systems
2.2. The Academic Perspective
The starting point is understanding the gaps in the tra-
Ben Shneiderman, one of the pioneers of HCI, proposes ditional approach to the development of AI systems to
trustworthiness as one of three principles, along with determine the changes to propose and the integration of
safety and reliability, of human-centered AI (HCAI) sys- new processes into the software lifecycle. This concep-
tems, which guarantee an appropriate balance of automa- tual framework aims to support designers and developers
tion and human control. Specifically: in creating and evaluating SAI systems. The objective
is to provide a standardized methodology to those who
• Trustworthiness concerns the property that makes
create AI-powered services that reduce the gap between
systems deserving of being trusted by humans.
technology and humans and decrease cognitive demand
• Reliability comes from the application of techni- when interpreting and understanding the outputs that
cal practices of software engineering that build systems produce.
systems that produce appropriate and/or ex- The objective of this work lies in defining a frame-
pected responses. work that considers and merges the two perspectives (i.e.
Ethics Guidelines for Trustworthy AI and AIA), while
• Safety is a strategy to guide the refinement of the identifying principles, guidelines, and techniques that
model performance to prevent potential failure belong to different disciplines by finding the appropriate
and improper use [13]. links. Figure 2 presents an initial version of the concep-
The three above mentioned properties are the most tual framework that consists in two layers, Design and
recurrent in the literature since they are the main areas Assessment, explained below.
of research and can encompass the other properties; nev-
ertheless, the state of the art concerning the human-AI
interaction, considers other 22 properties that can influ-
ence the design and development of any kind of system
Figure 2: Conceptual Comprehensive Framework for the design and the evaluation of Symbiotic AI
3.1. Design ploying AI as an instrument. The legal standpoint must
be considered for designing and developing AI systems
This layer embraces four main research areas that con-
to create products that comply with regulations and can
tribute equally: Human-Computer Interaction (HCI), Law
be released to the public. Currently, the main elements
& Ethics, Software Engineering (SE), and AI. The follow-
to consider are the AIA and the General Data Protec-
ing sections describe each component of the framework,
tion Regulation (GDPR); the first regulates the design,
illustrating its role in the SAI scenario.
development, and use of AI systems in the EU, while the
GDPR is a law that defines how data is handled, stored,
Human-Computer Interaction (HCI) HCI is one of and processed [15].
the pivotal components of this framework because the These regulations define the ethical principles that any
symbiotic relationship can be achieved if such systems kind of system should possess to be available to society.
allow users to reach their goals with effectiveness, effi-
ciency, and satisfaction, thus, by being usable and pro-
Artificial Intelligence (AI) This dimension refers to
viding a positive user experience. Other key elements
AI from a technical and algorithmic standpoint because
that HCI is responsible for are feedback and affordance,
the framework aims to suggest the appropriate tech-
enabling humans to understand how the system should
niques and practices to adopt depending on the require-
be used, making them feel at ease with proper commu-
ments of the systems to create. AI models, along with
nication [6]. Involving humans iteratively during each
high computational power, can be employed in multi-
phase of the system’s lifecycle implies performing inter-
ple domains, such as business, finance, healthcare, agri-
views, questionnaires, field studies, and focus groups to
culture, smart cities, and cybersecurity; however, they
perform quantitative and qualitative evaluations of the
cannot be used as a one-size-fits-all solution because, de-
systems and to obtain rich insights concerning the users’
pending on the activities, different tasks are needed - e.g.,
needs, preferences and cognitive models [6, 7].
classification, prediction, description -, raising the need
for context-specific models, parameters, and variables
Ethical & Legal Factors This dimension considers the [16]. The effectiveness of SAI systems is not guaranteed
regulatory, philosophical, and ethical standpoint since by simply obtaining high-performing models but rather
designers and developers must create products that pre- by systems that properly integrate Transparency, Explain-
serve users’ social, working, and personal well-being. ability, and Interpretability. This provides users with the
One of the main issues concerning AI, which becomes right instruments to comprehend the processes behind
particularly valid for the branch of SAI, consists of avoid- outputs, influencing their decisions, and what data is
ing biases and ensuring fairness. This element must be responsible for the system’s responses.
always considered because the root of biases is found
in how data is treated by AI models, for example, in the
Software Engineering (SE) This framework aims to
learning phase. This determines the unfair behavior of
guide design and developers in creating SAI systems, en-
systems that can influence humans’ decisions when em-
suring that they operate by following a human-centered
approach while complying with legal requirements and to assess the behavior and performance of such systems
implementing high-performing AI systems. Thus, the is crucial to ensure the proper deployment of AI, which is
objective is to integrate the Agile principles and the pro- part of the daily lives of countless individuals. As Trust-
cesses of the Agile Development Lifecycle with those worthiness plays a pivotal role in an effective human-AI
belonging to the SAI design, creating a mapping that interaction, the future of this research will focus on de-
does not exclude any discipline [17]. termining its complementary principles and its impact
on symbiosis by carrying out verticalized case studies
3.2. Assessment and performing in-depth investigations in the literature.
In this new scenario, where a strict correlation and con-
tamination exists between human and AI performance, Acknowledgments
it becomes essential to define novel metrics to assess the
human-AI symbiotic relationship. The research of Miriana Calvano and Antonio Curci is
Traditionally, human beings and AI have been viewed supported by the co-funding of the European Union -
as distinct and unrelated entities, causing UX and AI met- Next Generation EU: NRRP Initiative, Mission 4, Com-
rics to be defined independently to evaluate both human ponent 2, Investment 1.3 – Partnerships extended to
behavior and system performance. Considering them in universities, research centers, companies, and research
unison, it is possible to draft a preliminary set of met- D.D. MUR n. 341 del 15.03.2022 – Next Generation EU
rics that can be employed to assess the symbiosis. By (PE0000013 – “Future Artificial Intelligence Research –
integrating both the dataset and user information and FAIR” - CUP: H97G22000210007).
considering the user’s characteristics from the training
phase of the AI model, it is possible to foster symbio- References
sis, making the system’s behaviour as much as possible
adaptable to the user’s needs. [1] T. E. Commission, Proposal for a regulation of
Since Trustworthiness allows users to trust systems that the european parliament and of the council laying
operate safely and exhibit reliable behavior, it is contem- down harmonised rules on asrtificial intelligence
plated as one of the starting points of this research work (artificial intelligence act) and amending certain
[4]. Assessing this aspect is difficult since it varies across union legislative acts, 2024. URL: http://thomas.loc
many application contexts [4]; therefore, it is necessary .gov/cgi-bin/query/z?c102:H.CON.RES.1.IH.
to understand whether its evaluation should consider [2] C. Sanderson, D. Douglas, Q. Lu, E. Schleiger,
it as a stand-alone property or as an ensemble of other J. Whittle, J. Lacey, G. Newnham, S. Hajkowicz,
dimensions, such as safety, fairness, robustness, etc1 . C. Robinson, D. Hansen, Ai ethics principles in prac-
Two potential metrics are proposed to assess how tice: Perspectives of designers and developers, IEEE
Trustworthy an AI system is: Preventing Undesired Sys- Transactions on Technology and Society 4 (2023)
tem Behaviors, which refers to how effectively the system 171–187. URL: http://dx.doi.org/10.1109/TTS.2023.
avoids actions that could potentially harm the user or 3257303. doi:10.1109/tts.2023.3257303.
deviate from expected behavior; Correctness of Decisions, [3] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Gi-
which measures the extent to which system’s decisions annotti, D. Pedreschi, A Survey of Methods for
align with user expectations and desired outcomes. Explaining Black Box Models, ACM Computing
Surveys 51 (2019) 1–42. URL: https://dl.acm.org/doi
/10.1145/3236009. doi:10.1145/3236009.
4. Conclusions [4] B. Shneiderman, C. Plaisant, M. Cohen, S. Jacobs,
This paper presents preliminary considerations concern- N. Elmqvist, N. Diakopoulos, Designing the User
ing the novel field of Symbiotic AI with respect Trust- Interface: Strategies for Effective Human-Computer
worthiness. It presents the main challenges of identifying Interaction, 6 ed., Pearson Education, 2016. URL: ht
the principles of this field while stressing the need for a tps://books.google.it/books?id=PpItDAAAQBAJ.
human-centered approach when dealing with AI systems [5] I. O. for Standardization, Iso 9241:210 - ergonomics
of any kind. This research work is the starting ground for of human-system interaction, 2019. URL: https://
the definition of a comprehensive framework, presented www.iso.org/standard/77520.html.
in Section 3, that encompasses multiple disciplines and [6] H. Sharp, J. Preece, Y. Rogers, Interaction Design:
aims to guide designers and developers in creating SAI beyond human-computer interaction, 5 ed., John
systems. This framework is still in its early stages and at Wiley & Sons, Inc., 2019.
a conceptual state. Delineating a standardized approach [7] I. O. for Standardization, Iso 9241:210 - ergonomics
of human-system interaction: Human-centred de-
1
https://ec.europa.eu/futurium/en/ai-alliance-consultation.1.html
sign for interactive systems, 2019. URL: https:
//www.iso.org/standard/77520.html.
[8] P. Bourque, R. E. Fairley (Eds.), SWEBOK: guide to
the software engineering body of knowledge, ver-
sion 3.0 ed., IEEE Computer Society, Los Alamitos,
CA, 2014.
[9] S. S. Grigsby, Artificial intelligence for advanced
human-machine symbiosis, in: D. D. Schmorrow,
C. M. Fidopiastis (Eds.), Augmented Cognition: In-
telligent Technologies, Springer International Pub-
lishing, Cham, 2018, pp. 255–266.
[10] M. Vahabava, The risks associated with genera-
tive AI apps in the European Artificial Intelligence
Act (AIA), in: Workshops at the Second Interna-
tional Conference on Hybrid Human-Artificial In-
telligence (HHAI), CEUR Workshop Proceedings,
Munich, Germany, 2023, pp. 1–12.
[11] E. Commission, European commission - ethics
guidelines for trustworthy ai, 2021. URL: https:
//ec.europa.eu/f uturium/en/ai-alliance-consu
ltation.1.html.
[12] J. Laux, S. Wachter, B. Mittelstadt, Trustworthy ar-
tificial intelligence and the European Union AI act: On
the conflation of trustworthiness and acceptability
of risk, Regulation & Governance 18 (2024) 3–32.
URL: https://onlinelibrary.wiley.com/doi/10.1111/
rego.12512. doi:10.1111/rego.12512.
[13] B. Shneiderman, Human-Centered AI, 1 ed., Oxford
University PressOxford, 2022. URL: https://academ
ic.oup.com/book/41126. doi:10.1093/oso/9780
192845290.001.0001.
[14] B. Kitchenham, Procedures for Performing System-
atic Reviews, Technical Report, Keele University,
2004.
[15] Gazzetta Ufficiale dell’Unione Europea, General
Data Protection Regulation (GDPR): Regulation
(EU) 2016/679, 2018.
[16] I. Sarker, Ai-based modeling: Techniques, appli-
cations and research issues towards automation,
intelligent and smart systems, 2022. doi:10.20944
/preprints202202.0001.v1.
[17] D. Salah, R. F. Paige, P. Cairns, A systematic liter-
ature review for agile development processes and
user centred design integration, in: Proceedings
of the 18th International Conference on Evaluation
and Assessment in Software Engineering, ACM,
London England United Kingdom, 2014, pp. 1–10.
URL: https://dl.acm.org/doi/10.1145/2601248.26012
76. doi:10.1145/2601248.2601276.