<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Human-Centered Perspectives in Explainable Artificial Intelligence</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Muhammad Sufian</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Ilia Stepin</string-name>
          <email>ilia.stepin@usc.es</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Jose Maria Alonso-Moral</string-name>
          <email>josemaria.alonso.maral@usc.es</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Alessandro Bogliolo</string-name>
          <email>alessandro.bogliolo@uniurb.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="editor">
          <string-name>Artificial Intelligence, Explainable AI, Human-centered XAI, XAI Design Perspectives, Systematic Survey</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Pure and Applied Sciences, University of Urbino Carlo Bo</institution>
          ,
          <addr-line>Urbino</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2023</year>
      </pub-date>
      <fpage>0000</fpage>
      <lpage>0002</lpage>
      <abstract>
        <p>The widespread use of Artificial Intelligence (AI) in various domains has led to a growing demand for algorithmic understanding, transparency, and trustworthiness. The field of eXplainable AI (XAI) aims to develop techniques that can inspect and explain AI systems' behaviour in a way that is understandable to humans. However, the efectiveness of explanations depends on how users perceive them, and their acceptability is connected with the level of understanding and compatibility with users' existing knowledge. So far, researchers in XAI have primarily focused on technical aspects of explanations, mostly without considering users' needs, and this aspect is necessary to consider for a trustworthy AI. In the meantime, there is a growing interest in human-centered approaches that focus on the intersection between AI and human-computer interaction, what is termed as human-centered XAI (HC-XAI). HC-XAI explores methods to achieve user satisfaction, trust, and acceptance for XAI systems. This paper presents a systematic survey on HC-XAI, reviewing 75 papers from various digital libraries. The contributions of this paper include: (1) identifying common human-centered approaches, (2) providing readers with insights into design perspectives of HC-XAI approaches, and (3) categorising with quantitative and qualitative analysis of all the papers under study. The findings stimulate discussions and shed light on ongoing and upcoming research in HC-XAI.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Over the last two decades, Artificial Intelligence (AI) has received an overwhelming response
from daily life applications, and industries such as autonomous vehicles, financial services, and
healthcare [
        <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
        ]. As the utility of AI systems in every walk of life has increased, the call for
algorithmic understanding, transparency, and trustworthiness has become a matter of regulation
[
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. The European Union General Data Protection Regulation (GDPR) [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] refers to the “right
to explanation” of European citizens when their personal data are automatically processed
by AI systems. Moreover, the amendments to the new AI Act adopted at the first reading
in June 2023 [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] include the newly proposed Art. 68c “A right to explanation of individual
decision-making” which explicitly requires AI systems to be endowed with diferent levels of
transparency depending on the risk of their application; being mandatory to provide
explanations for algorithmic decisions when requested in case of high-risk applications. Accordingly,
transparency is considered to be a prerequisite for explainability. Thus, in eXplainable Artificial
Intelligence (XAI), researchers aim to develop techniques to inspect and explain the working of
AI systems that humans can understand easily [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. The goal is to increase the transparency,
understandability, and usability of such intelligent systems.
      </p>
      <p>
        Notice that, the development of XAI applications is complex because the efectiveness of an
explanation depends on how the user perceives it [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Making the model transparent does not
guarantee it will be fully understandable at the user end [7]. Hence, understandable explanations
are necessary, and the quality of an explanation is determined by its ability to provide relevant
information that can be understood and applied. Also, it should be compatible with the user’s
knowledge and needs [8]. However, researchers in the field of XAI have primarily focused on
the technical perspective of explanations, giving little attention to users’ needs [9, 10]. Many
algorithms and techniques are designed based on researchers’ intuition of what makes a “good”
explanation, but without considering its usability, efectiveness, and practicality for human
users [11].
      </p>
      <p>
        In recent years, the XAI community has shown a growing interest in human-centered
approaches [12, 13, 14, 15, 16]. To address the crucial requirements regarding transparency,
understanding and trustworthiness in XAI systems, a new area of research has emerged where
AI and human-computer interaction (HCI) intersect [17]. This research field is termed
humancentered XAI (HC-XAI) [18]. It is important to examine the existing theoretical and practical
frameworks for generation and evaluation of explanations, regarding both efectiveness and
impact of explanations on user’s trust and satisfaction. The consideration for user requirements
and contextual factors in the design and evaluation of explanations are deemed necessary [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ],
while tailoring explanations to specific user goals and preferences is challenging [ 19].
Numerous review articles on XAI have been published, exploring diferent aspects and highlighting
the challenges and potential research directions. Some surveys briefly touch upon evaluation
measures within a broader perspective of XAI [
        <xref ref-type="bibr" rid="ref1">20, 1</xref>
        ] or mainly discuss evaluation through user
studies [
        <xref ref-type="bibr" rid="ref7">21, 22, 23</xref>
        ], others have a narrower scope focusing on specific application domains or
subareas of XAI [
        <xref ref-type="bibr" rid="ref8">17, 24, 18</xref>
        ]. However, despite these eforts, the attention given specifically to
HC-XAI was limited and scattered, lacking a dedicated focus on the subject.
      </p>
      <p>In this paper, we conducted a systematic survey on HC-XAI. In contrast to previous surveys,
our approach paid major attention to research papers dealing with human-centered factors,
including artefacts and techniques which strive to address the human needs for explanations.
The survey focuses on reviewing 75 papers from Scopus, Web of Science, IEEE, and ACM digital
libraries. Our objectives are: 1) to lay out common approaches for HC-XAI, 2) to provide
insights on the design perspectives of common approaches, and 3) the categorisation of included
papers. To achieve these objectives, we conducted a systematic literature review driven by the
following specific research questions:
• RQ1: What are the most common human-centered approaches (including frameworks,
methods, theories, and artefacts) in XAI?
• RQ2: How is the co-design perspective considered in HC-XAI approaches?
• RQ3: How grounded are human-centered approaches on theoretical and practical aspects
of explanations?
Accordingly, we first categorise and provide insights into common approaches for HC-XAI.
Then, we do quantitative and qualitative analyses of the papers under study. Our overarching
goal is to bring attention to the ongoing and upcoming research related to HC-XAI. Therefore,
the rest of the paper is organised as follows. Section 2 introduces background and related work.
Section 3 details the methods used in the systematic literature review. Section 4 presents the
most outstanding quantitative results. Section 5 provides a qualitative analysis of the most
relevant papers. Section 6 discusses implications and research opportunities. Section 7 provides
readers with final remarks.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Background and Related Work</title>
      <p>
        Human-centered explanations are designed and tailored to meet the needs, understanding,
and cognitive abilities of human recipients [
        <xref ref-type="bibr" rid="ref7">23, 18</xref>
        ]. These explanations aim to enhance the
comprehensibility and accessibility of complex information for individuals by considering
their unique perspectives, backgrounds, and experiences. The central concept driving
humancentered explanations is to bridge the gap between specialised (technical) knowledge and the
broader (or specific) target audience. By prioritising the human recipient, explanations are
easier to understand, facilitate informed decision-making, and foster transparency, trust, and
engagement between experts and the wider public [
        <xref ref-type="bibr" rid="ref9">25, 17</xref>
        ].
      </p>
      <p>
        On the one hand, there are numerous reviews available on the expanding field of XAI. They
play a vital role in defining and establishing the concept of XAI [
        <xref ref-type="bibr" rid="ref1 ref10">26, 1</xref>
        ]. They delve into exploring
the interconnections between XAI and related fields of study [
        <xref ref-type="bibr" rid="ref10">26, 15</xref>
        ], categorising diferent
methodologies [
        <xref ref-type="bibr" rid="ref11 ref8">24, 27</xref>
        ], analysing the user’s perspective [21], examining evaluation practices
[
        <xref ref-type="bibr" rid="ref12">28</xref>
        ], and proposing future directions for research [
        <xref ref-type="bibr" rid="ref1 ref10">26, 1</xref>
        ]. However, it is worth noting that the
importance of human-centered approaches in the context of XAI is only briefly acknowledged
in previous reviews.
      </p>
      <p>
        On the other hand, in recent years, there is a growing interest to highlight the needs and
importance of human-centered approaches [12, 13, 14, 15, 16]. To address this aspect, Chromik
and Schuessler [
        <xref ref-type="bibr" rid="ref13">29</xref>
        ] proposed a taxonomy that incorporates human perspectives for evaluating
XAI. Additionally, Lai et al. [
        <xref ref-type="bibr" rid="ref9">25</xref>
        ] conducted a comprehensive review of studies focusing on
collaborative human-AI decision-making. Their review specifically explores the role of
explanations in evaluating the success of such collaborative eforts. Furthermore, Ferreira and
Monteiro [21] paid attention to the user experience in XAI applications. They carried out an
analysis of the users themselves, their motivations, and the contextual factors that influence the
presentation of explanations.
      </p>
      <p>In the following sections we will provide readers with a systematic literature survey to
categorise existing approaches which introduce, devise, or apply human-centered artefacts and
techniques for XAI.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Methods</title>
      <p>
        In this section, we outline our methodology for identification, screening, eligibility (reviewing),
and inclusion of papers. To ensure comprehensive insights into the HC-XAI domain, we
conducted a systematic collection of papers, regarding both quantitative and qualitative information.
Our methodology follows the guidelines established by Moher et al. [
        <xref ref-type="bibr" rid="ref14">30</xref>
        ] for preferred reporting
items for systematic reviews and meta-analyses (PRISMA).
      </p>
      <p>Publication Search Engine. In order to ensure a comprehensive and manageable selection
of papers, we selected interdisciplinary databases (search engines) such as Web of Science (WoS),
Scopus, ACM Digital Library, and IEEE digital Explorer for the initial selection of papers. These
databases not only encompass research publications in the field of Computer Science, but also
index studies in scientific fields such as AI, HCI, Social Sciences and Philosophy; enabling a
comprehensive review of the literature related to our research questions.</p>
      <p>Publication Year. We limited our selection to papers published between 2018 and mid-May,
2023. This time frame was chosen due to the significant surge of interest in XAI in recent years,
particularly following the implementation of regulations such as the GDPR and AI act in Europe,
as well as other AI-related acts in the UK and the US.</p>
      <p>Search Strategy. On May 15th, 2023, we conducted the advanced search using web tools
provided by the selected digital databases. These tools allow for the replication of the study
and ensure consistent querying across all the databases. We performed a query consisting of
multiple terms on the title, abstract, and author keywords.</p>
      <p>The query for the digital databases was as follows: ((expla* AI OR expla* Artificial Intelligence
OR Counterfactual expla*) OR (interpret* AI OR interpret* Artificial Intelligence)) AND (generat*
OR framework* OR develop* OR (human-centered OR human-centred) OR (user-centered
OR user-centred)). To maximise the diversity of the retrieved papers, we used word-stems
in our search query. For example, the search item “expla*” covered all word-forms such as
“explanation”, “explaining”, “explanatory”, etc. We used the search terms to gather the most
recent publications that mention explainable AI and interpretable AI, respectively, across all
subject areas. Also, the search query was overlapping to diferentiate between publications
covering human-centered and user-centered development frameworks (to cover the broader
perspective of human-centricity). The terms “generate”, “framework”, and “develop”, along with
their corresponding word-forms, were used to appropriately limit the pool of publications in
the unified set.</p>
      <p>Inclusion and Exclusion. We focused on papers within the domain of HC-XAI. To be
included, papers needed to present original work that introduced, applied, proposed, or evaluated
human-centered methods or theories for explaining AI techniques, models, or systems. The
initial filtering of papers (screening) was based on title, abstract and author keywords. Our
paper selection process focused on original research articles that underwent peer-review and
were written in English. Prior to evaluating the main content of each paper based on our
inclusion criteria (eligibility), we manually excluded papers that comprised extended abstracts,
doctoral consortium submissions, early career tracks, invited talks or tutorials, as well as all
records which were not written in English.</p>
      <p>PRISMA Guidelines. The flow diagram for the systematic literature review is shown in
Figure 1, in which we illustrate diferent phases in agreement with the PRISMA guidelines.</p>
      <p>Identification</p>
      <p>Screening</p>
      <p>Eligibility</p>
      <p>Included
Initial
Search</p>
      <p>Scopus (n=189)</p>
      <p>WoS (n=41)
ACM Digital Library (n=79)
IEEE Digital Library (n=18)</p>
      <p>Title and abstract screening
(n=238)</p>
      <p>Full text screening (n=82)
Duplicates removed (n=89)</p>
      <p>Papers excluded (n=156)</p>
      <p>Papers excluded (n=19)</p>
      <p>Papers covered in survey</p>
      <p>n=(75)
Papers included upon
snowballing (n=12)</p>
      <p>In the identification phase, we conducted an initial search across the selected digital libraries,
using the predefined search query mentioned earlier. The results from this search yielded 18
papers from the IEEE digital library, 79 papers from the ACM digital library, 41 papers from
WoS, and 189 papers from Scopus making a total of 327 papers. These relatively low numbers
can be attributed to the limited research activity on HC-XAI.</p>
      <p>In the screening phase, the relevant information (i.e., title, abstract, keywords, authors’
names, journal name, source, publisher, and year of publication) from the identified records was
exported to a Microsoft Excel spreadsheet. A total of 238 records were left after 89 duplicates
were removed. These 238 papers were then subjected to screening based on the inclusion
criterion. After the screening process, 156 papers were deemed ineligible and excluded from
further consideration. The remaining 82 papers entered the eligibility phase, where each paper
underwent a thorough analysis through full-text reading. In the eligibility phase, a decision was
made on each paper, determining whether it would proceed to the final phase. Unfortunately,
19 papers did not adequately address to our research questions and were excluded, leaving a
total of 63 papers for the final phase.</p>
      <p>
        During the final inclusion phase, 12 additional papers were included through a snowballing
process [
        <xref ref-type="bibr" rid="ref15">31</xref>
        ]: we identify and add other relevant publications manually from the bibliographies
of the already selected manuscripts and relevant papers suggested by the peers of the authors,
ensuring maximum coverage of the related subject areas. As a result, we had a set of 75 papers
for synthesis and detailed meta-analysis.
      </p>
      <p>
        Review Process. We conducted a thorough analysis of the final set of papers, following the
taxonomy proposed by Guidotti et al. [
        <xref ref-type="bibr" rid="ref8">24</xref>
        ]. To delve into the specifics of this taxonomy, we
kindly refer interested readers to Section 4 of Guidotti et al. survey [
        <xref ref-type="bibr" rid="ref8">24</xref>
        ]. In short, we examined
and categorised contributions from multiple perspectives. The process began by reviewing the
main content (excluding appendices and supplementary material) of each paper. Then, papers
were categorised across four main dimensions: (1) the main contribution—whether the papers
introduce, devise, or apply an approach, (2) the type of input data, (3) the specific type of task,
and (4) the type of explanation.
      </p>
    </sec>
    <sec id="sec-4">
      <title>4. Quantitative Analysis</title>
      <p>Before addressing the research questions and related answers, we conducted a bibliometric
analysis on the aggregated query results. This analysis allowed us to gain a comprehensive
understanding of the HC-XAI research field.</p>
      <sec id="sec-4-1">
        <title>4.1. Statistics of Included Papers</title>
        <p>
          Figure 3 illustrates the growth of papers in the field of HC-XAI over the past years (from 2018
to mid-May 2023). The small number of papers published in 2018 (5) and 2019 (6), could be
attributed to the relatively lower popularity of HC-XAI during those years (which coincide
with the initial years of application of the GDPR) or to the limited usage of the related terms
(i.e., “human-centered” and “user-centered”) at that time. Anyway, there has been a consistent
increase in the number of papers since 2020, with a very significant rise in 2022, even if we do
not observe in HC-XAI the same exponential growth noticed by Adadi and Berrada [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ] for XAI
in general. Notice that, the decrease in 2023 is due to the fact that only 4.5 months are taken
into account. All in all, the observed trend suggests a growing awareness of the importance and
necessity of HC-XAI methods in recent years. Finally, regarding a comparison among papers
that apply (blue line), evaluate (orange line), devise (green line), and other surveys and theories
(red line) for HC-XAI has been shown in 3. There is a significant trend for devising methods
which seem to be gaining more and more attention.
        </p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Categorisation of Included Papers</title>
        <p>The categorisation process considers various dimensions such as the type of data used, the
specific task being addressed, the approach taken in contributing to the field (whether it involves
application, evaluation, or development), and the specific type of explanation employed to clarify
the underlying predictions or decisions made by the intelligent system.</p>
        <sec id="sec-4-2-1">
          <title>4.2.1. Type of Data, Task, and Explanation</title>
          <p>
            Diferent data types are taken as input for predictive models such as image, text, and tabular
data. Our findings indicate that image data (8%) predominantly constitute the majority in
heatmap explanations, tabular data (57%) for feature scores and counterfactual explanations
while text input (6%) is primarily employed for textual explanations. The category “Other” (28%)
is introduced to account for such data types as time series, graphs, or logical rules (ontology) that
do not fit the predefined categories. Of course, it is important to note that these statistics may
be influenced by our choice of publication venues, and the complexity of the task itself can also
have an impact. Table 1 presents a quantitative analysis of various types of tasks, along with the
related data types for those tasks, in the included papers and their corresponding bibliographical
references. These are 49 papers out of 75 which solve any task in their contributing work.
As shown in Table 1, a significant portion (51%) of the papers focus on classification models.
This can be attributed to the broad nature of classification tasks and the popularity of posthoc
explanations, as explaining specific classification decisions serves as a compelling use case for
outcome explanations. We observed that classification tasks were performed by using mainly
tabular data, image or text, but also other data such as time series or graphs [
            <xref ref-type="bibr" rid="ref17 ref18 ref19 ref2">8, 2, 33, 34, 35</xref>
            ].
          </p>
          <p>
            The second largest category of papers (29%) addressed both classification and regression
tasks concurrently in their studies. Notice that, papers which only deal with regression tasks
consider only tabular data [55, 56, 57]. In addition, papers which account for recommendation
tasks, they work with tabular [
            <xref ref-type="bibr" rid="ref41 ref42 ref43">14, 70, 71, 72</xref>
            ] and other graph-based data [
            <xref ref-type="bibr" rid="ref44">73</xref>
            ]. In other tasks,
Jiao et al. [
            <xref ref-type="bibr" rid="ref45">74</xref>
            ] use textual data for investigating the task of question answering (QA) through
the scenario-based design, while Shruthi et al. devise an explanation supported by ontological
data [
            <xref ref-type="bibr" rid="ref46">75</xref>
            ].
          </p>
          <p>On the other hand, we found out a diversity in the types of explanations under study. We
categorised specific explanations in five main distinct categories which apply, devise, and
evaluate (49 out of 75 papers), as outlined in Table 2. Notably, the most prevalent explanation
type is the so-called hybrid approach which combines feature scores and counterfactual methods,
which was found in 37% of the papers. This is followed by the use of counterfactuals alone
(27%), feature scores alone (20%), and text highlighting (8%). Other explanation types include
saliency maps, heatmaps, ontology, and rules, each accounting for 2% of the papers.</p>
          <p>
            Moreover, the analysis also highlights the dominance of approaches for explaining black-box
models in current research on human-AI interaction. Local feature explanations, exemplified by
LIME [9] and SHAP [10], are commonly employed to answer the “why” question. Counterfactual
explanations, on the other hand, are used to tackle “what-if” or “why-not” questions. In the
context of recommendation systems, a hybrid approach combining text highlighting, rules,
and counterfactuals is also utilised, such as in a hybrid recommendation system [62]. Other
explanation types include text highlighting, saliency maps, heatmaps, rules, and an ontological
representation of explanations. Text highlighting, as referenced in papers [
            <xref ref-type="bibr" rid="ref42 ref45">14, 74, 64, 71</xref>
            ], allows
for emphasising specific textual elements to aid in understanding the explanation. Saliency
maps, as employed in one of the papers [
            <xref ref-type="bibr" rid="ref34">50</xref>
            ], provide visual representations highlighting the
most relevant areas or features. Heatmaps [
            <xref ref-type="bibr" rid="ref36">52</xref>
            ] ofer a graphical depiction of the intensity or
importance of diferent elements within the explanation. Rules, as discussed in another paper
[
            <xref ref-type="bibr" rid="ref44">73</xref>
            ], provide structured statements or conditions to explain AI decisions. Lastly, the utilisation of
an ontological representation [
            <xref ref-type="bibr" rid="ref46">75</xref>
            ] of explanations is also observed. These additional techniques
contribute to the diverse range of approaches used for generating explanations in the included
papers.
          </p>
        </sec>
        <sec id="sec-4-2-2">
          <title>4.2.2. Type of Contribution and HC-XAI Methods.</title>
          <p>The objective of this section is to provide insights to researchers and practitioners seeking
appropriate contributions in the field of HC-XAI. In Table 3, we present the categorisation
of papers that devise (introduce), apply, and evaluate the existing methods along with their
specific contribution. It is worth noting that when a method devises an HC-XAI approach,
it can be linked to multiple types of explanations, as there is no precondition of one-to-one
correspondence between a method and an explanation type in the set of papers under study.</p>
          <p>
            By examining the included papers, a categorisation of main contributions is presented in
Table 3. The first group of papers provides theoretical guidelines for designing HC-XAI
frameworks. The second group consists of user studies and interviews. We also refer to previous
user studies conducted on ML systems [
            <xref ref-type="bibr" rid="ref28">44</xref>
            ] or explainable interfaces [17], which can be used
for comparison or serve as templates for designing user studies [22]. The third group includes
design paradigms for explanations. The fourth group deals with cognitive and socio-technical
theories. For example, Miller [15] suggested building XAI on social sciences such as cognitive
science and psychology. Additionally, XAI aims to assist users in developing mental models of
AI systems [
            <xref ref-type="bibr" rid="ref35">51</xref>
            ]. The fith group highlights the needs for HC-XAI [
            <xref ref-type="bibr" rid="ref64 ref65">93, 94</xref>
            ]. Finally, the sixth
group includes systematic surveys [
            <xref ref-type="bibr" rid="ref66">13, 95, 96</xref>
            ].
          </p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Qualitative Analysis</title>
      <p>In this section, we go in depth with answering, from a qualitative viewpoint, the three research
questions that we posed in the introduction.</p>
      <sec id="sec-5-1">
        <title>5.1. XAI Objectives Identified in Included Papers</title>
        <p>RQ1 deals with frameworks, methods, theories and artefacts that support HC-XAI. First of all,
we paid attention to generic concerns in the context of XAI (see Table 4).</p>
        <p>The primary studies highlighted multiple objectives for XAI, with many papers
emphasising the general reasons and prerequisites for their implementation. Fostering user trust and
enhancing transparency emerged as the most prominent goals among the commonly
identiifed objectives. Another recurring theme observed in the studies revolved around catering to
the user satisfaction as well as user needs, user experience, and social needs across diverse
domains. This emphasis stems from the fact that diferent users (including domain experts,
end-consumers, AI engineers, legislators...) possess unique motivations for seeking explanations.
The purpose of XAI is to efectively assist each user in achieving their specific objectives by
tailoring the explanations to their individual needs. User-centered approaches ensure that XAI
accommodates properly diverse requirements and aspirations of users across various domains.
We only found a few papers to aid user future actions and promoting informed decision-making.
Other recurring objectives which require further attention are related to the improvement of
the human-AI collaboration and thus increasing the task performance.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. Interaction and Dialogue-based Explanation Methods</title>
        <p>RQ2 investigated XAI techniques that are supported by a co-design perspective, i.e., they
incorporate interaction or dialogue as a human-centered element and play a crucial role in
enhancing transparency and understanding of AI systems. These techniques aim to bridge
the gap between complex ML models and human users by enabling meaningful conversations
and explanations. Through the integration of dialogue, XAI techniques foster a more inclusive
and user-centered approach, ensuring that AI systems are not black boxes but rather tools that
empower individuals to make informed decisions. For example, by allowing users to engage in
a dialogue with an AI system, Stepin et al. [98] facilitate a two-way exchange of information
through an information-seeking dialogue, where the AI system provides explanations for its
decisions and users can seek clarification or express their concerns.</p>
        <p>
          In addition, Akula et al. [
          <xref ref-type="bibr" rid="ref36">52</xref>
          ] introduced a mind-based framework to enhance user’s trust in
counterfactual explanations for image data. The end user can interact to seek desired
counterfactual outcomes through a dialogue-based interface. Cabour et al. [
          <xref ref-type="bibr" rid="ref23">39</xref>
          ] also highlighted the
benefit of combining the technical development of XAI systems with a proper identification and
interpretation of the user needs. They proposed an architecture to define the explanation space
from a user-inspired perspective. This architecture mainly caters to five elements: the end-users’
mental models, cognitive process, interface, the human-explainer agent, and the agent process.
Sufian et al. [ 16] also remarked the need for human-involvement in the explanation generation
process by customising explanations with user feedback. Their framework customises
counterfactual explanations on demand. Finally, Zhu et al. [
          <xref ref-type="bibr" rid="ref39">68</xref>
          ] proposed an XAI framework for
Designers, specifically tailored for game designers. Their human-centered approach facilitates
game designers to co-create with AI techniques by focusing on specific users, their needs and
tasks. Human-AI interaction and dialogue promote trust, accountability, and collaboration, as
users gain insights into the decision-making process and can provide input and feedback.
        </p>
      </sec>
      <sec id="sec-5-3">
        <title>5.3. Theoretical and Practical Aspects of Explanations</title>
        <p>RQ3 investigated HC-XAI methods which are grounded on theoretical and practical evaluation
approaches. Evaluation of explanations can be supported by functionally-grounded,
humangrounded, or application-grounded approaches (see Table 5). We adhere to the categorisation of
metrics proposed by Doshi-Velez and Kim [19].</p>
        <p>Functionally-grounded metrics do not require any human feedback. They are based on
objective criteria that can be measured without human involvement. As a result, they measure
the formal properties of the explainer, while regarding functional aspects of explanations (e.g.,
ifdelity, accuracy, actionability, sparsity, or plausibility). Human-grounded metrics require direct
human involvement for their measurement. They emphasise the human perspective and consider
cognitive and psychological factors (e.g., satisfaction, persuasiveness, or novelty). These metrics
gather feedback from users to assess the quality and usefulness of explanations.
Applicationgrounded metrics focus on real-world applicability and impact. Their computation usually
involves domain experts who are asked to assess how well the intelligent system performs
within specific application domains, considering practical constraints and requirements.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Research Opportunities</title>
      <p>Insights from the systematic literature review carried out in the previous sections indicate
a pressing need to develop human-centered explanation approaches in various fields of AI.
Traditional methods of explaining AI decisions often fall short in meeting human requirements
for transparency and understanding. However, a promising avenue that has gained considerable
attention is the utilisation of human-perspective for explanation generation and evaluation.</p>
      <p>Human-centered explanations play a critical role in promoting transparency, trust, and
accountability in AI systems. Such explanations enable users to understand the reasoning
behind automated decisions and empower them to make informed choices. However, existing
research indicates a gap in actionable and human-centered explanations, what leads to some
research opportunities for the exploration of human-centered approaches in the field of HC-XAI:
• Bridging the explanation gap by handling user needs. The systematic literature
review reveals a persistent gap in providing actionable explanations that are meaningful
and relevant to users. The explanations driven by user feedback ofer a promising
approach to bridge the gap. By incorporating user preferences and context, explanations
can provide actionable insights that resonate with users and facilitate more informed
decision-making.
• Personalisation and Contextualisation. Diferent users may have diverse preferences,
values, and needs, and a one-size-fits-all explanation may not be suficient. By involving
users in the explanation generation process, explanations can adapt to individual user
characteristics and provide insights that are tailored to their specific context.
• Closing the Feedback Loop. The literature review indicates a need to close the feedback
loop between users and AI systems, enabling iterative improvements and
accountability. This iterative feedback loop can enhance system performance, address biases and
limitations, but also ensure the system evolves based on user needs and expectations.
• Human-AI Collaboration and Co-creation. Human-centred explanations promote
human-AI collaboration and co-creation. By involving users in the generation and
evaluation of explanations, we facilitate a meaningful dialogue between users and AI
systems. Users can: provide feedback, ask questions, actively participate in refining and
improving the system’s behaviour.</p>
    </sec>
    <sec id="sec-7">
      <title>7. Conclusion</title>
      <p>In this work, we have presented a systematic literature review from which we can conclude that
there is a need for more human-centered explanation approaches. Such HC-XAI approaches
can enhance human understanding, enable ethical evaluations, and foster user engagement,
thereby serving the “human-in-the-loop” cause. Moreover, building human-centered
explanations means bridging the explanation gap, enhancing user agency, enabling personalisation and
contextualisation, addressing trust and ethical concerns, closing the feedback loop, and
promoting human-AI collaboration. Consequently, we can bridge the gap between complex algorithms
and human intuition, empowering individuals to make informed decisions, identifying biases,
and actively taking part in the development of responsible AI technologies.</p>
      <p>Even though we conducted a medium-scale survey, it was more feasible and practical than
larger surveys, providing valuable insights into human-centered approaches, while requiring
fewer resources, such as time and human resources. Despite its limitations, this survey can
provide a starting point for further research and help determine the viability of pursuing
largerscale surveys. In conclusion, various factors can influence the efectiveness of explanations in
human-centered perspectives. These factors include the specific explanation technique used,
the characteristics of the dataset, and the nature of the task at hand.</p>
    </sec>
    <sec id="sec-8">
      <title>Acknowledgments</title>
      <p>Muhammad Sufian is a PhD researcher (Matricola N.309445). Ilia Stepin is an
FPI researcher (grant number: PRE2019-090153). This research was funded by
MCIN/AEI/10.13039/501100011033 (grants PID2021-123152OB-C21, TED2021-130295B-C33,
and RED2022-134315-T), the Galician Ministry of Culture, Education, Professional Training,
and University (grants ED431C2022/19 and ED431G2019/04). All grants were co-funded by the
European Regional Development Fund (ERDF/FEDER program).
[7] C. Rudin, Stop explaining black box machine learning models for high stakes decisions
and use interpretable models instead, Nature machine intelligence 1 (2019) 206–215.
[8] X. He, Y. Hong, X. Zheng, Y. Zhang, What are the users’ needs? design of a user-centered
explainable artificial intelligence diagnostic system, International Journal of Human–
Computer Interaction 39 (2023) 1519–1542.
[9] M. T. Ribeiro, S. Singh, C. Guestrin, Model-agnostic interpretability of machine learning,
arXiv preprint arXiv:1606.05386 (2016).
[10] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in:
Proceedings of the 31st International Conference on Neural Information Processing Systems,
NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 2017, p. 4768–4777.
[11] M. Ribera, A. Lapedriza, Can we do better explanations? a proposal of user-centered
explainable AI, in: IUI workshops, volume 2327, 2019, p. 38.
[12] U. Ehsan, M. O. Riedl, Human-centered explainable AI: Towards a reflective sociotechnical
approach, in: HCI International 2020-Late Breaking Papers: Multimodality and Intelligence:
22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020,
Proceedings 22, Springer, 2020, pp. 449–466.
[13] H. Chen, C. Gomez, C.-M. Huang, M. Unberath, Explainable medical imaging AI needs
human-centered design: guidelines and evidence from a systematic review, npj Digital
Medicine 5 (2022) 156.
[14] J. Graefe, S. Paden, D. Engelhardt, K. Bengler, Human centered explainability for intelligent
vehicles–a user study, in: Proceedings of the 14th International Conference on Automotive
User Interfaces and Interactive Vehicular Applications, 2022, pp. 297–306.
[15] T. Miller, Explanation in artificial intelligence: Insights from the social sciences, Artificial
intelligence 267 (2019) 1–38.
[16] M. Sufian, P. Graziani, J. M. Alonso, A. Bogliolo, FCE: Feedback based counterfactual
explanations for explainable AI, IEEE Access 10 (2022) 72363–72372.
[17] Q. V. Liao, M. Pribić, J. Han, S. Miller, D. Sow, Question-driven design process for explainable</p>
      <p>AI user experiences, arXiv preprint arXiv:2104.03483 (2021).
[18] U. Ehsan, P. Wintersberger, Q. V. Liao, M. Mara, M. Streit, S. Wachter, A. Riener, M. O.</p>
      <p>Riedl, Operationalizing human-centered perspectives in explainable AI, in: Extended
Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp.
1–6.
[19] F. Doshi-Velez, B. Kim, Towards a rigorous science of interpretable machine learning,
arXiv preprint arXiv:1702.08608 (2017).
[20] S. Mohseni, N. Zarei, E. D. Ragan, A multidisciplinary survey and framework for design
and evaluation of explainable AI systems, ACM Transactions on Interactive Intelligent
Systems (TiiS) 11 (2021) 1–45.
[21] J. J. Ferreira, M. S. Monteiro, What are people doing about XAI user experience? a survey
on AI explainability research and practice, in: Design, User Experience, and Usability.
Design for Contemporary Interactive Environments: 9th International Conference, DUXU
2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen,
Denmark, July 19–24, 2020, Proceedings, Part II 22, Springer, 2020, pp. 56–73.
[22] D. Wang, Q. Yang, A. Abdul, B. Y. Lim, Designing theory-driven user-centric explainable
AI, in: Proceedings of the 2019 CHI conference on human factors in computing systems,
do not behave like users expect, in: Proceedings of the 30th ACM Conference on User
Modeling, Adaptation and Personalization, 2022, pp. 110–120.
[55] H. Ma, K. McAreavey, R. McConville, W. Liu, Explainable AI for non-experts: Energy
tarif forecasting, in: 2022 27th International Conference on Automation and Computing
(ICAC), IEEE, 2022, pp. 1–6.
[56] C. Bove, J. Aigrain, M.-J. Lesot, C. Tijus, M. Detyniecki, Contextualization and exploration
of local feature importance explanations to improve understanding and satisfaction of
non-expert users, in: 27th international conference on intelligent user interfaces, 2022, pp.
807–819.
[57] B. Hayes, M. Moniz, Trustworthy human-centered automation through explainable AI
and high-fidelity simulation, in: Advances in Simulation and Digital Human Modeling:
Proceedings of the AHFE 2020 Virtual Conferences on Human Factors and Simulation,
and Digital Human Modeling and Applied Optimization, July 16-20, 2020, USA, Springer,
2021, pp. 3–9.
[58] M. Förster, P. Hühn, M. Klier, K. Kluge, User-centric explainable AI: design and evaluation
of an approach to generate coherent counterfactual explanations for structured data,
Journal of Decision Systems (2022) 1–32.
[59] D. Cirqueira, M. Helfert, M. Bezbradica, Towards design principles for user-centric
explainable AI in fraud detection, in: Artificial Intelligence in HCI: Second International
Conference, AI-HCI 2021, Held as Part of the 23rd HCI International Conference, HCII
2021, Virtual Event, July 24–29, 2021, Proceedings, Springer, 2021, pp. 21–40.
[60] A. A. Shrotri, N. Narodytska, A. Ignatiev, K. S. Meel, J. Marques-Silva, M. Y. Vardi,
Constraint-driven explanations for black-box ML models, in: Proceedings of the AAAI
Conference on Artificial Intelligence, volume 36, 2022, pp. 8304–8314.
[61] H. Baniecki, D. Parzych, P. Biecek, The grammar of interactive explanatory model analysis,</p>
      <p>Data Mining and Knowledge Discovery (2023) 1–37.
[62] A. Silva, M. Schrum, E. Hedlund-Botti, N. Gopalan, M. Gombolay, Explainable artificial
intelligence: Evaluating the objective and subjective impacts of XAI on human-agent
interaction, International Journal of Human–Computer Interaction 39 (2023) 1390–1404.
[63] Q. V. Liao, D. Gruen, S. Miller, Questioning the AI: informing design practices for
explainable AI user experiences, in: Proceedings of the 2020 CHI Conference on Human Factors
in Computing Systems, 2020, pp. 1–15.
[64] J. E. Sales, A. Freitas, S. Handschuh, A user-centred analysis of explanations for a
multicomponent semantic parser, in: Natural Language Processing and Information Systems:
25th International Conference on Applications of Natural Language to Information Systems,
NLDB 2020, Saarbrücken, Germany, June 24–26, 2020, Proceedings 25, Springer, 2020, pp.
37–44.
[65] F. Sovrano, F. Vitali, Explanatory artificial intelligence (YAI): human-centered explanations
of explainable AI and complex data, Data Mining and Knowledge Discovery (2022) 1–28.
[66] C. Panigutti, A. Beretta, D. Fadda, F. Giannotti, D. Pedreschi, A. Perotti, S. Rinzivillo,
Codesign of human-centered, explainable AI for clinical decision support, ACM Transactions
on Interactive Intelligent Systems (2023).
[67] D. H. Kim, E. Hoque, M. Agrawala, Answering questions about charts and generating
visual explanations, in: Proceedings of the 2020 CHI conference on human factors in
(COMPASS), 2022, pp. 439–452.
[96] A. Bertrand, R. Belloum, J. R. Eagan, W. Maxwell, How cognitive biases afect XAI-assisted
decision-making: A systematic review, in: Proceedings of the 2022 AAAI/ACM conference
on AI, ethics, and society, 2022, pp. 78–91.
[97] M. Sufian, M. Y. Khan, A. Bogliolo, Towards human cognition level-based experiment
design for counterfactual explanations, in: 2022 Mohammad Ali Jinnah University
International Conference on Computing (MAJICC), 2022, pp. 1–5. doi:10.1109/MAJICC56935.
2022.9994203.
[98] I. Stepin, K. Budzynska, A. Catala, M. Pereira-Fariña, J. M. Alonso-Moral,
Informationseeking dialogue for explainable artificial intelligence: Modelling and analytics, Argument
&amp; Computation (2023).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>A.</given-names>
            <surname>Adadi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Berrada</surname>
          </string-name>
          ,
          <article-title>Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)</article-title>
          ,
          <source>IEEE access 6</source>
          (
          <year>2018</year>
          )
          <fpage>52138</fpage>
          -
          <lpage>52160</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>D.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Song</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Shin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <article-title>How should the results of artificial intelligence be explained to users?-research on consumer preferences in user-centered explainable artificial intelligence</article-title>
          ,
          <source>Technological Forecasting and Social Change</source>
          <volume>188</volume>
          (
          <year>2023</year>
          )
          <fpage>122343</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>S.</given-names>
            <surname>Ali</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Abuhmed</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>El-Sappagh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Muhammad</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Alonso-Moral</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Confalonieri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Guidotti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. D.</given-names>
            <surname>Ser</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Díaz-Rodríguez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Herrera</surname>
          </string-name>
          ,
          <article-title>Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence</article-title>
          ,
          <source>Information Fusion</source>
          (
          <year>2023</year>
          )
          <article-title>101805</article-title>
          . URL: https://www.sciencedirect.com/science/article/pii/ S1566253523001148. doi:https://doi.org/10.1016/j.inffus.
          <year>2023</year>
          .
          <volume>101805</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>P.</given-names>
            <surname>Voigt</surname>
          </string-name>
          ,
          <string-name>
            <surname>A.</surname>
          </string-name>
          <article-title>Von dem Bussche, The EU general data protection regulation (GDPR), A Practical Guide</article-title>
          , 1st Ed., Cham: Springer International Publishing
          <volume>10</volume>
          (
          <year>2017</year>
          )
          <fpage>10</fpage>
          -
          <lpage>5555</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <article-title>[5] Parliament and Council of the European Union, Proposal for laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts</article-title>
          ,
          <year>2023</year>
          . https://www.europarl.europa.eu/doceo/document/TA-9
          <article-title>-2023-0236_EN</article-title>
          .pdf.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>T.</given-names>
            <surname>Ha</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y. J.</given-names>
            <surname>Sah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Park</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <article-title>Examining the efects of power status of an explainable artificial intelligence system on users' perceptions</article-title>
          ,
          <source>Behaviour &amp; Information Technology</source>
          <volume>41</volume>
          (
          <year>2022</year>
          )
          <fpage>946</fpage>
          -
          <lpage>958</lpage>
          .
          <year>2019</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>15</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Rong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Leemann</surname>
          </string-name>
          , T.-t. Nguyen,
          <string-name>
            <given-names>L.</given-names>
            <surname>Fiedler</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Seidel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Kasneci</surname>
          </string-name>
          , E. Kasneci,
          <article-title>Towards human-centered explainable AI: User studies for model explanations</article-title>
          ,
          <source>arXiv preprint arXiv:2210.11584</source>
          (
          <year>2022</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>R.</given-names>
            <surname>Guidotti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Monreale</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ruggieri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Turini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Giannotti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Pedreschi</surname>
          </string-name>
          ,
          <article-title>A survey of methods for explaining black box models, ACM computing surveys (CSUR) 51 (</article-title>
          <year>2018</year>
          )
          <fpage>1</fpage>
          -
          <lpage>42</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>V.</given-names>
            <surname>Lai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q. V.</given-names>
            <surname>Liao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Smith-Renner</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Tan</surname>
          </string-name>
          ,
          <article-title>Towards a science of human-AI decision making: a survey of empirical studies</article-title>
          ,
          <source>arXiv preprint arXiv:2112.11471</source>
          (
          <year>2021</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>A.</given-names>
            <surname>Abdul</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Vermeulen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. Y.</given-names>
            <surname>Lim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Kankanhalli</surname>
          </string-name>
          ,
          <article-title>Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda</article-title>
          ,
          <source>in: Proceedings of the 2018 CHI conference on human factors in computing systems</source>
          ,
          <year>2018</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>18</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>P.</given-names>
            <surname>Linardatos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Papastefanopoulos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kotsiantis</surname>
          </string-name>
          ,
          <string-name>
            <surname>Explainable</surname>
            <given-names>AI</given-names>
          </string-name>
          :
          <article-title>A review of machine learning interpretability methods</article-title>
          ,
          <source>Entropy</source>
          <volume>23</volume>
          (
          <year>2020</year>
          )
          <fpage>18</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>S. T.</given-names>
            <surname>Mueller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. R.</given-names>
            <surname>Hofman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Clancey</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Emrey</surname>
          </string-name>
          ,
          <string-name>
            <surname>G.</surname>
          </string-name>
          <article-title>Klein, Explanation in human-AI systems: A literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI</article-title>
          , arXiv preprint arXiv:
          <year>1902</year>
          .
          <year>01876</year>
          (
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>M.</given-names>
            <surname>Chromik</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Schuessler</surname>
          </string-name>
          ,
          <article-title>A taxonomy for human subject evaluation of black-box explanations in XAI, ExSS-ATEC@ iui 1 (</article-title>
          <year>2020</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [30]
          <string-name>
            <given-names>D.</given-names>
            <surname>Moher</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Liberati</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Tetzlaf</surname>
          </string-name>
          , D. G. Altman, t. PRISMA Group*,
          <article-title>Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement</article-title>
          ,
          <source>Annals of internal medicine 151</source>
          (
          <year>2009</year>
          )
          <fpage>264</fpage>
          -
          <lpage>269</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>R.</given-names>
            <surname>Streeton</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Cooke</surname>
          </string-name>
          , J. Campbell,
          <article-title>Researching the researchers: Using a snowballing technique</article-title>
          ,
          <source>Nurse researcher 12</source>
          (
          <year>2004</year>
          )
          <fpage>35</fpage>
          -
          <lpage>47</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [32]
          <string-name>
            <given-names>N. Van</given-names>
            <surname>Eck</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Waltman</surname>
          </string-name>
          ,
          <article-title>Software survey: VOSviewer, a computer program for bibliometric mapping</article-title>
          , scientometrics
          <volume>84</volume>
          (
          <year>2010</year>
          )
          <fpage>523</fpage>
          -
          <lpage>538</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [33]
          <string-name>
            <given-names>S.</given-names>
            <surname>Hanses</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <article-title>How do users interact with AI features in the workplace? understanding the AI feature user journey in enterprise</article-title>
          ,
          <source>in: CHI Conference on Human Factors in Computing Systems Extended Abstracts</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>7</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [34]
          <string-name>
            <given-names>E.</given-names>
            <surname>Veitch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O. A.</given-names>
            <surname>Alsos</surname>
          </string-name>
          ,
          <article-title>Human-centered explainable artificial intelligence for marine autonomous surface vehicles</article-title>
          ,
          <source>Journal of Marine Science and Engineering</source>
          <volume>9</volume>
          (
          <year>2021</year>
          )
          <fpage>1227</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [35]
          <string-name>
            <given-names>M.</given-names>
            <surname>Riveiro</surname>
          </string-name>
          , S. Thill, “
          <article-title>that's (not) the output i expected!” on the role of end user expectations in creating explanations of AI systems</article-title>
          ,
          <source>Artificial Intelligence</source>
          <volume>298</volume>
          (
          <year>2021</year>
          )
          <fpage>103507</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [36]
          <string-name>
            <given-names>M.</given-names>
            <surname>Afzaal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Zia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Nouri</surname>
          </string-name>
          , U. Fors,
          <article-title>Informative feedback and explainable AI-based recommendations to support students' self-regulation, Technology, Knowledge and Learning (</article-title>
          <year>2023</year>
          )
          <fpage>1</fpage>
          -
          <lpage>24</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [37]
          <string-name>
            <given-names>V.</given-names>
            <surname>Swamy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Du</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Marras</surname>
          </string-name>
          , T. Kaser,
          <article-title>Trusting the explainers: Teacher validation of explainable artificial intelligence for course design</article-title>
          ,
          <source>in: LAK23: 13th International Learning Analytics and Knowledge Conference</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>345</fpage>
          -
          <lpage>356</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [38]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bhattacharya</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Ooge</surname>
          </string-name>
          , G. Stiglic,
          <string-name>
            <given-names>K.</given-names>
            <surname>Verbert</surname>
          </string-name>
          ,
          <article-title>Directive explanations for monitoring the risk of diabetes onset: Introducing directive data-centric explanations and combinations to support What-If explorations</article-title>
          ,
          <source>in: Proceedings of the 28th International Conference on Intelligent User Interfaces</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>204</fpage>
          -
          <lpage>219</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [39]
          <string-name>
            <given-names>G.</given-names>
            <surname>Cabour</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Morales-Forero</surname>
          </string-name>
          , É. Ledoux,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bassetto</surname>
          </string-name>
          ,
          <article-title>An explanation space to align user studies with the technical development of explainable AI, AI &amp; SOCIETY (</article-title>
          <year>2022</year>
          )
          <fpage>1</fpage>
          -
          <lpage>19</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [40]
          <string-name>
            <given-names>C.</given-names>
            <surname>Bove</surname>
          </string-name>
          ,
          <string-name>
            <surname>M.-J. Lesot</surname>
            ,
            <given-names>C. A.</given-names>
          </string-name>
          <string-name>
            <surname>Tijus</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>Detyniecki, Investigating the intelligibility of plural counterfactual examples for non-expert users: an explanation user interface proposition and user study</article-title>
          ,
          <source>in: Proceedings of the 28th International Conference on Intelligent User Interfaces</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>188</fpage>
          -
          <lpage>203</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [41]
          <string-name>
            <given-names>F.</given-names>
            <surname>Cheng</surname>
          </string-name>
          , Y. Ming,
          <string-name>
            <given-names>H.</given-names>
            <surname>Qu</surname>
          </string-name>
          ,
          <article-title>Dece: Decision explorer with counterfactual explanations for machine learning models</article-title>
          ,
          <source>IEEE Transactions on Visualization and Computer Graphics</source>
          <volume>27</volume>
          (
          <year>2020</year>
          )
          <fpage>1438</fpage>
          -
          <lpage>1447</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [42]
          <string-name>
            <given-names>T.</given-names>
            <surname>Susnjak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. S.</given-names>
            <surname>Ramaswami</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Mathrani</surname>
          </string-name>
          ,
          <article-title>Learning analytics dashboard: a tool for providing actionable insights to learners</article-title>
          ,
          <source>International Journal of Educational Technology in Higher Education</source>
          <volume>19</volume>
          (
          <year>2022</year>
          )
          <fpage>12</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [43]
          <string-name>
            <given-names>N.</given-names>
            <surname>Mollaei</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Fujao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Silva</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Rodrigues</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Cepeda</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Gamboa</surname>
          </string-name>
          ,
          <article-title>Human-centered explainable artificial intelligence: automotive occupational health protection profiles in prevention musculoskeletal symptoms</article-title>
          ,
          <source>International Journal of Environmental Research and Public Health</source>
          <volume>19</volume>
          (
          <year>2022</year>
          )
          <fpage>9552</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [44]
          <string-name>
            <given-names>M.</given-names>
            <surname>Dikmen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Burns</surname>
          </string-name>
          ,
          <article-title>The efects of domain knowledge on trust in explainable AI and task performance: A case of peer-to-peer lending</article-title>
          ,
          <source>International Journal of Human-Computer Studies</source>
          <volume>162</volume>
          (
          <year>2022</year>
          )
          <fpage>102792</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [45]
          <string-name>
            <surname>W.-J. She</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          <string-name>
            <surname>Senoo</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          <string-name>
            <surname>Iwakoshi</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          <string-name>
            <surname>Kuwahara</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Siriaraya</surname>
          </string-name>
          ,
          <article-title>Af'fective design: Supporting atrial fibrillation post-treatment with explainable AI</article-title>
          ,
          <source>in: 27th International Conference on Intelligent User Interfaces</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>22</fpage>
          -
          <lpage>25</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [46]
          <string-name>
            <given-names>M.</given-names>
            <surname>Sufian</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Bogliolo</surname>
          </string-name>
          ,
          <article-title>Investigation and mitigation of bias in explainable AI</article-title>
          ,
          <source>in: CEUR WORKSHOP PROCEEDINGS</source>
          , volume
          <volume>3319</volume>
          ,
          <string-name>
            <surname>AIxIA</surname>
          </string-name>
          ,
          <year>2022</year>
          , pp.
          <fpage>89</fpage>
          -
          <lpage>94</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [47]
          <string-name>
            <given-names>N.</given-names>
            <surname>Spreitzer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Haned</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. van der Linden</surname>
          </string-name>
          ,
          <article-title>Evaluating the practicality of counterfactual explanations</article-title>
          ,
          <source>in: Workshop on Trustworthy and Socially Responsible Machine Learning</source>
          ,
          <source>NeurIPS</source>
          <year>2022</year>
          ,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [48]
          <string-name>
            <given-names>T. A.</given-names>
            <surname>Schoonderwoerd</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Jorritsma</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. A.</given-names>
            <surname>Neerincx</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Van Den</surname>
          </string-name>
          <string-name>
            <given-names>Bosch</given-names>
            ,
            <surname>Human-centered</surname>
          </string-name>
          <string-name>
            <surname>XAI</surname>
          </string-name>
          :
          <article-title>Developing design patterns for explanations of clinical decision support systems</article-title>
          ,
          <source>International Journal of Human-Computer Studies</source>
          <volume>154</volume>
          (
          <year>2021</year>
          )
          <fpage>102684</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          [49]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Nakao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Stumpf</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ahmed</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Naseer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Strappelli</surname>
          </string-name>
          ,
          <article-title>Toward involving end-users in interactive human-in-the-loop AI fairness</article-title>
          ,
          <source>ACM Transactions on Interactive Intelligent Systems (TiiS) 12</source>
          (
          <year>2022</year>
          )
          <fpage>1</fpage>
          -
          <lpage>30</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          [50]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Gao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. S.</given-names>
            <surname>Sun</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Zhao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. R.</given-names>
            <surname>Hong</surname>
          </string-name>
          ,
          <article-title>Aligning eyes between humans and deep neural network through interactive attention alignment</article-title>
          ,
          <source>Proceedings of the ACM on HumanComputer Interaction</source>
          <volume>6</volume>
          (
          <year>2022</year>
          )
          <fpage>1</fpage>
          -
          <lpage>28</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          [51]
          <string-name>
            <surname>K. Z. Gajos</surname>
          </string-name>
          , L. Mamykina,
          <article-title>Do people engage cognitively with AI? impact of ai assistance on incidental learning</article-title>
          ,
          <source>in: 27th International Conference on Intelligent User Interfaces</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>794</fpage>
          -
          <lpage>806</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          [52]
          <string-name>
            <given-names>A. R.</given-names>
            <surname>Akula</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Liu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Saba-Sadiya</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Todorovic</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Chai</surname>
          </string-name>
          , S.-C. Zhu,
          <article-title>CXToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models</article-title>
          ,
          <source>Iscience</source>
          <volume>25</volume>
          (
          <year>2022</year>
          )
          <fpage>103581</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          [53]
          <string-name>
            <given-names>J. Labaien</given-names>
            <surname>Soto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. Zugasti</given-names>
            <surname>Uriguen</surname>
          </string-name>
          ,
          <string-name>
            <surname>X. De Carlos Garcia</surname>
          </string-name>
          ,
          <article-title>Real-time, model-agnostic and user-driven counterfactual explanations using autoencoders</article-title>
          ,
          <source>Applied Sciences</source>
          <volume>13</volume>
          (
          <year>2023</year>
          )
          <fpage>2912</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          [54]
          <string-name>
            <given-names>M.</given-names>
            <surname>Riveiro</surname>
          </string-name>
          ,
          <string-name>
            <surname>S. Thill,</surname>
          </string-name>
          <article-title>The challenges of providing explanations of AI systems when they computing systems</article-title>
          ,
          <year>2020</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>13</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref39">
        <mixed-citation>
          [68]
          <string-name>
            <given-names>J.</given-names>
            <surname>Zhu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Liapis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Risi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Bidarra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. M.</given-names>
            <surname>Youngblood</surname>
          </string-name>
          ,
          <article-title>Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation, in: 2018 IEEE conference on computational intelligence and games (CIG)</article-title>
          , IEEE,
          <year>2018</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>8</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref40">
        <mixed-citation>
          [69]
          <string-name>
            <given-names>X.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. R.</given-names>
            <surname>Jonker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Todi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Qian</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Evangelista Belo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Mun</surname>
          </string-name>
          , et al.,
          <article-title>XAIR: A framework of explainable AI in augmented reality</article-title>
          ,
          <source>in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>30</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref41">
        <mixed-citation>
          [70]
          <string-name>
            <given-names>R.</given-names>
            <surname>Shang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. K.</given-names>
            <surname>Feng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <article-title>Why am i not seeing it? understanding users' needs for counterfactual explanations in everyday recommendations</article-title>
          ,
          <source>in: 2022 ACM Conference on Fairness, Accountability, and Transparency</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>1330</fpage>
          -
          <lpage>1340</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref42">
        <mixed-citation>
          [71]
          <string-name>
            <given-names>P.</given-names>
            <surname>Kouki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Schafer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Pujara</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. O'Donovan</surname>
            ,
            <given-names>L. Getoor,</given-names>
          </string-name>
          <article-title>User preferences for hybrid explanations</article-title>
          ,
          <source>in: Proceedings of the Eleventh ACM Conference on Recommender Systems</source>
          ,
          <year>2017</year>
          , pp.
          <fpage>84</fpage>
          -
          <lpage>88</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref43">
        <mixed-citation>
          [72]
          <string-name>
            <given-names>P.</given-names>
            <surname>Kouki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Schafer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Pujara</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. O'Donovan</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Getoor</surname>
          </string-name>
          ,
          <article-title>Personalized explanations for hybrid recommender systems</article-title>
          ,
          <source>in: Proceedings of the 24th International Conference on Intelligent User Interfaces</source>
          ,
          <year>2019</year>
          , pp.
          <fpage>379</fpage>
          -
          <lpage>390</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref44">
        <mixed-citation>
          [73]
          <string-name>
            <given-names>P.</given-names>
            <surname>Qian</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Unhelkar</surname>
          </string-name>
          ,
          <article-title>Evaluating the role of interactivity on improving transparency in autonomous agents</article-title>
          ,
          <source>in: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>1083</fpage>
          -
          <lpage>1091</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref45">
        <mixed-citation>
          [74]
          <string-name>
            <given-names>J.</given-names>
            <surname>Sun</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q. V.</given-names>
            <surname>Liao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Muller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Agarwal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Houde</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Talamadupula</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. D.</given-names>
            <surname>Weisz</surname>
          </string-name>
          ,
          <article-title>Investigating explainability of generative AI for code through scenario-based design</article-title>
          ,
          <source>in: 27th International Conference on Intelligent User Interfaces</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>212</fpage>
          -
          <lpage>228</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref46">
        <mixed-citation>
          [75]
          <string-name>
            <given-names>S.</given-names>
            <surname>Chari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Seneviratne</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. M.</given-names>
            <surname>Gruen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. A.</given-names>
            <surname>Foreman</surname>
          </string-name>
          ,
          <string-name>
            <surname>A. K. Das</surname>
          </string-name>
          ,
          <string-name>
            <surname>D. L. McGuinness</surname>
          </string-name>
          ,
          <article-title>Explanation ontology: a model of explanations for user-centered AI, in: The Semantic Web-ISWC</article-title>
          <year>2020</year>
          : 19th International Semantic Web Conference, Athens, Greece, November 2-
          <issue>6</issue>
          ,
          <year>2020</year>
          , Proceedings,
          <string-name>
            <surname>Part</surname>
            <given-names>II</given-names>
          </string-name>
          , Springer,
          <year>2020</year>
          , pp.
          <fpage>228</fpage>
          -
          <lpage>243</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref47">
        <mixed-citation>
          [76]
          <string-name>
            <given-names>S. S.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Meister</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V. V.</given-names>
            <surname>Ramaswamy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Fong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Russakovsky</surname>
          </string-name>
          ,
          <article-title>Hive: evaluating the human interpretability of visual explanations</article-title>
          , in: Computer Vision-ECCV
          <year>2022</year>
          : 17th European Conference, Tel Aviv, Israel,
          <source>October 23-27</source>
          ,
          <year>2022</year>
          , Proceedings,
          <string-name>
            <surname>Part</surname>
            <given-names>XII</given-names>
          </string-name>
          , Springer,
          <year>2022</year>
          , pp.
          <fpage>280</fpage>
          -
          <lpage>298</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref48">
        <mixed-citation>
          [77]
          <string-name>
            <given-names>J.</given-names>
            <surname>Novak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Maljur</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Drenska</surname>
          </string-name>
          ,
          <article-title>Transferring AI explainability to user-centered explanations of complex COVID-19 information</article-title>
          , in: HCI International 2022-Late Breaking Papers:
          <source>Interacting with eXtended Reality and Artificial Intelligence: 24th International Conference on Human-Computer Interaction, HCII</source>
          <year>2022</year>
          ,
          <string-name>
            <given-names>Virtual</given-names>
            <surname>Event</surname>
          </string-name>
          , June 26-July 1,
          <year>2022</year>
          , Proceedings, Springer,
          <year>2022</year>
          , pp.
          <fpage>441</fpage>
          -
          <lpage>460</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref49">
        <mixed-citation>
          [78]
          <string-name>
            <given-names>Q. V.</given-names>
            <surname>Liao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Gruen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Miller</surname>
          </string-name>
          ,
          <article-title>Questioning the AI: informing design practices for explainable AI user experiences</article-title>
          ,
          <source>in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems</source>
          ,
          <year>2020</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>15</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref50">
        <mixed-citation>
          [79]
          <string-name>
            <given-names>L.</given-names>
            <surname>Wiebelitz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Schmid</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Maier</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Volkwein</surname>
          </string-name>
          ,
          <article-title>Designing user-friendly medical AI applications-methodical development of user-centered design guidelines</article-title>
          ,
          <source>in: 2022 IEEE International Conference on Digital Health (ICDH)</source>
          , IEEE,
          <year>2022</year>
          , pp.
          <fpage>23</fpage>
          -
          <lpage>28</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref51">
        <mixed-citation>
          [80]
          <string-name>
            <given-names>S.</given-names>
            <surname>Naveed</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Ziegler</surname>
          </string-name>
          ,
          <string-name>
            <surname>Featuristic:</surname>
          </string-name>
          <article-title>An interactive hybrid system for generating explainable recommendations-beyond system accuracy</article-title>
          ., in: IntRS@ RecSys,
          <year>2020</year>
          , pp.
          <fpage>14</fpage>
          -
          <lpage>25</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref52">
        <mixed-citation>
          [81]
          <string-name>
            <given-names>A.</given-names>
            <surname>Kirsch</surname>
          </string-name>
          ,
          <article-title>Explain to whom? putting the user in the center of explainable AI</article-title>
          , in: Proceedings of the First International Workshop on Comprehensibility and
          <article-title>Explanation in AI and ML 2017 co-located with 16th International Conference of the Italian Association for Artificial Intelligence (AI* IA</article-title>
          <year>2017</year>
          ),
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref53">
        <mixed-citation>
          [82]
          <string-name>
            <given-names>L.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. A.</given-names>
            <surname>Deleris</surname>
          </string-name>
          ,
          <article-title>What does it mean to explain? a user-centered study on AI explainability</article-title>
          , in: Artificial Intelligence in HCI: Second International Conference,
          <source>AI-HCI</source>
          <year>2021</year>
          ,
          <article-title>Held as Part of the 23rd HCI International Conference</article-title>
          , HCII 2021,
          <string-name>
            <given-names>Virtual</given-names>
            <surname>Event</surname>
          </string-name>
          ,
          <source>July 24-29</source>
          ,
          <year>2021</year>
          , Proceedings, Springer,
          <year>2021</year>
          , pp.
          <fpage>107</fpage>
          -
          <lpage>121</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref54">
        <mixed-citation>
          [83]
          <string-name>
            <given-names>L.</given-names>
            <surname>Sanneman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. A.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <article-title>A situation awareness-based framework for design and evaluation of explainable AI</article-title>
          , in: Explainable, Transparent Autonomous Agents and
          <string-name>
            <surname>Multi-Agent</surname>
            <given-names>Systems</given-names>
          </string-name>
          : Second International Workshop, EXTRAAMAS 2020, Auckland, New Zealand, May 9-
          <issue>13</issue>
          ,
          <year>2020</year>
          ,
          <source>Revised Selected Papers 2</source>
          , Springer,
          <year>2020</year>
          , pp.
          <fpage>94</fpage>
          -
          <lpage>110</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref55">
        <mixed-citation>
          [84]
          <string-name>
            <given-names>M.</given-names>
            <surname>El-Assady</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Moruzzi</surname>
          </string-name>
          ,
          <article-title>Which biases and reasoning pitfalls do explanations trigger? decomposing communication processes in human-AI interaction</article-title>
          ,
          <source>IEEE Computer Graphics and Applications</source>
          <volume>42</volume>
          (
          <year>2022</year>
          )
          <fpage>11</fpage>
          -
          <lpage>23</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref56">
        <mixed-citation>
          [85]
          <string-name>
            <given-names>S. F.</given-names>
            <surname>Jentzsch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Höhn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Hochgeschwender</surname>
          </string-name>
          ,
          <article-title>Conversational interfaces for explainable AI: a human-centred approach</article-title>
          , in: Explainable, Transparent Autonomous Agents and
          <string-name>
            <surname>Multi-Agent</surname>
            <given-names>Systems</given-names>
          </string-name>
          : First International Workshop, EXTRAAMAS 2019, Montreal, QC, Canada, May
          <volume>13</volume>
          -14,
          <year>2019</year>
          ,
          <source>Revised Selected Papers 1</source>
          , Springer,
          <year>2019</year>
          , pp.
          <fpage>77</fpage>
          -
          <lpage>92</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref57">
        <mixed-citation>
          [86]
          <string-name>
            <given-names>U.</given-names>
            <surname>Ehsan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Saha</surname>
          </string-name>
          ,
          <string-name>
            <surname>M. De Choudhury</surname>
            ,
            <given-names>M. O.</given-names>
          </string-name>
          <string-name>
            <surname>Riedl</surname>
          </string-name>
          ,
          <article-title>Charting the sociotechnical gap in explainable AI: A framework to address the gap in XAI, Proceedings of the ACM on Human-Computer Interaction 7 (</article-title>
          <year>2023</year>
          )
          <fpage>1</fpage>
          -
          <lpage>32</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref58">
        <mixed-citation>
          [87]
          <string-name>
            <given-names>L.</given-names>
            <surname>Sanneman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. A.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <article-title>An empirical study of reward explanations with human-robot interaction applications</article-title>
          ,
          <source>IEEE Robotics and Automation Letters</source>
          <volume>7</volume>
          (
          <year>2022</year>
          )
          <fpage>8956</fpage>
          -
          <lpage>8963</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref59">
        <mixed-citation>
          [88]
          <string-name>
            <given-names>M.</given-names>
            <surname>Ghajargar</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Bardzell</surname>
          </string-name>
          ,
          <article-title>Making AI understandable by making it tangible: Exploring the design space with ten concept cards</article-title>
          ,
          <source>in: Proceedings of the 34th Australian Conference on Human-Computer Interaction</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>74</fpage>
          -
          <lpage>80</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref60">
        <mixed-citation>
          [89]
          <string-name>
            <given-names>L.</given-names>
            <surname>Capone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Bertolaso</surname>
          </string-name>
          , et al.,
          <article-title>A philosophical approach for a human-centered explainable AI</article-title>
          , in: XAI.it@AIxIA,
          <year>2020</year>
          , pp.
          <fpage>80</fpage>
          -
          <lpage>86</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref61">
        <mixed-citation>
          [90]
          <string-name>
            <given-names>A.</given-names>
            <surname>Kasirzadeh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Smart</surname>
          </string-name>
          ,
          <article-title>The use and misuse of counterfactuals in ethical machine learning</article-title>
          ,
          <source>in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency</source>
          ,
          <year>2021</year>
          , pp.
          <fpage>228</fpage>
          -
          <lpage>236</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref62">
        <mixed-citation>
          [91]
          <string-name>
            <given-names>U.</given-names>
            <surname>Ehsan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q. V.</given-names>
            <surname>Liao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Muller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. O.</given-names>
            <surname>Riedl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. D.</given-names>
            <surname>Weisz</surname>
          </string-name>
          ,
          <article-title>Expanding explainability: Towards social transparency in AI systems</article-title>
          ,
          <source>in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems</source>
          ,
          <year>2021</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>19</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref63">
        <mixed-citation>
          [92]
          <string-name>
            <given-names>G.</given-names>
            <surname>Margetis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ntoa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Antona</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Stephanidis</surname>
          </string-name>
          ,
          <article-title>Human-centered design of artificial intelligence, Handbook of human factors and ergonomics (</article-title>
          <year>2021</year>
          )
          <fpage>1085</fpage>
          -
          <lpage>1106</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref64">
        <mixed-citation>
          [93]
          <string-name>
            <given-names>U.</given-names>
            <surname>Schmid</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Wrede</surname>
          </string-name>
          ,
          <article-title>What is missing in XAI so far? an interdisciplinary perspective</article-title>
          , KI-Künstliche
          <string-name>
            <surname>Intelligenz</surname>
          </string-name>
          (
          <year>2022</year>
          )
          <fpage>1</fpage>
          -
          <lpage>13</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref65">
        <mixed-citation>
          [94]
          <string-name>
            <surname>C. M. Navarro</surname>
          </string-name>
          , G. Kanellos, T. Gottron,
          <article-title>Desiderata for explainable AI in statistical production systems of the european central bank</article-title>
          ,
          <source>in: Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD</source>
          <year>2021</year>
          ,
          <string-name>
            <given-names>Virtual</given-names>
            <surname>Event</surname>
          </string-name>
          ,
          <source>September 13-17</source>
          ,
          <year>2021</year>
          , Proceedings,
          <string-name>
            <surname>Part</surname>
            <given-names>I</given-names>
          </string-name>
          , Springer,
          <year>2022</year>
          , pp.
          <fpage>575</fpage>
          -
          <lpage>590</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref66">
        <mixed-citation>
          [95]
          <string-name>
            <given-names>C. T.</given-names>
            <surname>Okolo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Dell</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Vashistha</surname>
          </string-name>
          ,
          <string-name>
            <surname>Making</surname>
            <given-names>AI</given-names>
          </string-name>
          <article-title>explainable in the global south: A systematic review</article-title>
          ,
          <source>in: ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies</source>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>