Examining the Nexus between Explainability of AI Systems and User's Trust: A Preliminary Scoping Review

Abstract

EXplainable AI (XAI) systems are designed to provide clear explanations of how the system arrived at a decision or prediction, which increases users' trust. However, the factors that promote trust among XAI users, the different dimensions of trust, and how they affect the human-AI relationship are still under exploration. Through a preliminary literature review, this paper aims to collect the most recent empirical evidence (n=13) that investigates the nexus between XAI and users' trust, highlighting the most salient factors shaping this relationship. The studies measured XAI, including understandability, informativeness, and system design factors. Different scales were used, such as Likert scales and preexperimental surveys, as well as more nuanced approaches like image classification AI and focus groups. Trust in AI was evaluated through criteria like trustworthiness and scales for agreement with statements about trust, even if some studies adopted methods like latent trust evaluations, observational measures, and usability tests. The studies collectively suggest that various factors such as clear explanations, perceived understanding of AI, transparency, reliability, fairness, user-centeredness, emotional responses, and design elements of the system influence trust in AI. Low-fidelity explanations, feelings of fear or discomfort, and low perceived usefulness can decrease trust, with systems displaying medium accuracy or utilizing visual explanations not adversely affecting user trust. Explainability methods like PDP and LIME appear effective at increasing user trust, while SHAP explanations perform less well. To foster trust, AI developers should prioritize designs considering both cognitive and affective trust-building aspects.t.

1. Introduction

AI explainability and user trust remain ambiguous. Therefore, this study aims to analyze the most significant empirical studies investigating the relationship between AI explainability and users' trust to identify the key factors.

1.1. User's Trust in XAI Systems: The Psychological Perspective

Trust is pivotal in human-technology interactions and denotes a user's readiness to depend on an automated system for goal attainment [ 4 ]. In the AI era, user trust is integral for effective system utilization. Insufficient trust can precipitate disengagement, while excess trust can engender overreliance and frustration [ 5 ]. Consequently, AI should foster calibrated trust— generated by an alignment between expectations and system competence—to optimize user engagement [ 6 ]nand to avoid misuse, abuse, or disuse of the technology [ 7 ]. From a psychological standpoint, trust in human-AI interactions features cognitive and affective forms [ 8 ]. Cognitive trust emerges from rational assessments of system capabilities and performance, whereas affective trust derives from emotional components such as comfort and familiarity [ 8 ]. Both variants influence user behavior and decision-making with different antecedents and repercussions. As Gillath et al. [ 9 ] indicated, AI has a more substantial impact on cognitive trust, which typically establishes initial user trust. Over time, affective components gain prominence for maintaining long-term AI system relationships. The literature also highlights latent trust, offering a method to study user trust in AI through observed behavior and emotional responses [ 10 ]. This concept unveils implicit trust elements and yields insights into the emotional facet of trust, thereby facilitating system design enhancements. Thus, latent trust can potentially improve user interactions with AI systems. The multifaceted nature of trust enhances our understanding of human-AI interactions. Each aspect provides a distinct perspective on user behavior and decision-making concerning XAI. Yet, further exploration of these dimensions remains an active research area. In this context, our study aspires to enrich the nuanced comprehension of trust dynamics in the evolving AI landscape.

2. Method

We conducted a scoping literature review in three stages to identify critical factors influencing the relationship between XAI systems and user trust. Initially, we used databases like Scopus, Web of Science, and Google Scholar, employing keywords like Trust, Artificial Intelligence, Explainable AI, XAI, Transparen*, and Explainab*, resulting in 41 records. Next, our team utilized Ryann.ai [ 11 ], an AI-powered tool aiding remote collaboration for literature reviews. This software streamlined the selection process with features such as tagging, inclusion/exclusion functions, and selection rationale recording. Four researchers screened the sources during this stage based on title, abstract, keywords, content, relevance, research outcomes, and recency (considering the past five years). Lastly, additional authors examined the selected sources for research methodology and quality, ensuring no pertinent experimental studies were omitted. This procedure resulted in 13 sources included in our review, offering a comprehensive exploration of the relationship between XAI systems and user trust.

3. Results

The 13 selected studies summarized in Table 1, reveal recurrent themes within the experimental design, data collection, and outcomes of AI system evaluation. Trust measurement techniques frequently involve self-reported questionnaires that assess system features such as accuracy, reliability, transparency, and usability. Two studies also considered latent trust [ 12 ][ 13 ], examining user engagement, trust, and emotional responses to AI systems. The AI systems scrutinized vary widely yet display comparable patterns. Different methodologies are employed to measure explainability; in each study, users are asked to comprehend the predictions of AI or express their agreement with the system's internal functioning. The findings from 13 studies indicate that user trust in AI positively correlates with their perceived understanding of the algorithm influenced by factors like transparency, reliability, fairness, and the system's usercenteredness [ 14 ]. The results underscore the delicate balance necessary in providing explanations. Low-fidelity explanations [ 15 ], perceptions of low usefulness [ 16 ] and feelings of fear or discomfort can diminish trust [ 13 ]. Conversely, certain factors do not adversely affect trust, such as medium-accuracy systems [ 15 ] or those utilizing visual explanations [17][18]. Visual explanations can, in fact, help users achieve calibrated trust by providing additional information that can be trusted without over-trusting the system [19]. Furthermore, explanation methods like Partial Dependence Plot (PDP) and Local Interpretable Model-agnostic Explanations (LIME) garnered elevated levels of concurrence among participants, suggesting an enhancement in trust. Conversely, Shapley Additive Explanations (SHAP) elicited participant responses marked by neutrality and disagreement, demonstrating less effectiveness in fostering trust [20]. Hence, the usefulness of the XAI framework emerges as a clear theme in bolstering user trust. Despite initial difficulties users may have in interpreting AI results, clear explanations of AI functionality and decision-making processes increase user trust [21][22][23]. Additionally, anthropomorphic design can positively impact user acceptance and trust in XAI conditions [24]. This design approach generates affective trust, enhancing the user's and AI's emotional responses.

4. Conclusions

This scoping review's primary objective was to discern the key factors that shape the relationship between AI explainability and users' trust. According to the findings from the 13 selected studies, users are more likely to perceive AI systems as fair, dependable, and user-oriented when they can comprehend the rationale and logic underpinning these systems' decisions [ 14 ]. Conversely, factors such as low-fidelity explanations [ 15 ], perceived limited utility [ 16 ], and emotions of fear or discomfort [ 13 ] can rust trust. The role of the XAI in augmenting user trust is confirmed as a prominent theme that emerges from these observations. Even though users may initially face difficulties in interpreting AI outcomes, supplying clear explanations about AI functionality and decision-making processes bolsters user trust [21][22][23]. Even amid perceptions of unsatisfactory system performance, the XAI interface can aid in achieving an appropriate calibration of trust [23]. These findings pave the way for subsequent work and suggest several initial recommendations for developers aiming to enhance users' trust in XAI systems. As a first step, it is advisable for developers to prioritize the design and development of AI models that are intrinsically explainable. Incorporating interpretability features into the system architecture and decision-making processes facilitates comprehension and cognitive trust. Tools such as rulebased systems, decision trees, and model-agnostic explanations (e.g., PDP, LIME) can provide users with meaningful explanations [19]. Furthermore, it is vital to ensure that the explanations provided by AI systems are unambiguous, concise, and easily comprehensible to non-experts. One challenge would be to balance overly technical explanations, which may confuse users, and lowfidelity explanations, which may limit users' ability to make informed judgments about system outputs. Both communication strategies can lead to perceptions of limited usefulness. Mediumaccuracy systems and visual aids may enhance user engagement without significantly impacting affective trust. Lastly, user feedback is crucial in refining XAI systems and fine-tuning trust calibration. User input can help pinpoint areas where explanations are inadequate or fail to address specific concerns. Continuous evaluation and iterative improvements of system explanation mechanisms maintain the "human-in-the-loop." Ultimately, developers can tailor XAI systems to optimize trust calibration processes and system performance. As AI becomes further entrenched in our daily lives, user-centric explainability will assume a critical role in leveraging the full potential of AI technologies while mitigating societal apprehensions. Even though this scoping review represents preliminary work, it provides a foundation for future research to discover additional strategies to fortify the link between AI explainability and users' trust.

References

[1] A.M. Antoniadi , Y. Du , Y. Guendouz , L. Wei , C. Mazo , B.A. Becker , C. Mooney , Current challenges, and future opportunities for XAI in machine learning-based clinical decision support systems: a systematic review , Appl. Sci . 11 ( 2021 ) 5088 . doi: 10 .3390/app11115088.

[2] X. Wang , T. Ahonen , J. Nurmi , Applying CDMA technique to network-on-chip , IEEE Transactions on Very Large Scale Integration (VLSI) Systems 15 .10 ( 2007 ) 1091 - 1100 . doi: 10 .1109/TVLSI. 2007 .903914

[3] TUPLES , 2023 . URL: https://tuples.ai.

[4] E. Glikson , A.W. Woolley , Human trust in artificial intelligence: Review of empirical research , Acad. Manag. Ann . 14 ( 2020 ) 627 - 66 . doi: doi.org/10.5465/annals. 2018 . 0057 .

[5] J.D. Lee , K.A. See , Trust in automation: Designing for appropriate reliance , Hum. Factors 46 ( 2004 ) 50 - 80 . doi: 10 .1518/hfes.46.1.50_ 30392 .

[6] J. Kraus , D. Scholz , D. Stiegemeier , M. Baumann , The more you know: trust dynamics and calibration in highly automated driving and the effects of take-overs, system malfunction, and system transparency , Hum. Factors 62 ( 2020 ) 718 - 736 . doi: 10 .1177/0018720819853686.

[7] A. Jacovi , A. Marasovic , T. Miller , Y. Goldberg , Formalizing Trust in Artificial Intelligence: Prerequisites , Causes and Goals of Human Trust in AI , in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21 , ACM Press, 2021 , pp. 624 - 635 . doi: 10 .1145/3442188.3445923.

[8] N. Kyung , H. E. Kwon , Rationally trust, but emotionally? The roles of cognitive and affective trust in laypeople's acceptance of AI for preventive care operations, Prod . Oper. Manag. ( 2022 ) pp. 1 - 20 . doi: 10 .1111/poms.13785.

[9] O. Gillath , T. Ai , M. S. Branicky , S. Keshmiri , R. B. Davison , R. Spaulding , Attachment and trust in artificial intelligence , Comput. Hum. Behav . 115 ( 2021 ) 106607 . doi: 10 .1016/j.chb. 2020 . 106607 .

[10] J. T. Luo , P. McGoldrick , S. Beatty , K. A. Keeling , On-screen characters: Their design and influence on consumer trust , J. Serv. Mark . 20 ( 2006 ) 112 - 124 . doi: 10 .1108/08876040610657048.

[11] M. Ouzzani , H. Hammady , Z. Fedorowicz , A. Elmagarmid , Rayyan-a web and mobile app for systematic reviews , Syst. Rev. 5 ( 2016 ) 210 . doi: 10 .1186/s13643-016-0384-4.

[12] E. Bernardo , R. Seva , Affective Analysis of Explainable Artificial Intelligence in the Development of Trust in AI Systems , in: T. Ahram, W. Karwowski , P. Di Bucchianico , R. Taiar , L. Casarotto , P. Costa (Eds.), AHFE International Conference 2023 , LNCS, vol. 69 , AHFE International, San Francisco, 2023 , pp. 417 - 427 .

[13] E. Bernardo , R. Seva , Exploration of Emotions Developed in the Interaction with Explainable AI , in: Proceedings of the 15th International Symposium on Computational Intelligence and Design , IEEE, New York, 2022 , pp. 143 - 146 .

[14] B. Ewerz , M. Hoefler , C. Marx , P. Moertl , How to Facilitate Explainability of AI for Increased User Trust: Results of a Study with a COVID-19 Risk Calculator , in: Proceedings of the 44th International Convention on Information, Communication and Electronic Technology , IEEE , New York, 2021 , pp. 1076 - 1080 .

[15] A. Papenmeier , G. Englebienne, C. Seifert , How model accuracy and explanation fidelity influence user trust , in: Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence , arXiv preprint arXiv: 1907 .12652, 2019 , pp. 94 - 100 . doi: 10 .48550/arXiv. 1907 . 12652 .

[16] J. Druce , M. Harradon , J. Tittle , Explainable artificial intelligence (XAI) for increasing user trust in deep reinforcement learning driven autonomous systems , in: Proceedings of the 33rd Conference on Neural Information Processing Systems , arXiv preprint arXiv: 2106 .03775, 2021 . doi: 10 .48550/arXiv.2106.03775