Artificial Intelligence (AI) in clinical decision support systems has considerable potential to improve medical care, but its application in clinical practice is still limited. A lack of transparency and human-oversight builds trust and acceptance barriers. It is often claimed that Explainable AI (XAI) is a promising method for overcoming those barriers. However, current XAI methods often fail to meet the diverse needs of diferent stakeholders. This research proposal aims to address this issue by developing human-centered explanations tailored to the individual requirements of various stakeholders. Therefore, my research employs a design science research approach, implementing and iteratively evaluating multiple promising XAI concepts, such as concept-based or glocal explanations. This approach will identify and refine the most promising methods based on stakeholder feedback. The results will contribute significantly to the development of human-centered explanations, advancing towards more responsible AI in clinical settings.
In a clinical setting, using artificial intelligence ( AI)-based clinical decision support system (CDSS)
promises to support physicians in making complex decisions and thus improve patient care. However,
few of these AI systems are used in practice because they fail due to a lack of interpretability which
can lead to trust barriers among physicians [
Many researchers suggest putting the end-users and their needs at the center of attention when
developing XAI [
Considering the varied levels of expertise among users — ranging from domain expertise to AI expertise — there is a distinct need for personalized explanations. For instance, a senior physician may require a less complex explanation in intensive care units than an assistant physician. At the same time, an AI developer necessitates a diferent form of explanation altogether. One common issue with current XAI methodologies is their failure to address the diverse desiderata of stakeholders, thereby failing to achieve their intended goals. The challenge lies in delivering the appropriate type of explanation to the right group of users, as suggested by Mohseni et al. [4].
To address these complexities, developing human-centered explanations, also known as “usercentered” or “personalized” explanations, is recommended [5]. “Human-centered” explanations should meet the individual needs of various stakeholders, particularly in high-risk areas like healthcare [6, 7].
This can be achieved through user-centered design of XAI [8] or by choosing the most suitable technical XAI methodology.
Functionallygrounded Evaluation Complexity Faithfulness Robustness
and their
weighted metrics 50% 30% 20%
centered
(a) The analysis of stakeholders and their desiderata influence the choice of the human-centered explanation approach, aswell as the given attributionbased.
Stakeholder Concepts
Concept
based Explanations 50 % 30 % 20 %
explanations
(b) User-controlled Explanations.
(c) Concept-based Explanations.
(d) Glocal Explanations.
Therefore, my research aims to address this problem in detail. My goal is to achieve AI systems that diferent stakeholders accept and create an appropriate, reliable interaction between the human and the AI. I will apply, assess, combine, and create new methods with promising approaches to creating human-centered explanations (see Figure 1). The diferent XAI approaches are described in Section 2. These methods will be applied and investigated in a clinical setup. I assume I can transfer the knowledge to other less-risky domains from the results obtained in this high-risk clinical domain. I am convinced that, especially in high-risk domains like healthcare, we must integrate the socio-technical perspective and the individual stakeholder desiderata to build trustworthy AI.
In this section, I provide a brief overview of related work that provides promising approaches for humancentered XAI and provide the background information for the key human-centered XAI approaches. Hereby, I also highlight papers that take the role of stakeholders and their desiderata in XAI into account. Table 1 summarizes and assigns the related work to main categories.
Before providing more background information about XAI approaches, I highlight the study from Calisto et al. [9] that explores the impact of personalized AI communication on clinical outcomes in breast cancer diagnosis. The study involved 52 varying expertise-level clinicians who used conventional and assertiveness-based AI communication styles to diagnose patient cases. Results showed that personalized AI communication significantly reduced diagnostic time and errors, particularly for less experienced clinicians, without compromising accuracy. The findings highlight the importance of adaptable AI communication to build trust, reduce cognitive load, and streamline clinical workflows, ofering valuable insights for designing efective AI systems in high-stakes domains. My work aims to extend these insights in healthcare by evaluating the impact of diferent XAI concepts. The following paragraphs describe three promising XAI approaches and highlight related work for the concrete approach.
Optimizing Explanation by Explanation Properties. This approach leverages XAI evaluation methods to assess how well explanations meet various XAI goals. Functionally-grounded evaluation methods serve as proxies for stakeholder desiderata, enabling the assessment of explanations based on diferent properties. This information can be used to directly optimize explanations towards those properties or aggregate explanations based on their properties. Users can personalize explanations by weighting diferent properties according to their preferences. Decker et al. [10] propose a method to enhance the reliability of feature attributions by combining multiple attribution methods to derive optimal convex combinations, improving robustness and faithfulness. Tadesse et al. [11] introduce a direct optimization approach that reliably produces explanations with optimal properties, allowing users to control trade-ofs between diferent properties.
Concept-based Explanations. This approach shifts the focus from feature-level explanations to concept-level explanations, aiming to express AI predictions in semantically human-understandable concepts. This allows for personalization towards stakeholder desiderata through pre-explaining concept elicitation or interactive adjustments. Das et al. [12] introduce the State2Explanation (S2E) framework, which provides concept-based explanations for AI decision-making, enhancing both AI agent learning and end-user understanding. The framework defines criteria for concepts in sequential decision-making and learns a joint embedding model between state-action pairs and concept-based explanations, significantly improving user task performance.
Glocal Explanations. Combining local XAI methods (explaining individual predictions) and global XAI methods (explaining the whole model), this approach enriches explanations semantically, making them more human-centered. Known as “Glocal” explanations, this method matches global model concepts to individual predictions, bringing stakeholders into the loop of AI prediction. Achtibat et al. [13] introduce the Concept Relevance Propagation (CRP) approach, which combines local and global perspectives by extending Layer-wise Relevance Propagation (LRP).
My research aims to develop XAI methods that personalize explanations to various stakeholder desiderata. Hereby, the plan is to use evaluation methods and human-centered concepts that improve the explanations by satisfying the end-user desiderata. Using functionally-grounded evaluation methods creates an initial understanding of the explanation quality. Promising approaches like property-optimized explanations, concept-based explanations, or glocal explanation frameworks should be further investigated and improved. I will apply the diferent XAI methods in CDSS’ with diferent stakeholders, to assess the methods applicability and human-ai-collaboration.
The research goal leads to the following general research question:
GRQ: How should AI explanations be created to satisfy various stakeholder desiderata in clinical decision support systems?
If related to current promising approaches, the following subquestions arise from this general research question: • RQ1: How can functionally-grounded evaluation methods adapt explanations to individual stakeholder desiderata? • RQ2: How can concept-based explanations enhance user understanding by aligning with individual user mental models? • RQ3: How can glocal explanation methods be implemented to enrich local explanations with global model insights and thus satisfy stakeholder desiderata?
The primary objective of this research is to develop human-centered explanations for AI systems through the exploration of various Explainable AI (XAI) methodologies. To achieve this, a mixedmethods research approach will be employed, combining design science research (DSR) with qualitative and quantitative techniques to comprehensively address the research questions (see Figure 2).
As a core component of DSR, my research will involve the iterative design and development of XAI approaches as artifacts. The development will be iterative, with continuous feedback integration from stakeholders of varying expertise levels. Through cycles of awareness of problem, suggestion, development, and evaluation, the explanations will be progressively aligned with each stakeholder group’s cognitive styles and domain-specific knowledge [ 14]. Each artifact iteration will be evaluated based on its alignment with stakeholder desiderata and its efectiveness in enhancing understanding. Hereby, with each cycle the number of selected XAI approaches will be reduced, to finally focus on one approach.
The single research methods, that are used in the process steps in each DSR cycle are described in the following and are divided into qualitative (see Section 4.1) and quantitative analysis (see Section 4.2) methods. 4.1. Qualitative Analysis As part of the DSR method, the following qualitative analysis research methods are used: Stakeholder Analysis through Interviews and Surveys. I will be gathering interviews and surveys to capture the diverse desiderata of diferent stakeholder groups and qualitative data. These will target stakeholders with varying roles, such as decision-makers, developers, and domain experts while considering their respective levels of expertise. This approach will ensure a comprehensive understanding of stakeholder-specific desiderata and mental models regarding AI systems. Thematic Analysis. The collected data will undergo thematic analysis, focusing on identifying themes that reflect the varied expectations and requirements of diferent stakeholder groups. This analysis will guide the development of explanation frameworks that are adaptable to diferent user expertise levels and cognitive styles. 4.2. Quantitative Analysis Additionally, the following quantitative analysis research methods are used within the DSR cycles: Experimental Design with Stratified Sampling. Controlled experiments will utilize stratified sampling to ensure representation across stakeholder groups and expertise levels. Participants will engage with various explanations (feature-based, concept-based, and glocal) to assess their impact on understanding, appropriate reliance, and satisfaction across diverse user profiles. Knowledge
Development Awareness of
Problem Suggestion Evaluation Conclusion
Third+ Cycle Interviews and Surveys: Capture diverse desiderata of different
stakeholders
Literature review
Thematic Analysis: Analyse collected data and identify themes that reflect varied desiderata of different stakeholder groups Artifact Version 1: Implementation of multiple human-centered XAI approaches that address desiderata
Qualitative and Quantitative
Evaluation: Application- and functionally-grounded evaluation of
XAI methods Show general feasability of humancentered XAI approaches
Gather feedback from stakeholders
on Artifact Version 1. Analyze evaluation results to identify strengths
and weaknesses.
Refine XAI approaches based on
feedback and evaluation.
Develop criteria for narrowing down approaches.
Analyze feedback from second cycle.
Plan optimization of top-performing XAI approach. Establish criteria for
final selection.
Artifact Version 2: Implement refined XAI approaches.
Artifact Version 3: Implement final version of selected XAI approach.
Qualitative and Quantitative
Evaluation: Application- and functionally-grounded evaluation of
XAI methods Identify top-performing XAI
approaches.
Summarize findings and prepare for next cycle.
Rigorous evaluation of final XAI approach. Validate effectiveness and
usability with stakeholders.
Finalize selection of best-fitting
XAI approach.
Evaluation Metrics. Functionally-grounded evaluation metrics such as faithfulness, sensitivity, and complexity will be employed to measure explanation quality quantitatively. 4.3. Mixed-Methods Research Approach The chosen research approach strategically incorporates the diverse needs of multiple stakeholders to answer the research questions and achieve the final contribution. Therefore, the research uses thematic analysis and stratified experimental design to customize explanation approaches to meet stakeholder groups’ specific needs and preferences. It aims to assess how technical implementation afects explanation properties and user satisfaction based on stakeholders’ expertise levels. Functionally-grounded evaluation methods are used to objectively assess various XAI approaches and optimize explanations to address diverse stakeholder requirements, directly tackling research question RQ1. Here, I assume that functionally-grounded evaluation methods are promising tools for improving the quality and efectiveness of explanations. Additionally, the research seeks to enhance interaction through humancentered explanations by developing frameworks tailored to stakeholders’ knowledge and cognitive styles, exploring the alignment of explanations with user desiderata (RQ2), and balancing global and local perspectives (RQ3), ensuring technical robustness and adaptability to various stakeholders. I expect that concept-based explanations improve decision-making in human-AI teams more efectively than feature-based explanations and that glocal explanations provide a more comprehensive understanding of AI models than purely global or local explanations.
My previous research work has been mainly directed towards gaining domain-specific knowledge in the ifeld of AI in healthcare and attaining a comprehensive understanding of XAI. The foundation of this research was an exhaustive literature review, that included an evaluation of various XAI methodologies to determine their applications and limitations. This examination incorporated an exploration of AI and XAI methods on healthcare applications, identifying critical areas where these technologies can improve clinical outcomes. Furthermore, I started to explore the role of XAI in building trust and improving collaboration between human practitioners and AI systems by participating workshops with clinicians and regular project meetings.
Year 1
Year 2
Year 3 Design Science
Research
To translate theoretical insights into practical applications, I participated in stakeholder workshops with our research project partners from a university hospital that focused on developing CDSS’ specific to the intensive care unit (ICU) environment and the collection of user requirements. These workshops were conducted with practicing physicians, ensuring that the AI systems align with clinical desiderata.
I engaged in a dataset exploration process, initiating research with the MIMIC-IV dataset and subsequently securing access to data from the ICU station at a local university hospital. This work contained data preprocessing, feature engineering, and implementing multivariate time series models to predict the patient’s length of stay or mortality.
To further assess the eficacy of XAI methods, I systematically evaluated various attribution-based XAI methods, including SHAP [15], LIME [16] and others. This evaluation was based on a multivariate time series data model to assess explanation properties such as faithfulness, complexity, and sensitivity to efectively measure each XAI method’s utility.
Moreover, my research extended into an interdisciplinary examination of the implications of XAI within the context of the EU AI Act, thereby contributing to socio-technical discussions and regulatory analysis. Through these foundational eforts, I have established a robust groundwork for my current research works to advance human-centered explanations of AI predictions in healthcare settings.
The research roadmap which will guide the development and validation of human-centered explanations, by highlighting the milestones to address the main research question and its final contribution. As visualized in Figure 3, the DSR method is planned for three years.
Research Milestones. My research is segmented into three main milestones that align with the primary research goal of developing human-centered explanations. Each milestone marks an end of a DSR cycle and thus, concludes the goal of the respective cycle.
1. First Milestone: Completed initial development and evaluation of multiple XAI approaches, establishing a foundation for refinement and selection in subsequent cycles. This includes creating a requirements backlog and selecting promising XAI concepts and methods, with initial results to guide further development. 2. Second Milestone: Refined and evaluated XAI approaches, identifying promising methods for ifnal selection. This involves confirming the selection of the final XAI approach with stakeholders, ensuring alignment with project goals and stakeholder needs. 3. Third Milestone: Successfully selected and optimized the best-fitting XAI approach for implementation. This milestone includes optimizing and approving a human-centered XAI approach, followed by final testing with stakeholders to ensure efectiveness and satisfaction.
Final Contribution. My research identifies and optimizes the most efective human-centered XAI approaches that address various stakeholder desiderata for CDSS’. Through a systematic DSR process, my work refines multiple XAI methods, evaluates their performance, and selects the best-fitting approach. My work not only advances the field of XAI by demonstrating the feasibility and efectiveness of human-centered solutions but also provides a comprehensive framework for future XAI development and implementation, ensuring enhanced interpretability and stakeholder satisfaction across diverse stakeholder groups. Thereby, the interdisciplinary exchange with socio-technical research disciplines is crucial.
My research uses a mixed-methods approach to enhance and implement novel XAI approaches that target the desiderata of various stakeholders in clinical decision support systems. Applying a DSR approach, the research incorporates attribution-based explanations to optimize and implement new human-centered XAI methods and evaluate them iteratively with various stakeholders. By this, my research significantly contributes to advancing human-centered XAI, thereby promoting the overall development of human-centered AI.
I express my sincere gratitude to my academic supervisor, Niklas Kühl, and my industrial supervisor, Jan-Philipp Steghöfer, for their exceptional guidance and support throughout my research. This research is sponsored by the German Federal Ministry of Education and Research under grant number 16SV9031.
During the preparation of this work, the author used ChatGPT-4o and Grammarly to: Grammar and spelling check, paraphrase and reword. After using these tools, the author reviewed and edited the content as needed and takes full responsibility for the publication’s content.