This paper explores the role of AI in data-driven decision-making across sectors like healthcare, retail, and energy, highlighting the challenges of ML models' complexity and opacity. It focuses on improving explanation understandability and trust through a study involving expert and layman feedback on different explanation types. Although the study focuses on developing a genetic programming (GP) tool to aid decision-making in these fields, the findings are relevant for any machine learning algorithm. This strategy enhances user trust and transparency across various ML models, providing applicable insights for AI applications.
Research in explainable AI (XAI) aligns AI system explanations with user expectations and needs. Key studies, such as [1], highlight identifying crucial stakeholders in AI explainability and the development of a framework to meet these needs. Tools like the System Causability Scale [2] and the System Usability Scale [3] have been introduced to assess ML explanation interfaces and their effectiveness. Furthermore, a novel questionnaire leveraging psychometrics [4] aims to reliably evaluate XAI method explanations, addressing explainability's complex nature. This body of work underpins our effort to craft AI tools that meet the diverse requirements of professionals in fields such as medicine, retail, and energy, proposing a cross-disciplinary approach to enhance user satisfaction and trust in AI applications. In their literature review, the authors in [5] define five primary goals for AI system interactions with end users: understandability, trustworthiness, transparency, controllability, and fairness. They recommend designing XAI systems to achieve these objectives and suggest guidelines for creating explanations focusing on crucial system components. Additionally, they highlight the necessity for compromises in AI explanations, underlining the absence of a one-size-fits-all solution.
The paper is organized as follows: we begin with an overview of related work. This is followed by introducing the three distinct use cases and their unique characteristics. In Section 3, we elaborate on the methodology employed in conducting the surveys. The paper concludes with a discussion of our findings and presents conclusions, including recommendations for developing a GP tool to support practitioners across three use cases. The developed questionnaires are publicly available to facilitate the dissemination of knowledge in the field of XAI.
The medical scenario explores GP models for paraganglioma and diabetes, aiming to predict the tumor's progression and diabetes presence. The model for paraganglioma seeks to guide physicians on treatment timing, enhancing shared decision-making, optimizing treatments, and reducing unnecessary interventions without substituting clinical judgment. For diabetes, the model uses a well-known dataset [6] to predict if a patient has or does not have diabetes.
Retail use case Grocery stores use Dynamic Timeslot Pricing to balance customer satisfaction with efficiency in home delivery. They offer flexible delivery times while keeping costs low. This AI-based approach sets fair and clear prices by looking at customer data and delivery logistics to estimate how much customers are willing to pay and the cost to serve. An algorithm then matches customer preferences with delivery efficiency to find the best times and prices.
The method, which sets slot prices using a specific formula (Prescriptive Model), depends on two support models-the Willingness to Pay (WTP) and Cost to Serve (CTS) models.
Energy use case To recommend savings, the energy use case predicts household energy consumption by analyzing weather, historical usage, building dynamics, pricing, and indoor temperatures. It aims to offer users clear explanations to support informed decisions and to integrate these insights into business strategies for improved energy efficiency. Key considerations include weather conditions, past consumption patterns, building characteristics, pricing strategies for managing demand, and indoor temperature monitoring for energy conservation. The challenge is making these forecasts understandable and actionable, facilitating efficient energy use and decision-making in practical settings.
This section outlines the survey methodologies applied to the three investigated use cases. Our approach incorporated two methods: conducting interviews with domain experts and distributing questionnaires to practitioners who may not have expert knowledge.
Details of the surveyed experts are available at this link: Interviewed Experts Document. Links to the questionnaires for each use case can be found in the following subsections. Three medical doctors completed the medical use case questionnaires, while the retail questionnaires were filled out by the interviewed expert and six additional respondents. For the energy case, six respondents completed the questionnaires, four of whom were the experts interviewed.
The questionnaire, which focused on diabetes risk estimation and was developed for the medical scenario, aimed to explore the type of AI model explanations doctors need. Key areas explored included the trade-off between accuracy and explainability, various presentation formats (such as symbolic regression graphs, genetic programming protocols, SHAP feature importance graphs, coefficients tables, and textual explanations), and their impact on understandability and decision-making effectiveness. Doctors were asked to rate each format's interpretability and effectiveness on a 1 to 5 scale.
Additionally, an interview focusing on the paraganglioma case collected insights on tumor identification, statistical prediction models, genetic factors, training protocols for new doctors, expectations from AI tools in managing paraganglioma, and the specific explanations needed for comprehending this condition. The questionnaire and interview outcomes are intended to guide the development of AI tools that effectively meet doctors' informational needs and preferences.
The questionnaire for the medical scenario can be explored here: Diabetes Questionnaire
The retail use case questionnaire was designed to delve into several key areas. First, they explored price breakthroughs to gauge the significance of location and demand and how clear the explanations were to customers. Next, the questionnaire sought to identify which types of explanations customers preferred and how well they understood them. Lastly, there was a focus on summarization assessment to evaluate the need for summaries in conjunction with detailed pricing information. This part aimed to assess how these summaries affected clarity and influenced decision-making. Participants rated explanations on interpretability and effectiveness from 1 (least) to 5 (highest), aiming to understand the extent to which explanations helped in decision-making and their clarity to customers. For this use case, two questionnaires have been devised for two categories of users.
1. Decision-makers Seek a comprehensive understanding of feature contributions to model predictions for system optimization. With their expert background, they prefer detailed, technical explanations to build trust and validate the model's use based on its accuracy. Decision-Makers Questionnaire 2. Customers Favor straightforward, accessible explanations that still convey essential information, aiding in understanding the rationale behind received offers without overwhelming technical detail.Customers Questionnaire
The interview, which was recorded as a video file, explored issues such as finding a balance between accuracy and explainability in e-commerce models, the incorporation of graphs and mathematical formulas into explanations, understanding customer behavior through the dynamic relationship between slot availability and pricing, and designing a dynamic dashboard to manage the interaction between operational efficiency and customer behavior effectively.
The questionnaire targets operational managers and customers, aiming to identify their preferred formats (tables, charts, interactive graphics, text) and types of explanations (causal, contrastive, counterfactual) for model predictions. Operational managers, the primary audience, must provide detailed feedback based on their expertise. They will focus on how model features affect predictions and optimization opportunities to enhance their trust and model endorsement through accurate and complex explanations. In contrast, customers likely prefer simpler, straightforward explanations that clarify the rationale behind offers. The energy questionnaire delves into key areas like the accuracy-explainability trade-off, the value of explanations in forecasting, the role of what-if scenarios in understanding model outcomes, and the specific needs of facility managers for detailed explanations and visualization tools such as SHAP graphs, highlighting preferences for explanation frequency and detail level.
All interviewed experts and five additional energy experts have completed the questionnaire. Energy Questionnaire
The interviews explored the energy problem from various angles, each tailored to the interviewee's expertise. Discussions ranged from addressing market challenges in energy solutions and the importance of clear explanations for end-users to exploring energy consumption disaggregation and the role of genetic programming in enhancing analysis. Insights were also shared on leveraging machine learning for water consumption monitoring to optimize resource management and identify inefficiencies. Additionally, the design and usability of user interfaces for energy management systems were examined, emphasizing the need for intuitive and engaging interfaces to manage energy consumption better.
Figure 1 summarizes key findings from the diabetes questionnaire.
Doctors prefer AI explanations that balance a slight decrease in accuracy for better clarity, find complex graphs challenging, and favor clear, intuitive details like protocols and SHAP graphs. Simplification and clarity were highlighted as essential for effectively conveying model logic, with counterfactual explanations being particularly valued for their potential to improve patient understanding and therapy compliance. Feature importance graphs were most favored, followed by textual explanations and rulebased protocols. Graphs and coefficient tables were least preferred due to concerns about understandability.
Interview insights highlight the novelty of our paraganglioma models due to a lack of benchmarks to measure the accuracy of our models, the critical role of genetic data in personalized medicine, and the need for tools to monitor tumor growth. The value doctors place on model predictions for patient communication emphasizes the importance of accurate, explainable models to foster trust and informed decisions. Initial tests on GP models for paraganglioma are documented in [7], providing detailed outcomes.
The decision-makers seek explanations across various dimensions: customer behavior, transportation costs, and strategies for maximizing profits. The questionnaires findings are summarized in the figure 2 In feedback from decision-makers on AI system explanations, there's an openness to sacrificing a portion of model performance for enhanced explainability, with preferences for detailed yet intuitive insights into model workings. This encompasses a broad interest in customer behavior, cost analysis, and profit strategies, highlighting a desire for interactive tools and visualizations that facilitate deeper understanding and strategic adjustments. There's a notable emphasis on practical application, with decision-makers valuing features like counterfactual explanations and the ability to interpret and act upon complex information, all aimed at optimizing operational efficiency and customer engagement.
The interview highlighted a preference for explainability over accuracy, with caution advised due to limited machine learning expertise. Simple visual explanations and mathematical formulas are preferred to avoid complexity. Graphical dashboards are recommended for assessing operational efficiency and customer behavior, enhancing interpretability and interaction. Counterfactual explanations are valued for demonstrating the impact of decisions such as new scheduling slots. Developing models that identify customer characteristics and behaviors by region is essential for deeper business insights.
The insights from operational and facility managers are summarized in figure 3. Operational managers favor a balance between accuracy and transparency, adjusting the trade-off based on the audience. They prefer visual and simple mathematical explanations to suit various stakeholder technical levels. Graphical dashboards are effective for insights into efficiency and customer behavior, with counterfactual explanations providing useful scenario analysis. Strategic analyses, such as regional behavior modeling and what-if scenarios, highlight the value of feature importance graphs and counterfactuals in delivering clear, actionable insights for decision-making and management.
Insights from the interviews demonstrate a preference for explanatory forecasting models over basic ones, with methods applicable across sectors like gas and energy. Ease of use and interactive elements are advised for the graphical interface, alongside a smartphone component for energy applications to enable notifications. For detailed analyses of GP models in energy, see [8] and [9].
The table 1 summarizes the overarching guidelines derived from the survey findings. Drawing from these insights, the design of the explanatory tool should incorporate two essential modules: a Counterfactual Module, which calculates the minimal changes required to shift the model's decision towards a desired outcome, thereby enabling "What-if" scenarios based on user queries, and a Global Importance Module, which provides visualization of the significant feature contributions to the model's predictions, in line with findings from the user studies. Both modules should be integrated within the tool, ensuring that the inputs, outputs, and connections between modules are well-defined.
This study identifies foundational components for an XAI framework intended for various applications through comprehensive questionnaires and interviews with domain experts in three distinct use cases. The envisioned XAI tool incorporates a Counterfactual Module to facilitate "What-if" scenarios, allowing users to see how minimal changes could lead to desired outcomes. Additionally, a Global Importance Module is designed to visually represent the most influential features in model predictions, resonating with the XAI literature emphasizing the critical role of feature importance and counterfactual explanations. While aiming for shared applicability, the framework also acknowledges the unique requirements of each specific case, although the detailed exploration of these unique case aspects was beyond this paper's scope. This approach informs the ongoing development of the AI tool, leveraging insights gathered from user studies to ensure the tool's effectiveness across different domains. Our tool is now prepared for evaluation by experts across the three fields. We will integrate their feedback into an updated version of the tool. For future research, the interest in online retail and energy sectors for customizable and user-specific explanations points towards a growing trend. This trend leans towards integrating NLP interactivity into explanations, an area we are beginning to explore.
This research was conducted under the Transparent, Reliable, and Unbiased Smart Tool for AI (Trust-AI) project, with Grant Agreement ID: 952060, funded by the EU Commission.