This study presents insights gathered from surveys and discussions with specialists in three domains, aiming to find essential elements for an explanation framework that could be applied to these and possibly other use cases. The applications analyzed include a medical scenario (involving predictive ML), a retail use case (involving prescriptive ML), and an energy use case (also involving predictive ML). We interviewed professionals from each sector, transcribing their conversations for further analysis. Additionally, experts and non-experts in these fields filled out questionnaires designed to probe various dimensions of explanatory methods. The findings indicate a universal preference for sacrificing a degree of accuracy in favor of greater explainability. Additionally, we highlight the significance of feature importance and counterfactual explanations as critical components of such a framework. Our questionnaires are publicly available to facilitate the dissemination of knowledge in the field of XAI.
This paper explores the role of AI in data-driven decision-making across sectors like healthcare, retail, and energy, highlighting the challenges of ML models’ complexity and opacity. It focuses on improving explanation understandability and trust through a study involving expert and layman feedback on diferent explanation types. Although the study focuses on developing a genetic programming (GP) tool to aid decision-making in these fields, the findings are relevant for any machine learning algorithm. This strategy enhances user trust and transparency across various ML models, providing applicable insights for AI applications.
Research in explainable AI (XAI) aligns AI system explanations with user expectations and
needs. Key studies, such as [
The paper is organized as follows: we begin with an overview of related work. This is followed by introducing the three distinct use cases and their unique characteristics. In Section 3, we elaborate on the methodology employed in conducting the surveys. The paper concludes with a discussion of our findings and presents conclusions, including recommendations for developing a GP tool to support practitioners across three use cases. The developed questionnaires are publicly available to facilitate the dissemination of knowledge in the field of XAI.
Medical Scenario The medical scenario explores GP models for paraganglioma and diabetes,
aiming to predict the tumor’s progression and diabetes presence. The model for paraganglioma
seeks to guide physicians on treatment timing, enhancing shared decision-making, optimizing
treatments, and reducing unnecessary interventions without substituting clinical judgment. For
diabetes, the model uses a well-known dataset [
Retail use case Grocery stores use Dynamic Timeslot Pricing to balance customer satisfaction with eficiency in home delivery. They ofer flexible delivery times while keeping costs low. This AI-based approach sets fair and clear prices by looking at customer data and delivery logistics to estimate how much customers are willing to pay and the cost to serve. An algorithm then matches customer preferences with delivery eficiency to find the best times and prices.
The method, which sets slot prices using a specific formula (Prescriptive Model), depends on two support models—the Willingness to Pay (WTP) and Cost to Serve (CTS) models.
Energy use case To recommend savings, the energy use case predicts household energy consumption by analyzing weather, historical usage, building dynamics, pricing, and indoor temperatures. It aims to ofer users clear explanations to support informed decisions and to integrate these insights into business strategies for improved energy eficiency. Key considerations include weather conditions, past consumption patterns, building characteristics, pricing strategies for managing demand, and indoor temperature monitoring for energy conservation. The challenge is making these forecasts understandable and actionable, facilitating eficient energy use and decision-making in practical settings.
This section outlines the survey methodologies applied to the three investigated use cases. Our approach incorporated two methods: conducting interviews with domain experts and distributing questionnaires to practitioners who may not have expert knowledge.
Details of the surveyed experts are available at this link: Interviewed Experts Document. Links to the questionnaires for each use case can be found in the following subsections. Three medical doctors completed the medical use case questionnaires, while the retail questionnaires were filled out by the interviewed expert and six additional respondents. For the energy case, six respondents completed the questionnaires, four of whom were the experts interviewed.
The questionnaire, which focused on diabetes risk estimation and was developed for the medical scenario, aimed to explore the type of AI model explanations doctors need. Key areas explored included the trade-of between accuracy and explainability, various presentation formats (such as symbolic regression graphs, genetic programming protocols, SHAP feature importance graphs, coeficients tables, and textual explanations), and their impact on understandability and decision-making efectiveness. Doctors were asked to rate each format’s interpretability and efectiveness on a 1 to 5 scale.
Additionally, an interview focusing on the paraganglioma case collected insights on tumor identification, statistical prediction models, genetic factors, training protocols for new doctors, expectations from AI tools in managing paraganglioma, and the specific explanations needed for comprehending this condition. The questionnaire and interview outcomes are intended to guide the development of AI tools that efectively meet doctors’ informational needs and preferences.
The questionnaire for the medical scenario can be explored here: Diabetes Questionnaire
The retail use case questionnaire was designed to delve into several key areas. First, they explored price breakthroughs to gauge the significance of location and demand and how clear the explanations were to customers. Next, the questionnaire sought to identify which types of explanations customers preferred and how well they understood them. Lastly, there was a focus on summarization assessment to evaluate the need for summaries in conjunction with detailed pricing information. This part aimed to assess how these summaries afected clarity and influenced decision-making. Participants rated explanations on interpretability and efectiveness from 1 (least) to 5 (highest), aiming to understand the extent to which explanations helped in decision-making and their clarity to customers. For this use case, two questionnaires have been devised for two categories of users.
1. Decision-makers Seek a comprehensive understanding of feature contributions to model predictions for system optimization. With their expert background, they prefer detailed, technical explanations to build trust and validate the model’s use based on its accuracy.
Decision-Makers Questionnaire 2. Customers Favor straightforward, accessible explanations that still convey essential information, aiding in understanding the rationale behind received ofers without overwhelming technical detail.Customers Questionnaire
The interview, which was recorded as a video file, explored issues such as finding a balance between accuracy and explainability in e-commerce models, the incorporation of graphs and mathematical formulas into explanations, understanding customer behavior through the dynamic relationship between slot availability and pricing, and designing a dynamic dashboard to manage the interaction between operational eficiency and customer behavior efectively.
The questionnaire targets operational managers and customers, aiming to identify their preferred formats (tables, charts, interactive graphics, text) and types of explanations (causal, contrastive, counterfactual) for model predictions. Operational managers, the primary audience, must provide detailed feedback based on their expertise. They will focus on how model features afect predictions and optimization opportunities to enhance their trust and model endorsement through accurate and complex explanations. In contrast, customers likely prefer simpler, straightforward explanations that clarify the rationale behind ofers. The energy questionnaire delves into key areas like the accuracy-explainability trade-of, the value of explanations in forecasting, the role of what-if scenarios in understanding model outcomes, and the specific needs of facility managers for detailed explanations and visualization tools such as SHAP graphs, highlighting preferences for explanation frequency and detail level.
All interviewed experts and five additional energy experts have completed the questionnaire. Energy Questionnaire
The interviews explored the energy problem from various angles, each tailored to the interviewee’s expertise. Discussions ranged from addressing market challenges in energy solutions and the importance of clear explanations for end-users to exploring energy consumption disaggregation and the role of genetic programming in enhancing analysis. Insights were also shared on leveraging machine learning for water consumption monitoring to optimize resource management and identify ineficiencies. Additionally, the design and usability of user interfaces for energy management systems were examined, emphasizing the need for intuitive and engaging interfaces to manage energy consumption better.
Feature importance graphs were most favored, followed by textual explanations and rulebased protocols. Graphs and coeficient tables were least preferred due to concerns about understandability.
Interview insights highlight the novelty of our paraganglioma models due to a lack of
benchmarks to measure the accuracy of our models, the critical role of genetic data in personalized
medicine, and the need for tools to monitor tumor growth. The value doctors place on model
predictions for patient communication emphasizes the importance of accurate, explainable
models to foster trust and informed decisions. Initial tests on GP models for paraganglioma are
documented in [
The decision-makers seek explanations across various dimensions: customer behavior, transportation costs, and strategies for maximizing profits. The questionnaires findings are summarized in the figure 2
In feedback from decision-makers on AI system explanations, there’s an openness to sacrificing a portion of model performance for enhanced explainability, with preferences for detailed yet intuitive insights into model workings. This encompasses a broad interest in customer behavior, cost analysis, and profit strategies, highlighting a desire for interactive tools and visualizations that facilitate deeper understanding and strategic adjustments. There’s a notable emphasis on practical application, with decision-makers valuing features like counterfactual explanations and the ability to interpret and act upon complex information, all aimed at optimizing operational eficiency and customer engagement.
The interview highlighted a preference for explainability over accuracy, with caution advised due to limited machine learning expertise. Simple visual explanations and mathematical formulas are preferred to avoid complexity. Graphical dashboards are recommended for assessing operational eficiency and customer behavior, enhancing interpretability and interaction. Counterfactual explanations are valued for demonstrating the impact of decisions such as new scheduling slots. Developing models that identify customer characteristics and behaviors by region is essential for deeper business insights.
The insights from operational and facility managers are summarized in figure 3.
Operational managers favor a balance between accuracy and transparency, adjusting the trade-of based on the audience. They prefer visual and simple mathematical explanations to suit various stakeholder technical levels. Graphical dashboards are efective for insights into eficiency and customer behavior, with counterfactual explanations providing useful scenario analysis. Strategic analyses, such as regional behavior modeling and what-if scenarios, highlight the value of feature importance graphs and counterfactuals in delivering clear, actionable insights for decision-making and management.
Insights from the interviews demonstrate a preference for explanatory forecasting models
over basic ones, with methods applicable across sectors like gas and energy. Ease of use and
interactive elements are advised for the graphical interface, alongside a smartphone component
for energy applications to enable notifications. For detailed analyses of GP models in energy,
see [
The table 1 summarizes the overarching guidelines derived from the survey findings.
Drawing from these insights, the design of the explanatory tool should incorporate two essential modules: a Counterfactual Module, which calculates the minimal changes required to shift the model’s decision towards a desired outcome, thereby enabling "What-if" scenarios based on user queries, and a Global Importance Module, which provides visualization of the significant feature contributions to the model’s predictions, in line with findings from the user studies. Both modules should be integrated within the tool, ensuring that the inputs, outputs, and connections between modules are well-defined.
This study identifies foundational components for an XAI framework intended for various applications through comprehensive questionnaires and interviews with domain experts in three distinct use cases. The envisioned XAI tool incorporates a Counterfactual Module to facilitate "What-if" scenarios, allowing users to see how minimal changes could lead to desired outcomes. Additionally, a Global Importance Module is designed to visually represent the most influential features in model predictions, resonating with the XAI literature emphasizing the critical role of feature importance and counterfactual explanations. While aiming for shared applicability, the framework also acknowledges the unique requirements of each specific case, although the detailed exploration of these unique case aspects was beyond this paper’s scope. This approach informs the ongoing development of the AI tool, leveraging insights gathered from user studies to ensure the tool’s efectiveness across diferent domains. Our tool is now prepared for evaluation by experts across the three fields. We will integrate their feedback into an updated version of the tool. For future research, the interest in online retail and energy sectors for customizable and user-specific explanations points towards a growing trend. This trend leans towards integrating NLP interactivity into explanations, an area we are beginning to explore.
This research was conducted under the Transparent, Reliable, and Unbiased Smart Tool for AI (Trust-AI) project, with Grant Agreement ID: 952060, funded by the EU Commission.