As machine learning becomes increasingly integral to autonomous decision-making processes involving human interaction, the necessity of comprehending the model's outputs through conversational means increases. Most recently, foundation models are being explored for their potential as post hoc explainers, providing a pathway to elucidate the decision-making mechanisms of predictive models. In this work, we introduce traceable question-answering, leveraging an external knowledge repository to inform the responses of Large Language Models (LLMs) to user queries within a scene understanding task. This knowledge repository comprises contextual details regarding the model's output, containing highlevel features, feature importance, and alternative probabilities. We employ subtractive counterfactual reasoning to compute feature importance, a method that entails analysing output variations resulting from decomposing semantic features. Furthermore, to maintain a seamless conversational flow, we integrate four key characteristics - social, causal, selective, and contrastive - drawn from social science research on human explanations into a single-shot prompt, guiding the response generation process. Our evaluation demonstrates that explanations generated by the LLMs encompassed these elements, indicating its potential to bridge the gap between complex model outputs and natural language expressions.
CEUR ceur-ws.org
As we become increasingly reliant on AI applications in our daily lives, it becomes imperative
to interact with autonomous decision-making systems in human-understandable terms [
The increasing demand for transparent and interpretable AI models has prompted the
exploration of various explanation methods, resulting in significant progress in both inherently
interpretable and post-hoc explainability methods [
Additionally, to cultivate a seamless conversational experience, we incorporated key elements derived from social science research on human explanations into the system prompt, directing the response generation process. We present our vision for creating this integrated approach, outlining each component, and illustrating its viability through a real-world dataset. This integration has the potential to bridge the gap between intricate model outputs and user comprehension towards advancing the development of human-understandable explanations.
The demand for transparent and interpretable AI models has led to exploring diverse explanation
methods for vision tasks [
Recent strides in Large Language Models (LLMs), exemplified by GPT-4 (OpenAI), Bard (Google),
Claude-2 (Anthropic), and Llama-2 (Meta), mark a transformative era in Natural Language
Processing (NLP) research. These models have become pervasive across diverse applications,
ranging from machine translation and question-answering to text generation [
In the context of XAI, the audience for explanations is predominantly human; thus,
understanding what makes an explanation human-friendly is paramount. While accuracy is important,
the clarity and presentation of explanations are equally crucial in ensuring comprehensibility.
According to Miller [
An explanation represents an interactive exchange between two roles: the explainer and the
explainee, and is governed by certain ‘rules’ [
Moreover, explanatory agents employing anthropomorphic traits like politeness markers
(e.g., “Thank you!”), warm and friendly tones, and empathetic expressions (e.g., “I understand.”)
are pivotal for efective human interaction [
Causal explanations aim to elucidate the cause-and-efect relationship underlying a decision.
When explaining a prediction made by a machine learning model, identifying the most influential
features can provide insight into the key causal connections to the prediction. This process,
also known as causal inference, starts with observations (e.g., what if a feature had a diferent
value) and selecting some of those causes as the explanation. Miller divides casual inference
into two parts: counterfactuals and abductive reasoning [
The reasoner uses a model of hypothetical counterfactual cases to derive an explanation, such
as perturbing inputs, to see how they causally impact the prediction (See Section 3.1). Abductive
reasoning is inferring the causes from those observations to form the most probable explanation.
One can think of abductive reasoning as the following process: observing unexpected or
surprising events, generating one or more hypotheses about these events, judging the plausibility
of the hypotheses, and selecting the ‘best’ hypothesis as the explanation [
Explanations are contextual by nature. While an event may have numerous causes, the recipient
of the explanation typically cares about only a specific subset relevant to the context [
Abnormality is a key criterion for explanation selection; as such, people select unusual causes
to explain events [
Studies have shown that people only request contrastive explanations, essentially ‘why’
questions framed to imply the diferences between two possible outcomes [
Similarly, in the context of ML interpretability, people are not specifically interested in all
the factors that led to the prediction but instead in the factors that need to change so that the
prediction would also change. Explanations that present some contrast between the instance to
explain and a reference point (i.e., a hypothetical instance) are preferable because the cognitive
burden of complete explanations is too high. However, most existing work considers contrastive
questions but not contrastive explanations, providing two complete explanations individually.
This could be because the contrastiveness is application-dependent, given that a reference point
may not be apparent in all tasks [
This section outlines the key elements of the traceable question-answering methodology. We detail the proposed explainability approach for extracting feature importance and the prompting technique to generate accurate and human-friendly explanations. Additionally, we provide an application example illustrating how we establish a connection between the model output, the explainability technique, and language models to produce natural language explanations in an interactive format.
Subtractive counterfactual reasoning is the process of removing an event to understand its causal
influence on an outcome [
This study proposes a decomposition-based approach to examine the output variations by decomposing input values. By measuring such variations, we determine the degree of impact on an outcome and reason about the importance of each feature value. As a function, the model assigns a class label and prediction probability ∶ → () given input . To measure the efect of each input value, we observe the model’s prediction for without the knowledge of event = , where is the value of feature . To explicitly represent the absence of information, the feature value is replaced with an undefined value, i.e., NaN (Not a Number). NaN values are treated as invalid or masked elements and are not considered in the computation. We attain an array containing probabilities for all perturbations resulting from decomposition. In order to determine the importance of an individual feature , we begin by identifying the and values within this array, which establishes the decision boundaries for the given input. Next, we compute the feature importance as the position of the probability– after decomposing the feature– relative to the and probabilities. The computation is expressed as follows: ( ) (1) ) =
(| ∖ (| ∖ )
− (| ∖ − (| ∖ ) )
This approximates an importance value between zero and one for each input feature. Considering that evaluating fractions has a higher cognitive load, the values are rounded to the nearest integer to increase comprehensibility. Moreover, this analysis is carried out for the categories following the main prediction simultaneously to generate explanations for the contrastive cases.
Traceable question-answering, through a process known as Retrieval-Augmented Generation
(RAG), integrates external knowledge sources to furnish responses to user inquiries. This
method improves the performance of LLMs by referencing an external knowledge source prior
to generating a response (See Figure 1). This approach mitigates the problem of ‘hallucination’
and ensures factual consistency, particularly in scenarios where facts may evolve over time [
In our work, the knowledge repository contains tabular information about model output, features, contrastive cases, and feature importance values obtained through the decompositionbased explanation method described in Section 3.1. This information is saved as the output of the explanation module as a .csv file. Then, the file, the system prompt (See Figure 3), and the user input query are fed to the LLM model for response generation. The resulting responses contextualise the feature importance grounded in the models’ inner workings, thereby rendering the explanations traceable to the underlying model.
Furthermore, our aim is to imbue LLM with the characteristics of explanations discussed in Section 2.3 to foster a human-friendly interaction. Ideally, LLM-generated responses should adhere to conversational norms, encompassing opening, clarification, and closing statements while appropriately integrating social cues. The model is expected to deduce causal relationships from the observations within the provided knowledge repository, selecting the most pertinent causes to form an explanation and presenting it alongside a contrastive case. To accomplish this goal, we devised a system prompt that provides explicit instructions to the model to generate eficient responses to user queries. The system prompt includes a single-shot example illustrating the desired interaction, potential user inquiries and outlining how the model can efectively address them, enabling the language model to produce consistent and predictable outcomes, as presented in Figure 3.
To demonstrate our approach to traceable question-answering, we selected a visual scene understanding task. This involves scene classification, categorizing images based on their content and inferring high-level semantic scenes from low-level visual features. Many critical applications of scene classification include human-robot collaboration, autonomous driving, and other autonomous physical systems that rely on semantic segmentation to inform decisionmaking processes [40, 41, 42]. We experimented with the GoogleNet Places365 model designed specifically for scene recognition and classification tasks [ 43]. The model is trained on the Places365 dataset, containing more than 365 scene categories. We focus on semantic urban scene understanding with the road segmentation dataset CamVid (Cambridge-driving Labelled Video Database). The dataset helps to understand driving scenes, comprising images capturing street-level views from urban driving scenarios. It includes manual annotations across 32 classes, encompassing elements such as building, driveway, pavement, tree, trafic sign, car, pedestrian, and bicyclist. The Deeplab v3+ deep learning model is trained on the CamVid dataset for semantic segmentation [44]. Figure 2 shows the segmentation result for the example presented in the next section. Since we utilized of-the-shelf models for illustration purposes, the emphasis is not on these models’ performances.
Semantic information plays a critical role in autonomous vehicle (AV) decision-making processes, as it captures the meaning and context of the physical surroundings observed in the road scene. This includes understanding the appearance of various elements such as buildings and roads, identifying shapes such as cars and pedestrians, and recognizing spatial relationships to contextualize the scene [45]. Particularly in unstructured of-road environments, the accurate understanding of the surroundings concerning semantic classes such as trail, grass, or rock is important for safe and deliberate navigation [46]. Moreover, semantic segmentation provides a way to represent and comprehend the details of the environment in the language space. By leveraging the labelled information within the image scene, we can semantically map the crucial components based on their feature importance. This granularity enables a more insightful analysis of feature importance, contributing to a nuanced understanding of the model’s decisionmaking process. Such insights can be helpful for post-incident forensic analysis in case of misreading a scene causing collisions or accidents. In this work, we suppose situations where the user checks in with an AI assistant in an AV system in non-critical driving scenarios while engaging in non-driving related tasks.
In this section, we present our experimental findings. We analyzed nine distinct scene classes — parking lot, street, residential neighbourhood, crosswalk, highway, industrial area, gas station, shopfront, and general store (outdoor)— across diferent scenarios (totalling 42), where the model displayed varying degrees of confidence in its output. We evaluated the responses in terms of sociability, causality, selectiveness, and contrastiveness generated by GPT-3.5 (gpt-3.5-turbo) and GPT-4 (gpt-4-1106-preview) models.
SYSTEM: You are an AI model designed to adapt to various scenarios in autonomous driving. You have the details like predicted class, alternative predictions (contrastive cases), probabilities, feature importance, and how certain factors afect predictions. When engaging with user queries, focus on the specific question, provide concise explanations within a 50-word limit, and use friendly language. Identify key causal connections and highlight abnormal values that significantly contribute to the prediction, explaining their relevance in driving scenarios. Here’s an example of how you could respond to user questions: USER: The display panel just showed ‘residential neighbourhood’ on the screen. It didn’t seem pretty confident.
YOU (accessing relevant data): Hey there! It looks like we’re in a ‘residential neighbourhood’ right now with a probability of 40 percent. This means we’re on a road surrounded by features like driveways, sidewalks, trees, and cars, which are highly influential (with high importance values) to this observation.
USER: Cool! I am just curious, what happens if there was no sidewalk?
If the environment looked more like a freeway with no sidewalks, the ‘residential neighbourhood’ would be less likely (drop to 20 percent); in contrast, it would increase the probability for ‘highway’ (alternative prediction) to 13 percent.
USER: Interesting! Can you tell me how ‘residential neighbourhood’ and ‘highway’ difer in their features? YOU (comparing most important features): Sure! Both places have buildings and cars, but the sidewalks and trees are strong clues for a neighbourhood.
USER: How many cars are there? YOU: I’m sorry, but I’m currently unable to provide the exact number of cars. However, based on the detected features, we’re in a residential area where I should be extra cautious for pedestrians and potentially slower speeds compared to major roads. Is that a suficient explanation? USER: Yes, thanks! YOU: You’re welcome! If you have any more questions or need assistance with anything else, feel free to ask. Enjoy the ride!
In examining the sociability within the responses, our initial focus centres on the overall tone of the responses. We employed the Valence Aware Dictionary and Sentiment Reasoner (VADER) to gauge the sentiment conveyed within the text, encompassing dimensions of positivity, negativity, and neutrality. Sentiment scores for each response are aggregated to delineate overall sentiment trends. Figure 5 illustrates a noticeable positive trendline by both models, indicating a prevalent USER: The display panel just showed ‘parking lot’ on the screen.
ChatXAI: Hey! It appears we’re in a ‘parking lot’ with a likelihood of 52%. Notably, cars have the highest feature importance (10), which drastically reduces the probability to 17% if they’re absent. That’s a key indicator for us to be mindful of our surroundings, especially of vehicles moving in or out.
USER: Cool! What could it be otherwise if it wasn’t a parking lot? ChatXAI: If it weren’t a parking lot, the system suggests an ‘industrial area’ with a probability of 11%, or a ‘motel’ at 9%, or even a ‘gas station’ at 6%. Each has a distinct pattern of features impacting their likelihood.
USER: Can you tell me how parking lot and industrial area difer in their features? ChatXAI: Certainly! A ‘parking lot’ and an ‘industrial area’ difer mainly in buildings and poles, which are more critical in an industrial context (with importance ratings of 10 and 4, respectively). Also, the presence of driveways is a vital feature for an industrial area (importance 10) compared to a parking lot.
USER: Is there an empty space? ChatXAI: I’m sorry, but I don’t currently have the ability to visually confirm the presence of an empty space in real-time. My function is to assist with predictive analytics based on the features detected in the driving scenarios. If you need help with anything else, feel free to ask! USER: Ok, thanks! ChatXAI: You’re welcome! If you have any more questions or need assistance with anything else, feel free to ask. Safe travels! inclination towards positivity in the sentiment expressed. A qualitative inspection of the text reveals a prevalence of polite, helpful, and informative social cues across all responses, supporting this observation. Consequently, the neutral sentiment score remains notably low, underscoring the prevalence of emotional expression throughout the corpus. Exploring negative sentiment scores unveils instances where the models acknowledge their limitations, particularly when unable to retrieve answers from the external knowledge repository.
Considering the three foundational phases of dialogue—opening, clarification, and closing statements—next, we analyzed the indicators such as politeness markers and emotional expressions within each phase. To facilitate this analysis, we created a distinct dictionary for each phase and computed the term frequency of social cues present in each phase.2 Opening statements include greetings such as ‘hey’ and ‘hello.’ Clarification statements are characterized by social cues indicative of helpfulness, continued support, and awareness of its limitations, 2Term frequency is defined as the number of times a term occurs in the document over the total number of terms in a document. Positive
Neutral
Negative including terms such as ‘absolutely’, ‘sorry’, ‘further’, ‘curious’, ‘questions’, ‘help’, ‘feel’, and ‘free’. Closing statements commonly conveyed sentiments of well-wishing or safety, with phrases including terms such as ‘welcome’, ‘enjoy’, ‘safe’, ‘great’, ‘pleasant’, and ‘glad’.
Figure 6 shows that both models utilize these terms fairly across the respective phases of the dialogue. Our qualitative assessment of the responses further confirms the contextual relevance of these terms, which align with conversational maxims (See Section 2.3). The higher frequency value observed in closing statements can be attributed to their typically brief nature, which encapsulates most of the social cue terms. Additionally, both models occasionally repeat certain phrases in their responses, albeit with the application of a frequency and presence penalty (0.3).3 In addition to analysing social cues, we examined part-of-speech (POS) pronouns as social interaction markers, encompassing terms such as ‘I’, ‘you’, ‘we’, ‘my’, ‘our’, and ‘your.’ Quantitative analysis and qualitative evaluation of the results indicate that both models adequately and appropriately applied POS pronouns.
Opening
Statement Clarification
Statement
Closing Statement
PoS Pronouns 0 3Frequency penalty discourages repetitive use of words or phrases, while presence penalty promotes diversity in the generated text. 13 13 9 5 3 50 40 30 20 10 0 Figure 7: Frequency of causatGioPnT-3t.5erms observed.
To evaluate the expression of causality, we initially extracted responses indicating cause and efect, primarily found within clarification statements. We then identified key terms related to causality, such as ‘because’, ‘if’, ‘then’, ‘albeit’, ‘due’, ‘contribute’, ‘influence’, ‘afect’, ‘impact’, and ‘efect’, and tallied their occurrences within each answer to measure the presence of causal reasoning. In Figure 7, we compare the frequency of these terms used by two models in the response set. Notably, ‘if’ emerges as the most frequent term used by both models, often utilized for expressing counterfactual reasoning (e.g., ‘if it was not’, ‘if there is/were’), while ‘influence’ typically indicates the cause of a decision. Overall, GPT-4 demonstrated a higher frequency of causality terms and showcased greater diversity in expression compared to GPT-3.5.
Through the qualitative review, we examined existing patterns of dependency parsing signifying causal relationships. Two primary grammatical structures emerged, suggesting cause-efect relationships within the text. The first structure pertains to how the presence or absence of certain features influences the likelihood of specific scenarios. The second structure involves a cause-efect relationship between feature importance and probability, indicating that features with higher importance scores exert a greater impact on probability. It is important to acknowledge that while these grammatical structures and terms are widely used, their accuracy is not always consistent across models. In the subsequent section, we will further evaluate the selection and accuracy of inferred causal relationships.
In this section, we evaluate whether the models efectively detect unusual values within the provided knowledge repository, infer potential causes from those observations, and select the most relevant ones to construct an explanation. Our evaluation encompasses analysing responses across the number of causes given, graded selection, and sort order to assess whether expected causes are addressed correctly. First, we specified the maximum number of uniquely suficient causes for each case, which are feature importance values >5. The number of causes varies across scenarios; some exhibit only a single highly important feature >5, while others might involve multiple high-importance features with varying degrees of significance. Graded selection is crucial for discerning important values such that the language model must prioritize the most relevant causes when encountered with multiple features of comparable importance levels. Additionally, graded selection entails disregarding features with lower importance scores (<5). Finally, the sort order concerns the sequence of selected causes, which should follow a hierarchy from the most significant and relevant features to those of lesser importance. Thereafter, responses are categorized as fulfilled, partial, subpar, or unfulfilled according to their arrangement with our specifications.
The findings presented in Table 2 indicate that GPT-4 outperformed GPT-3.5 in all categories. Notably, GPT-4 had a much higher success rate (76%) than GPT-3.5 (24%) in identifying the most relevant cause (fulfilled). Additionally, GPT-4 had a lower rate of subpar and unfulfilled responses. GPT-3.5 referenced features not present in the knowledge source as potential causes in two instances. These results suggest that GPT-4 is more accurate and selective than GPT-3.5 in identifying the most relevant cause(s) from a set of possible causes.
Furthermore, an additional criterion for assessing explanation selection is simplicity, which involves utilizing concise language, avoiding technical terminology, and referencing fewer causes. Table 3 evaluates responses by comparing three metrics: the frequency of technical jargon, response length, and the number of causes cited. Lower values across all three metrics signify simpler responses.
To measure the term frequency of technical jargon, we compiled a list of technical terms in the responses and calculated their frequency within the generated explanations. The list contains terms such as ‘prediction’, ‘feature importance’, ‘score’, ‘contrastive cases’, and ‘confidence’, which may not be easily comprehensible to all users. Table 3 indicates that GPT-3.5 exhibits a slightly higher term frequency (0.0162) compared to GPT-4 (0.0157), suggesting that GPT-3.5 utilizes technical jargon marginally more frequently. The length metric refers to the median lengths of the responses generated by each model, which are quite similar (45 tokens for GPT-3.5 and 44 tokens for GPT-4). In terms of the number of causes cited, GPT-4 cites fewer causes (median of 2) than GPT-3.5 (median of 3), suggesting that GPT-4’s responses tend to be simpler in terms of causal reasoning. Overall, the results suggest that GPT-4 generates simpler responses than GPT-3.5 across all three assessed metrics.
Regarding the out-of-scope questions posed, such as the number of cars or pedestrians, the intent was to assess whether the model would fabricate an answer. Both models have expressed their inability to provide such information. This demonstrated an awareness of their limitations rather than attempting to generate a response.
Ideally, initial explanations should include a contrast with an alternative case to justify the output. Then the user would investigate it further if desired. In our experiment, we observed that in most cases, models, particularly with GPT3.5, failed to mention alternative scenarios unless explicitly prompted (e.g., inquiries such as “What could it be otherwise if it wasn’t X?” or “How does X difer from other options?”). Nevertheless, to measure the presence of contrastiveness across models, we analysed the occurrence of contrastive terms and comparative phrases within the generated responses. We compiled a dictionary consisting of terms such as ‘distinguish’, ‘diferent’, ‘contrast’, ‘compared to’, ‘in contrast’, ‘while’, ‘both’, ‘on the other hand’, ‘whereas’, and ‘conversely’, which introduce contrasting explanations. Figure 8 shows that ‘distinguish’ is the most frequently used term by GPT-3.5, whereas ‘diferent’ predominates in GPT-4’s responses. Other frequently encountered terms include ‘compared to’, ‘contrast’, and ‘diferentiate.’ Overall, the results suggest that GPT-4 employs a wider array of contrastive terms than GPT-3.5 in its responses. Our qualitative observations further confirm that while GPT-3.5 tends to adhere to one type of contrastive explanation, GPT-4 explores diverse ways to express such contrasts.
18 16
Although both models frequently employ these terms, indicating the generation of alternative viewpoints in their responses, the efectiveness of contrastive explanations varies considerably. Our qualitative review found that well-developed contrasting explanations included alternative scenarios with counterfactuals to strengthen the case. Conversely, weakly constructed contrastive explanations failed to explore ‘what if’ scenarios to elucidate the disparity between the two situations under comparison.
This study contributes to advancing post-hoc explainability by proposing a traceable questionanswering approach. By integrating LLMs with the decomposition-based explainability technique, we generate natural language explanations in an interactive format. Our approach promotes transparency and interpretability in scene-understanding tasks by fusing LLMs with semantic feature importance. The proposed approach has the potential for application across various domains, ofering comprehensible explanations for decision-making processes, including critical areas such as medical applications. While this is a promising development, there are still areas for further exploration and improvement. One area that requires attention is the overuse of social cues by LLMs. While polite and helpful phrases can enhance user experience, their misuse can become vexing over time. It is important to evaluate such interactions through human subject studies to understand user perceptions of the explanations provided by the LLM. Future research could also explore dialogue management strategies to address issues related to interaction and personalization. On another note, the dynamic nature of the external knowledge source ofers the potential for further expansion with additional observations. This not only ensures that the system remains relevant and up to date but also allows the system to answer diferent types of questions. Additionally, incorporating multimodal elements, such as combining text with visual explanations, can enhance the efectiveness of the responses (e.g., highlighting the specific elements contributing to the prediction). Another consideration is the estimation of feature importance values, which are subject to perturbations and computational processes. It is important to assess and reconcile diferences in feature importance across algorithms to ensure stability and consistency.
This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation. generation for large language models: A survey, arXiv preprint arXiv:2312.10997 (2023). [40] B. Li, S. Liu, W. Xu, W. Qiu, Real-time object detection and semantic segmentation for autonomous driving, in: MIPPR 2017: Automatic Target Recognition and Navigation, volume 10608, SPIE, 2018, pp. 167–174. [41] I. R. Rodrigues, G. Barbosa, A. Oliveira Filho, C. Cani, M. Dantas, D. H. Sadok, J. Kelner, R. S.
Souza, M. V. Marquezini, S. Lins, Modeling and assessing an intelligent system for safety in human-robot collaboration using deep and machine learning techniques, Multimedia Tools and Applications (2021) 1–27. [42] Z. Wang, L. Wei, L. Wang, Y. Gao, W. Chen, D. Shen, Hierarchical vertex regression-based segmentation of head and neck ct images for radiotherapy planning, IEEE Transactions on Image Processing 27 (2017) 923–937. [43] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, A. Torralba, Places: A 10 million image database for scene recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (2017). [44] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schrof, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 801–818. [45] S. Hao, Y. Zhou, Y. Guo, A brief survey on semantic segmentation with deep learning,
Neurocomputing 406 (2020) 302–321. [46] B. Li, Y. Shi, Z. Qi, Z. Chen, A survey on semantic segmentation, in: 2018 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, 2018, pp. 1233–1240.