For humans, explanations are conversations, while explanations of models are one-directional result presentations. To make model explanations conversational, previous work presented a framework for conversational XAI agents. However, this early research prototype heavily relied on template-based natural language responses and a Jupyter Notebook user interface, limiting its application to a lab environment. In this work, we present XAgent, a conversational XAI agent addressing these limitations. We designed XAgent as Web application with a chat interface, harnessing the power of the Llama-2 language model to leverage the template-based design to natural conversations. The inclusion of LLAMA2 further extends the scope of the agent to also answer questions beyond XAI, such as for example factual questions or questions about terminology and supports in prompting the user for additional information in case of incomplete requests. With these improvements, we lift the previous research prototype to a deployment-ready conversational XAI agent, bringing us closer to our goal of creating a real-world XAI agent that explains machine learning models in a conversational style. The source code for XAgent is available at https://github.com/aix-group/XAGENT/.
albeit Conversational XAI has gained increasing attention from the research community
recently [
in a Jupyter Notebook environment, thereby limiting the installation to a lab environment. To address these limitations, we introduce XAgent, a conversational agent using Large Language
framework [
and (2) enable the system to answer questions about factual knowledge, such as the meaning of specific terminologies. Additionally, we (3) demonstrate a real-world system with a graphical user interface (GUI) and hosting capabilities, facilitating the deployment and accessibility of
learning models in a conversational manner.
User Question XAI question phrase bank Reference Question Retrieval NLU
QPG Reference Question Method Selection QX Answer XAI Method Libary XAI Method Output Answer
Generation NLG
model on questions sourced from the XAI question bank [
user’s actual intent from a wide range of XAI utterances. The NLU model is trained on the question phrase bank from QPG. We extend this component by adding a feature that suggests alternative questions to choose in case NLU fails to identify the intent.
method (XAI method output). In this work, we incorporate factual questions requiring external knowledge and employ a Large Language Model (LLM) to provide explanations.
(a) Introduction (b) Information Collection
(e.g., a table, graph, or number) to a natural language answer. We integrate LLMs in the final step of the textual answer generation in the NLG component to improve the answer’s text quality from a pure template-driven answer to natural language.
consists of two phases: the Input Phase and the Explanation Phase. The Input Phase refers
to the process of data collection from the user. Once all relevant data has been successfully
gathered, the system transitions to the Explanation Phase, wherein the agent explains the model
and its prediction to the user.
3.1. Input Phase
Before users can inquire about the model and request explanations for the predictions, they
must first provide the necessary information about the instance to be predicted. We refer to
this as the Input Phase. In this phase, we introduce the agent to the user with an example
instance, but also provide the option for the user to change the instance by using the command
/change instance. The introduction is illustrated in Figure 2a. For this demonstration, we
use the German-credit dataset2 as a fixed dataset for interaction. However, technically, we
can change the dataset by typing /change dataset. We also provide a pre-trained Random
Forest classifier with the accuracy of 85% on this dataset. If the user chooses to provide their
own instance, we implement a series of questions to gather the required information. Each
question is accompanied by a constraint to ensure that the user provides valid input, such as
selecting from predefined categories for categorical features or entering a numeric value for
numerical features. If the user provides unexpected input, they are prompted to repeat the
entry until it meets the specified requirements. Once all the necessary information has been
collected, the system uses a pre-trained classifier to generate the prediction results, which are
then presented to the user. All of the information of the instance and the prediction for it is
stored in the memory. Importantly, the questions and answers provided by the agent during
this input phase are enhanced using a LLAMA-2 [
predefined templates to the NLG component. The templated answers are then returned to the
Natural Language Question NLU Improved Answer LLM Templated Answer Selected Question NLG Method Output Dialog Manager Information
Question Method Output Instance Information
QX
(a) Explanation Process (b) Questions Suggestion 3.2.1. Natural Language Understanding (NLU)
phrase bank within the QPG component to train the NLU by framing the task of matching user
questions to reference questions as multi-class classification. Sentence embeddings are generated
for user and reference questions using SimCSE [
output identifies the reference question that best fits the user’s intent. If the probability falls below a threshold, , the question is considered an unknown variation or paraphrase.
In such scenarios, our preceding system declines to provide an answer or a suggestion to the question and the conversation halts. In this work, we enhance the NLU components by presenting the user with a curated list of questions for selection. The order of questions in this list is based on their cosine similarity values within the question bank. Initially, the list presents the top 5 questions with the highest similarity, with subsequent options provided if the desired question is not among the initial selection. Moreover, users are provided with the opportunity to rephrase their questions. An example of this question selection case is depicted in Figure 4b. 3.2.2. Dialog Manager The Dialog Manager (DM) component serves as the central processing unit, orchestrating information processing and decision-making within the system. Specifically, within the DM, we define four states corresponding to four situations that the agent might encounter: the Input state, Explanation state, Suggestion state, and Missing Data state. The Input and Explanation states correspond to the Input and Explanation phases, while the Suggestion and Missing Data states represent two situations that may arise during the Explanation Phase. The dialog begins with the Input state. Once all information has been collected, the DM transitions the state to
to comprehend the user’s question. In cases of missing data, the DM sets the state to Missing
user asks How does this feature impact its prediction? without explicitly specifying the feature, the DM will ask the user to provide the name of the feature. For situations where the question is unclear, the DM sets the current state to the Suggestion stage, in which the NLU returns a list of suggested questions.
3.2.3. Question-XAI (QX)
For the QX component, we employ the mapping established in the original work [
Question: {Question}. Explain it in 100 tokens.
as there exists a general question, What does [ML terminology] mean? If the NLU matches the user’s question to this query, the QX will activate LLAMA-2 to respond to the user’s inquiry.
3.2.4. Natural Language Generation (NLG)
are defined within a JSON file. To accommodate a new dataset, a corresponding new template is required. This template encapsulates information about the features, as well as the responses for each question and case. The final response, derived from the template, is subsequently processed through the LLAMA-2 model using the specific prompt below: Improve the language of the text but keep the original intent of the below text. Do not add or omit any information, only adapt the language. Please keep the information in square brackets unchanged. Only return the text without the double quotes. Do not ask for anything else. Do not add any comments. {Text}.
Improved text: 3.3. User Interface
we implement session handling. Specifically, we initialize all components only once except for the Dialog Manager. For each user session, a new agent object is created, equipped with its own Dialog Manager initialized with default states. This approach ensures that each user interacts with their respective agent object, leveraging the shared NLU, LLAMA-2, QX, and
Figure 2 and Figure 4 show screenshots of a conversation from the web application of XAgent.
user-agent interaction. Additionally, LLAMA-2 generates the answer to the question what is the definition of ’accuracy’? . Generally, LLAMA-2 demonstrates benefits in terms of language improvement and answering factual knowledge questions. However, it also has its own shortcomings. For example, in Figure 2b, LLAMA-2 generates “Give me your Sex [male, female].
after the request, not before. Additionally, in the same figure, we observe inconsistencies in the presentation of the list of values, as they sometimes appear in square brackets and sometimes in parentheses.
the Streamlit framework, we lift the previous strongly constrained research prototype to a deployment-ready conversational XAI agent. Future work will involve tuning the prompts for
a promising direction for future exploration involves extending the application to other data types, particularly visual data, and utilizing multimodal architectures like Vision-Language models for explanation.