When people are subject to high-stakes decisions informed by computer models, they have a legitimate interest in understanding the basis for the model's judgements and whether actions can be taken to turn a dispreferred decision into a preferred one. For example, if an application for a loan is denied by the model, the applicant has an interest in understanding the conditions that would yield an approval. In this paper, we argue that these kinds of counterfactual (or contrastive) explanations rest on domain-specific and commonsensical principles that can be negotiated, and sketch a method for incorporating such principles in an explanatory dialogue system using enthymematic reasoning.
Consider a conversation between a bank clerk and a customer. The clerk says: “Your loan application has been declined.” The customer asks “Why?” Now, how should the clerk respond? There are several options, for example describing the bank’s policy for credit assessment, or highlighting some specific aspect of the customer’s status that contributed to the decision.
When people ask for an explanation for an event, they are often interested in knowing why
some other event did not occur instead [
Most existing methods for generating CEs have been designed for single-shot (i.e.
noninteractive) settings [
Reasoning in dialogue involves non-logical common-sense inferences, in rhetoric referred to
as enthymemes [
Consider for example, two colleagues walking to work, when one of them says “Let’s walk along Walnut Street. It’s shorter.” [10]. At first glance we may not recognise this utterance as involving inference. However, if we don’t assume the topos “if you are going somewhere you want to get there as fast as possible” and the topos “if a route is shorter it is faster”, the utterance hardly seems meaningful, let alone valid. Rules of thumb like these are principles that we live by, and they allow us to communicate eficiently as not everything needs to be made explicit.
However, topoi do not necessarily apply to all situations – on a sunny day we might want to take the scenic route, and the shortest route is not always the fastest. In interaction we can easily reject and question topoi, and a successful answer to a why-question regarding a suggestion should tap into a topos which is acceptable to the addressee in the context.
An enthymeme might serve to persuade or mislead a listener, but the same mechanism can also make it easier for a conversational participant to accept an honest and constructive proposal made by another agent. This is particularly pertinent in the context of XAI, since the opacity of many ML models makes it impossible to communicate the causes for a prediction in a way that is both logically complete and understandable for humans. Section 3 exemplifies how enthymematic explanations of model predictions can point the user’s attention in a constructive direction, while at the same time allowing the user to reject particular topoi.
To manage interactive counterfactual reasoning between a user and a system, we propose a method for selecting a relevant CE in a given situation, as well as strategies for enthymematic reasoning.
We illustrate the collaborative reasoning that our proposed method supports using the following hypothetical interaction between a user (u) and dialogue system (s):1 s.1: u.1: s.2: u.2: s.3: u.3: s.4: u.4:
Your loan application has been declined.
Why? Had you earned more than €2500 and had a more qualified job, it would be accepted. Well, I probably can’t get a more qualified job very easily.
Ok. The application would also be accepted if you earned more than €3000, with your current type of job.
Hmm. That is also out of reach, unfortunately.
I see. There is also age. If you were 5 years younger, we would grant you the loan.
Actually, that’s really useful to know! Perhaps my partner would be eligible.
The example assumes a predictive model with three independent variables: income, age and job status. Initially, the system assumes that reducing age is not feasible, modelled as a topos. Consequently, in s.2, the system presents a CE that does not involve changing age. In u.2, the user implicitly rejects the system’s example by asserting a constraint concerning job status. Subsequently, in s.3, the CE presented by the system is filtered by both age and job status. In u.3, the user conveys that changing income is also not feasible, thus imposing a new constraint. This time, the filtering based on all three variables disqualifies all generated CEs. This triggers a fallback strategy: to make the topos (in this case concerning age) explicit, and to communicate a CE conditioned on inhibiting the topos (s.4). In this example, suppressing the age-related topos happens to be relevant to the user.
The technique for selecting the most relevant CE rests on the following assumptions: 1) a predictive model which maps situations described by an input vector to a target label and 2) a generator that samples statistically relevant and similar counterexamples, probabilistically. For example, given a loan application represented by a vector describing the applicant’s income, age and job status, returns a c c e p t or d e c l i n e ; if returns d e c l i n e then we should generate CEs for = a c c e p t .
By “statistically relevant” we mean that CEs should reflect typical characteristics in the data that was trained on, e.g. that certain combinations of income and job states are more likely than others. An example of such a method is described by Dombrowski et al. [11].
By “similar” we mean that CEs should involve a small number of feature modifications. For example, a CE that involves modifying only income is preferred over one that involves modifying both income and job status. This desideratum is particularly important for datasets with a large number of variables, since it favours CEs that are easier to communicate and comprehend [12].
On this basis, we propose to filter the CEs generated by by (a) topoi conveying domainspecific and/or commonsensical principles, and (b) constraints gathered from the interaction with the user. For example, the topoi may convey that generally, becoming younger or earning 5 times more are unachievable. Constraints are user-specific information harvested from earlier interaction, or other data such as a previously filled in form. In the above example, viewing an income of more than €3000 as not feasible is a constraint arising from u.3.
1Partly adopted from [
Below we present a tentative list of dialogue strategies that can support the desired behaviour. These can be implemented as update rules in an information-state update approach [13, 14]. Counterfactual question-answering To address a user’s why-question about a model-based decision, the system can reply with a CE.
an enthymeme, as illustrated by all system utterances in the dialogue example above. Constraint integration A user’s stated feasibility conditions are integrated into the information state as a constraint. A statement can be explicit as in “I can’t earn more than X”, or implicit/deictic as in “That is out of reach”, referring to a previous system utterance. Topos elicitation When a CE cannot be given with the current set of topoi, and disregarding a topos would enable a CE, then make the topos explicit and present the CE that it enables.
This strategy underlies s.4.
As described in section 3, our sketch for managing enthymematic counterfactuals assumes a generator of counterfactual candidates with two desiderata: statistical relevance (with respect to the domain as such) and similarity (with respect to the specific situation at hand). Optimizing these desiderata jointly is not trivial, since they sometimes contradict each other. For example, assuming a loan applicant with a relatively unqualified job and an income of €1500, earning 10 times more would be a small change (in the sense that only one feature needs to be modified), but statistically irrelevant (since such a high income is unusual for unqualified jobs). Since this paper primarily focuses on the dialogical reasoning process, we assume a generator with the desired properties without proposing a specific generator. Nevertheless, in order to implement and evaluate our approach in the future, a concrete generator needs to be chosen (either by selecting a previous method from the literature or by proposing a novel one).
While similarity is an important aspect of CEs [12], dialogue may ofer possibilities to increase the acceptable number of feature changes. For example, a CE that involves 5 feature changes may be communicated using a stepwise information presentation strategy such as “First of all, you would need to earn more than €2500” [15]. It would be informative to study the conversational strategies that humans use in these cases.2
More broadly, the challenges associated with explaining predictions made by opaque models
– such as deep neural networks – can be taken as a reason to simply not use such models
for high-stakes decisions [17]. Nevertheless, to the extent that opaque models remain in use,
explanation methods are needed. Furthermore, explaining simpler and more transparent models,
such as decision trees or logistic regression, is not trivial either, particularly when targeting
users who are not ML experts [18]. In fact, popular XAI techniques such as LIME [19] and
SHAP [20] explain an opaque model using a simpler model which approximates the behaviour
2For a discussion about informing the design of XAI by collecting and analysing human dialogues, see [
In this paper, we have outlined a model-agnostic approach to explaining predictions using enthymematic counterfactuals. One of the main benefits is that it can direct the user’s attention towards counterexamples that are actionable, while at the same time allowing the principles (topoi) that underpin the reasoning to be questioned and rejected in a dialogue. In future work, we plan to formalise the approach and implement and evaluate a prototype. Another interesting avenue of research concerns the question of how to acquire the topoi, which can be encoded by the system developers, but may be harvestable from interactions [21]. More generally, we believe that enthymemes ofer a useful avenue for further research in XAI and hope that the approach outlined here can stimulate further work in this direction.
This work was supported by the Swedish Research Council (VR) grant 2014-39 for the establishment of the Centre for Linguistic Theory and Studies in Probability (CLASP) at the University of Gothenburg. [10] M. A. Walker, The efect of resource limits and task complexity on collaborative planning in dialogue, Artificial Intelligence 85 (1996) 181–243. [11] A.-K. Dombrowski, J. E. Gerken, K.-R. Müller, P. Kessel, Difeomorphic counterfactuals with generative models, 2022. URL: https://arxiv.org/abs/2206.05075. [12] S. Wachter, B. Mittelstadt, C. Russell, Counterfactual explanations without opening the black box: Automated decisions and the GDPR, Harv. JL & Tech. 31 (2017) 841. [13] S. Larsson, Issue-based dialogue management, Ph.D. thesis, 2002. [14] J. Ginzburg, The interactive stance: Meaning for conversation, Oxford University Press, 2012. [15] V. Demberg, J. D. Moore, Information presentation in spoken dialogue systems, in: 11th
Conference of the EACL, 2006, pp. 65–72. [16] A. Berman, C. Howes, “Apparently acousticness is positively correlated with neuroticism”.
Conversational explanations of model predictions, in: Proceedings of SemDial, 2022. [17] C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence 1 (2019) 206–215. [18] J. Dieber, S. Kirrane, Why model why? Assessing the strengths and limitations of LIME, 2020. URL: https://arxiv.org/abs/2012.00093. doi:1 0 . 4 8 5 5 0 / A R X I V . 2 0 1 2 . 0 0 0 9 3 . [19] M. T. Ribeiro, S. Singh, C. Guestrin, “Why should I trust you?”: Explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, ACM, 2016, p. 1135–1144. [20] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, Advances in neural information processing systems 30 (2017). [21] V. Maraev, E. Breitholtz, C. Howes, J.-P. Bernardy, Why should I turn left? Towards active explainability for spoken dialogue systems., in: Proceedings of the Reasoning and Interaction Conference (ReInAct 2021), ACL, Gothenburg, Sweden, 2021, pp. 58–64.