Counterfactual explanation has become a popular and promising method of explaining black-box AI systems and their decisions in recent years. However, a lack of rigorous psychological research means that little is known about what constitutes a 'good' counterfactual explanation, or how they facilitate user understanding of the underlying system. My doctoral research aims to examine how these sorts of explanations are understood and evaluated by users, identify desirable characteristics of counterfactual explanations, and investigate how current state-of-the-art counterfactual explanation techniques satisfy these criteria. These insights will guide the development of a novel explanation method designed to meet the psychological requirements of users. In order to address these research questions, to date I have conducted three large-scale, well-controlled user studies using materials drawn from an existing case-base. These studies have yielded novel findings about the impact of counterfactual explanation on users objective understanding and subjective judgments of an AI system. Based on these results, we have proposed an extension of a case-based counterfactual method that produces psychologically-valid explanations, which is to our knowledge, the first method designed with this specific criterion in mind.
Explaining opaque AI systems and their decisions using contrastive counterfactual examples
has gained considerable traction in recent years (see [
My doctoral research seeks to address these issues by examining how counterfactual explanations are evaluated by human users using both objective and subjective measures, and identifying psychological desiderata of these sorts of explanations. This is achieved by conducting large-scaled, controlled user studies with materials drawn from an existing case-base. These insights will guide an analysis of existing computational methods to assess how well they meet these psychological criteria, as well as the design of a novel, psychologically-grounded case-based approach to counterfactual explanation.
Counterfactual explanations have received significant attention in recent years as a means of
elucidating decisions made by black box AI systems to users. Over 100 methods have been
proposed to generate counterfactual explanations [
The core objectives of my research are to examine how counterfactual explanations impact users’ understanding and perceptions of an AI system, and identify the optimal characteristics of these explanations, in order to guide the design of a novel, user-centric counterfactual method that produces psychologically-valid explanations. Specifically, I investigate how counterfactual explanations of AI predictions improve users’ objective accuracy in a prediction task, and subjective judgments of explanation satisfaction and trust in the system. In addition, I examine how focusing on certain feature-types appears to increase user accuracy, and hence, help users more readily understand the AI system. These insights will guide both the development of a counterfactual explanation method that meets users’ psychological requirements, as well as shed new light on counterfactual explanation in human cognition. The key research questions I have identified are: • What are the optimal characteristics of a counterfactual explanation from a psychological perspective? • Which counterfactual methods produce the best explanations in terms of computational metrics (e.g., sparsity, proximity, plausibility)? • How can a counterfactual method produce explanations that meet users’ psychological criteria of explanations?
User Studies. In order to investigate the efects of counterfactual explanations on users’
understanding and evaluations of an AI system, we conducted a series of user studies designed
to assess the impact of counterfactual explanation on users’ task accuracy and subjective
judgments. We compare these efects to those of causal explanations and a control condition
(in which participants receive only descriptions of the system’s decisions). Participants in the
studies were presented with materials in the form of case-instances, each consisting of five
features used to predict blood alcohol content: gender (male/female), weight (in kg), amount
of alcohol consumed by the person (in units), duration of drinking period (in minutes), and
stomach-fullness (full/empty). Users were shown the output of a simulated AI system presented
as an application, designed to predict whether someone is over the legal blood alcohol content
limit to drive. Materials were selected from a case-base of instances of normally-distributed
values of the feature-set. In the training phase of the experiments, participants were shown
examples of tabular data for diferent individuals, and asked to make a judgment about whether
each individual was under or over the limit on each screen. After giving their response, feedback
was given on the next page, along with an explanation, the content of which was dependent on
the experimental condition (see Figure 1 for a sample of the material used in the counterfactual
condition). Upon completing the training phase, participants began the testing phase, in which
they were shown more example instances referring to individuals and again asked to judge
if each individual was over or under the legal limit to drive. For each instance, participants
were asked to consider a specific feature in making their prediction; for instance, “Given this
person’s WEIGHT, please make a judgment about their blood alcohol level.” After submitting
their response, no feedback or explanation was given. In addition to measuring task accuracy,
participants were also asked to provide judgments of explanation satisfaction and trust, measured
using the DARPA Explanation Satisfaction and Trust scales [
Towards a Psychologically-valid Counterfactual Method. A key result from the user
studies discussed above was that users were significantly more accurate when making
predictions about categorical features (stomach fullness and gender) than continuous features (units,
weight and drinking duration). This finding is supported by evidence from the counterfactual
reasoning literature that people do not spontaneously change continuous variables when
generating counterfactuals for past events [
To date, I have conducted three large-scale, well-controlled user studies (total N = 474) which
have revealed novel insights into how counterfactual and causal explanations are understood
and perceived by users. While counterfactual explanations are judged as more satisfying and
trustworthy than causal explanations, they appear to be only slightly more efective in improving
objective performance in a prediction task. This disconnect between objective and subjective
measures suggests that it is critical to examine how explanations aid user understanding rather
than merely improve subjective perceptions. Furthermore, users appear to understand the
impact of categorical features on the system’s decision more readily than that of continuous
features, a distinction that current computational methods do not account for. Findings from the
ifrst user study were presented at the Cognitive Aspects of Knowledge Representation workshop
at IJCAI’22 [
Based on the finding that counterfactuals that change categorical features are more readily
understood than those focusing on continuous features, we developed a counterfactual method
that accounts for this feature-type distinction. An analysis of common UCI datasets suggests
that sparse counterfactuals with categorical feature-changes are relatively rare, and so our
method adapts Keane and Smyth’s [
The main focus of my research at present is the design of a second series of psychological experiments examining the role of simplicity (or sparsity) in counterfactual explanation, and how it impacts user understanding and subjective judgments. In tandem, I am working on the implementation and evaluation of popular counterfactual computational methods, in order to identify those methods which are most successful (i.e. have the best average performance) in generating counterfactuals that meet given criteria of an explanation over a set of representative problems. These criteria include conventional metrics (e.g., proximity, sparsity, plausibility) as well as novel properties derived from user testing (such as whether a counterfactual makes continuous or categorical feature-changes). The final phase of my Ph.D. research will involve synthesising the insights from these two strands of work in order to develop a novel, psychologically-grounded method for counterfactual explanation.