With the increasing demand for understanding the decision-making processes of artificial intelligence applications, explainable AI (XAI) systems have become increasingly important. Counterfactual explanations, a promising approach in XAI, take advantage of human counterfactual reasoning mechanisms to ofer intuitive explanations of how a model's predictions could have been diferent. This Ph.D. project focuses on the design of post-hoc XAI techniques to generate counterfactual explanations that utilize case-based reasoning. It highlights the benefits of post-hoc explanation systems in improving our understanding of black-box models and explores the unique advantages of counterfactual explanations as an instance-based method. Furthermore, this report presents an overview of my doctoral studies and current state. It contributes to the growing body of research on XAI by presenting novel insights into the design of post-hoc XAI systems. Additionally, the report identifies areas in the existing literature that require further investigation and suggests potential directions for future research. Overall, this report ofers valuable insights for researchers and practitioners interested in the design of XAI systems and highlights the importance of transparency and interpretability in artificial intelligence.
The increasing prevalence of artificial intelligence models in various aspects of our daily lives
has created a growing need to understand how these models make decisions. However, the
complexity of these models has made it challenging to comprehend the factors that contribute
to their predictions. For instance, consider two individuals with similar backgrounds applying
for a home loan from a bank that uses a black-box model to assess loan applications. As shown
in Figure 1, one applicant is declined, while the other is approved, leaving the rejected applicant
wondering about the reasons behind their application’s rejection. Counterfactual explanations,
which are a type of post-hoc explanation, can provide highly satisfactory explanations in such
situations [
Counterfactual explanations aim to answer the "what if ?" question by presenting hypothetical
examples that demonstrate how a model’s prediction can be changed with minimal efort. For
example, counterfactual explanations answer the "What if the rejected applicant had earned
$500 more? Would their application have been accepted?" questions. These explanations ofer
valuable insights that can help users understand the predictions made by black-box models,
particularly when the factors influencing the model’s decision are unclear [
Explanation generation in Explainable AI (XAI) applications has diferent purposes and
approaches. Nunes and Jannach’s study [
Another challenge in XAI applications is generating or selecting the best explanation for a
case, which requires essential quality metrics, such as trustworthiness, understandability,
informativeness, suficiency, and unbiasedness. There are various approaches from diferent fields
to meet these quality requirements, including the Case-Based Reasoning (CBR) methodology.
CBR is a problem-solving methodology that has four steps (retrieve, reuse, revise, retain) and
benefits from past experiences with high interpretability [
This report focuses on post-hoc counterfactual XCBR techniques and identifies areas in the existing literature that require further investigation while presenting an overview of the author’s doctoral studies and current state.
This section serves as a foundation for outlining the various points of contribution and aims of my doctoral work. The points that are already published and in progress currently are elaborated upon in detail in the next section (See Section 3).
1. Literature review: A comprehensive literature review to establish the theoretical foundation for the research in XAI and counterfactual explanation generation. 2. Combining global and local explanations: Global explanations help to provide an overall understanding of how a model is working and can identify important features that are driving its predictions. This can be particularly useful for identifying biases or areas where the model could be improved. Local explanations, on the other hand, provide insight into individual predictions and can help to build trust in the model by giving users a clear understanding of how the model arrived at a particular decision. Without local explanations, it can be dificult for users to understand why a particular decision was made and this may lead to mistrust in the model. By combining both global and local explanations, machine learning models can be made more transparent, trustworthy, and ultimately more useful to their intended users. 3. High-quality counterfactuals: Counterfactuals allow hypothetical scenarios to be generated and evaluated, thus promoting model transparency and interpretability. Moreover, counterfactuals can be used to test model robustness and evaluate the efect of input changes on model output. Therefore, the generation of robust, diverse, trustworthy counterfactuals is essential for the XAI systems that provide counterfactual explanations. 4. Flexible XAI system: Flexibility in XAI systems refers to the ability to adapt in many aspects like diferent user needs and preferences, data types, amount of the dataset, and application areas. 5. Domain knowledge integration: Domain knowledge refers to the knowledge and expertise that is specific to a particular application domain and is often held by domain experts such as clinicians, engineers, or financial analysts. Incorporation of expert knowledge from a particular domain into explanation systems is expected to improve their performance, relevance, and interpretability. 6. Explanation representation: Explanation representation is a critical component of XAI systems, as it enables users to understand and trust the decisions made by these models. In instance-based explanations, the explanations can be presented as instances, texts, tables, graphics in diferent forms, or diferent combinations of them. 7. Evaluation: The evaluation of XAI systems can be done in two ways: user evaluations, which measure the efect on users, and quantitative evaluations, which rely on statistical calculations. An ideal XAI system is able to evaluate the generated explanations using both methods and learn from the outcomes.
In the previous section, the research objectives and points that we aimed to contribute are listed and explained. This section presents the methodology for contributing to some of the listed objectives to date.
In the initial stages of the research, a thorough literature review was undertaken, which has been continuously updated and expanded throughout the course of the doctoral studies. The primary contribution lies in the provision of a comprehensive and in-depth analysis of the state-of-the-art approaches in the field. This review encompasses a wide range of perspectives and characteristics, ofering valuable insights into the diverse landscape of research in the domain. The final version of the literature review will be published until completion of the doctoral studies, ensuring its availability to the academic community and further enriching the existing body of knowledge.
The first published work[
In another sub-project, we proposed a novel approach for generating twin XAI systems that
utilize CBR to explain black-box models in multi-class classification tasks. The preliminary
work has been presented at the XCBR challenge during the 2022 International Conference on
Case-Based Reasoning (ICCBR-2022), and details of the preliminary experiments can be found
in the proceedings [
The twin XAI system consists of a multi-agent CBR system (MA-CBR system), where each
agent is developed for a specific class and is modeled separately. The system incorporates
feature attributions and data distribution to project the diferent characteristics of classes
through separately calculated SHAP values for each class over the black-box model. To develop
the local similarity functions, a data-driven similarity measure development method is employed
[
One of the key contributions of this work is the facilitation of expert knowledge incorporation into the XAI system, which enhances the reliability and trustworthiness of the explanations provided. Additionally, the multi-agent structure enables the generation of instance-based explanations that incorporate both local and global features, providing a comprehensive understanding of model outputs. An evaluation metric is also introduced called "rigidity" which measures the adaptability and flexibility of the black-box model’s performance through the proposed explanation system. This metric helps assess the quality and reliability of the system’s explanations. The proposed system was tested on diverse datasets with varying characteristics, diferent performance levels of black box models, and varying degrees of expert knowledge. It was observed that the system exhibited a high degree of flexibility. Furthermore, reproducible benchmarking experiments and open-source implementation of the approach and evaluation metric are provided 1, promoting transparency and further research in the field.
We are currently engaged in ongoing research focused on the development of a perturbationbased counterfactual generation method, PertCF, that leverages feature attributions generated by SHAP values. This approach combines the strengths of perturbation-based counterfactual generation and feature attribution to produce counterfactuals that are of high quality, stable, and interpretable. Unlike conventional approaches that employ predefined distance metrics such as Euclidean distance, PertCF adopts specialized metrics tailored to the specific problem at hand. This specialization of distance metrics ofers two distinct advantages. Firstly, it utilizes SHAP values calculated for each class individually, enabling the projection of diferent class characteristics through feature attribution, in a similar way to the previous work. Secondly, it facilitates the seamless incorporation of domain or expert knowledge, thereby efectively representing the semantics of the data. Preliminary evaluations indicate that PertCF demonstrates measurable advancements over state-of-the-art methods. However, further development and refinement of the results are necessary to enhance its performance and eficacy.
Currently, there are several ongoing tasks and next steps in the research: • Further development and enhancement of the local-global attribution method to improve its efectiveness. Specifically, exploring data-driven techniques that are independent of the model to strengthen its capabilities. • Refinement and improvement of the counterfactual generation method that is being worked on, followed by publication to share the findings with the research community. • Completion of the survey, which aims to provide a comprehensive overview of the approaches developed in recent years. This involves evaluating and comparing these approaches based on various characteristics and presenting them from diferent perspectives. • Expanding the applicability of the research, such as exploring the integration of multimodal data to enhance the capabilities and scope of the proposed methods. • Addressing the challenges related to interactable and customized explanations, seeking solutions that can efectively handle complex scenarios and accommodate specific user requirements.
These tasks collectively aim to advance the understanding and application of explainable artificial intelligence, improving the interpretability and trustworthiness of machine learning models.