Explainable Artificial Intelligence (XAI) is a field usually dedicated to ofering insights into the decisionmaking mechanisms of AI models. Its purpose is to enable users to comprehend the reasoning behind the results provided by these models, going beyond mere outputs. In addition, one of the main goals of XAI is to improve the performance of AI models by exploiting the explanations of their decisionmaking processes. However, a predominant portion of XAI research concentrates on elucidating the functioning of AI systems, with comparatively fewer studies delving into how XAI techniques can be leveraged to enhance the performance of an AI system. This underlines a potential area for further exploration and development in the field of XAI. In this paper we focus on the possibility to enhance the performance of an already trained AI model. To this aim we propose a new scheme of interaction between explanations provided by SHAP XAI method and computations of the responses of a given AI model. This new proposal was tested using the well-known CIFAR-10 dataset and EficientNet-B2 model, showing promising results.
Explainable Artificial Intelligence (XAI) aims to provide an understanding of how AI models
work and reasons beyond the decisions they make, allowing users to understand their results.
This is particularly important as AI becomes more integrated into everyday life and critical
decision-making processes such as healthcare and finance. A large part of the past and current
literature [
For example, by explaining their decision-making processes, XAI techniques can help AI researchers better understand the mechanisms behind AI outputs, allowing them to identify errors in their design and/or implementation.
Thus, our goal is to establish an automated process by which the explanations of the ML system’s behaviour are used to improve the system’s performance.
The core idea is that explanations can facilitate the discernment of pivotal input features
influencing specific outputs. This acquired knowledge can then be applied to fine-tune or refine
the ML system itself. In particular, we focus on the possibility to enhance an already trained ML
system with slight changes to the system itself. In this work, we leverage the SHAP (SHapley
Additive exPlanations, [
Two experimental setups were designed to assess the efectiveness of our approach. The ifrst one aims to investigate whether SHAP provides explanations with informative content significant enough to enhance the model’s performance. In other words, we propose a strategy to experimentally find an upper bound on the capability of the SHAP explanations to improve the system performance in a given problem. In the second experimental setup, we show how the SHAP explanations can be efectively incorporated into the processing of the ML system responses in order to improve them. In this work, we will provide the following contributions: i) We consider a model already trained, aiming to introduce minimal modifications while leveraging its existing feature extraction ability, ii) we propose a simple strategy to experimentally compute the extent to which explanations can improve the already trained model, and iii) we present a straightforward methodology for seamlessly integrating explanations with the output of a pre-trained model, ofering a practical and efective framework for model improvement.
The paper is organized as follows: in Sec. 2 the current literature about XAI used to improve ML models is reported; in Sec. 3 the proposed method is described while in Sec. 4 and 5 experimental assessment and the obtained results are described and reported, respectively. Finally, in Sec. 6 we conclude the paper with final remarks.
The internal mechanisms of modern ML approaches, such as Deep Learning, are typically
opaque, posing challenges for AI scientists attempting to comprehend the underlying processes
governing their behaviors. Consequently, establishing a clear understanding of the relationships
between inputs and outputs can prove to be challenging. As a result, the utilization of eXplainable
AI (XAI) methods span various domains, including but not limited to images [
In [
We assume that a dataset = {(x(), ())}=1 of labeled data is available, where x() is a sample point in a given feature space and () is the corresponding class in a class label space {1, 2, . . . , }.
In this work, we adopt SHAP explanations to improve the performance of a given classifier on an input x(). The model can be formalized as a function : → R . The model output can be viewed as a score function for each possible class, and the predicted class can be computed as ^() = arg max{ (x())}. For each input x() and output ^() by , SHAP provides explanations about all the possible classes. In this work, we will refer to the explanation provided by SHAP on an input x() respect to a generic class ∈ {1, 2, . . . , } as x(),, instead the SHAP explanation respect to the inferred class ^() provided by will be indicated by x() , and finally we will refer to the SHAP explanation respect to the real target class () as *x() .
In our proposal, we consider a neural network model as composed of a feature extractor
followed by a classification layer , i.e. (x()) = (︀ (x()))︀ . We want to exploit both the
knowledge given by explanations x() on the model encoding (x()) together with features
extracted by (x()), to obtain an improved ML model (︀ (x()), (x()))︀ able to provide
improved performance respect to . The overall model is shown in Fig. 1. Summarizing, the
proposed framework is composed of three main components: i) a feature extractor , taken from
a model with a given performance that we want to improve, ii) an explanation Encoder
which extract the relevant information from the provided explanations, and iii) a final classifier
. The input x() and explanations x() are fed to and respectively. The resulting outputs
(x()) and (x() ) are then concatenated together and fed to , which will provide the final
classification. In other words, will replace in the classification step, taking into account
information provided by SHAP explanations. Diferently from similar work such as [
The main goal of this work is to improve the classification performance of a model exploiting SHAP explanations. To this aim, we want first to investigate if SHAP provides explanations containing enough and useful information to improve the classification performance of . Therefore, initially we investigate if and how much SHAP surrogates can help the classification of a given model. Once it has been established that SHAP surrogates are useful to improve classification performance, they can be used for the efective classification system. Therefore, the experimental assessment is composed of two sets of experiments, the former to investigate if SHAP explanations could be used as additional input to enhance efectiveness of the classification model, the latter to adopt them in a real scenario. Since the first experimental scenario is purely exploratory, we assume that the explanations *x() about the real class for each input x() are available, while in the second experimental scenario we adopt the efective explanation *x() provided by the SHAP method, therefore assuming the real case where information about the efective input class may not be available.
CIFAR-10 [35] was used as benchmark dataset. CIFAR-10 is a collection of 60,000 color images grouped into ten categories, that are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. The dataset ofers 50,000 training images and 10,000 test images, all of size 32 × 32.
As base model we employed EficientNet-B2 model [ 36], pre-trained on the ImageNet dataset [37]. The model is then fine-tuned on CIFAR-10 training data, resulting in an accuracy of 97.15% on the test set. Since EficientNet is trained on ImageNet with images having dimensions of 224 × 224, bicubic interpolation was employed on CIFAR-10 to scale images from the original resolution of 32 × 32 to 224 × 224. The best model hyperparameters have been found with a grid-search strategy, where hyperparameters and variation ranges are reported in Tab. 1.
The resulting baseline model was then used to generate SHAP explanations for a given input.
SHAP explanations were generated using the method described in [
As explained in Sec. 3, the proposed framework (Fig. 1) is a double-branch neural network architecture, where the first branch consists in the pre-trained feature extractor fed with an input x(), while the second branch consists in an encoder , whose architecture is shown in Fig. 2. The encoder processes a SHAP value-based explanation with the aim of capturing meaningful information and relations about both the visual content of the input image and the associated explanations. The outputs of and , that are the features extracted by x() and the surrogate explanation respectively, are then concatenated together and fed to the final classifier . In this work, we adopt as classifier a simple shallow fully-connected neural network with a number of neurons equals to the number of classes. Keeping the inner parameters of fixed, parameters of and are learned during the training stage. The best hyperparameters have been found adopting a grid-search strategy with ranges listed in Tab. 1. Two distinct types of experiments were conducted, the former to investigate if SHAP explanations could be used as additional input to enhance efectiveness of the classification model, therefore exploiting *x() about the true classes (), while the latter to adopt them in a real scenario, therefore no knowledge of the true class of the input was considered available. In the second case the following two strategies to build explanations have been adopted: A. Most probable class explanation x() : the explanation x() of the most probable ^() class according to is generated.
B. Weighted average explanation (x): An averaged explanation x () is computed from the set of explanations generated by SHAP on the input x() for the model , with each explanation’s contribution weighted by the classification scores provided by model , as shown in Equation 1. x is the input image, represents the total number of classes, (x()) is the score assigned to class by , and x(), is the SHAP values attributed to the class on the input x().
x() = ∑︁ (x()) · x(), =1 (1)
In this section, the results of the experimental assessments are reported.
In Table 2 performance accuracy on the test set of the CIFAR-10 dataset is shown. Specifically, the baseline model , without additional information provided by SHAP, achieved an accuracy of 97.15%. Instead, using the adopted framework and explanations *x (that are explanation on the correct classes), an higher accuracy of 98.64% was achieved. Diferently, relying on the most probable class explanation x or the weighted average explanations x, accuracy drops to 97.06% and 97.21%, respectively. First results suggest that SHAP explanations on the model performance, in general, can have a significant impact, leading to performance improvements. Despite this, last experiments showed that the proposed approaches adopting efective explanations x lead to performance comparable with the baseline.
Figure 3 provides predictions for some of the images in the test data. It is relevant to note that the use of explanations corresponding to the real classes * leads to correct predictions in several cases, revealing a considerable information value in such explanations. In some cases, the weighted average technique is equally efective, also contributing to correct predictions.
Explainable Artificial Intelligence (XAI) plays a crucial role in unraveling the decision-making processes of AI models, shedding light on their inner workings and enhancing transparency. While the literature has predominantly focused on explaining AI systems, this work takes a step further to investigate how these explanations can be harnessed to improve AI system performance. In an ideal scenario, we envision an automated process where AI system explanations are leveraged to autonomously enhance system performance and understanding. Our assumption is that insights gained from explanations can enhance the adaptability of AI models to a wide range of inputs. In this study, we employ the SHAP (SHapley Additive exPlanations) method to develop a system aimed at improving classifier performance. SHAP, a widely recognized explainability technique, provides insights into the contribution of diferent features in a model’s predictions. Our approach involves using a surrogate of SHAP explanations obtained through an encoder, distilling only the pertinent information from the explanation. This ensures a focus on crucial insights while maintaining the eficiency of our classifier improvement framework. We conduct two sets of experimental setups to assess the efectiveness of our approach. The ifrst set aims to investigate whether SHAP explanations contain suficient and useful information to enhance model performance. This setup primarily evaluates SHAP’s explanatory power in isolation. In the second setup, we actively incorporate these explanations into our framework to demonstrate their real-world impact and efectiveness in enhancing classifier performance. This setup represents a practical application of SHAP insights. The results are encouraging. In the first experimental setup, when utilizing explanations corresponding to the true classes, we achieved the highest accuracy of 98.64%. This highlights the significant impact of explanations on model performance, especially when true class information is available. In real-world scenarios where this information is not available, we have shown that solutions can be found where the model performance is at least comparable to the baseline. This highlights the potential of explanations to improve the performance of AI systems. Further investigations to improve the proposed method will be made and proposed in future works. In conclusion, this work provides valuable insights into the potential of XAI methods, such as SHAP, to not only explain AI systems but also to enhance their functionality and adaptability. As AI continues to play a pivotal role in various domains, understanding and harnessing the power of explanations for improvement becomes increasingly important.
This work is supported by the European Union - FSE-REACT-EU, PON Research and Innovation 2014-2020 DM1062/2021 contract number 18-I-15350-2, and was partially supported by the Ministry of University and Research, PRIN research project "BRIO – BIAS, RISK, OPACITY in AI: design, verification and development of Trustworthy AI.", Project no. 2020SSKZ7R, by the Ministry of Economic Development, “INtegrated Technologies and ENhanced SEnsing for cognition and rehabilitation” (INTENSE) project, and by Centro Nazionale HPC, Big Data e Quantum Computing (PNRR CN1 spoke 9 Digital Society & Smart Cities, CUP: E63C22000980007 ). Furthermore, we acknowledge financial support from the PNRR MUR project PE0000013-FAIR (CUP: E63C22002150007 ). feature selection mechanism, in: 2020 33rd SIBGRAPI conference on Graphics, Patterns and Images (SIBGRAPI), Ieee, 2020, pp. 340–347. [33] K. Roshan, A. Zafar, Utilizing xai technique to improve autoencoder based model for computer network anomaly detection with shapley additive explanation (shap), arXiv preprint arXiv:2112.08442 (2021). [34] L. Antwarg, C. Galed, N. Shimoni, L. Rokach, B. Shapira, Shapley-based feature augmentation, Information Fusion 96 (2023) 92–102. [35] A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images (2009). [36] M. Tan, Q. Le, Eficientnet: Rethinking model scaling for convolutional neural networks, in: International conference on machine learning, PMLR, 2019, pp. 6105–6114. [37] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.