The usage of eXplainable Artificial Intelligence (XAI) methods has become essential in practical applications, given the increasing deployment of Artificial Intelligence (AI) models and the legislative requirements put forward in the latest years. A fundamental but often underestimated aspect of the explanations is their robustness, a key property that should be satisfied in order to trust the explanations. In this study, we provide some preliminary insights on evaluating the reliability of explanations in the specific case of unbalanced datasets, which are very frequent in high-risk use-cases, but at the same time considerably challenging for both AI models and XAI methods. We propose a simple evaluation focused on the minority class (i.e. the less frequent one) that leverages on-manifold generation of neighbours, explanation aggregation and a metric to test explanation consistency. We present a use-case based on a tabular dataset with numerical features focusing on the occurrence of frost events.
Nowadays, Artificial Intelligence (AI) and Machine Learning (ML) models have become essential tools for practitioners tackling real and complex problems. These include high-risk applications such as in healthcare, climate, and fraud detection, which require highly reliable models due to the serious consequences that incorrect or biased predictions may have. However, many real-world datasets in these domains are intrinsically unbalanced as the critical events, e.g. rare diseases, natural catastrophes, or fraudulent transactions, occur less frequently than the normal conditions. Dataset unbalance introduces significant challenges for the AI and ML models, often leading to biased model predictions. In this context, gaining insights on what lies behind a model’s prediction is particularly valuable for enhancing its trustworthiness. Despite this, ML models are usually very complex and are often treated by practitioners as black boxes.
Explainable Artificial Intelligence (XAI) methods aim at improving the transparency of ML models and ofer a set of tools that can be used to
open the black box either via local or global explanations. In
practical applications, while eforts are made towards the development of highly accurate ML models,
the explainability aspect may at times be overlooked. This is in part an undesirable consequence
of the new legislative requirements that have been proposed for high-risk applications, both in the
GDPR [
CEUR
ceur-ws.org to high-risk scenarios may reflect into unintended, yet harmful, consequences. In this respect, the robustness of the explanations is often left unconsidered, whereas it is a decisive aspect to increase model trustworthiness and reliability of the provided explanation. The robustness of explanations (also referred to as stability) can be broadly defined as the ability of an explanation method (or explainer) to produce similar explanations for similar inputs.
When evaluating unbalanced datasets, assessing the reliability of explanations becomes even more important. To better understand the problem, we can consider a simple example. Assume that a model (⋅) reaches an accuracy of 99% on the training dataset for a real-world problem. If the dataset is balanced or with a limited factor of unbalance, we can conclude that the model is behaving appropriately. If instead we assume to be working with a highly unbalanced dataset, e.g. ( = 0) = 0.99 and ( = 1) = 0.01 , we cannot come to the same conclusion. In fact, a 99% accuracy could be easily obtained with a model that always predicts the class 0, which is obviously not the expected behaviour. This example sparks an important consideration on explanation reliability: if the majority class, that is the most frequently occurring one, is easily predicted by the model, the derived explanations cannot be deemed trustworthy. Moreover, considering that most of the times the training of ML models with unbalanced data is specifically adjusted to concentrate on the minority class, we cannot take for granted that the model has correctly learned the specific facets of the majority class, even if the accuracy is high.
In this study, we address what has been discussed so far by presenting a preliminary evaluation of
if and how explanations can be trusted in the context of unbalanced datasets. For this purpose, we
take inspiration from the framework recently introduced in [
An overview of the proposed method is illustrated in Figure 1. Let = ( X, y) be a dataset with points and features where y is the vector of class labels. Let = 0 indicate the majority class and = 1 the minority class, such that ( = 0) ≫ ( = 1) . Let be divided into training, validation and test sets and let (⋅) be a neural network trained on the training set. Let be an explanation method and ( x) the explanation associated to the data point x and model . If is a feature importance method, the explanation will be also referred to as feature attribution.
Let x be a data point of interest and let x̃ be a perturbation such that ( x, x̃) < with > 0 with a distance metric, e.g. the Euclidean distance. A local neighbourhood of point x is defined as the set of perturbed data points that are suficiently close to the original data point and for which a model predicts the same class as the original point, .̂
In the context of XAI, a local explanation aims at understanding the reasoning behind a model’s prediction for a specific input vector, say x. A key property of local explanations is their robustness, as introduced in Section 1. By definition, locality plays a central role in the evaluation of local explanations but it can lead to misleading results when the data manifold is not properly taken into account. A neighbourhood should not only be made of datapoints which are close to the original one, but also be faithful to the observed data distribution: only in this scenario the evaluation can be trustworthy.
Leveraging the manifold hypothesis, we are able to construct neighbourhoods which are on-manifold
as in [
with ←̄ ( ⋅ 100, (1 − ) ⋅ 100) ℛ̂(x) = 1
∑ (( x), ( x̃)) | | x̃∈ ( ̂ x) = (( x), (̄ x))
The second score ( ̂ x), referred to as consistency, computes the rank correlation between the original explanation and the locally weighted averaged one (Equation 2).
Given a neighbourhood of size , costructed via Equation 1, we apply an explanation method to both the original data point and its perturbations, retrieving the local explanations ( x) and ( x̃), with x̃ ∈ .
For points belonging to the minority class = 1 , we can compute a local weighted attribution in order to retain local information. In particular, the averaged explanation is: ē(x) = ∑x̃∈ e(x̃) ⋅ ( x, x̃) ∑x̃∈ ( x, x̃) where ( x, x̃) = ( 1 x, x̃)
Using the aggregated explanation allows us to take into account the results of a data augmentation on the minority class, which is ensured to be faithful as it lays on-manifold. Explanations are weighted according to the distance between the original data point and the perturbed one: same-class perturbations which are more distant will be given smaller weight in the aggregated explanation.
A neighbourhood created following this perturbation scheme should be at least of size = 100 . A ifnal filtering step is then performed to discard all perturbations that change the predicted class label.
We assess the reliability of the minority class explanations by verifying if the locally weighted attribution is robust. In particular, we compare two locally-computed scores on .
The first score ℛ̂(x) - local robustness [
The consistency score allows us to investigate if the aggregated explanation is locally faithful to the original one, ensuring that it summarizes useful information on the model decision making.
To get some preliminary insight on if and how explanations on unbalanced datasets can be trusted, we apply the described methodology (Figure 1) to a real-world use-case. The case study was chosen to reflect what might be the needs of a practitioner, who finds himself working with an unbalanced dataset and having to apply XAI techniques, and thus wanting to determine whether the explanations received can be trusted or not. The code implementation is available on Github.1
We selected the problem of identifying the occurrence of frost events, using atmospheric data as
predictors. Frost is critical for agriculture, but rare and dificult to predict. A model that quantifies the
occurrence of frost can be the first step for practitioners to estimate agricultural business policies in
regions where data are limited. We used the the publicly available ERA5 reanalysis data [
The model’s architecture is a simple fully connected neural network, which takes the input features
and processes them through five consecutive linear layers. The first four layers are followed by a ReLU
activation function, while the final output layer applies a sigmoid activation to produce a probability
score for classification. Data were split in three subsets: train ( 75%), validation (15%) and test (10%).
The split was performed using a stratified approach based on municipalities, ensuring that all days of
a municipality are assigned to the same set, either training or validation/test. We trained the model
using the focal loss (FL), first introduced by [
() = −(1 − ) log() (5)
We trained the model for 100 epochs with a batch size of 256, using the RAdam [
As model-agnostic methods such as LIME and SHAP have been proved to exhibit poor robustness,
as shown in [
We evaluated the data points belonging to the test set. We firstly investigated the impact of the neighbourhood generation on both the majority and minority class. In particular, we considered both a random neighbourhood generation, in which Gaussian noise ∼ (0, 2) is added the numerical variables, and the medoid-based approach presented in Subsection 2.1. By comparing the robustness of Integrated Gradients (Figure 2) we show that there is indeed a diference between the two neighbourhood generating approaches and that the minority class is the most afected one. In fact, the randomgenerating scheme is associated with a larger distribution shift in the minority class than in the majority class. The negative efect of using a random neighbourhood is also evident if we consider the other explainability methods analysed. These results support our hypothesis that, in highly unbalanced datasets, the minority class is particularly vulnerable and requires careful analysis.
We then focused on the minority class data points ( = 1 ) and we built a neighbourhood of size = 100 for each point, following the medoid-based generation scheme with hyperparameters = 5 and = 0.05 . The hyperparameters were set through a random search to ensure that, on the validation set, at least 95% of the generated data points were classified as their corresponding original ones.
Figure 3 shows that the explanations may be more or less locally robust, depending on the considered method. We observe that the less stable method in this case-study is LRP, as confirmed by the lowest overall mean value and large standard deviation presented in Table 2. This technique at times sufers from the vanishing gradient problem, returning zero-vector attributions as a result. To limit the efect of uninformative explanations, we selected the epsilon propagation rule, as it was the one minimizing the vanishing gradient problem on this dataset. The choice of the propagation rule can significantly impact the results and should be carefully selected as a dataset-specific feature. When comparing the ensemble method robustness to the other two explainers, we need to take into account that its robustness is also influenced by LRP.
We showed that the individual explanations lie in a robust area of the data-manifold, as proven by the robustness scores derived on the test set for the four explainers (the larger the score, the more robust the area). However, this is not enough as we also need to ensure that the explanations are meaningful.
Our aim is to compute the locally weighted attribution ̄ via Equation 2 and to test if it is consistent with the original point’s explanation. This is done by computing the consistency score ( ̂ x) via Equation 4. Since we showed that the data points lie in a robust area of the data space, we can leverage on-manifold information to enrich the set of data points predicted to be in class = 1 and produce an individual explanation which takes into account also the neighbouring points. Increasing the amount of information considered when providing an individual explanation for a point of the minority class helps in understanding the decision making process of the model.
Figure 4 presents the consistency scores of the four explainers on the test set: we observe that the two methods which have higher robustness scores ℛ̂(x), namely Integrated Gradients and DeepLIFT, are associated to larger consistency scores ( ̂ x). This was expected as the local explanations are more similar to one another and the averaged explanation remains highly correlated with the original one. In contrast, it is interesting to note that both LRP and the ensemble aggregation, despite having a lower robustness score due to the influence of the LRP method, are still able to produce satisfyingly consistent aggregated explanations within the generated neighbourhood.
Table 2 presents the mean robustness and consistency scores with the corresponding standard deviation reported within brackets. It can be seen that DeepLIFT is the most robust and most consistent method in the considered use-case, being associated with both the highest mean value and the lowest standard deviation. According to the practitioner’s needs, the ensemble could also be deemed to be a valid candidate as it jointly considers the efects of multiple explanations, addressing possible inconsistencies by construction, and proposing satisfying robustness and consistency scores.
The consistency scores show that retaining local information from carefully crafted on-manifold neighbourhoods can be beneficial for the minority class explanations. In particular, the locally averaged explanation maintains more local information than the individual explanation on a given data point and it has been proved to encapsulate explanations of a robust area of the data manifold.
Despite being in its early stages of development, we argue that this line of research could bring to light useful insights in dealing with unbalanced datasets in practical use-cases. We believe that the analysis could be further enriched by defining a metric to quantify the quality of an explanation in such complex use-cases, considering both the robustness and the consistency as defined in Section 2. Moreover, it would be interesting to test how an uncertainty analysis on the minority class could be beneficial for the correct evaluation of these datasets from both a technical and a practitioner point of view.
As next steps, we plan to investigate how the results change as the performance of the neural network model varies and, at the same time, investigate the efects of a varying unbalancing ratio between the two classes. The analysis will be extended to diferent datasets, both public and private, in order to maintain the real-world aspect of interest to practitioners, but at the same time favour the reproducibility of the results. In future research stages, we also plan to investigate diferent machine learning models for improving performance with unbalanced dataset (e.g. by explicitly taking into account the spatio-temporal dimensions that often characterise many datasets) and adapt the choice of XAI methodologies and reliability reasoning accordingly.
We wish to thank Assicurazioni Generali Spa for their support and interest in our work. The authors have not employed any Generative AI tools.