The use of Artificial Intelligence (AI) models in real-world and high-risk applications has intensified the discussion about their trustworthiness and ethical usage, from both a technical and a legislative perspective. The field of eXplainable Artificial Intelligence (XAI) addresses this challenge by proposing explanations that bring to light the decision-making processes of complex black-box models. Despite being an essential property, the robustness of explanations is often an overlooked aspect during development: only robust explanation methods can increase the trust in the system as a whole. This paper investigates the role of robustness through the usage of a feature importance aggregation derived from multiple models (-nearest neighbours, random forest and neural networks). Preliminary results showcase the potential in increasing the trustworthiness of the application, while leveraging multiple model's predictive power.
The role of Machine Learning (ML) and Artificial Intelligence (AI) has become ever so prominent in the
latest years. The widespread use of these tools, especially in high-risk applications [
In light of these considerations, we aim at tackling two research questions on explanation
trustworthiness. The first research direction posits that the disagreement problem may be mitigated by considering
an aggregation of multiple explanations. The second research direction instead aims at increasing
system’s trustworthiness by computing a local robustness score to determine whether an explanation
can be trusted. Previous attempts at answering these questions were made in [
Taking inspiration from these results, we propose a natural generalization of the approach to multiple
classes of models, aiming at answering both research questions. In particular, we investigated if an
ensemble approach could be used even in cases in which multiple decision making processes are being
considered at once, and tested the trustworthiness of the derived explanations. We have focused on
-nearest neighbours (-NNs), random forests (RFs) and NNs. The choice of these three models was
governed by the diferences in which the models make a prediction, having -NN a distance-base
method and RF an ensemble of rule-based base learners, significantly difering with respect to NNs in
terms of complexity, accuracy and inherent interpretability. As in [
The remainder of this paper is structured as follows: Section 2 will present the proposed methodology, Section 3 the preliminary results on tabular datasets and binary classification tasks and Section 4 future research directions.
Despite being considered an interpretable method, -NNs still lack a proper feature importance approach to explain their decisional process. In particular, while it is easy to understand the distance-based reasoning when predicting a new datapoint’s class, it is not as simple to identify which features were the most influential for the given prediction.
Leveraging its inner mechanisms, we have developed a new feature importance approach to locally explain the decisional process of -NNs. Having selected a value for the hyperparameter , a new datapoint is classified based on the most frequent class which is encountered among its nearest neighbours within the training set. The diference between the two possible classes can mainly be attributed to the features that exhibit larger distances between the two sets of corresponding points. This reasoning can be further extended when considering a point for which all the -nn are associated to the same class. Locally, there are no distinctions to be emphasised, but the classification can be imputed to the fact that the datapoint is closer to those of the predicted class rather than the opposing one. This concept aligns with the reasoning of -NN on a greater scale: datapoints are classified based on the most similar class, and the similarity is defined with distance-based metrics.
Our proposal makes use of this characteristics and produces a feature-importance explanation as described in Algorithm 1. Consider two sets of datapoints, namely and ¬, which are comprised of the nearest neighbours selected, respectively, between the training set’s datapoints belonging to class and those of the other class ¬. The average feature-wise distance within each set of points and the datapoint of interest x is computed and stored as and ¬. Assuming x is predicted to belong to class , the explanation is proposed as the feature-wise diference = ¬ − . To ensure comparability, the resulting vector is normalized to have norm one.
Algorithm 1 -NN explanation: x is the input point and is the -nearest neighbours model function average_distance(x, ) ← 1/| | · ∑︀x′∈ (x, x′)
return end function ← ¬ ← ← ¬ ← ← ¬ − ← /||||2
return end function function Explain_knn(x, ) = (x) k-nn(, x|)
k-nn(, x|¬) average_distance(x, ) average_distance(x, ¬) ◁ Class predicted by the -nn model ◁ nearest neighbours belonging to class ◁ Average feature-wise distance ◁ Normalize to have norm 1
Random Forest explanations are usually proposed in the form of single decision trees or sets of rules, to align with the binary decisions that occur at each node split. While explanations of this form can be useful, it is hard to compare them to feature attributions derived from other classes of models. For this reason, we have developed a new feature attribution method for random forests (presented in Algorithm 2), based on the computed node impurity at each node of the traversed paths in the forest. Algorithm 2 RF explanation: x is the input point and is the random forest model ← [] ← end for
return end function function sum_node_impurity(path, ) for node ∈ path do node.feature
add(node.impurity) ← else function Explain_rf(x, ) , ¬ ← (x) ← arg max( (x)) , ¬ ← array(0, dim = ) for tree ∈ do path ← decision_path(tree, x) if tree.predict(x) = then
sum_node_impurity(path, ) ¬ ← end if end for ← (¬ + ) × − × ¬ ← /||||2
return end function sum_node_impurity(path, ¬) ◁ Predicted class probabilities
◁ Predicted class ◁ Follow the traversed path in the tree ◁ Normalize to have norm 1
Consider a datapoint x which is passed through a random forest composed of trees. Assume that the random forest predicts point x belonging to class , having predicted probabilities [, ¬]. For each of the trees in the forest, the datapoint’s prediction is made by traversing the tree following a single decision path. Depending on whether the individual tree’s prediction aligns with the random forest majority vote, the decision path includes features that contribute to the RF predicted class or to the opposite one. By construction, each node in a decision tree is selected as the one with minimal node impurity, practically measured by the Gini index or the cross entropy. We propose to use the node impurities as feature importances, with due adjustments, considering their relevance in the training phase of the model itself. Consider two feature attribution vectors, namely and ¬, of length equal to the number of features in the input vector x. For each tree in the forest, we retrieve the decision path followed by point x and check if the prediction of the individual tree is the same as that of the random forest or not. For a tree which agrees with the RF prediction, we retrieve the feature importance vector and, for all the features used at the split nodes of the decision path, sum the node impurity to the corresponding element of . The vector ¬ is otherwise used to store the node impurities in a similar manner. A final calibration is performed to properly take into account the efect of positively and negatively influencing features. In particular, the feature importance coeficients defined above are weighted by the opposite class probability scores, returning a vector of the form = (¬ + ) · − · ¬. The parameter = 0.01 yields non-null coeficients in the case in which ¬ = 0.00. As in the -NN case, a final normalization step is performed to ensure comparability.
As mentioned in the Introduction, we propose an aggregation of explanations derived from diferent
models. While new approaches have been proposed in the previous subparagraphs, for NNs we will
be using DeepLIFT [
Assume that the feature importance vectors for -NN (Algorithm 1), RF (Algorithm 2) and NN
(DeepLIFT [
=1
Among the advantages of using the average as an aggregation is its ability to penalize strong sign discordances between the individual methods, while still producing attributions with both positive and negative signs. It also takes into account instances in which one of the methods is uncertain about the importance of a feature , that is, when it is associated to a coeficient |()| < , ( > 0).
Robustness estimation is a critical aspect of this research and, while multiple works have introduced
robustness metrics for explanations, we selected the estimator proposed in [
1 ∑︁ ((x), (x˜)) where = {x˜|x˜ = x + with ∈ R, (x, x˜) < ( > 0) and (x) = (x˜)}, is the number of features and is the Spearman’s rho rank correlation coeficient.
In literature, there is a clear distinction between perturbation-based robustness and adversarial
robustness. The first one refers to the ability of an explainer to produce similar explanations for similar
inputs under the expected data distribution. The latter one instead refers to malicious attacks to either
the model or the explanation, and encompasses the ability of an explainer to produce similar outputs
under such attacks. Our research focuses on non-adversarial perturbations and, as shown in [
This scheme yields perturbations which are on-manifold and therefore more truthful to the observed data distribution on which the model was trained. A final step is performed to remove the perturbations that change predicted class label with respect to the original point, ensuring comparability explanationwise.
As we are considering three diferent models being used at the same time, the requirement that the predictions should be the same for both the original datapoint and its perturbation has to be satisfied for all three models concurrently. More specifically, (x) = (x˜) ∀ ∈ (1, . . . , ).
Dataset and Model Details We have tested our proposal on five publicly-available tabular datasets addressing binary classification problems 1. Some preprocessing steps were performed on the data, such as the standardization of numerical variables, the one-hot encoding of categorical ones as well as the removal of highly correlated features. The dataset were split into a training, a validation and a test sets. For all the use-cases, we have trained a -NN classifier, a random forest and a neural network. The -NN models were trained with a number of neighbours set at = 15 for the adult and bank dataset and = 5 for the other cases. The random forest were built with = 25 base learners for all cases. All models exhibit an adequate performance as the accuracy scores are above the 80% mark, as summarized in Table 1, with the exception of the -NN model on the heloc dataset. As can be expected, the accuracy of the -NN models is often the lowest while the neural networks showcase larger values in the most complex datasets (adult, bank, heloc) and the random forest in the two simpler ones (cancer, wine).
neighbourhood was generated with hyperparameters selected such that, for the three models, at least 95% of the generated datapoints were kept on average. Default choices for the hyperparameters are = { = 5, = 0.05, = 0.05}. For the individual models the filtering is based on the constraint (x) = (x˜) while for the aggregation it holds that (x) = (x˜) ∀ ∈ (1, . . . , ).
Note that for the aggregation, the condition includes both the case in which all models agree on the prediction and that in which one of the three models predicts the opposing class. In this case, to correctly identify the feature importance vectors expressing the decision behind the models’ predictions, it is possible to leverage the binary nature of the problems and change sign to the explanations associated to the disagreeing model before computing the aggregation. It is essential that the three explanations are referring to the same output class, to ensure comparability.
Let be a model associated to a feature importance vector a with ∈ (1, . . . , ). Assume that, for a given datapoint x, it holds that 1(x) ̸= 2(x), 2(x) = 3(x). Then, we will consider as feature attributions the vectors of the form (− a1, a2, a3) when computing the aggregation method a.
For each model and corresponding explanation method then, the feature importances for the test set and the neighbourhoods are retrieved. The aggregation is performed following Section 2.3 taking into account the possible sign change for some attributions, as discussed. The robustness scores are then derived for the three individual model-explanation pairs and the aggregation, following Eq. 2. The complete framework is summarized in Figure 1.
Qualitative Evaluation of the Aggregation Let us briefly discuss an example of the derived
aggregation, analysing the explanations of one datapoint x from the test set of adult dataset. The
results are depicted in Figure 2 where, for each of the twelve features, the three individual explanations
(coloured circles) and the aggregation (red star) are presented. This example illustrates two key aspects
of the aggregation: first, cases in which the three explanations have coeficients very close to zero (and
possibly difering in sign) are shrunk towards the zero (that is, the feature is considered non important)
- Features 3, 9, 10, 11. Secondly, when features are difering in sign and the magnitudes are large, the
aggregation penalizes the disagreement but produces a coeficient which is coherent with the stronger
signals - Features 1, 2, 4, 6, 8, 12. This is an expected (and desirable) behaviour as consistent information
is kept and disagreements are penalized more strongly according to the magnitude.
Robustness Estimation Table 2 presents the average robustness scores computed on the test set
for the individual models and the aggregation. As can be seen also from Figure 3, -NN appears to
be the least robust method on average, while NN the most. The aggregated explanation acts as an
average - in terms of robustness - between the three individual components. This is to be expected as
the robustness score was shown in [
Additionally, it is possible to note that the robustness scores derived from -NN and RF align with both the model’s inner reasoning and the explanation methods we have described in Subsections 2.1 and 2.2. For the -NN explanations, we expect the robustness to be lower on average as, by construction, the explanation method deviates the most from a local neighbourhood of the datapoint of interest. In particular, as we are always considering the distance to the nearest neighbours of class ¬, it may be that - especially when the datapoint is far from the observed class boundary - the variability between the distances is higher. This reflects into lower robustness scores even if it locally mimics the decisional reasoning of the underlying model. On the other hand, RF exhibits greater robustness scores as it is harder to generate small perturbations that sensibly change the decision path traversed in each tree in the forest, maintaining lower explanation variability on average.
Model Concordance and Neighbourhood Size An interesting information which is easily derivable in this setting is the neighbourhood size as an additional metric of robustness. As for the aggregation multiple predictions are being considered at once, the local agreement between the models can be used as further confirmation of the robustness. The assumption is that, as suggested by our previous results, a robust explanation usually lies in an area of the data manifold which is robust explanation-wise. This can be verified by assessing if multiple models flag the same datapoint as being robust or not and are agreeing in the point’s prediction, efectively verifying that the explanations are referred to the same class. In this context, the local agreement prediction-wise can indicate if the manifold area is robust also from an explanation point of view.
Results from Table 3 show the average size retained in the neighbourhoods associated to the aggregation, starting from = 100 datapoints and after filtering out the perturbations that change class label with respect to the original one. It is possible to note that the average neighbourhood size varies greatly depending on the dataset being analysed (column Test - %). The remaining columns of the table show the number of observations in test set and corresponding average neighbourhood size for the datapoints over which the three models agree on the prediction (Agree) and those for which one model (which names the column) predicts the opposite class with respect to the other two. Values in bold in the columns % are associated with the largest observed neighbourhood size after the filtering step. It can be noted that, in most cases, the largest size is achieved under the full concordance of the three models, efectively supporting our assumption. Note that, for the cancer dataset, there are 0 datapoints for which -NN is the disagreeing model, therefore the corresponding value N/A indicates that no data was available for the specific scenario during our experimental evaluation.
Validation Assessment The results from Table 3 suggest that, even when considering multiple
models of diferent types, it is possible to discuss the relationship between explanation robustness and
model agreement, as we have previously observed in [
TPR = #{Robust & Agree} #{Agree}
FPR = #{Robust & Disagree} #{Disagree} (4) where the agreement and disagreement are defined according to the model’s concordance in the prediction and the robustness is defined, according to a threshold ℎ, as I[ℛ^ (x) ≥ ℎ].
Varying the threshold value, it is possible to obtain a ROC curve as in Figure 4, where the dotted gray line represents the bisecting line. The Area Under the Curve (AUC) can be used as a goodness measure: its values are reported in Table 4. The greater the value of the AUC, the more preferable is the given model-explanation pair under the agreement assumption and the considered dataset. The results from both Figure 4 and Table 4 suggest that the aggregation can act as a good explanation but not all models are performing in the expected manner. In particular, it seems that the -NN attributions are the least efective in the selected scenarios.
What we have presented in this contribution are the first steps towards a multi-model and multiexplanation aggregation to increase trustworthiness. The aim of our work is proposing an aggregation of explanations and models that can increase the trustworthiness in the system, by leveraging the predictive power of multiple models and their diference in the reasoning process. The robustness score and associated neighbourhood size indicate if the aggregated explanations lie in a robust area of the feature space and can therefore be trusted. Being a work in progress, our approach could still benefit from improvements on the following points: • An aggregation approach more complex than the arithmetic mean, able to deal with cases in which, for example, the attributions are of the form (, , − 2 ) with > 0. The current aggregation would results in an attribution equal to 0, compared to a magnitude (in absolute value) of at least . Taking this aspect into account could benefit the reliability of the explanations. • A new approach for -NN explanations, as the proposed one is dependant on the hyperparameter and - even if from a theoretical point of view is consistent with the -NN reasoning - it exhibits too large a variability. • A more complete validation assessment of the proposal, to test its potential in real use-cases from both an explanation quality and trustability point of view. This would also allow to jointly consider the predictive power of the composing models more appropriately. • The extension of the approach to other classes of models and diferent XAI approaches. This analysis would allow for an extensive evaluation of the eficacy of ensemble approaches under varying circumstances.
We wish to thank Assicurazioni Generali Spa for their support and interest in our work.
The authors have not employed any Generative AI tools.
This appendix presents a comparative example of LIME and SHAP’s robustness. Figure A1 depicts the robustness scores of LIME, SHAP and DeepLIFT for the bank and heloc dataset (as in Figure 3), with the yellow dotted-dashed line representing DeepLIFT’s robustness in both figures. For comparability, LIME and SHAP (which are model-agnostic methods by nature) have been applied to the same model in this example - the neural network. It is clear from Figure A1 that both LIME and SHAP are unstable methods as their robustness scores are, on average, much lower than the DeepLIFT counterpart. Critically, they lie below the 0.50 threshold (gray dashed line), denoting high instability.
Table A1 presents the average robustness scores derived on all the considered datasets for LIME, SHAP and DeepLIFT over the neural network. Note that the column "DeepLIFT" corresponds to the column "NN" of Table 2. Again, it is clear that the robustness scores of LIME and SHAP underperforms with respect to other techniques.
In light of these considerations, as well as further support from the literature, LIME and SHAP were excluded from our analysis as their inner instability may damage both the individual explanations and any ensemble approach that may include them.
17.77 33.11 22.80 56.11 53.99
13.50 10.29 9.34 17.33 36.08
85.03 78.74 84.23 98.40 92.96