Local additive explanation methods are increasingly used to understand the predictions of complex Machine Learning (ML) models. The most used additive methods, SHAP and LIME, sufer from limitations that are rarely measured in the literature. This paper aims to measure these limitations on a wide range (304) of OpenML datasets, and also evaluate emergent coalitional-based methods to tackle the weaknesses of other methods. We illustrate and validate results on a specific medical dataset, SA-Heart. Our ifndings reveal that LIME and SHAP's approximations are particularly eficient in high dimension and generate intelligible global explanations, but they sufer from a lack of precision regarding local explanations. Coalitional-based methods are computationally expensive in high dimension, but ofer higher quality local explanations. Finally, we present a roadmap summarizing our work by pointing out the most appropriate method depending on dataset dimensionality and user's objectives.
Machine Learning (ML) represents a real revolution in various
domains, such as finance, insurance, healthcare, biomedical.
However, machine learning models give a prediction without
necessarily being accompanied by an understandable explanation. These
models, often referred as "black-boxes", raise the challenging
question of how humans can understand the determinants of
the prediction. Explainability is also more than a technological
problem, it involves among other ethical, societal and legal issues.
In healthcare, this may involve the professional being able to
explain to the patient how the algorithm works and the criteria for
the decision process. The results of ML models must therefore be
expressed in a way that can be understood by domain-experts,
like medical practitioners [
The paper is organised as follows. Section 2 reviews the existing work, classifying and comparing explanation methods for tabular data. Section 3 describes the four additive methods to be compared in this paper. The experiments are presented in Section 4 where we study the explanation characteristics, the impact of the predictive model on explanation profiles, highlighting the behavior of explanation methods based on a practical medical use case. Conclusive lessons-learned are then detailed in Section 5. 2
Few works [
Comparative studies between local explanation methods are
also available, such as [
The latest results are therefore indications for the absence
of a single method that would provide the best explanations in
all situations. However, none of these previous works clearly
indicates in which situation a method should be preferred to
another one. Consequently, our aim is to give the key factors to
make an informed decision among the existing additive methods.
As indicated in [
Additive methods are described as explanation models that
produce a vector of weights to represent the influence of each feature,
the sum of which approximates the output of the original model.
Explanations can be computed for a single instance, so for
every instance of a data set, hence the term "local". In this section,
we explore several existing methods that fit this definition. We
focus on post-hoc methods that deliver their explanations for
a given model already trained. Methods used in this study are
all agnostic, meaning that they can be applied to any kind of
machine learning model, except for the TreeSHAP method [
LIME method is a well-known local explanation method described
in [
The full implementation of LIME is available on GitHub : https://github.com/marcotcr/lime. 3.2
To explain individual predictions, a method based on Shapley
values is described in [
Several more recent methods, including SHAP [
SHAP (SHapley Additive exPlanations) [
The full implementation of SHAP is available in GitHub : https://github.com/slundberg/shap. 3.4
Another agnostic explainer based on Shapley values, the
coalitional method, was introduced to take into account the
interdependence of features and solve some restrictions of SHAP. It uses
grouping methods such as Principal Component Analysis (PCA),
Spearman correlation factor (Spearman) and Variance Inflation
Factor (VIF) to pre-compute groups of features for explanations
[
The full implementation of Coalitional-based method is available on GitHub : https://github.com/kaduceo/coalitional_explanation_methods. 4
In this section, we propose experiments comparing the explanation methods presented in the previous section. The goal is to identify the general behavior of each method and how this behavior eventually difers according to a predictive model (learned from data) and the dimensionality of the data (number of features). 4.1
All experiments are run on an Intel Xeon Gold 6230 processor with 125 GB of RAM using Python 3.9.7. All runs are performed on a single core of CPU for optimization and reproducibility. To compare explanation methods, we apply them to a wide range of 304 datasets available on OpenML (www.openml.org). Due to computational constraints of explanation methods, we only considered datasets with at most 13 features, and at most 10 000 instances. We also only considered classification tasks to use comparable predictive models and metrics. We describe the amount and size of datasets per number of features in Table 1.
As an explanation method needs a model to be applied to, we choose four widely used types of ML models for classification: Logistic Regression (LR), Support Vector Machines (SVM), Random Forests (RF) and Gradient Boosted Machines (GBM). For the ifrst three, we use the implementation of Python library scikitlearn version 1.0.1. For GBM, we use the Python library XGBoost version 1.5. We use default values for models hyperparameters. For explanations methods, we use Python libraries shap 0.40 and lime 0.2.0.1.
Then, to be able to compare the explanations, we need to define metrics of interest. In Section 4.2, we present three metrics that we will use for this study.
Section 4.3 aims to compare the four additive methods introduced in Section 3. In particular, we use two distinct coalitionalbased methods: the Complete method, which serves as reference for an influence deviation measurement (second metric), and the Spearman method with a threshold of 25% of all groups of features. Regarding SHAP, we use the model-agnostic KernelSHAP on all datasets. As this method is very slow to execute if we use the whole dataset as background samples for permutations, we choose to follow SHAP’s recommendation1 by doing a K-Means clustering on the input dataset, and then taking the centroids as background samples. We choose = 10 clusters for each dataset, thus naming the method KernelSHAP10. In addition, for the two tree-based predictive models XGBoost and Random Forests, we use the model-specific explainer TreeSHAP by two implementations. The first one determines SHAP values with background samples, similarly to KernelSHAP but optimised for tree-based methods. We use the whole dataset as background samples for this method. The second one approximates SHAP values by considering the trees structures, and does not need background samples in input, so we name it TreeSHAPapprox. Last, we consider LIME, which requires a number of perturbed samples to be created for explaining each instance. We choose to set this number to 100 samples for all datasets.
With similar methodology, Section 4.4 identifies the impact of the predictive model on specific explanation methods.
Lastly, we present in Section 4.5 a practical example of the diferent explanations methods applied to a specific dataset, SAHeart. This dataset is chosen for its medical context (coronary heart diseases), a suficient number of instances (462) and features (10) to train a coherent model and compute the explanations in acceptable computational times. The underlying idea is to illustrate the highlighted behaviors by taking a concrete example as it could be used by an end user (e.g. a physician). 4.2
Because of the subjective nature of explanations, there is no consensus on objective mathematical ways to evaluate the explanations. Therefore, to evaluate explanation methods performances and compare them over a high number of datasets, we define three diferent metrics that only need the influence values given by the method. The first one is the computational time per instance, which is the amount of time taken by a given method to compute the local influences of a whole dataset, divided by the number of instances in the dataset. The second one is a quantification of the average deviation of the influence given by a method from the Complete method (see Section 3). This error rate is defined as: 1 Õ 1 Õ =1 =1 (, ) = ( ) − ( ) where, for a given a dataset, the number of instances, the number of features, the features vector for the instance , ( ) the influence of a feature for a given instance , a given explanation method and a given machine learning model, and ( ) the influence given by the Complete method for the same model, 1KernelSHAP documentation includes recommendation to use K-Means algorithm to speed up computation time https://shap-lrjball.readthedocs.io/en/latest/generated/ shap.KernelExplainer.html same feature and same instance. The third metric evaluates the distribution of feature importance assigned by a given explanation. The raw value being not necessarily comparable between explanation methods, the cumulative importance proportion of features given by a method was considered. This metric shows whether an explanation method favours the attribution of great importance to a few features or, on the contrary, a more homogeneous distribution among a larger number of features. The importance of a feature is defined here as the mean absolute value of influence assigned to instances for such feature. For example, in a dataset with 2 features, if a method gives 80% of the importance to the most important feature (and so 20% to the second), it would have a cumulative importance proportion vector of [0, 0.8, 1]. We can then compute the (normalised) Area Under Curve (AUC) of such a vector with : (, ) = 1 Õ−1 + +1 =0 2 where is the total importance proportion taken by the most important features.
As this cumulative sum is sorted by construction from most important to least important features, this value is bound between 0.5 and 1. A value of 0.5 means that the explanation method gives the same importance to all features, while a value of 1 means that the explanation method gives non-zero influences only to a single feature, explaining the model’s predictions with a single feature. 4.3
We show in Figure 1 the evolution of the execution time of each method for each predictive model, averaged over datasets that share the same number of features. LIME, having a linear complexity with the number of features, is computationally expensive compared to other methods in low dimension (few features), but is less expensive than coalitional-based methods and KernelSHAP in higher dimensions. LIME also seems to have very low interdataset time variability, resulting in smaller error bars on the graph. Coalitional-based methods show an exponential complexity with the number of features, having high execution time in high dimension, but have a similar execution time with other methods in low dimension. Spearman method execution time seems naturally correlated to the Complete method execution time, taking a fraction of the time (roughly 25%) of the Complete method. KernelSHAP, despite a limitation on the amount of background samples, has a high execution time in high dimension, comparable to coalitional-based methods for non-tree based methods. For tree-based methods, KernelSHAP is slower in low dimension, but faster in high dimension than coalitional-based methods. Last, tree-based explainers seem to have constant execution time per instance no matter the number of features, and the approximate tree path dependent version of TreeSHAP has the lowest execution time per instance.
Regarding the second metric, Figure 2 shows the average absolute diference in influence between each method and the Complete method (reference). First, we can see that overall, the more features there are in a dataset, the closest (measured by the second metric) the influences are to the Complete method. This is probably due to the fact that usually, the more features there are, the less influence amplitude each individual feature has in the prediction. We also note that no matter the model, common methods are ranked in the same way. In low dimension (less than 6 features), KernelSHAP is the closest to the Complete method, followed by Spearman, while LIME is the the farthest. In higher dimensions, Spearman becomes more precise than KernelSHAP. TreeSHAP (both the approximate and the data dependent version) is more precise than KernelSHAP, but still less precise than Spearman in high dimensions. Note that the approximate version of TreeSHAP is not showed on the graph for XGBoost because its implementation forces its SHAP values to be in log odds instead of probabilities, making it impossible to compare to other methods.
Finally, we show in Figure 3 an example of the graphical representation of the cumulative feature importance proportion. The ifgure shows the averaging of the cumulative importance proportion of the most-important features for the 37 datasets having 10 features. This way, for each predictive model and for each method, we obtain a curve from which we compute the third metric: the AUC of the curve. We see on the figure that some methods present steeper curves than others. For example, with Logistic Regression and SVM, LIME gives less proportion of the total importance to the few first most-important features, compared to coalitional-based and SHAP methods. For tree-based models, we see that SHAP, no matter the method, gives much more importance to the first few most-important features than the other methods.
According to the method for computing AUC illustrated in Figure 3, we represent the average values of AUC for datasets from 2 to 13 features for each ML model and explanation method in Figure 4. For all models, we can see that SHAP methods tend to produce influences with a higher AUC compared to other methods. This means that SHAP methods tend to assign most of the feature importance to fewer most-important features, while other methods tend to distribute the feature importance more uniformly over all features. The two coalitional-based methods seem to generate similar AUCs for the features importance. Finally, LIME tends to produce influences with lower AUCs for non-tree-based methods, while it produces AUCs closer to the coalitional-based methods for tree-based methods. 4.4
We show in Figure 5 the computational time per instance needed to compute the explanations of each predictive model, for each explanation method.
We can see that LIME’s execution time has almost no intermodel variability: the computation time per instance is the same no matter the model. For the other methods, the ranking of the method’s computational performances according to the model is roughly the same, from slowest to fastest: Random Forests, XGBoost, SVM and Logistic Regression. SVM has overall higher variability, presenting steeper curves and higher error bars. SVM even presents outlying results when applied to KernelSHAP in higher dimensions. Overall, we do not observe specific behavior of method’s computation time in regards to the model used, except for TreeSHAPapprox where Random Forests are faster to compute. This may be related to the fact that TreeSHAPapprox only considers tree structures, as Random Forests tree structures are simpler than XGBoost’s. In general, the faster a model is to train and predict values and the simpler it is, the faster the explanations are to compute, no matter the method, We present in Figure 6 the mean absolute diference between each method applied to each model and the Complete method applied to each model. The figure does not present the results for TreeSHAPapprox because the only relevant model for this method is Random Forests, there is no other model to compare the results with.
For the three model-agnostic methods (LIME, KernelSHAP and Spearman), the Logistic Regression and SVM models generate the most precise explanations compared to the Complete method on the same models. We can see that the explanations based on Logistic Regression are usually more precise than SVM’s, especially in low dimensions. XGBoost explanations are less precise than Random Forest’s, except for the Spearman method (similar results observed). Overall, it seems that the simpler the model, the more precise it is in regards to the Complete method.
Finally, regarding the AUC, we present all the results in Figure 7. We observe that for LIME and KernelSHAP, there is no significant diference between the AUC of the model’s explanations. However, for the coalitional-based methods, we can see a clear separation between tree-based methods and non tree-based methods: the latter have higher AUC than the others. This means that, when using coalitional-based methods, one should be aware that diferent models may yield a diferent importance distribution over the features. For the tree-specific methods, we can see that XGBoost generates explanations with slightly higher AUCs than Random Forests on average. 4.5
Amongst the OpenML datasets previously studied, we choose a medical dataset, SA-Heart, to compare the explanations given by the diferent additive methods on an example. This way, we aim to both illustrate and validate the conclusions of the previous sections regarding explanation methods characteristics. We also aim to highlight practical diferences that we can see on the influences of diferent methods for the same model and dataset.
SA-Heart is a dataset extracted from a larger database of
SouthAfricans detailed in a 1983 study [
For conciseness in this paper, we limit the analysis to a single machine learning model. We choose Random Forests, as every explanation method that we benchmark is applicable to it. We present the results with SVM, Logistic Regression and XGBoost models in supplementary data.
To compare the explanations of the diferent additive methods, we look at global explanations given by each method. We use SHAP-like representations to visualize global explanations by aggregating local explanations on the same representation. This way, we build diferent figures. The first one, in Figure 8, represents a global explanation of the predictive model, given by each explanation method, by plotting the explanation profile of each feature on a separated line. For each method, the features are sorted in decreasing feature importance, the top one being the most contributing feature on average, while the bottom one being the least contributing feature on average. For each feature, each dot represents an individual from the dataset, its color representing the value of the associated feature. Its position on the x-axis represents the contribution of the feature to the prediction of this individual, and overlapping dots are jittered on the y-axis.
We can see that most of the features have similar ranking among the diferent methods: tobacco and age are the two most important features except for the Spearman method which ranks age 5th. On the opposite side, alcohol, spb, and type-A are always in the 4 least important features. These features have also similar explanation profiles. Conversely, some other features exhibit more marked diference depending on the methods. The most important diference is observed on the binary feature family history of heart disease. This feature is assigned fairly low importance by the coalitional-based method, relatively high importance (3rd most important feature) by SHAP methods, and very high importance by LIME (most important feature). Obesity and adiposity have also diferent influences depending on the method: obesity is ranked second least contributing by LIME and SHAP, but more important by the coalitional-based methods. It is important to note that obesity and adiposity are highly correlated (Pearson’s correlation r=0.72). We hypothesize that it may be the reason for such diferences. Overall, the three SHAP methods give similar explanations and have almost identical ranking of the features. From a global perspective, we can also see that SHAP and LIME present a more homogeneous "gradient" of colors for the explanations, where coalitional-based methods present mixed up colors in the explanations. This means that LIME and SHAP’s explanations are more locally monotonic, in the sense that the influence value of a feature for an individual is more locally correlated to the value of the feature for LIME and SHAP than it is for coalitional-based methods.
The second visualization that we present are Partial Dependence Plots (PDP). PDPs focus on the relationship between a feature and the influence of this feature on the model’s prediction by plotting each pair of feature value and influence value on a 2-dimensional axis. We compare the PDPs of several important features in Figure 9.
Looking at the PDPs for the age feature, we show that LIME seems to form clusters of points around specific cut-of age values. To a lesser extent, this phenomenon can also be seen on the other SHAP methods. Conversely, coalitional-based methods have similar PDPs, and do not seem to find such cut-ofs. However, it seems to be a special behavior of the explanation at specific ages. For example, subjects around 50 years have a marked lower contribution of this feature to the prediction of the presence of coronary heart disease than people even slightly younger or older. This may hint at an over-fitting of the machine learning model that would not have been captured by the other explanation methods. The explanation of the tobacco feature also largely difers among explanation methods. Where all the methods agree on attributing a low value to non-smoking individuals, the evolution of the contribution varies with the quantity of tobacco. Once again, LIME and SHAP explanations seem to find a cut-of value for tobacco consumption, of around 7 and 9 respectively, while coalitional-based methods capture a non-monotonic, more complex relationship.
We also look at adiposity PDPs. Once again, the three SHAP explanations are close to each other. Interestingly, they capture a non-monotonic relationship between the feature and the outcome, giving people around 30% of adiposity a higher influence for this feature (in absolute value) than people close to this value. This relationship seems to be captured in a lesser extent by coalitional-based methods, but not captured at all by LIME. We also note that the Complete and Spearman influences are more scattered, which means that more variance exists amongst subjects of the same adiposity for these methods than for the others.
Lastly, looking at obesity PDPs, LIME and SHAP methods find a negative relationship between obesity and the chd prediction. This seems counter intuitive, as obesity is a strong known comorbidity factor of heart diseases. As previously mentioned, obesity and adiposity are strongly correlated (r=0.72), and it may be the reason for such observation. Furthermore, we have mentioned in section 3 that SHAP works under the hypothesis that features are independent, but with such correlation, it is very unlikely that obesity and adiposity are independent. To better understand the relationship between these two features, as found by the methods, we plot in Figure 10 the influence values of adiposity and obesity given by each method.
The Complete and Spearman methods seems to find a positive correlation between the influences of the two features: when an individual is assigned a high influence value for obesity, a high influence value for adiposity is usually assigned, and conversely. We can even distinguish two clusters of individuals: one for individuals that have a high influence value for both features, and one for individuals that have a low influence value for both features. Such pattern is not found by LIME or SHAP, thus confirming the lack of ability of these methods to consider dependent features.
On a more global scale, we see that LIME and SHAP produce explanations that are easier to read at a first glance compared to Complete and Spearman explanations. However, LIME and SHAP seem to capture diferent cut-ofs and relationships, and it is hard to confirm such values without further biological knowledge. Coalitional-based methods seems to produce explanations that are harder to read on a global scale, but more precise at an individual level and able to take into account the dependencies between features. PDPs for all features are available at https://github.com/ EmmanuelDoumard/local_explanation_comparative_study. 5
Table 2 summarizes advantages and drawbacks of each method studied in this paper. Overall, we highlight the fact that coalitionalbased methods should be better at producing precise local explanations while SHAP should be better at producing coherent and easily interpretable global explanations. It is also confirmed by the fact that SHAP tends to assign more importance to few features than other methods, producing global explanations that are easier to read, but potentially hiding other features contributions and inter-dependences. Technically, KernelSHAP gives access to hyper-parameters to balance between execution time and explanation precision, but they are less accessible than Spearman’s and LIME’s parameters. Indeed, without extensive KernelSHAP knowledge or documentation readout, users can easily miss on these parameters.
We use all the results presented in this paper to show a simplified roadmap in the form of a decision tree in Figure 11 with the intent to help readers finding the most suitable explanation method according to their datasets and objectives.
On this figure, high dimension represents the number of features present in the studied dataset. Indeed, there is no "hard" cut-of to define when it goes from low to high dimension, but with our experiments, we can consider this cut-of somewhere between 11 and 15 features, depending on the dataset complexity and the user computational time and material available. "Accurate tree-based model" represents the ability of training a satisfactory (defined by the user’s objectives) tree-based model on the dataset. The model can then be explained thanks to the optimization done in TreeSHAP. If the desired model is not tree-based, we advise the user to look at KernelSHAP and LIME’s parameters to reduce the number of background samples and perturbation samples respectively, until the explanations are computed in a reasonable time. However, we warn the user about the loss of precision induced by such method approximations.
Finally, we show that SHAP and LIME can make important approximations in some cases, and that coalitional-based methods cannot be executed in reasonable time in high dimension. This leaves an empty space for high dimension precise explanations that is not yet addressed to our knowledge. 6
In this paper we performed a practical analysis of several local explainability methods for tabular data. Our findings indicate that there is not a single method that is the most appropriate for every usage. Such usages include the need of a high precision for local explanations or on the contrary the need of explanations that can be aggregated to produce a better and clearer global understanding, while taking into account the complexity level of data especially concerning the high dimension case. Therefore, this thorough analysis allowed to identify strengths and limitations of each method along with practical recommendations on which method is most suitable for the use case of the user. The Complete is of course the most accurate but sufer for very long computational time. Nevertheless, Coalitional based methods allow an acceptable computational time while maintaining a strong precision of explanations. On the contrary, LIME and SHAP methods ofer a more intelligible global view of feature efects. The greatest problem arises when high dimension ( i.e., high number of features) is involved, as it is often the case in statistics and Machine Learning. In this case, the exponential complexity of Coalitional-based methods make them too long to compute. Indeed, the worst case scenario is the need for high precision local explanations in high dimension since there is a clear lack of methods addressing this problem in the current literature. However, it is still possible to have local explanations with limited quality in high dimension, with the level of quality mostly depending on the time available for the user to generate such explanations. It is thus a very interesting future axis of work to benchmark the performances, in terms of precision of local explanations, of every local explainability method in a high dimension context under the constraint of a time limit. This would add value to our recommendations by filling out the ’high-precision in high-dimension’ gap identified in our study. It would also be interesting to look into other machine learning models, especially deep neural networks which are more and more used. The very high complexity of this type of models hints at a diferent behavior for the explanation methods, but also an increase in computation time.
This study has been partially supported through the grant EUR CARe N°ANR-18-EURE-0003 in the framework of the Programme des Investissements d’Avenir.
We also thank the French National Association for Research and Technology (ANRT) and Kaduceo company for providing us with PhD grants (no. 2020/0964).