Local Interpretable Model-Agnostic Explanations (LIME) is a popular method to perform interpretability of any kind of Machine Learning (ML) model. It explains one ML prediction at a time, by learning a simple linear model around the prediction. The model is trained on randomly generated data points, sampled from the training dataset distribution and weighted according to the distance from the reference point - the one being explained by LIME. Feature selection is applied to keep only the most important variables, their coeficients are regarded as explanation. LIME is widespread across diferent domains, although its instability - a single prediction may obtain diferent explanations - is one of the major shortcomings. This is due to the randomness in the sampling step, as well and determines a lack of reliability in the retrieved explanations, making LIME adoption problematic. In Medicine especially, clinical professionals trust is mandatory to determine the acceptance of an explainable algorithm, considering the importance of the decisions at stake and the related legal issues. In this paper, we highlight a trade-of between explanation's stability and adherence, namely how much it resembles the ML model. Exploiting our innovative discovery, we propose a framework to maximise stability, while retaining a predefined level of adherence. OptiLIME provides freedom to choose the best adherence-stability trade-of level and more importantly, it clearly highlights the mathematical properties of the retrieved explanation. As a result, the practitioner is provided with tools to decide whether the explanation is reliable, according to the problem at hand. We extensively test OptiLIME on a toy dataset - to present visually the geometrical findings - and a medical dataset. In the latter, we show how the method comes up with meaningful explanations both from a medical and mathematical standpoint.
sions?”) are some of the main topics XAI tries to
address. To achieve the explainability, quite a few
techNowadays Machine Learning (ML) is pervasive and niques have been proposed in recent literature. These
widespread across multiple domains. Medicine makes approaches can be grouped based on diferent criterion
no diference, on the contrary it is considered one of [11], [12] such as i) Model agnostic or model specific
the greatest challenges of Artificial Intelligence [
Unfortunately, ML methods are hardly perfect and, portant variables and a linear model is trained on the especially in the medical field where human lives are weighted dataset. The model coeficients are regarded at stake, Explainable Artificial Intelligence (XAI) is ur- as explanation. LIME has already been employed sevgently needed [10]. Medical education, research and eral times in medicine, such as on Intensive Care data accountability (“who is accountable for wrong deci- [14] and cancer data [15],[16]. The technique is known to sufer from instability, mainly caused by the ranPGraolcweeadyi,nIrgeslaonfdthe CIKM 2020 Workshops, October 19-20, 2020, domness introduced in the sampling step. Stability is a " giorgio.visani2@unibo.it (G. Visani) desirable property for an interpretable model, whereas 0000-0001-6818-3526 (G. Visani); 0000-0003-3913-7701 (E. Bagli); the lack of it reduces the trust in the explanations re0000-0003©-1206206C4o-p9y6rig3h2t f(oFrt.hiCsphapeesrabyniits)authors. Use permitted under Creative trieved, especially in the medical field. CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmUmoRns WLiceonrsekAsthtriobuptioPnr4o.0cIneteerdnaitniognasl ((CCC EBYU4R.0)-.WS.org) In our contribution, we review the geometrical idea on which LIME is based upon. Relying on statistical description can be found in [13]. We may consider the theory and simulations, we highlight a trade-of be
how much LIME’s simple model resembles the ML model.the dependent variable. tween the explanation’s stability and adherence, namely spanned by the independent variables 1, ...,
and tiLIME: a framework to maximise the stability, while ML surface, in the point we want to explain. This task retaining a predefined level of adherence. OptiLIME is analytically unfeasible, since we don’t have a paraprovides both i) freedom to choose the best adherencemetric formulation of the function, besides the ML surstability trade-of level and ii) it clearly highlights the face may have a huge number of discontinuity points, mathematical properties of the explanation retrieved. preventing the existence of a proper derivative and
tangent. To find an approximation of the tangent, LIME decide whether each explanation is reliable, according uses a Ridge Linear Model to fit points on the ML surto the problem at hand.
explanations both from a medical and mathematical dataset, where the method comes up with meaningful the standpoint. In addition, a toy dataset is employed to present visually the geometrical findings.
2.1. LIME Framework A thorough examination of LIME is provided from a in regions with only few training points. geometrical perspective, while a detailed algorithmic Avoid the sampling step step using the training units only and a combination of Hierarchical Clustering and K-Nearest Neighbour techniques. Although this method achieves stability, it may find a bad approximation of the ML function, face, in the neighbourhood of the reference individual.
Points all over the ℝ space are generated, sampling
values from a Normal distribution inferred from the training set. The coordinate values are obtained by ML predictions, so that the generated points are guaranteed to perfectly lie on the ML surface. The concept of neighbourhood is introduced using a kernel function (RBF Kernel), which smoothly assigns higher weights to points closer to the reference. Ridge Model is trained on the generated dataset, each point weighted by the kernel function, to estimate the linear relationship ( ) = + ∑ =1
garded as LIME explanation.
. The coeficients are re[13] and its extension using decision rules [25] rely on In [29] the authors propose to bypass the sampling this technique for providing local approximations. 2.2. LIME Instability
Explanations derived from repeated LIME calls, under the same conditions, are considered stable when statistically equal [26]. In [27] the authors provide insight about LIME’s lack of robustness, a similar notion to the above-mentioned stability. Analogous findings also in drawback or difident about the method because of its unreliability. By all means, unambiguous explanations are a key desiderata for the interpretable frameworks.
For the sake of shortness, in the following review we consider only model agnostic techniques, which are efective on any kind of ML model by construction. A popular approach is to exclude a certain feature, or group of features, from the model and evaluate the loss incurred in terms of model goodness. Such value quantifies the importance of the excluded feature: an high loss value underlines an important variable for the prediction task. The idea has been first introduced by Breiman [17] for the Random Forest model and has
power of the ML models has been decomposed into single variables contribution in PDP [19], ICE [20] and
explanations in SHAP [22], where the decomposition is obtained through a game-based setting.
model mimicking the behaviour of the ML model. In this vein, approximations on the entire input space are provided in [23] and [24] among others, while LIME been generalised to a model-agnostic framework, named [28]. Often, practitioners are either not aware of such the ML model. The same idea is exploited also for local the sampling step, when new observations are ranEvaluate the post-hoc stability
conditions, and test whether the results are equivalent. Among the various propositions on how to conduct the test, in [30] the authors compare the standard deviations of the Ridge coeficients, whereas [31] examines the stability of the feature selection step - whether the selected variables are the same - . In [26] two complementary indices have been developed, based on statistical comparison of the Ridge models generated by repeated LIME calls. The Variables Stability Index (VSI) checks the stability of the feature selection step, whereas the Coeficients Stability Index (CSI) asserts the equality of coeficients attributed to the same feature.
highest reachable level of stability, constrained to the ifnding of a relevant local explanation. From a geometrical perspective, the relevance of the explanation corresponds to the adherence of the linear plane to the ML surface. To evaluate the stability we rely on the CSI and VSI indices [26], while the adherence is assessed using the 2 statistic, which measures the goodness of the linear approximation through a set of points [32].
All the figures of merit above span in the range [
To fully explain the rationale of the proposition, we ifrst cover three important concepts about LIME. In the geometrical ideas about LIME may be well reprethis section we employ a Toy Dataset to show our the- sented in a 2d plot. oretical findings. 3.1. Kernel Width defines locality Toy Dataset
The dataset is generated from the Data Generating Pro- fault is the RBF Kernel (Formula 1). It is applied to each
cess: point ( ) generated in the sampling step, obtaining an
= ( ) ∗ + 10 individual weight. The formulation provides smooth
weights in the range [
In Figure 2, LIME generated points are displayed as The dataset is intentionally one dimensional, so that green dots and the corresponding LIME explanations (red lines) are shown. The points are scattered all over the ML function, however their size is proportional to the weight assigned by the RBF kernel. Small kernel widths assign significant weights only to the closest points, making the further ones almost invisible.
3.2. Ridge penalty is harmful to LIME dataset.
of white noise, so that the standard formulation of the problem is
= ( ) + , where ∼ (0, 2). The aim of each statistical model is to retrieve the best specification of the DGP function ( ), given the noisy
Ridge Regression [33] assumes a linear DGP, namely unique solution. ( ) = + ∑ =1 tional to the norm , and applies a penalty proporof the
coeficients, enforced during the estimation process through the penalty parameter . This technique is useful when dealing with very noisy datasets (where the stochastic component exhibits high variance 2) [34]. In fact, the noise makes various sets of coeficients as viable solutions. Instead, tuning to its proper value allows Ridge to retrieve a
DGP, while the sampled points are the dataset. Recalling that the coordinate of each point is given by ML prediction, it is guaranteed they lie exactly on the ML surface by construction. Hence, no noise is present in our dataset. For this reason, we argue that Ridge penalty is not needed, on the contrary it can be harmful and distort the right estimates of the parameters, as shown in Figure 3.
distorted by the penalty, proven also by the 2 values. This happens especially for small kernel width values, since each unit has very small weight and the weighted residuals are almost irrelevant in the Ridge loss, which is dominated by the penalty term. To minimize the penalty term the coeficients are shrunk towards 0. (a) Ridge Penalty = 0 (b) Ridge Penalty = 1
of LIME, we wish to understand how Stability and Adherence vary wrt to it.
From the theory, we have few helpful results: • Taylor Theorem [32] gives a polynomial approximation for any diferentiable function, calculated in a given point. If we truncate the formula to the first degree polynomial, we obtain a linear function, its approximation error depends on the distance from the point in which the error is evaluated and the given point.
Thus, if we assume the ML function to be differentiable in the neighbourhood of ( ), the adherence of the linear model is expected to be inversely proportional to the width of the neighbourhood, i.e. to the kernel width. This is true since the approximation error depends on the distance from the two points, namely the neighbourhood size. 3.4. OptiLIME
stability are monotonous noisy functions of the kernel width: for increasing kernel width we observe, on average, decreasing adherence and increasing stability.
Our proposition consists in a framework which enables the best choice for the trade-of between stability and adherence of the explanations. OptiLIME sets a desired level of adherence and finds the largest kernel width, matching the request. At the same time, the best kernel width provides the highest stability value, constrained to the chosen level of adherence. At the end of the day, OptiLIME consists in an automated way of finding the best kernel width. Moreover, it empowers the practitioner to be in control of the trade-of between the two most important properties of LIME • in Linear Regression, the standard deviation of Local Explanations.
the coeficients is inversely correlated to the stan- To retrieve the best width, OptiLIME converts the dard deviation of the variables [32]. decreasing 2 function into (, ̃ 2), by means of ForThe stability of the explanations depends on the mula 2: spread of the variables in our weighted dataset.
We then expect the kernel width and Stability to be directly proportional.
diferent kernel width values and evaluate both 2 and CSI metrics (VSI is not considered in the Toy Dataset, where ̃ 2 is the requested adherence. since only one variable is present). In Figure 4 the re- For a fixed ̃ 2, chosen by the practitioner, the function sults of such experiment, for the reference unit, are (, ̃ 2) presents a global maximum. We are particushown. larly interested in the arg max (, ̃ 2), namely the
Both the adherence and stability are noisy functions best kernel width.
of the kernel width: they contain some stochasticity, In order to solve the optimum problem, Bayesian
due to the diferent datasets generated by each LIME Optimization is employed, since it is the most suitable
call. Despite this, it is possible to detect a clear pat- technique to find the global optimum of noisy
functern: monotonically increasing for the CSI Index and tions [
For numerical evidence of these properties, we fit with random values, , number of iterations of the the Logistic function [35], which retrieves the best monotosenaorucsh refinement strategy. Increasing the parameters approximation to a set of points. The goodness of the ensures to find a better kernel width value, at the cost logistic approximation is confirmed by a low value of of longer computation time. the Mean Absolute Error (MAE).
To corroborate our assumption, the same process has been repeated on all the units of the Toy Dataset, obtaining average MAE for the 2 approximation of 0.005 and for the CSI of 0.026. The logistic growth rate has also been inspected: 2 highest growth rate is -10.78 and CSI lowest growth rate is 7.20. These results ensure the monotonous relationships of adherence and stability with the kernel width, respectively decreasing and increasing.
erence unit of the Toy Dataset is presented. ̃ 2 has been set to 0.9, = 20 and = 40. The points in the plot represent the distinct evaluations performed by the Bayesian Search in order to find the optimum. Comparing the plot with Figure 4, we observe the effect of Formula 2 on the left part of the 2 and (, ̃ 2) functions. In Figure 5 the search has converged to the maximum, evaluating various points close to the best kernel width. At the same time, it is evident the stochastic nature of the CSI function: the several CSI (, ̃ 2) = { 2( ), if 2( ) ≤ ̃ 2 2 ̃ 2 − 2( ) if 2( ) > ̃ 2 (2) (a) Best LIME Explanation, Unit 100 measurements, performed in the proximity of 0.3 value of the kernel width, show a certain variation. Nonetheless, it is possible to recall the increasing CSI trend.
(b) Best LIME Explanation, Unit 7207 Dataset to achieve the same adherence, we are forced to con- sciously. sider a smaller portion of the ML model, hence a small We exploit these findings in order to tackle LIME neighbourhood. Smaller kernel width implies also a weak points. The result is the OptiLIME framework, reduced Stability, testified by small values of the VSI which represents a new and innovative contribution to and CSI indices. Whenever the practitioner desires the scientific community. OptiLIME achieves stability more stable results, it is possible to re-run OptiLIME of the explanations and automatically finds the proper with a less strict requirement for the adherence. It kernel width value, according to the practitioner’s needs. is important to remark that low degrees of adherence The framework may serve as an extremely useful will make the explanations increasingly more global: tool: using OptiLIME, the practitioner knows how much the linear surface retrieved by LIME will consist in an to trust the explanations, based on their stability and average of many local non-linearities of the ML model. adherence values.
The computation time largely depends on the Bayesian Nonetheless, we acknowledge that the optimization Search, controlled by the parameters and . In our framework may be improved to allow for a faster and setting, = 10 and = 30 produce good results for more precise computation. both the units in Figure 6.
On a 4 Intel-i7 CPUs 2.50GHz laptop, the OptiLIME evaluation for Unit 100 and Unit 7207 took respectively Acknowledgments 123 and 147 seconds to compute. For faster, but less accurate results, the Bayesian Search parameters can be We acknowledge financial support by CRIF S.p.A. and reduced. Università degli Studi di Bologna.
In Medicine, diagnostic computer algorithms providing accurate predictions have countless benefits, notably they may help in saving lives as well as reducing medical costs. However, precisely because of the importance of these matters, the rationale of the decisions must be clear and understandable. A plethora of techniques to explain the ML decisions has grown in recent years, though there is no consensus on the best in class, since each method presents some drawbacks. Explainable models are required to be reliable, thus stability is regarded as a key desiderata.
We consider the LIME technique, whose major drawback lies in the lack of stability. Moreover, it is dificult to tune properly its main parameter: diferent values of the kernel width provide substantially diferent explanations.
The main contribution of this paper consists in the clear decomposition of the LIME framework in its relevant components and the exhaustive analysis of each one, starting from the geometrical meaning through the empirical experiments to validate our intuitions.
We showed that Ridge penalty is not needed and LIME works best with simple Linear Regression as explainable model. In addition, smaller kernel width values provide a more adherent LIME plane to the ML surface, therefore a more realistic local explanation. Eventually, the trade-of between the adherence and stability properties is extremely valuable since it empowers the practitioner to choose the best kernel width conthe 21th ACM SIGKDD International Conference tional expectation, Journal of Computational and on Knowledge Discovery and Data Mining, 2015, Graphical Statistics 24 (2015) 44–65. pp. 507–516. [21] D. W. Apley, J. Zhu, Visualizing the efects of pre[8] T. A. Lasko, J. C. Denny, M. A. Levy, Computa- dictor variables in black box supervised learning tional phenotype discovery using unsupervised models, arXiv preprint arXiv:1612.08468 (2016). feature learning over noisy, sparse, and irregular [22] S. M. Lundberg, S.-I. Lee, A unified approach clinical data, PloS one 8 (2013). to interpreting model predictions, in: Advances [9] E. J. Topol, High-performance medicine: The in Neural Information Processing Systems, 2017, convergence of human and artificial intelligence, pp. 4765–4774.
Nature medicine 25 (2019) 44–56. [23] M. Craven, J. W. Shavlik, Extracting tree[10] A. Holzinger, From machine learning to explain- structured representations of trained networks, able AI, in: 2018 World Symposium on Digital in: Advances in Neural Information Processing Intelligence for Systems and Machines (DISA), Systems, 1996, pp. 24–30.
IEEE, 2018, pp. 55–66. [24] Y. Zhou, G. Hooker, Interpreting models [11] C. Molnar, Interpretable Machine Learning, Lulu. via single tree approximation, arXiv preprint com, 2020. arXiv:1610.09036 (2016). [12] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, [25] M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: F. Giannotti, D. Pedreschi, A survey of methods High-precision model-agnostic explanations, in: for explaining black box models, ACM comput- Thirty-Second AAAI Conference on Artificial Ining surveys (CSUR) 51 (2018) 93. telligence, 2018. [13] M. T. Ribeiro, S. Singh, C. Guestrin, Why should [26] G. Visani, E. Bagli, F. Chesani, A. Poluzzi, D. Cai trust you?: Explaining the predictions of any puzzo, Statistical stability indices for LIME: Obclassifier, in: Proceedings of the 22nd ACM taining reliable explanations for Machine LearnSIGKDD International Conference on Knowl- ing models, arXiv preprint arXiv:2001.11757 edge Discovery and Data Mining, ACM, 2016, pp. (2020).
1135–1144. [27] D. Alvarez-Melis, T. S. Jaakkola, On the robust[14] G. J. Katuwal, R. Chen, Machine learning model ness of interpretability methods, arXiv preprint interpretability for precision medicine, arXiv arXiv:1806.08049 (2018).
preprint arXiv:1610.09045 (2016). [28] A. Gosiewska, P. Biecek, IBreakDown: Un[15] A. Y. Zhang, S. S. W. Lam, N. Liu, Y. Pang, L. L. certainty of model explanations for nonChan, P. H. Tang, Development of a Radiology additive predictive models, arXiv preprint Decision Support System for the Classification of arXiv:1903.11420 (2019).
MRI Brain Scans, in: 2018 IEEE/ACM 5th Interna- [29] M. R. Zafar, N. M. Khan, DLIME: A deterministic tional Conference on Big Data Computing Appli- local interpretable model-agnostic explanations cations and Technologies (BDCAT), IEEE, 2018, approach for computer-aided diagnosis systems, pp. 107–115. arXiv preprint arXiv:1906.10263 (2019). [16] C. Moreira, R. Sindhgatta, C. Ouyang, P. Bruza, [30] S. M. Shankaranarayana, D. Runje, ALIME: A. Wichert, An Investigation of Interpretability Autoencoder Based Approach for Local InterTechniques for Deep Learning in Predictive Pro- pretability, in: International Conference on Intelcess Analytics, arXiv preprint arXiv:2002.09192 ligent Data Engineering and Automated Learn(2020). ing, Springer, 2019, pp. 454–463. [17] L. Breiman, Random forests, Machine learning [31] C. Molnar, Limitations of Interpretable Machine 45 (2001) 5–32. Learning Methods, 2020. [18] J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, [32] W. H. Greene, Econometric Analysis, Pearson L. Wasserman, Distribution-free predictive in- Education India, 2003. ference for regression, Journal of the American [33] A. E. Hoerl, R. W. Kennard, Ridge Regression: Statistical Association 113 (2018) 1094–1111. Biased Estimation for Nonorthogonal Problems, [19] J. H. Friedman, Greedy function approximation: Technometrics 12 (1970) 55–67. doi:10.1080/ A gradient boosting machine, Annals of statistics 00401706.1970.10488634.
(2001) 1189–1232. [34] W. N. van Wieringen, Lecture notes on ridge re[20] A. Goldstein, A. Kapelner, J. Bleich, E. Pitkin, gression, arXiv preprint arXiv:1509.09169 (2019).
Peeking inside the black box: Visualizing sta- [35] P.-F. Verhulst, Correspondance mathématique et tistical learning with plots of individual condi- physique, Ghent and Brussels 10 (1838) 113.