=Paper=
{{Paper
|id=Vol-2699/paper01
|storemode=property
|title=Guided-LIME: Structured Sampling based Hybrid
Approach towards Explaining Blackbox Machine Learning Models
|pdfUrl=https://ceur-ws.org/Vol-2699/paper01.pdf
|volume=Vol-2699
|authors=Amit Sangroya,Mouli Rastogi,C. Anantaram,Lovekesh Vig
|dblpUrl=https://dblp.org/rec/conf/cikm/SangroyaRAV20
}}
==Guided-LIME: Structured Sampling based Hybrid
Approach towards Explaining Blackbox Machine Learning Models
==
https://ceur-ws.org/Vol-2699/paper01.pdf
Guided-LIME: Structured Sampling based Hybrid
Approach towards Explaining Blackbox Machine
Learning Models
Amit Sangroya, Mouli Rastogi, C. Anantaram and Lovekesh Vig
TCS Innovation Labs, Tata Consultancy Services Ltd., Delhi, India
Abstract
Many approaches to explain machine learning models and interpret its results have been proposed. These include shadow
model approaches, like LIME and SHAP; model inspection approaches like Grad-CAM and data-based approaches like Formal
Concept Analysis (FCA). Explanations of the decisions of blackbox ML models using any one of these approaches has their
limitations as the underlying model is rather complex. Running explanation model for each sample is not cost-efficient. This
motivates to design a hybrid approach for evaluating interpretability of blackbox ML models. One of the major limitations
of widely-used LIME explanation framework is the sampling criteria that is employed in SP-LIME algorithm for generating
a global explanation of the model. In this work, we investigate a hybrid approach based on LIME using FCA for structured
sampling of instances. The approach combines the benefits of using a data-based approach (FCA) and proxy model-based
approach (LIME). We evaluate these models on three real-world datasets: IRIS, Heart Disease and Adult Earning dataset.
We evaluate our approach based on two parameters: 1) by measuring the prominent features in the explanations, and 2)
proximity of the proxy model to the original blackbox ML model. We use calibration error metric in order to measure the
closeness between blackbox ML model and proxy model.
Keywords
Interpretability, Explainability, blackbox Models, Deep Neural Network, Machine Learning, Formal Concept Analysis
1. Introduction erated. In the proxy model approach, the data corpus
needs to be created by perturbing the inputs of the tar-
Explainability is an important aspect for an AI system get blackbox model and then an interpretable shadow
in order to increase the trustworthiness of its decision- model is built, while in the model inspection approach
making process. Many blackbox deep learning mod- the model architecture needs to be available for in-
els are being developed and deployed for real-world spection to determine the activations, and in the data-
use (an example is Google’s Diabetic Retinopathy Sys- based approach the training data needs to be available.
tem [1]). For such blackbox systems neither the model Local shadow models are interpretable models that
details nor its training dataset is made publicly avail- are used to explain individual predictions of blackbox
able. Explanations of the predictions made by such machine learning models. LIME (Local Interpretable
blackbox systems has been a great challenge. Model-agnostic Explanations [9]) is a well-known ap-
Apart from post-hoc visualization techniques [2] (e.g., proach where shadow models are trained to approxi-
feature dependency plots), feature importance tech- mate the predictions of the underlying blackbox model.
niques based on sensitivity analysis, there have been LIME focuses on training local shadow models to ex-
three main approaches for explainability of AI systems: plain individual predictions, wherein a prediction of
i) Proxy or Shadow model approaches like LIME, SHAP interest 𝑦𝑖 of the target blackbox deep learning model
ii) Model inspection approaches like Class Activation is considered and its related input features 𝑥𝑖 ’s are
maps (CAM), Grad-CAM, Smooth-Grad-CAM, etc. and perturbed within a neighborhood proximity to mea-
iii) Data based approaches like Decision sets and For- sure the changes in predictions. Based on a reasonable
mal Concept Analysis [3, 4, 5, 6, 7]. Most of the re- sample of such perturbations a dataset is created and
search work on explainability has followed one of the a locally linear explainable model is constructed. To
above approaches [8]. However, each of these approachescover the decision-making space of the target model
have limitations in the way the explanations are gen- , Submodular Pick-LIME (SP-LIME) [9] generates the
global explanations by finding a set of points whose
CIKM 2020 Workshops, October 19-20, 2020, Galway, Ireland explanations (generated by LIME) are varied in their
email: amit.sangroya@tcs.com (A. Sangroya); mouli.r@tcs.com
(M. Rastogi); c.anantaram@tcs.com (C. Anantaram); selected features and their dependence on those fea-
lovekesh.vig@tcs.com (L. Vig) tures. SP-LIME proposes a sampling way based on
orcid: sub-modular picks to select instances such that the in-
© 2020 Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0). terpretable features have higher importance.
CEUR Workshop Proceedings (CEUR-WS.org)
Figure 2: Example for Calibration of ML model and proxy
explanation models
ysis to explain the outcomes of a machine learning
model. We use LIME to interpret locally by using a
linear shadow model of the blackbox model, and use
Formal Concept Analysis to construct a concept lat-
Figure 1: Example output of LIME after adding noisy fea- tice of the training dataset, and then extract out impli-
tures in the Heart Disease dataset cation rules among the features. Based on the impli-
cation rules we select relevant samples for the global
instances that we feed to SP-LIME. Therefore, rather
Figure 1 shows a sample explanation output of LIME than using all instances (which is very costly for deep
for a binary classification problem on Heart Disease networks) or random sampling (which never guaran-
dataset. The prediction probabilities are shown in the tees optimal behavior), we use a FCA guided approach
left using different colors and prominent features that for selecting the instances. Therefore, we call our frame-
are important for classification decision are shown in work as Guided-LIME.
the right. Important features are presented in a sorted We show that Guided-LIME results in better cover-
manner based on their relevance. Note that some noisy age of the explanation space as compared to SP-LIME.
features are also injected in the dataset and therefore Our main contributions in this paper are as follows:
are present in the explanation (af1, af2, af3 and af4)
• We propose a hybrid approach based on LIME
as well. In an ideal scenario, noisy features should
and FCA for generating explanation by exploit-
not be the most relevant features for any ML model
ing the structure in training data. We demon-
and therefore should be least important from an ex-
starte how FCA helps in structured sampling of
planation point of view. However, due to proxy model
instances for generating global explanations.
inaccuracies and unreliability, sometimes these noisy
features can also come as the most relevant features • Using the structured sampling, we can choose
in explanations. In figure 2, we show an example sce- optimal instances both in terms of quantity and
nario that compares the calibration level of two proxy quality to generate explanations and interpret
models with a machine learning model. The x axis in the outcomes.Thereafter, using calibration error
this figure is the confidence of model and y axis is the metric we show that Guided-LIME is a closer ap-
accuracy. Assuming that we have a blackbox machine proximate of the original blackbox ML model.
learning model and a proxy model that explains this
model, we argue that these models should be closer to
each other in terms of their calibration levels. 2. Background and Preliminaries
Ideally, a proxy model which is used for explaining a
machine learning model should be as close as possible 2.1. Blackbox Model Outcome
to the original model Explanation
Motivated by the design of an optimized explana-
tion model, we design a hybrid approach where we A blackbox is a model, whose internals are either un-
combine the shadow model approach proposed by LIME known to the observer or they are known but uninter-
with the data-based approach of Formal Concept Anal- pretable by humans. Given a blackbox model solving
a classification problem, the blackbox outcome expla-
nation problem consists of providing an interpretable
explanation for the outcome of the blackbox. In other
words, the interpretable model must return the pre-
diction together with an explanation about the rea-
sons for that prediction. In this context, local inter-
pretability refers to understanding only the reasons for
a specific decision. In this case, only the single pre-
diction/decision is interpretable. On the other hand, a
model may be completely interpretable when we are Figure 3: Example of a formal context using samples from
IRIS dataset
able to understand the global prediction behavior (dif-
ferent possible outcomes of various test predictions).
2.2. LIME Approach for Global
Explanations
SP-LIME algorithm provides a global understanding
of the machine learning model by explaining a set of
individual instances. Ribeiro et al. [9] propose a bud-
get 𝐵 that denotes the number of explanations to be
generated. Thereafter, they use Pick Step to select 𝐵 in-
stances for the user to inspect. The aim of this is to ob-
tain non-redundant explanations that represent how
the model behaves globally. This is done by avoiding Figure 4: Example of a concept lattice related to formal con-
instances with similar explanations. However, there text in Figure 3
are some limitations of this algorithm [10]:
• The SP-LIME algorithm is based on a greedy ap- (1982) to study how objects can be hierarchically grouped
proach which does not guarantee an optimal so- together according to their common attributes. FCA
lution. deals with the formalization of concepts and has been
applied in many disciplines such as software engineer-
• The algorithm runs the model on all instances
ing, machine learning, knowledge discovery and on-
to maximize the coverage function.
tology construction during the last 20-25 years. Infor-
Data points are sampled from a Gaussian distribu- mally, FCA studies how objects can be hierarchically
tion, ignoring the correlation between features. This grouped together with their common attributes. A for-
can lead to unlikely data points which can then be mal context 𝐾 = (𝐺, 𝑀, 𝐼 ) consists of two sets 𝐺 and
used to learn local explanation models. In [11], au- 𝑀 and a relation 𝐼 between 𝐺 and 𝑀. The elements
thors study the stability of the explanations given by of 𝐺 are called the objects and the elements of 𝑀 are
LIME. They showed that the explanations of two very called the attributes of the context. A formal concept
close points varied greatly in a simulated setting. This of a formal context 𝐾 = (𝐺, 𝑀, 𝐼 ) is a pair (𝐴, 𝐵). The
instability decreases the trust in the produced expla- set of all formal concepts of a context K together with
nations. The correct definition of the neighborhood the order relation 𝐼 forms a complete lattice, called the
is also an unsolved problem when using LIME with concept lattice of 𝐾 .
tabular data. Local surrogate models e.g. LIME is a Figure 3 and 4 are examples from IRIS dataset (more
concrete and very promising implementation. But the details in Section 4). In figure 3, we show a collection
method is still in development phase and many prob- of some objects and their attributes. For simplicity, we
lems need to be solved before it can be safely applied. choose only those objects where a particular attribute
is present or not. In real-world objects can have very
complex relationships with fuzzy values. Figure 4 is an
2.3. Formal Concept Analysis example concept lattice generated using this sample
Formal Concept Analysis (FCA) is a data mining model data.
that introduces the relation among attributes in a vi-
sual form. It was introduced in the early 80s by Wille
Figure 5: Overall workflow of Guided-LIME
3. Guided-LIME Framework: uses these instances to generate a set of local expla-
nation models and covers the overall decision-making
Guiding sampling in SP-LIME space. FCA provides a useful means for discovering
using FCA extracted concepts implicational dependencies in complex data [12, 13].
In previous work, FCA-based mechanism has been
In [9] SP-LIME has been used to generate global ex- used as an approach to explain the outcome of a black-
planations of a blackbox model. SP-LIME carries out box machine learning model through the construction
submodular picks from a set of explanations generated of lattice structure of the training data and then using
for a given set X of individual data instances. The SP- that lattice structure to explain the features of predic-
LIME algorithm picks out explanations based on fea- tions made on test data [4]. In this proposed hybrid
ture importances across generated explanations. How- approach, we use the power of FCA to determine im-
ever, the data instances X from which explanations plication rules among features and using that to guide
are generated, are either the full dataset (called Full- the submodular picks for LIME in order to generate
LIME) or data points sampled from a Gaussian distri- local explanations. It provides the benefits of using
bution (SP-LIME random), and ignore the correlation data-based approach and proxy model based approach
between features in the dataset. Carrying out SP-LIME in a unified framework.
for the full dataset (Full-LIME) is very time consuming
especially when the dataset is large. Carrying out SP-
LIME random on the dataset may end up considering
3.1. FCA-based selection of Instances
data points that are implied by other data points in the The goal of our FCA-based instances selection is to
explanation space. Thus it is important to analyze the take advantage of the underlying structure of data to
full data set and choose only those points for SP-LIME build a concise and non-redundant set of instances.
such that the selected data points are representative of We hypothesize that most of the state-of-the-art ap-
the data space. In this work, we propose a mechanism proaches do not consider this information (to the best
to determine the implication of features to guide the of our knowledge). We shortlist sample instances us-
selection of the instances X from the training dataset. ing the following process:
We use Formal Concept Analysis (FCA) to analyze the
training data and discover feature implication rules. 1. We first binarize the training data in an ad-hoc
Using these feature implication rules, we pick appro- way. The binarization technique is applied to
priate instances to feed into SP-LIME. SP-LIME then discretize the continuous attribute values into
only of two values, 0 or 1. The binarization pro- ples are chosen randomly using a Gaussian distribu-
cess can be done in a more formal manner e.g. tion. On the other hand, full approach make use of all
chiMerge algorithm [14] which ensures that bi- the instances. We extend the LIME implementation to
narization method does not corrupt the gener- integrate another method “FCA" that takes the samples
ated lattice. In the scope of current work, we generated using lattice and implication rules.
keep this process simple enough. Thereafter, we Algorithm 1 explains the steps to perform structured
generate concept lattice using standard FCA-based sampling using training data and pass to SP-LIME for
approach. Each concept in the lattice represents generating global explanations. The input to Guided-
the objects sharing some set of properties; and LIME is training data used to train the blackbox ML
each sub-concept in the lattice represents a sub- model. Data processing for finding the best samples
set of the objects. for Guided-LIME involves binarization of data. There-
2. We use ConExp concept explorer tool to gener- after, a concept lattice is created based on FCA ap-
ate lattice from the training data [15]. proach [4]. Using the concept lattice, we derive im-
plication rules. These rules are then used to select test
3.1.1. Generating Implication Rules from instances for Guided-LIME.
Training Data
Algorithm 1 Sample selection algorithm using FCA
In order to find an optimal subset of samples, we gen- for Guided-LIME
erate implication rules from the given training data. Require: Training dataset 𝐷
One of the challenge in generating implication rules is Ensure: Samples and their ranking
that for a given domain and training data, the number for a given Training dataset 𝐷 consisting of data
of rules can be very large. Therefore, we shortlist rules samples 𝑠 do
based on their expressiveness e.g. we select the subset Binarize numeric features
of rules that have the highest coverage and lowest re- Generate concept Lattice using FCA
dundancy. Find implication rules 𝑟
When we generate association rules from the dataset, Generate samples and their ranking
conclusion does not necessarily hold for all objects. Select top 𝑘 samples from each rule
However, it is true for some stated percentage of all end for
objects covering the premise of rule. We sort the rules for all top 𝑘 samples from each rule do
using this percentage and select the top 𝑘 rules. The Select samples using redundancy and coverage
value of 𝑘 is emperically calculated based on a given criteria
domain. end for
3.1.2. Generating Lattice Structure and As we mentioned previously, there are various ex-
selecting Instances amples of using a single approach for explanation. This
Using the lattice structure and implication rules, we can be done using any of the proposed techniques i.e.
select instances for guiding SP-LIME. We identify all proxy model, activation based or perturbation based
the instances that follow the implication rules. For approach. However, we argue that none of these ap-
each rule in the “implication rules list", we calculate proaches provides a holistic view in terms of outcome
if a given sample “pass" or “fail" the given criteria i.e. explanation. Whereas, if we use a hybrid approach
if a particular sample 𝑠 follows implication rule 𝑟 or such as a combination of proxy model and data-based
not. Finally, we produce a sorted list of the instances approach, it can provide a better explanation at a much
that are deemed more likely to cover maximally and reduced cost.
are non-redundant as well. One of the question that arise in our hybrid approach
is whether the approach is still model agnostic such as
LIME. We argue that sampling step do not affect the
3.2. Guided-LIME for Global model agnosticity in any manner. It just adds a sam-
Explanations pling step which helps in choosing the samples in a
We propose structured data sampling based approach systematic manner.
Guided-LIME towards a hybrid framework extending
SP-LIME. SP-LIME normally has two methods for sam-
pling: random and full. In the random approach, sam-
Dataset Classes # of in- # of fea- Features
stances tures
IRIS 3 150 4 sepal length, sepal width, petal length, petal width
Heart Disease 2 303 14 age of patient, sex, chest pain type, resting blood pressure,
serum cholesterol, fasting blood sugar, resting ECG, maxi-
mum heart rate achieved, exercise induced angina, ST de-
pression induced by exercise relative to rest, peak exercise
ST segment, number of major vessels colored by fluoroscopy,
Thal, Diagnosis of heart disease
Adult Earning 2 30000 14 age, workclass, fnlwgt, education, education-num, mari-
tal status, occupation, relationship, race, sex, capital-gain,
capital-loss, hours-per-week, native-country
Table 1
Summary of Datasets
4. Experiments and Results 20
SP-Lime
#Artificial Feature Count
4.1. Experimental Setup Guided-LIME
15
We use the following publicly available datasets to eval-
uate the proposed framework: IRIS, Heart Disease and
Adult Earning dataset (See Table 1). IRIS dataset con- 10
tains 3 classes of 50 instances each, where each class
refers to a type of iris plant [16]. There are a total 5
of 150 samples with 5 attributes each: sepal length,
sepal width, petal length, petal width, class (Iris Se-
tosa, Iris Versicolor, Iris Virginica). Similarly, Heart 0
Disease dataset contains 14 attributes; 303 samples and AF-1_Imp-1 AF-1_Imp-2
two classes [17]. Adult Earning dataset contains 48000
samples, 14 features across two classes. The machine Figure 6: FDR (False discovery rate) for IRIS dataset
learning task for all three datasets is classification. We
use random forest blackbox machine learning model
in all our experiments. generated explanations. Ideally, the noisy features should
not occur among the important features. Therefore a
lower FDR suggests a better approach for explanation.
4.2. Results
We present the discovery of number of noisy features
The goal of this experiment is to compare the proposed for each explanation averaged over 100 runs. Each ex-
Guided-LIME approach with random sampling of SP- planation consists of a feature importance vector that
LIME. In the scope of this work, we do not compare shows the importance of a particular feature. As we
the proposed hybrid approach with full sampling of see in Figures 6, 7, and 8, y axis is the number of noisy
SP-LIME. We perform a case study to find out which features and x axis is index of noisy feature. We in-
approach is better in selecting important features for a clude the cases where a noisy feature is at first or sec-
given blackbox model. As shown in Table 1, we main- ond place in the feature importance vector. AF-1_Imp-
tain ground truth oracle of important features as do- 1 represents artificial/noisy feature occurring at first
main knowledge [18, 19]. We train random forest clas- place in feature importance vector whereas AF-1_Imp-
sifier with default parameters of scikit-learn. In this 2 represents artificial/noisy feature occurring at sec-
experiment, we add 25% artificially “noisy” features in ond place. Guided-LIME sampling approach is consis-
the training data. The value of these features is cho- tently better than basic SP-LIME.
sen randomly. In order to evaluate the effectiveness
of approach we use FDR (false discovery rate) metric
which is defined as the total number of noisy features
selected as important features in the explanation.
We calculate the occurrence of noisy features in the
Table 2
Expected Calibration Error of Blackbox Model and proxy models
With artificial features Without artificial features
Datasets
blackbox Guided- SP- blackbox Guided- SP- Full-
LIME LIME LIME LIME LIME
Adult 0.061 0.065 0.041 0.056 0.065 0.041 0.059
Earning
Heart 0.149 0.167 0.216 0.125 0.165 0.169 0.136
Disease
IRIS 0.106 0.042 0.033 0.038 0.006 0.08 0.031
Table 3
Maximum Calibration Error of Blackbox Model and proxy models
With artificial features Without artificial features
Datasets
blackbox Guided- SP- blackbox Guided- SP- Full-
LIME LIME LIME LIME LIME
Adult 0.187 0.372 0.428 0.19 0.353 0.219 0.344
Earning
Heart 0.428 0.485 0.326 0.681 0.546 0.475 0.297
Disease
IRIS 0.307 0.311 0.178 0.134 0.009 0.406 0.408
30
SP-Lime SP-Lime
Guided-LIME Guided-LIME
#Artificial Feature Count
#Artificial Feature Count
40
20
20
10
0 0
AF-1_Imp-1 AF-1_Imp-2 AF-1_Imp-1 AF-1_Imp-2
Figure 7: FDR (False discovery rate) for Heart disease Figure 8: FDR (False discovery rate) for Adult earning
dataset dataset
4.3. Validating Guided-LIME using ror provide a better estimate of reliability of ML mod-
calibration level els [21, 22]. Moreover, the focus of our experiment is to
estimate the proximity of the shadow model w.r.t the
The objective of this experiment is to validate which original blackbox model. Calibration error values are
proxy model is a closer approximation to original black- therefore used to compare which model is the better
box model with respect to the prediction probabilities approximation of the original model. We hypothesize
of each model. In order to measure this closeness, var- that the proxy model with a ECE closer to the original
ious distance metric can be used e.g. KL divergence, blackbox ML model shall be a closer approximate.
cross entropy etc. We use the well established ECE We perform experiment in two settings: 1) with orig-
(expected calibration error) and MCE (maximum cali- inal data 2) by adding noisy features in the data. As
bration error) as the underlying metric to detect the shown in Tables 2 and 3, in both scenarios, ECE and
calibration of both the models [20]. Calibration er-
MCE of Guided-LIME is closer to the original ML model SHAP need to run for every instance. This generates a
in comparison to the random SP-LIME. This justifies matrix of Shapley values which has one row per data
the benefit of structured sampling. We also run ex- instance and one column per feature. We can inter-
periments with full samples of LIME. Although, this pret the entire model by analyzing the Shapley values
can be a better approximate of original model, but tak- in this matrix.
ing all the samples in the proxy model is not a practi- In CAM and Grad-CAM approaches, explanation is
cal and economic choice for real world huge datasets. provided by using a Saliency Mask (SM), i.e. a subset
Guided-LIME has a closer ECE to the original black- of the original record which is mainly responsible for
box model. Hence, Guided-LIME is a better choice as the prediction. For example, as salient mask we can
a proxy model to explain the original ML model. consider the part of an image or a sentence in a text.
A saliency image summarizes where a DNN looks into
an image for recognizing their predictions. Although
5. Related Work these solutions are not just limited/agnostic to black-
box NN, but it requires specific architectural modifica-
Various approaches for explainability of blackbox mod-
tions.
els have been proposed [8]. Broadly the existing tech-
Feature importance is well known approach to ex-
niques can be classified into Model Explanation ap-
plain blackbox models. More recently, instance-wise
proaches; outcome Explanation approaches; Model In-
feature selection methods are proposed to extract a
spection approaches. There are also example of works
subset of features that are most informative for each
that focus on designing transparent design of models.
given example in deep learning network. [29]. In [30]
In this work, we focus only on the outcome explana-
authors make use of a combination of neural networks
tion approaches. In the category of outcome explana-
to identify prominent features that impact the model
tion, CAM, Grad-CAM, Smooth Grad-CAM++, SHAP,
accuracy. These approaches are based on subset sam-
DeepLIFT, LRP and LIME are the main approaches [23,
pling through back-propagation.
24, 25, 9, 26, 27, 28]. These methods provide a locally
Ribeiro et. al. [9] present the Local Interpretable
interpretable shadow model which is able to explain
Model-agnostic Explanations (LIME) approach which
the prediction of the blackbox in understandable terms
does not depend on the type of data, nor on the type
for humans.
of blackbox b to be opened. In other words, LIME can
Most popular shadow model approaches for black-
return an understandable explanation for the predic-
box ML model explanations are Local Interpretable Model-
tion obtained by any blackbox. The main intuition of
Agnostic Explanations (LIME) and SHAP. LIME can
LIME is that the explanation may be derived locally
explain the predictions of any classifier in “an inter-
from the records generated randomly in the neighbor-
pretable and faithful manner, by learning an interpretable
hood of the record to be explained. As blackbox the
model locally around the prediction. In order to make
following classifiers are tested: decision trees, logistic
the predictions easily interpretable, LIME have two de-
regression, nearest neighbors, SVM and random for-
sign goals: Easy to interpret and Local fidelity: This
est.
means that outcomes of shadow model are easily inter-
In [31], authors find the global importance intro-
pretable and the explanation for individual predictions
duced by Local Interpretable Model-agnostic Explana-
are locally faithful, i.e. it correspond to how the model
tions (LIME) unreliable and present approach based on
behaves in the vicinity of the individual observation
global aggregations of local explanations with the ob-
being predicted.
jective to provide insights in a model’s global decision
In contrast, SHAP (SHapley Additive exPlanations)
making process. This work reveal that the choice of
is distinctly built on the Shapley value. The Shapley
aggregation matters regarding the ability to gain reli-
value is the average of the marginal contributions across
able and useful global insights on a blackbox model.
all permutations. The Shapley values consider all pos-
We find this work as motivation to propose an hybrid
sible permutations, thus SHAP is a united approach
approach where aggregations can be generated using
that provides global and local consistency and inter-
knowledge of data through FCA-based system.
pretability. However, its cost is time — it has to com-
In contrast to model explanation approaches such
pute all permutations in order to give the results. SHAP
as LIME and SHAP [9, 26], our approach is comple-
approach has speed limitations as it has to compute all
mentary which can guide these approaches for select-
permutations globally to get local accuracy whereas
ing the optimal instances for explanation. Extracting
LIME perturbs data around an individual prediction
rules from neural networks is also a well studied prob-
to build a model. For generating a global explanation,
lem [32]. These approaches depend on various factors
such as: Quality of the rules extracted; Algorithmic Proceedings of the 7th International Workshop
complexity; Expressive power of the extracted rules; "What can FCA do for Artificial Intelligence"? co-
Portability of the rule extraction technique etc. Our located with IJCAI 2019, Macao, China, ????
approach also uses the knowledge of structure in data [5] V. Petsiuk, R. Jain, V. Manjunatha, V. I. Morariu,
however it is not dependent on the blackbox model. A. Mehra, V. Ordonez, K. Saenko, Black-box ex-
Moreover, formal concept analysis based data analysis planation of object detectors via saliency maps,
provides a solid theoretical basis. 2020. arXiv:2006.03204.
[6] J. Pfau, A. T. Young, M. L. Wei, M. J. Keiser, Global
saliency: Aggregating saliency maps to assess
6. Conclusions and Future Work dataset artefact bias, 2019. arXiv:1910.07604.
[7] R. Iyer, Y. Li, H. Li, M. Lewis, R. Sundar, K. Sycara,
In this paper,we proposed a hybrid approach for eval-
Transparency and explanation in deep reinforce-
uating interpretability of blackbox ML systems. Al-
ment learning neural networks, in: Proceedings
though Guided-LIME do not guarantee an optimal so-
of the 2018 AAAI/ACM Conference on AI, Ethics,
lution, yet we observe that a single approach like LIME
and Society, AIES ’18, Association for Comput-
is not sufficient to explain the AI system thoroughly.
ing Machinery, New York, NY, USA, 2018, p.
There are limitations of deciding an optimal sampling
144–150. URL: https://doi.org/10.1145/3278721.
criteria in SP-Lime algorithm. Our approach combines
3278776. doi:10.1145/3278721.3278776.
the benefits of using a data-based approach (FCA) and
[8] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini,
proxy model based approach (LIME). Overall, our ap-
F. Giannotti, D. Pedreschi, A survey of meth-
proach is complementary to SP-LIME as we provided
ods for explaining black box models, ACM Com-
a structured way of selecting right instances for global
put. Surv. 51 (2018). URL: https://doi.org/10.1145/
explanations. Our results on real world datasets shows
3236009. doi:10.1145/3236009.
that false discovery rate is much lower with Guided-
[9] M. T. Ribeiro, S. Singh, C. Guestrin, "why should
LIME in comparison to random SP-LIME. Moreover,
I trust you?": Explaining the predictions of any
Guided-LIME has a closer ECE and MCE to the orig-
classifier, in: Proceedings of the 22nd ACM
inal blackbox model. In future, we would like to per-
SIGKDD International Conference on Knowl-
form extensive experiments with diverse datasets and
edge Discovery and Data Mining, San Francisco,
complex deep learning models.
CA, USA, August 13-17, 2016, 2016, pp. 1135–
1144.
References [10] C. Molnar, Interpretable Machine Learn-
ing, 2019. https://christophm.github.io/
[1] M. Miliard, Google, verily using ai to screen interpretable-ml-book/.
for diabetic retinopathy in india, 2019. URL: [11] D. Alvarez-Melis, T. S. Jaakkola, On the ro-
https://www.healthcareitnews.com/news/ bustness of interpretability methods (2018).
URL:
google-verily-using-ai-screen-diabetic-retinopathy-india. http://arxiv.org/abs/1806.08049, cite
[2] J. Wagner, J. M. Kohler, T. Gindele, L. Het- arxiv:1806.08049Comment: presented at 2018
zel, J. T. Wiedemer, S. Behnke, Interpretable ICML Workshop on Human Interpretability
and fine-grained visual explanations for convo- in Machine Learning (WHI 2018), Stockholm,
lutional neural networks, in: Proceedings of the Sweden.
IEEE/CVF Conference on Computer Vision and [12] S. O. Kuznetsov, Fitting pattern structures to
Pattern Recognition (CVPR), 2019. knowledge discovery in big data, in: P. Cellier,
[3] H. Lakkaraju, S. H. Bach, J. Leskovec, Inter- F. Distel, B. Ganter (Eds.), Formal Concept Anal-
pretable decision sets: A joint framework for de- ysis, Springer Berlin Heidelberg, Berlin, Heidel-
scription and prediction, in: Proceedings of the berg, 2013, pp. 254–266.
22nd ACM SIGKDD International Conference on [13] X. Benavent, A. Castellanos, E. de Ves, A. García-
Knowledge Discovery and Data Mining, Associ- Serrano, J. Cigarrán, Fca-based knowledge
ation for Computing Machinery, New York, NY, representation and local generalized linear
USA, 2016. models to address relevance and diversity
[4] A. Sangroya, C. Anantaram, M. Rawat, M. Ras- in diverse social images, Future Generation
togi, Using formal concept analysis to explain Computer Systems 100 (2019) 250 – 265. URL:
black box deep learning classification models, in: http://www.sciencedirect.com/science/article/
pii/S0167739X18307271. doi:https://doi.
org/10.1016/j.future.2019.05.029. ence on Computer Vision (ICCV), 2017.
[14] R. Kerber, Chimerge: Discretization of numeric [25] D. Omeiza, S. Speakman, C. Cintas, K. Welder-
attributes, in: Proceedings of the Tenth National mariam, Smooth grad-cam++: An enhanced in-
Conference on Artificial Intelligence, AAAI’92, ference level visualization technique for deep
AAAI Press, 1992, p. 123–128. convolutional neural network models, 2019.
[15] S. Yevtushenko, J. Tane, T. B. Kaiser, S. Obied- arXiv:1908.01224.
kov, J. Hereth, H. Reppe, Conexp - the con- [26] S. M. Lundberg, S.-I. Lee, A unified ap-
cept explorer (2000-2006). URL: http://conexp. proach to interpreting model predictions, in:
sourceforge.net. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach,
[16] R. Fisher, Iris data set, 2019. URL: https://archive. R. Fergus, S. Vishwanathan, R. Garnett (Eds.),
ics.uci.edu/ml/datasets/iris. Advances in Neural Information Processing
[17] D. W. Aha, Heart disease data set, 2019. URL: Systems 30, Curran Associates, Inc., 2017, pp.
https://archive.ics.uci.edu/ml/datasets/Heart+ 4765–4774. URL: http://papers.nips.cc/paper/
Disease. 7062-a-unified-approach-to-interpreting-model-predictions.
[18] R. El-Bialy, M. A. Salamay, O. H. Karam, M. E. pdf.
Khalifa, Feature analysis of coronary artery heart [27] A. Shrikumar, P. Greenside, A. Kundaje, Learn-
disease data sets, Procedia Computer Science 65 ing important features through propagating acti-
(2015) 459 – 468. URL: http://www.sciencedirect. vation differences, 2017. arXiv:1704.02685.
com/science/article/pii/S1877050915029622. [28] A. Binder, G. Montavon, S. Bach, K.-R. Müller,
doi:https://doi.org/10.1016/j.procs. W. Samek, Layer-wise relevance propagation for
2015.09.132, international Conference on neural networks with local renormalization lay-
Communications, management, and Information ers, 2016. arXiv:1604.00825.
technology (ICCMIT’2015). [29] J. Chen, L. Song, M. J. Wainwright, M. I.
[19] D. D. Sarkar, Hands-on machine Jordan, Learning to explain: An information-
learning model interpretation, 2018. theoretic perspective on model interpretation,
URL: https://towardsdatascience.com/ 2018. arXiv:1802.07814.
explainable-artificial-intelligence-part-3-hands-on-machine-learning-model-interpretation-e8ebe5afc608.
[30] J. Yoon, J. Jordon, M. van der Schaar, IN-
[20] C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, VASE: instance-wise variable selection using
On calibration of modern neural networks, in: neural networks, in: 7th International Confer-
Proceedings of the 34th International Confer- ence on Learning Representations, ICLR 2019,
ence on Machine Learning - Volume 70, ICML’17, New Orleans, LA, USA, May 6-9, 2019, Open-
JMLR.org, 2017, p. 1321–1330. Review.net, 2019. URL: https://openreview.net/
[21] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, forum?id=BJg_roAcK7.
D. Tran, Measuring calibration in deep learn- [31] I. van der Linden, H. Haned, E. Kanoulas, Global
ing, in: Proceedings of the IEEE/CVF Confer- aggregations of local explanations for black box
ence on Computer Vision and Pattern Recogni- models, ArXiv abs/1907.03039 (2019).
tion (CVPR) Workshops, 2019. [32] R. Andrews, J. Diederich, A. Tickle, Survey and
[22] V. Kuleshov, N. Fenner, S. Ermon, Accu- critique of techniques for extracting rules from
rate uncertainties for deep learning using cal- trained artificial neural networks, Knowledge-
ibrated regression, in: J. Dy, A. Krause Based Systems 6 (1995) 373–389. doi:10.1016/
(Eds.), Proceedings of the 35th International 0950-7051(96)81920-4.
Conference on Machine Learning, volume 80
of Proceedings of Machine Learning Research,
PMLR, Stockholmsmässan, Stockholm Sweden,
2018, pp. 2796–2804. URL: http://proceedings.
mlr.press/v80/kuleshov18a.html.
[23] B. Zhou, A. Khosla, L. A., A. Oliva, A. Torralba,
Learning Deep Features for Discriminative Lo-
calization., CVPR (2016).
[24] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedan-
tam, D. Parikh, D. Batra, Grad-cam: Visual expla-
nations from deep networks via gradient-based
localization, in: The IEEE International Confer-