Despite the widespread use, machine learning methods produce black box models. It is hard to understand how features in uence the model prediction. We propose a novel explanation method that explains the predictions of any classi er by analyzing the prediction change obtained by omitting relevant subsets of attribute values. The local internal logic is captured by learning a local model in the neighborhood of the prediction to explain. The explanations provided by our method are e ective in detecting associations among attributes and class label.
Machine learning algorithms are widely applied in every aspect of our society.
Their growing popularity and their widespread use have made it increasingly
important to understand why a classi cation model take a particular decision. The
call for more explainable predictions comes also for institution. The European
Union approved the GDPR, a regulation for ensuring personal data protection.
It states that individuals have the right to receive \meaningful information about
the logic involved" in case of automated decision-making. For some authors, this
requirement legally mandates a \right to explanation" [
We propose a novel model-agnostic explanation method that explains the predictions made on single instances by any classi er. The explanation highlights the internal logic of the model in a neighborhood of the prediction. It is based on the knowledge of the local behavior of the model, captured by an interpretable local model. Many algorithms have been proposed for improving the interpretability of already existing classi cation models. Model-dependent solutions are proposed for handling only speci c models. Model-agnostic solutions instead treat the machine learning model as a black box. Our research is focused on these approaches for their general applicability and the advantages derived from it. These methods in fact are applicable to any classi cation methods without making any assumption on their internal logic. Thus, the comparison among di erent techniques in terms of model interpretability is possible.
While some approaches try to explain the original model globally, others
propose a general method for explaining individual predictions, i.e. why particular
decisions are made. Ribeiro et al. [
Several works study how a prediction changes if parts of the input
components are omitted. Lemaire et al. [
We propose a novel method applicable to explain individual predictions of any classi cation method. Given the particular prediction that we want to explain, we omit one or more attribute values at a time and we measure how the prediction changes. The relevance of the change is estimated as a di erence of prediction probabilities with respect to a particular target class. The greater is this di erence, the more the omitted attribute values are relevant for the prediction.
With respect to existing approaches, we are interested in understanding not only how each single attribute value is signi cant for the prediction but also how it interacts with the others. An attribute value can determine the prediction alone or only if it is in conjunction with others. In the latter case, we need to omit more attributes at the time for observing how the prediction changes. Omitting sets of attribute values allows also to deal with the disjunction case. The disjunction condition occurs when more than one con gurations of attributes values determines the prediction. We can observe a change of the prediction probability only if we considered the omission of the attribute values together. a=1 b=1 c=2 d=3 e=1 f=2 a, b a, b, e a=1 b=1 c=2 d=3 e=1 f=2 Deriving Local Internal Logic for Black Box Models
The feature values that in conjunction or disjunction are relevant for a prediction are highlighted by a local interpretable model. The local model is an associative classi er learned in the neighborhood of the prediction that we want to explain. The local rules, being understandable, provide preliminary insights of why a decision is made by the considered model. Each rule is in AND form. Thus, it gives the information of what attributes together determine the prediction. Moreover, a prediction may be determined by more rules. We can deal with the disjunction condition considering jointly the omission of the subsets highlighted by the local rules. Only the relevant attribute subsets provided by the local model are considered, instead of the complete power set of all attribute combinations. Hence, our approach overcomes the exponential time complexity. 4
Preliminary Results dataset=monks-1 model=ANN p(class=1jx)=0.999 true class=1 dataset=monks-1 model=NB p(class=1jx)=1 true class=1 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 4 - target class=1 4 - target class=1
(a) (b) Fig. 1: Explanation of (a) the neural network and (b) the Naive Bayes prediction of a particular instance of the monks-1 data set.
In this section, preliminary outcomes of our novel explanation method are
presented. The Monk1 data set is an arti cial data set composed by 6 discrete
attributes a,b,c,d,e,f and the class label can take value 1 or 0 [
If we explain the prediction for the same instance, made by another model, we may obtain a di erent result. The explanation should capture how the model behaves in the locality of the instance. Di erent models work di erently. This di erence may be on the predicted class label, but also on the feature values that drive the prediction. Consider the explanation of the same instance x and still built with respect to class 1, but classi ed by the Naive Bayes classi er (NB). The local model returns a single relevant rule: fe = 1g ! class = 1. The results are shown in Figure 1b. The NB classi er assigns correctly the instance x to class 1, but only because e=1. The local model and the explanation highlight that the Naive Bayes classi er has not learned the association that if a=b then class=1. Because of its assumption of independence between features, it is not able to learn the importance that a and b have together. Hence, the local model and the explanation in this case successfully re ect the model behavior. 5
Preliminary tests show that our technique is able to capture the diverse internal logic of classi cation techniques. Di erently than existing approaches, the importance of relevant subsets of feature values to the prediction is computed. Thus, our method provides to end users the information of what attributes together determine the prediction and the quanti cation of their in uence.
As future work we plan to (i ) formalize our approach proposing formal de nitions of the prediction change estimation, (ii ) evaluate the e ect of the neighborhood in the local rules and the resulting explanations and (iii) apply the proposed method to real-world data sets, validating explanations through the assistance of domain experts.
Acknowledgements This work is partially funded by SmartData@PoliTO.