Introduction
Nowadays, Arti cial Intelligence is one of the most important scienti c and
technological areas, with a huge socio-economic impact and a pervasive adoption in
every eld of the modern society. High-pro le applications such as autonomous
vehicles, medical diagnosis, spam ltering, image recognition, and voice
assistants are based on Arti cial Intelligence (AI) systems. Modern AI is mainly
based on Machine Learning models that allow AI systems to reach impressive
performance in emulating human behavior. The most e ective ML models are
black-box models [
18
], i.e., obscure decision-making or predictive methods that
\hide" the logic of their internal decision processes to humans, either because not
human-understandable or because not directly accessible. Examples of black-box
models include Neural Networks and Deep Neural Networks, SVMs, Ensemble
classi ers such as Random Forest, but also compositions of expert systems, data
mining, and hard-coded software. The choice for the adoption of these obscure
models is driven by the high performance in terms of accuracy [
36
]. As a
consequence, the last decade has witnessed the rise of a black-box society [
27
].
The lack of explanations of how these black-box models make decisions is
a restriction for their adoption in safety-critical contexts and socially sensitive
domains such as healthcare or law. Moreover, the problem is not only for lack of
transparency but also for possible biases inherited by black-box models from
artifacts and preconceptions hidden in the training data of the ML algorithms.
Predictive ML models learned on biased datasets may inherit such biases, possibly
leading to unfair and wrong decisions. Consequences of biased misclassi cations
can damage decision-makers and put certain societal groups at risk [
9, 28, 39
]
For instance, the AI software used by Amazon to determine the areas of the US
to which Amazon would o er free same-day delivery, unintentionally restricted
minority neighborhoods from participating in the program (often when every
surrounding neighborhood was allowed)1. Another example is relative to
propublica.org. Their journalists have shown that the COMPAS score, a predictive
model for the \risk of crime recidivism" (proprietary secret of Northpointe),
has a strong ethnic bias. Indeed, according to this score, a black who did not
re-o end was classi ed as \high risk" twice as much as whites who did not
reo end. On the other hand, white repeat o enders were classi ed as \low risk"
twice as much as black repeat o enders2. In [
9
] is shown that the neural network
used to train the English language words was encoding biases towards gender
and stereotypes. The authors show that for the analogy \Man is to computer
programmer as woman is to X", the variable X was replaced by \homemaker"
by the neural network. Consequently, the research in eXplainable AI (XAI) and
on the study of explanation methods for obscure ML models has recently caught
much attention [
1, 5, 18, 26, 40
].
In addition, an innovative aspect of the General Data Protection Regulation
(GDPR) promulgated by the European Parliament, which has become law in
May 2018, are the clauses on automated decision-making. The GDPR, for the
rst time, introduces, to some extent, a right of explanation for all individuals to
obtain \meaningful explanations of the logic involved" when automated decision
making takes place. Despite con icting opinions among legal scholars regarding
the real scope of these clauses [
15, 24, 37
], there is a joint agreement on the need
for the implementation of such a principle is imperative and that it represents
today a huge open scienti c challenge. However, without technology capable of
explaining the logic of black boxes, the right to explanation will remain a \dead
letter". How can companies trust their AI services without understanding and
validating the underlying rationale of their ML components? Furthermore, in
turn, how can users trust AI services? It will be impossible to increase the trust
of people in AI without explaining the rationale followed by these models. These
are the reasons why explanation is now at the heart of responsible, open data
science across multiple industry sectors and scienti c disciplines.
2
Explanation Methods
A black-box predictor is a ML obscure model, whose internals are unknown to
the observer, or they are known but uninterpretable by humans [
18
]. Therefore,
1 http://www.techinsider.io/how-algorithms-can-be-racist-2016-4
2 http://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
in ML to interpret means to give or provide the meaning or to explain in
understandable terms the predictive process of a model to a human [
5, 13
]. It is
assumed that the concepts composing an explanation are self-contained and do
not need further explanations [
18
]. The most widely used approach to explain
black-box models and return interpretations is a sort of reverse engineering : the
explanation is learned by observing the changes in the black-box output by
varying the input. A set of dimensions are identi ed to analyze ML interpretability,
and explanation methods and, in turn, re ect on existing types of explanations.
Explanation Problems. In the literature, we recognize two types of
problems: black box explanation and explanation by design [
18
]. The black-box
explanation idea is to couple a ML black-box model with an explanation method
able to interpret the black-box decisions. The underlying strategy is to maintain
the high performance of the obscure model and to use an explanation method to
retrieve the explanations [
12, 23, 29
]. The explanation methods generally try to
approximate the black-box behavior with an interpretable predictor, also named
surrogate model. This kind of approach is the one more addressed nowadays in
the XAI research eld. On the other hand, the explanation by design consists of
directly designing a transparent model that is interpretable by design and aims
at replacing the obscure ML model with the new transparent one [
32, 33
].
In the literature, there are various models recognized to be interpretable.
Examples are decision tree, decision rules, and linear models [
14
]. These models
are considered easily understandable and interpretable for humans. They
sacrice performance for interpretability. Besides, most of them cannot be applied to
data types such as images or text, but only on tabular data.
Explanation Targets and Strategy. We recognize global and local
explanation methods depending on the target of the explanation. A global explanation
consists in providing an explanation that allows understanding the whole logic
of a black-box model and interpreting any possible decision. Global explanations
are di cult to achieve, and in the literature are provided only for tabular data.
On the other hand, a local explanation consists in retrieving the reasons for the
prediction returned by a black-box model for a speci c case. While for a global
explanation, the interpretable surrogate approximates the whole black-box, for a
local explanation, the interpretable surrogate model is used to approximate the
black-box behavior only in a \neighborhood" of the instance analyzed. The idea
is that, in such a neighborhood, it is easier to explain the decision boundary [
29
].
In addition, we distinguish between model-speci c and model-agnostic
explanation method depending on the strategy adopted. An explanation method
is model-speci c, or not generalizable [
25
], if it can be used to interpret only
particular types of black-box models. If an explanation method is designed to
interpret a Random Forest [
36
] and internally use a distance between trees, such
a method cannot be used to explain the predictions of a Neural Network. On the
other hand, a generalizable or model-agnostic explanation method can be used
independently from the black-box model being explained because the internal
characteristics of the black-box are not exploited to retrieve the explanation [
29
].
Desiderata of Explainable Methods. A set of desiderata should be
considered when designing and using explanation methods [
14
]. The interpretability
aspect should measure to what extent a given explanation is
human-understandable. Interpretability is generally evaluated with the complexity of the
interpretable surrogate model. For example, the complexity of a rule can be measured
with the number of clauses in the condition, for linear models with the number
of non-zero weights, while for decision trees with the depth of the tree. The
performance of the interpretable surrogate model form which explanations are
extracted is generally called delity and measures to which extent it accurately
imitates the black-box prediction. The delity is practically measured in terms
of Accuracy score, F1-score, etc. [
36
] with respect to the prediction of the
blackbox model. Moreover, an interpretable model should satisfy guarantee fairness
by protecting minorities against discrimination [
31
], and privacy by not revealing
sensitive information [
2
]. Also, an explanation methods must return robust and
stable explanations: similar instances should have similar explanation for a given
black-box model [
19
]. In addition, since the meaningfulness of an explanation
depends on the stakeholder [
7
], the explanation returned must consider the user
background : common users require simple clari cations, while domain experts
can be able to understand complex explanations. Finally, the time that a user
is allowed to spend on understanding an explanation is another crucial aspect.
In contexts where the decision time is not a constraint, one might prefer a more
exhaustive explanation, while when the user needs to quickly make a decision, it
is preferable to have an explanation \easy to read". Thus an explanation method
must consider time limitations.
Types of Explanations. Research on XAI is producing various alternatives.
Explanation methods di er one from another depending on the type of
explanation returned. In the following, we illustrate the most used types of explanations
and highlights how explanation methods build them.
{ List of Rules. An explanation returned in the form of a list of rules implies
that rules are read one after the other, and the rst rule for which the
conditions are veri ed is used for prediction. Rules are in form of if-then rule:
if conditions, then consequent the consequent corresponds to the prediction,
while the conditions explain the factual reasons for the consequent. The
CORELS method [
3
] is a transparent by design method able to build a list
of rules with the aim of globally replacing the black-box model. A compact
set of rules is returned by the transparent predictive method proposed in [
21
].
{ Single Tree Approximation. The black-box predictor is approximated
with a decision tree that represents all the possible decisions. The TREPAN
explanation method [
12
] allows to globally explore a Neural Network through
a tree structure that, starting from a root, shows for every path the
conditions driving the decision process. TREPAN retrieves the decision tree by
maximizing a gain ratio [
36
] calculated on the delity with respect to the
predictions of an obscure Neural Network.
{ Rule-based Explanation. A single if-then rule is used for local
explanations. The conditions of the rule explain the factual reasons for the
prediction. The LORE explanation method [
17
] builds a local decision tree in
the neighborhood of the instance analyzed, and then extracts from the tree a
single rule revealing the reasons for the decision on the speci c instance. The
ANCHOR method [
30
] returns if then rules called anchors. An anchor
contains a set of attributes with the values which are fundamental for obtaining
a certain prediction.
{ Features Importance. A feature importance-based local explanation
consists of attributes equipped with positive and negative values. The
explanation consists of both the sign and the magnitude of the contribution of the
attributes for a speci c prediction. If the value is positive, then it contributes
by increasing the model's output, if the sign is negative, it decreases the
output of the model. LIME [
29
] adopts a linear model as the interpretable local
surrogate and returns the importance of the features as an explanation
exploiting the regression's coe cients. SHAP [
23
] provides the local unique
additive feature importance for a speci c record exploiting shapely values.
{ Saliency Maps. In image processing, typical explanations consist of saliency
maps, i.e., images that show the positive (or negative) contribution of each
pixel to the black-box prediction. Saliency maps are built for locally
explaining DNN models by gradient [
34, 35
] and perturbation-based [
6
] attribution
methods. These explanation methods assign a score to each pixel such that it
is maximized the probability of returning the same answer without
considering irrelevant pixels. Under appropriate image transformations that exploit
the concept of \superpixels" also methods such as LORE and LIME can be
employed to explain black-box working on images.
{ Prototype-based Explanations. An explanation based on prototypes
returns specimens similar to the instance analyzed, which makes clear the
reasons for the prediction. Prototype-based explanations can refer to any type of
data. In [
11, 22
], image prototypes are used as the foundation of the concept
for interpretability [
8
]. In [
20
] is discussed the concept of counter-prototypes
called criticisms for tabular data, i.e., prototype showing what should be
different to obtain another decision. Exemplar and counter-exemplars synthetic
images are generated by the ABELE explanation method [
16
] to augment
the interpretability of local saliency maps.
{ Counterfactual Explanations. A counterfactual explanation shows what
should have been di erent to change the prediction of the black-box model.
Counterfactuals help people in reasoning on the cause-e ect relations
between observed features and classi cation outcomes [
4, 10
] and reveal what
should change in a given instance to obtain a di erent prediction [
37
]. The
explanation method proposed in [
38
] returns counterfactual explanations that
describe the smallest change that can be made to a given instance to obtain
a certain outcome by solving an optimization problem. The aforementioned
LORE [
17
], besides a factual explanation rule, also provides a set of
counterfactual rules extracted from the local decision tree, while ABELE [
16
]
returns synthetically generated counter-exemplar images.