From the user's perspective, interaction in active learning is very opaque: the user only sees a sequence of instances to be labeled, and has no idea what the model believes or how it behaves. Explanatory active learning (XAL) tackles this issue by making the model predict and explain its own queries using local explainers. By witnessing the model's (lack of) progress, the user can decide whether to trust it. Despite their promise, existing implementations of XAL rely on post-hoc explainers, which can produce unfaithful and fragile explanations, which misrepresent the beliefs of the predictor, confuse the user, and affect the quality of her supervision. As a remedy, we replace post-hoc explainers with selfexplainable models, and show how these can be actively learned from both labels and corrected explanations. Our preliminary results showcase the dangers of post-hoc explanations and hint at the promise of our solution.
Explainable machine learning has so far mostly focused on black-box models
learned offline [
Explanatory active learning (XAL) tackles this issue by injecting
explanations into the learning loop [
or revoke trust to it. This is analogous to trust between individuals, which
requires to develop appropriate expectations through interaction [
Existing implementations of XAL make use of post-hoc local explainers—
for instance, caipi [
As a remedy, we propose replacing post-hoc explainers with self-explainable
neural networks (SENNs), a recently proposed class of models that automatically
explain their own predictions. Intuitively, SENNs combine the transparency of
linear models with the flexibility of neural nets [
Summarizing, we: 1) highlight the risks of unfaithful explanations in interactive learning; 2) propose a novel implementation of XAL based on selfexplainable neural nets; 3) propose a joint loss to learn SENNs from labels and corrections directly; 4) report on preliminary experiments that showcase the behavior of post-hoc explainers and the promise of our solution. 2
In the following, we will stick to binary classification and indicate instances as x ∈ X and labels as y ∈ Y = {0, 1}. Our observations can be easily generalized to the multi-class case. 2.1
Given a classifier f : X → Y, for instance a neural network or a random forest,
post-hoc local explainers explain individual predictions without looking at the
exact inference steps performed by f [
In order to explain a prediction y0 = f (x0), lime learns an interpretable
local model g0 that mimics f in the neighborhood of x0, and then reads off an
explanation from it. The local model is a sparse linear predictor or a shallow
decision tree [
More formally, the process amounts to:
1. Sampling s interpretable instances ξ1, . . . , ξs by randomly perturbing the
interpretable representation of x0, namely ξ0 = ψ(x0);
2. Labeling each instance ξi using the target model yi = f (xi), where xi =
ψ−1(ξi) is the pre-image of ξi;
3. Weighting each example (ξi, yi) by its similarity to ξ0, i.e., k(ξi, ξ0); the
kernel function k determines the size and shape of the neighborhood of ξ0;
4. Fitting a local model g0 on the synthetic dataset via cost-sensitive learning,
so that examples outside of the neighborhood do not have much of an impact;
this is a form of model translation [
Pool-based active learning (AL) is designed for settings where labels are scarce
and expensive to obtain [
In order to make the interaction more transparent and directable,
explanatory active learning (XAL) injects explanations into the learning loop and
enables the user to interact with them [
By combining interactions and explanations, XAL helps the user to build a mental model of the predictor being learned, which is necessary for justifiably according trust to it. In addition, by virtue of learning from explanation corrections, XAL makes it less likely that the model learns to predict the right labels for the wrong reasons. We will see, however, that these promises can be difficult to keep when post-hoc explainers are involved. 2.3
Sparse linear models over interpretable features are canonically considered to be highly interpretable. These models have the form1 f (x) = σ(w>φ(x)), where σ is a sigmoid function, w ∈ Rn is constant, and φ : X → Rn is a fixed, interpretable feature map. The contribution of the jth feature to the output of the predictor is determined by the corresponding weight. Despite their interpretability, sparse linear models are limited to relatively simple learning tasks.
Self-explainable neural networks (SENNs) upgrade linear models by
substantially increasing their capacity and flexibility while preserving their
interpretability [
This is guaranteed by constraining w to vary slowly with respect to φ.
More formally, w is required to be locally difference bounded by φ, as per the
following definition:
1 Here and below, the bias term is left implicit.
Definition 1 ([
kx − x0k < δ =⇒ kf (x) − f (x0)k ≤ Lkg(x) − g(x0)k In other words, for every point x0 there is a neighborhood within which2 the change in f is bounded by the change in g. Notice that the local Lipschitz “constant” L is allowed to vary with x0.
This requirement is enforced during learning by penalizing the model for any deviations from linearity through the following regularization term: Ω(f ) d=ef k∇xf (x) − w(x)>Jxk where Jx is the Jacobian matrix. The model f is learned by minimizing the empirical risk of some classification loss `Y plus the above regularizer, i.e.: minf Eˆ(x,y) [`Y (f (x), y) + αΩ(f )] (2) Here Eˆ indicates expectation over mini-batches and α ≥ 0 is a hyperparameter. As usual with neural networks, minimization is performed with gradient descent techniques. Of course, an additional term encouraging the output of w(x) to be sparse can be considered. We will see later on that this is automatically the case in our extension of SENNs to explanatory active learning.
It is worth pointing out that SENNs are quite different from attention models, which are yet another way of explaining (to some extent) neural networks. Indeed, the former explain exactly which features contribute to the prediction, as well as their polarity. The features themselves are arbitrary (interpretable) functions of the inputs. Attention models, in contrast, identify which input features (e.g. pixels) are relevant, but not how they contribute to the decision nor whether they are against/in favor. 3 3.1
Let us start by discussing the case of lime, which is at the core of existing XAL implementations. In this case, the accuracy of the model translation process (described above) depends critically the choice of model class of g0, interpretable features ψ, kernel k, and number of samples s.
For instance: 1. If the kernel k is not chosen correctly (i.e. if it is too small, too broad, or has the wrong topology), then the synthetic dataset may fail to capture label changes around x0, as shown in Figure 1. 2 This can be limiting in classification scenarios, because the output should change abruptly when crossing the decision boundary. The formulation can be modified to account for this, but we keep it as is, for simplicity.
In all these cases, lime produces high-variance explanations that substantially misrepresent the behavior of f . Increasing the number of samples may improve the situation, but also the runtime, and may not be enough to stabilize the explanations anyway.
Crucially, the same high-level argument applies to all post-hoc explainers, because all of them treat f as a black-box and thus—by construction—must include an inexact model translation step.
From the standpoint of XAL, unfaithfulness has two major consequences. First, high-variance explanations portray the model as behaving semi-randomly, regardless of whether it is good or not. The user may also perceive that her supervision has no effect, and feel lack of control. In rare cases, caipi may also accidentally persuade the user into trusting a bad model. Both cases are problematic in sensitive applications, like the ones that XAL is designed for. More generally, unfaithful explanations misrepresent the learned model, defeating the purpose of explanatory active learning. Second, unfaithful explanations compromise the usefulness of the corrections. If the user is confused by the explanations, her corrections will be not as informative. Further, the correction may specify not to use a feature that the model is not using anyway (or vice versa). Depending on the model being learned, correcting the same feature too many times may also lead to learning instabilities.
It is therefore desirable to fix the XAL pipeline to rely on faithful and trustable explanations. In the next section, we show how to do so. Algorithm 1 Pseudocode of cali: L is the set of labeled examples, U is the set of unlabeled instances, and T is the query budget. Our proposed algorithm—dubbed Calimocho, or cali for short—follows closely the XAL learning loop; the pseudocode is listed in Algrithm 1. Notice that f here is a SENN. Explanation corrections are collected in a set C, initially empty. In each iteration, the algorithm chooses an instance x ∈ U (using uncertainty sampling, as caipi, for simplicity), predicts its class yˆ, and generates an explanation zˆ for the prediction (line 5). The explanation zˆ is simply read off of w(x), i.e., zˆ = w(x), without any projection or sampling step. All components are presented to the user (lines 6–7), who replies with the true label y of x and optionally with an improved explanation z¯. Finally, the dataset is updated and the preference z¯ zˆ is added to the corrections C.
The learning step requires to strategy to train SENNs using corrections too.
One option is to follow caipi, an use counter-examples. However, these only
approximately capture the constraint imposed by the correction, e.g., that the
label should not depend on a particular interpretable feature. Depending on the
application, there is also a (slim) chance that the counter-examples are actually
wrong. Feature influence supervision solves this issue by asking the user [
Instead, we opt for learning SENNs directly from corrections, as follows. Recall that, given an explanation zˆ, a correction specifies, e.g., which features are erroneously being used by the model. Applying the correction to zˆ leads to a corrected explanation z¯. It is therefore natural to impose that w(x) generates explanations that are closer to the corrected explanation rather than the predicted on. This can be accomplished by, e.g., minimizing the squared Euclidean distance kw(x) − z¯k22 while maximizing kw(x) − zˆk22. It is easy to see that this is equivalent to imposing a ranking loss:
kw(x) − z¯k22 − kw(x) − zˆk22 = 2hw(x), zˆ − z¯i + kz¯k22 − kzˆk22 Notice that the features that were not corrected by the user (i.e. z¯j − zˆj = 0) do not contribute to the loss, as expected. Our solution also generalizes the input
gradient constraints introduced in [
Letting the dataset include instances x, labels y, and preferences z¯ zˆ
(represented with C in the pseudo-code), and denoting the ranking loss as `Z (w(x), z¯
zˆ) = hw(x), zˆ − z¯i, cali fits f by extending Eq. 2 with the above loss:
minf Eˆ(x,z¯ zˆ,y) [λ`Y (f (x), y) + (1 − λ)`Z (w(x), z¯
zˆ) + αΩ(f )]
(3)
Noisy corrections can be handled by tweaking the hyperparameter λ ∈ [
A few remarks are in order. First, cali trades off the model-agnosticism of
caipi for exactness and efficiency. This is desirable, since faithfulness is necessary
to justifiably establish trust, especially in sensitive applications. In addition,
despite their apparent simplicity, SENNs are very flexible non-linear models [
It is natural to ask what kind of SENN architectures can be learned in a label-scarce framework like active learning. On the one hand, shallow SENNs can be learned efficiently from few labels alone. On the other, by supplementing label information, we expect explanation corrections do help to actively learn deep(er) models. Our preliminary results suggest that this might be the case. 4
We address the following research questions:
Q1 Are the explanations output by LIME faithful?
Q2 Does cali learn from corrections?
Q3 Does explanatory feedback help learn deeper models?
In order to do so, we implemented cali3 and ran a preliminary experiment
with the synthetic color dataset used in [
Q1: Are the explanations output by caipi faithful? We trained a SENN for 1000 epochs and every 250 iterations computed the recall@k of the explanations produced by LIME on 10 random (but fixed) test examples. This was done by looking at how many of the k highest scoring features in the SENN explanation (which is exact) appear among the k highest scoring features found by LIME. In practice, caipi stabilizes high-variance explanation by running LIME r times and averaging the obtained explanations. To check whether this technique is effective, we repeated LIME r = 5, 10, 25 times and then measured the pair-wise Euclidean distance between the r explanations. The results, in Figure 2, show that the LIME explanations are never completely faithful, regardless of the number of samples s and repeats r, thus confirming our arguments about post-hoc explainers. In addition, while increasing s does have a clear beneficial effect, r is surprisingly not beneficial. Regardless, increasing either does have an effect on runtimes, as shown by the right plot. In comparison, SENN explanations are exact and robust by construction and also essentially free to compute, because w(x) must be evaluated anyway whenever doing inference. In practice, evaluating w amounts to forward propagation and takes a tiny fraction of a second (data not shown). This can be a substantial advantage in interactive applications to avoid making the user wait.
Q2: Does cali learn from corrections? In this experiment, we select each rule in turn and use cali to actively learn a SENN from explanation corrections. The ground-truth explanation highlight the 4 or 3 pixels that the decision actually depends on, and the corrections identify the c = 1, . . . , 4 pixels whose predicted weight wi(x) is farthest away from the correct weight. Figure 3 shows the label loss `Y and the explanation loss `Z as more queries are made, up to 300. Turning
on learning from corrections brings a huge benefit, as expected. Indeed, when no
corrections are provied (i.e. λ = 1), the SENN slowly learns the target concept,
as shown by the leftmost plot, while corrections help the SENN to converge
much faster. Even more importantly, unless corrections are enabled, the model
fails to be “right for the right reasons”, as the explanation loss diverges as
more labels are obtained (middle plot). Finally, the rightmost plot shows that
active learning cali is very efficient, requiring less than 0.2 seconds per iteration
on average. One surprising result is that increasing the number of corrections c
does not monotonically increase performance. This needs to be validated further;
however, the overall trend is clear and is consistent with the findings of [
Despite the surge of interest on explainable AI and machine learning, most
research has focused on passive learning, and specifically on (1) designing
interpretable predictors [
We specifically study explanatory active learning (introduced in [
The importance of explanation faithfulness has long been recognized [
The present paper highlights the dangers of post-hoc explainers for explanatory active learning (XAL). Post-hoc explainers are prone to generating unfaithful explanations that misrepresent the target predictor and prevent the user from appropriately allocating trust to it. In order to solve this issue, we extend existing XAL implementations by replacing post-hoc explainers with self-explainable neural networks (SENNs). SENNs automatically generate exact and robust explanations for their own predictions. In order to integrate them with XAL, we show how to learn SENNs from labels and explanation corrections by combining classification and ranking losses. Our preliminary experiments showcase the fragility of post-hoc explainers and the potential of SENNs in explanatory active learning.
Of course, our results need further validation, including a more direct comparisong with caipi, which we plan to carry on soon. They also hint at the
promise of corrections for learning deeper networks even when labels are scarce. We plan to investigate this direction in future work.