Introduction

In applied machine learning, data often evolves over time, which necessitates changes to prediction models. Ensuring the reliability of such time-dependent models is increasingly important in high-stake applications, such as financial services [1], sensor [2,3] and network [4] analysis. In recent years, eXplainable Artificial Intelligence (XAI) has targeted such time-dependent explanations of predictions that react to changes in the underlying data distributions and prediction models [5]. In extreme cases, where data is observed sequentially over time from a data stream, models are updated incrementally with each now observation, known as online learning or incremental learning [6]. In this context, re-computing XAI methods from scratch can become computationally infeasible, where incremental variants have been proposed [7,8,9,10,11].

In this work, we present a unified framework that allows to efficiently compute incremental variants of model-agnostic global explanations (MAGEs). We demonstrate that existing incremental XAI techniques are summarized in the incremental MAGE framework. Furthermore, static MAGEs cover a wide range of existing model-agnostic XAI methods, including Shapley interactions [12], which expand the range of efficient incremantal XAI techniques for interpretability of black-box online learning models.

Background

We first introduce background on model-agnostic global explanations (Section 2.1), as well as online learning from data streams (Section 2.2). We consider a trained black-box model 𝑓 ∶ 𝒳 → 𝒴 with input domain 𝒳 equipped with a 𝑑-dimensional feature representation 𝒟 = {1, … , 𝑑}, e.g. 𝒳 = ℝ 𝑑 , and output domain 𝒴. We do do not make any further assumption on the model architecture and instead only allow access to the model by predicting instances. This is known as model-agnostic explanations [13].

Model-Agnostic Global Explanations

A global explanation of a black-box model considers the behavior of 𝑓 across a whole labeled dataset (𝑥 𝑗 , 𝑦 𝑗 ) ∈ 𝒳 × 𝒴 with 𝑗 = 1, … , 𝑛. Global feature importance (FI) is an instance of global explanations that outputs an importance score 𝜙 FI ∶ 𝒟 → ℝ for every feature 𝑖 ∈ 𝒟 [14]. Global FI measures a change in a model's performance, if the model's access to this feature's information is restricted. Permutation FI (PFI) [15,16] is computed by permuting the values of the target feature and measuring the change in performance across a dataset. By permuting the feature's value, the model's access to this information is limited and, thus, PFI yields an efficient way to compute global FI. However, a feature's information provided to the model's performance might strongly depend on other features. Therefore, perturbing a single feature's value in the presence of all remaining features is a limitation of PFI. Shapley additive global importance (SAGE) [14,17] accounts for this limitation by computing the increase in loss across sampled permutations 𝜋 ∶ 𝒟 → 𝒟 over the feature space. For such a permutation 𝜋, each feature 𝑖 ∈ 𝒟 appears at a certain position. SAGE proposes to measure the average increase in loss for the preceding features of 𝑖 with and without 𝑖. By sampling over several permutations, an approximation of the Shapley Value (SV) [18] is obtained, a concept from cooperative game theory that guarantees that the SAGE values fairly decompose the overall loss. While global FI quantifies the impact of individual features, it is limited in its expressivity.

To understand Feature Effects (FE), Partial Dependence Plots (PDPs) [19] visualize the effect of imputing a specified feature's value cross all observations and compute the average prediction of all observations, when this feature's value is set. The PDP visualizes this average prediction across a range of different values, which allows to globally interpret the effect of changing this feature's value on average [20]. Besides PDP, there exist other FE methods [20,21,22] with extensions to regional explanations [20,23]. Another way of quantifying FEs is by using interaction indices that distribute contributions to all individuals and groups of features up to a maximum group size 𝑘. In recent work, several Shapley-based interaction indices have been proposed [24,25,26] as well as their efficient computation in a model-agnostic setting [12,24,25,27,28,29]. Model-agnostic global explanations were widely applied in static environments [30], however, in practice, data is often of dynamic nature, where explanations become outdated when models are adapted over time.

Online Learning From Data Streams

In many real-world applications [2] data is observed sequentially over time. In an extreme setting, we observe a data stream (𝑥 0 , 𝑦 0 ), … , (𝑥 𝑡 , 𝑦 𝑡 ), where at time 𝑡 the data point (𝑥 𝑡 , 𝑦 𝑡 ) is observed. The goal of online learning [6] is to train a time-dependent model 𝑓 𝑡 by using the current observation (𝑥 𝑡 , 𝑦 𝑡 ) once to obtain an updated model 𝑓 𝑡+1 , i.e.

IncrementalUpdate(𝑓 𝑡 , 𝑥 𝑡 , 𝑦 𝑡 ) ⟶ 𝑓 𝑡+1 .

Prominent instances of online learning algorithms include Hoeffding adaptive trees [31] and adaptive random forests [32], where splits and tree-structures are replaced, if they become outdated. Other training schemes, such stochastic gradient descent, inherently allow for incremental updates [6]. Online learning is especially important, if the underlying data distribution changes over time. This phenomenon is known as concept drift and occurs in many applications [33]. Detecting concept drift and reacting adequately by updating the model accordingly is one of the major applications of incremental learning [6]. A common approach to detect concept drift is via accuracy-based drift detectors, where a sudden change in accuracy of the model indicates a change of distributions [33]. Recently, it was proposed to enhance such detection schemes using global FI methods [5]. However, the computation of such methods is a challenging problem that has been mainly considered in static scenarios.

A Unified Framework for Explaining Change in Models and Data

We now present unified framework that allows to efficiently explore incremental variants of modelagnostic global explanations in an online learning setting. In a static setting, a global explanation is typically computed for individual features (global FI) or groups of features (global FE), which we summarizes in the following definition.

Definition 1 (ℰ).

Global explanations are computed for every element in the explanation domain ℰ, which is a collection of features and interactions ℰ ⊆ 2 𝒟 . Given an explanation domain, the explanation can be computed for each element in a static setting.

Definition 2 (Static MAGE). A static Model-Agnostic Global Explanation (MAGE) 𝜙𝑓 ∶ ℰ → ℝ for a set of features 𝑆 is 𝜙 𝑓 (𝑆) ∶= 1 𝑛 𝑛 ∑ 𝑗=1 𝜆 𝑓 (𝑥 𝑗 , 𝑦 𝑗 , 𝑆, 𝒫 𝑥 𝑗 ,𝑆 ) .

Here, 𝒫 𝑥 𝑗 ,𝑆 is a set of data points and 𝜆 𝑓 is a method-specific explanation function.

Typically, the perturbation data 𝒫 𝑥,𝑆 is constructed by using a combination of the data point 𝑥 and another sampled data point x , where the feature's values from 𝑆 and −𝑆 ∶= 𝒟 ∖ 𝑆 are taken from either 𝑥 or x [14]. Thereby, the sampling of x may be done dependently or independently of 𝑥. Instantiations of static MAGEs include PFI [15] where ℰ contains individual features, and 𝜆 𝑓 measure the increase in loss. Therein, 𝒫 𝑥 𝑗 ,𝑆 includes a single data point constructed by the values of 𝑥 𝑗 for features in −𝑆 and the values of 𝑆 from another data point obtained from the dataset using a permutation. SAGE [14] is also covered in this framework by choosing 𝜆 𝑓 as the average over sampled permutations over 𝒟, as described in Section 2.1. Lastly, PDPs [19] are contained in this framework, where 𝜆 𝑓 is chosen as the prediction of a combination of 𝑥 𝑗 and x ∈ 𝒫 𝑥 𝑗 ,𝑆 , where 𝒫 𝑥,𝑆 contains the data points for which the PDP is visualized.

Having established a unified view on static MAGEs, we now turn our focus to an online learning setting as described in Section 2.2. Using the observed data points at time 𝑡 a naive way to compute MAGEs is via Definition 2 as

𝜙 𝑡 (𝑆) ∶= 1 𝑡 𝑡−1 ∑ 𝑠=0 𝜆 𝑓 𝑡 (𝑥 𝑠 , 𝑦 𝑠 , 𝑆, 𝒫 𝑥 𝑠 ,𝑆 ) .(1)

Re-computing Eq. 1 at every time step 𝑡 is an exhaustive operative, since static MAGEs are already time-consuming when computed once [14]. Moreover, Eq. 1 requires to store the full data stream, which is typically considered infeasible. As a remedy, practitioners might restrict the computation to a time window of fixed size [5]. However, reducing the number of observations lowers the quality of the explanation, which increases the variance. In the following, we propose a framework for an incremental computation of 𝜙 𝑡 , similar to the incremental update of the model 𝑓 𝑡 . Our goal is to leverage the previously calculated MAGE and update this explanation using the currently available datapoint, i.e.

IncrementalUpdate(𝜙 𝑡 , 𝑓 𝑡 , 𝑥 𝑡 , 𝑦 𝑡 ) ⟶ 𝜙 𝑡+1 By introducing a smoothing parameter 0 < 𝛼 < 1, we define the incremental MAGE.

Definition 3 (Incremental MAGE).

Let 0 < 𝛼 < 0. We define an incremental MAGE as 𝜙 𝑡 (𝑆) ∶= (1 − 𝛼) ⋅ 𝜙 𝑡−1 (𝑆) + 𝛼 ⋅ 𝜆 𝑓 𝑡 (𝑥 𝑡 , 𝑦 𝑡 , 𝑆, 𝒫 𝑥 𝑡 ,𝑆 ) .

The incremental MAGE computes a single term of the sum in Eq. 1 at each time step and exploits the previously computed MAGE values. This drastically reduces the computational complexity, which is at time 𝑡 equal to computing MAGE once with Eq. 1. However, the incremental MAGE allows to obtain 𝜙 𝑡 for every time step 𝑡 without sacrificing computational resources. Incremental variants of PFI [7] and SAGE [8], as well as PDP [9] have been recently proposed. They can be viewed as an instantiation of incremental MAGEs.

A major challenge in computing incremental MAGEs is the maintenance of the perturbation dataset 𝒫 𝑥,𝑆 over time, i.e. efficiently constructing perturbed data points that adhere to the data distribution. Reservoir sampling [34] has been adapted to efficiently store the data distribution with minimum resources [7]. Geometric sampling [7] proposes to store a reservoir of fixed lengths, where data points are replaced over time and more recent observations have a higher probability to be present in the reservoir compared to older observations. This mechanism allows to maintain a time-dependent marginal data distribution with limited resources. More advanced techniques maintain conditional distributions using online decision trees and allow for conditional sampling as required for instance in conditional SAGE [8]. It has been shown that both sampling techniques yield substantially different explanations [35]. Geometric sampling with marginal distributions highlights the structure of the model, whereas observational approaches via conditional sampling include the data distribution in the explanation [35].

Conclusion and Future Work

We summarized popular model-agnostic global explanation techniques, such as FI-based PFI and SAGE, as well as FE-based PDPs, into the MAGE framework for static learning environments. We then proposed the incremental MAGE framework to directly compute these explanations for online learning on data streams. Incremental MAGE allows to incrementally update previous estimates of MAGEs at each time step using minimal resources. We have shown that incremental variants, such as iPFP, iSAGE and iPDP can be summarized in the incremental MAGE framework. Incremental MAGE offers opportunities to expand the range of incremental variants of MAGE-techniques. For instance, recently proposed methods to estimate Shapley interactions [12,25,29] may be placed in the incremental MAGE framework to discover for complex interactions beyond isolated FE. Moreover, with increasing variety of explanations using different complexity levels, human-centered presentations and visualizations are important future work.