Feature importance plays a fundamental role in machine learning and serves as a cornerstone of explainable machine learning. In temporal settings, where data accumulates sequentially, the relevance of features may evolve, introducing challenges for interpretation. While temporal variation in feature importance is increasingly relevant for applications such as clinical monitoring and time-series prediction, it remains underexplored in the literature. In this paper, we propose a novel methodology for quantifying the temporal stability of local feature attributions. Our approach combines exponentially weighted moving average (EWMA) model with performance metrics. The goal is to compute a feature-wise stability metric that reflects how consistently a feature contributes to model predictions over time. To complement this, we introduce a distributional drift score based on the Wasserstein distance, capturing shifts in the underlying feature distributions. Together, these two signals form a diagnostic framework that distinguishes between shifts due to data dynamics and those arising from model behavior. We evaluate our approach on a simulated dataset reflecting mental health monitoring scenario, as well as a publicly available benchmark time-series dataset. In both cases, the proposed metrics uncover nuanced patterns of feature behavior, enabling practitioners to identify features that are not only important but also temporally reliable. Our results demonstrate that assessing both the stability of explanations and the drift of features provides a more robust foundation for trustworthy model interpretation in dynamic environments.
Although methods of explainable AI (XAI) have been advancing significantly in recent years [
Recent eforts have begun to explore explanation dynamics in time-dependent settings [
Feature attribution stability is interpreted as the temporal consistency of a feature’s importance, as
measured by an explanation method. Capturing these fluctuations is expected to yield new insights
into model robustness, data drift, or redundant feature use. For instance, a feature whose importance
lfuctuates erratically over time may signify a model that is excessively sensitive to noise or evolving
distributions. While the notion of stability has been examined in static contexts, such as in feature
selection [
To address this gap, we propose a novel approach that quantifies fluctuations in feature importance
over time, accounting for the performance and changes of predictions as more data becomes available,
thereby improving the interpretability of temporal machine learning models. Moreover, this approach
takes into account the drift of features to enhance the understanding of changes in the features
themselves over time. We validate our approach using a diverse set of datasets, including a case study
in mental health monitoring and benchmark dataset from the UCI Machine Learning Repository [
The structure of the paper is as follows. In the next section, we present a brief description of related works. This is followed by a presentation of the proposed approach in Section 3. The experimental results using simulated data and benchmark datasets are presented in Section 4. Finally, Section 5 outlines the conclusions and discusses directions for future work.
Comprehension of the role that input features play in machine learning models’ predictions constitutes
a fundamental principle of explainable AI [
On the contrary, model-agnostic methods are characterised by their ability to ofer greater flexibility.
Permutation importance is a method of assessing a feature’s relevance. It does so by evaluating the
impact on model performance when feature values are randomly shufled [
While these approaches have been the focus of extensive research for static datasets, their application
to temporal or sequential data remains limited. The majority of explanation methods treat time steps
independently, neglecting to consider how the importance of features evolves or fluctuates over time
[
Concurrently, the evaluation of explanation methods has emerged as a prominent research subject
in the field [
In this work, we quantify the evolution of feature attributions through two complementary metrics: a stability metric and a distribution drift score. Unlike static assessments, our approach decomposes explanation dynamics into interpretable signals that reeflct both model behavior and data characteristics over time. The stability metric captures deviations of feature importance from a smoothed historical baseline, weighted by model performance, thereby accounting for fluctuations in predictive reliability. In parallel, the distribution drift score quantifies changes in the underlying feature distributions using the Wasserstein distance.
In this section, we will present two complementary metrics developed to capture the temporal dynamics inherent in the feature importances of multivariate time-series data. These metrics include a stability metric based on Exponentially Weighted Moving Average (EWMA) and a drift score based on Wasserstein distance. Collectively, these measures facilitate comprehensive monitoring of the consistency and distributional evolution of explanatory signals over time.
Let X ∈ R× denote a sample of multivariate time series with features and time points. Let also x := X· , ∈ R be the vector of all feature observations at a time point 1 ≤ ≤ , where > 1. Feature importance scores are computed based on the realizations x of the time series. Let () ∈ R denote the importance of the feature ∈ {1, 2 . . . , } at the time point , derived from a model-specific feature attribution method, with the full sequence over time represented by () = {1(), 2(), . . . , ()}.
We propose a methodology for deriving a feature-wise stability score, a metric that quantifies the discrepancy between a feature’s perceived importance and its exponentially weighted historical trend.
The stability metric is then defined as: where: () = 1 − −1 1 ∑︁ () =2
^())︁ 2
︁( () −
|()| + |^()| +
• () ≥ 0 is a weight function assigned at time point = 2, . . . , , incorporating model
performance of the model. If () = 1, ∀, no weighting is applied.
• ̂︀() is the EWMA of past feature importance scores, recursively defined as:
^() = ^(−)1 + (1 − )(),
where ∈ [
• > 0.
The proposed approach ensures that importance deviations are scaled in relation to their respective magnitudes, thus preventing excessive penalization of low-importance features. A high () indicates that the importance of feature exhibits a close correspondence with its historical trend, thereby suggesting temporal reliability.
The parameter is a component in the analysis, which governs the rate at which past observations decay. When is set to a larger value, it places more emphasis on the most recent data points, yielding a smoother baseline. Conversely, when is set to a smaller value, it gives greater weight to older observations. (1) (2)
The well-known EWMA is a time series analysis technique that assigns exponentially decreasing weights to past observations. This property renders the EWMA particularly useful for detecting trends and anomalies in noisy data. In contrast to the simple moving average, which assigns equal weight to all past observations, the EWMA assigns greater weight to more recent data points. This characteristic renders the EWMA more responsive to changes in the time series.
The squared diference between the feature importance value and the EWMA of past feature importance scores is aiming to capture the variability in feature importance over time. This formulation gives more weight to recent time points and models with higher predictive performance, making it particularly responsive to evolving patterns. Additionally, the weight function adjusts for model performance, assigning greater importance to better-performing models while diminishing the influence of models with lower accuracy.
The stability metric is generalizable and can be applied with any feature importance method as defined in Eq. (1). In our experiments, we selected Shapley values, which provide an attribution-based measure of feature importance.
While the stability metric is capable of capturing fluctuations in importance, it does not have the capacity to detect whether the feature itself is undergoing a distributional shift. To address this limitation, we propose a drift score, computed as the Wasserstein-1 distance between the empirical distribution of a feature at the current time point and a reference distribution estimated from a kernel density estimate over a rolling window of past values, capturing recent trends in the feature’s distribution. Both distributions are approximated using kernel density estimation.
Let () denote the empirical distribution of the feature at a time point , and let ̂︀() denote a smoothed estimate of its past distribution. Then the drift score is defined as:
Drift() =
1 ∑︁ 1((), ̂︀()). − 1 =2 (3)
In this context, 1(· , · ) denotes the first-order Wasserstein distance originally introduced in [
Employing the stability metric () and the drift score Drift(), can facilitate the origin of temporal variations. For example, when high () and low Drift() are observed, the feature exhibits both stable importance and a stationary distribution over time. This suggests that the feature is robust and consistently relevant for the model across time, making it a strong candidate for long-term interpretability and reliable decision-making. On the contrary, when low () and high Drift() are reported, the feature’s importance fluctuates over time in conjunction with significant distributional changes. This indicates that the model’s reliance on the feature is adapting to underlying shifts in the data-generating process, potentially reflecting concept drift or context-sensitive importance. Another scenario would be to observe low () and low Drift(). The feature’s distribution remains largely stable, yet its attributed importance varies. This scenario may signal model instability, such as overfitting or sensitivity to transient patterns, since fluctuations in explanatory power are not explained by input variation. Alternatively, high () and high Drift() would mean that despite the feature undergoing distributional changes, its importance remains stable. This indicates model robustness to input drift, suggesting that the feature maintains its predictive role under varying conditions — a desirable trait in non-stationary environments.
We begin by validating the proposed stability and drift metrics using a simulated dataset derived from
real-world clinical observations, followed by experiments on a publicly available benchmark dataset
[
Our evaluation is motivated by a real-world clinical problem concerning the remote sensor-based monitoring of bipolar disorder patients. In this particular applied problem, considering the feature’s importance for a specific time point does not provide comprehensive insights into the temporal influence of the feature. If a feature’s importance may change over time significantly, ranging from the most important to least important, without a clear pattern, the inference based on such a feature might not be as reliable as assumed. Our approach aims to identify features that demonstrate temporal stability in both importance and distributional behavior, thus enhancing interpretability and trust in model-based inference.
The real-world dataset comprises acoustic and psychiatric data collected from a patient diagnosed
with bipolar disorder. For further details see [
To reduce complexity and enable stable classification under limited data, the original multiclass labeling was binarized mapping states into two categories: euthymia, which is considered the healthy state, and non-euthymia.
Considering the limited size of the real data and its complex nature, we simulated a larger, controlled
dataset that permits more rigorous and repeatable validation of the proposed metrics. The resulting
dataset is publicly available for further investigation [
We simulated observations from patient-specific distribution over five-year period. Each observation corresponds to a specific day, and the dataset was divided into two categories of periods: ground truth periods, which spanned nine days around a psychiatrist appointment where the patient’s mental state was known, and inter-appointment periods, representing the intervals between psychiatrist visits where the patient’s mental state was inferred. Each inter-appointment period consisted of 64 days, resulting in five meetings per year.
Then, we simulated 180 observations from the base distribution, with 60 observations from each state. This dataset was used as the prior knowledge dataset, while the five-year dataset included 1 825 observations with varying label distributions.
As the next step, we trained XGBoost classifiers [
For each window, model performance was evaluated using the area under the receiver operating characteristic (ROC) curve (AUC), or the 1 score if ROC AUC was unavailable, from the Scikit-learn package. Distribution drift was calculated for each window using Kernel Density Estimation. The drift between the current window and a smoothed estimate of the past distribution was then computed using the Wasserstein distance. Shapley values, computed using the SHAP package’s Explainer interface, were calculated for the prediction of the class representing the non-euthymic state. To calculate the stability metric, a vaue of = 0.7 was used in all experiments (see Eq. (2)).
For the simulated medical dataset, Fig. 3 presents Shapley values over time for five-year simulations, along with stability metrics incorporating model weighting by performance. Most stability metric values exceed 0.875, suggesting either minimal fluctuations in Shapley values or that the corresponding features have little influence on the model’s predictions. However, a closer inspection reveals that jitter, despite having the highest absolute Shapley values, exhibits significant variability in both magnitude and direction over time. This instability is captured in the stability metric, where jitter consistently shows the lowest stability values across both simulation periods.
Fig. 4 illustrates temporal changes in Shapley values for jitter and energy, alongside model performance at each time point. Notably, energy shows considerable fluctuations in its Shapley values, but these gradually stabilize at low positive values, with occasional negative spikes. This leads to an approximately 0.9 stability metric. On the other hand, jitter experiences both magnitude and direction lfuctuations before stabilizing at positive values. However, due to more significant fluctuations and a later stabilization, jitter achieves a lower stability metric of around 0.85, before eventually reaching 0.9. The performance-based weighting smooths some Shapley value changes, particularly between time points 2, 3, and 4 where energy’s Shapley values show noticeable changes. Notably, feature F0 maintains a high stability metric, reflecting its consistently low Shapley values relative to the other features throughout the time period.
Table 1 presents the stability metric and distribution drift values for the features considered in the simulated dataset. Since no universal thresholds exist for interpreting these metrics, we analyze them relative to one another. The cell color intensity reflects these internal comparisons and helps surface outliers and patterns.
We observe that jitter, shimmer and energy exhibit relatively low distribution drift (all below 0.03) and moderate-to-high stability scores, ranging from 0.89 to 0.96. These features maintain consistent importance over time and stem from relatively stable data distributions – indicating strong potential as interpretable and dependable predictors in a temporal context.
In contrast, the F0 feature stands out with the highest distribution drift (0.26), substantially exceeding that of the other features. Despite this, it shares the highest stability score (0.96), indicating that although the underlying distribution of this feature changes substantially, its relevance to model predictions remains stable – suggesting model robustness or invariance to feature drift.
A particularly instructive case is jitter. Although it frequently appears as one of the most influential features (based on raw Shapley values), its lower stability metric indicates substantial temporal variability. This highlights the risk of over-interpreting raw importance scores without considering temporal consistency. Features like energy and shimmer, with slightly lower but more stable contributions, may ofer more reliable insight for longitudinal interpretation or downstream decision-making.
This example illustrates the core utility of the proposed approach: it enables a nuanced decomposition of explanation quality, capturing both temporal stability and drift sensitivity. Such decompositions are essential when local explanations are used to guide clinical insight or intervention strategy. Traditional feature importance analysis cannot ofer this level of granularity – particularly when dealing with sequential or drifting data.
A current limitation of the analysis is the omission of long-term average Shapley trends, which could ofer additional insights into persistent feature relevance. This omission could result in incomplete inferences about the features, as the evolution of the overall importance of features across the entire time period is not taken into account. The incorporation of this information could facilitate a more comprehensive understanding of feature behavior and stability.
To assess the generalizability of our proposed metrics beyond the medical domain, we extend our
experimental framework to a publicly available time-series dataset, namely the Rocket League dataset[
The temporal division strategy and the structural framework (XGBoost) utilized in the medical simulation were employed in this investigation. The performance-weighted stability metric and distribution drift computations were also adopted.
Fig. 5 illustrates temporal changes in mean Shapley values and the corresponding stability metrics for selected features in the Rocket League dataset. We highlight three features that reflect diverse dynamic behaviors over time.
First, consider BallAcceleration. While its Shapley values oscillate in both magnitude and sign during the early intervals, the feature eventually stabilizes with increasingly consistent positive contributions. Despite initial fluctuations, the combination of late-stage consistency and strong positive attribution boosts its overall stability metric to around 0.92, placing it in the relative high-stability category.
In contrast, accelerate shows highly variable Shapley values with no consistent trend – swinging from strong positive to negative contributions across diferent intervals. This behavior results in a low stability metric of approximately 0.69, even though its distribution drift is minimal. Such a combination – low stability with low drift – may suggest overfitting or excessive context dependence, limiting the interpretability and reliability of this feature.
Finally, PlayerSpeed provides a third example, showing persistent relevance across the entire timeline. While its SHAP values exhibit some fluctuation in magnitude, their sign and overall importance remain stable. This yields a relatively high stability score (above 0.93) despite the feature undergoing substantial distributional shift, as reflected in one of the highest drift values in the dataset. This resilience implies that the model reliably incorporates PlayerSpeed despite changes in its input distribution – a sign of potential robustness or invariance. Figure 5: (a) Changes over time in mean Shapley values for features in Rocket League dataset and (b) values of stability metric for Rocket League dataset, with weighting by performance in Eq. (1) and = 0.7 in Eq. (2).
This study introduces a novel framework for evaluating the temporal stability of feature importance, while jointly accounting for distributional drift in the underlying data. We operationalize this idea through a performance-weighted stability metric and a drift score based on the Wasserstein distance, applied to local explanations derived from Shapley values. By incorporating exponentially weighted moving averages and performance-based weights, the proposed stability metric captures deviations in feature importance over time in a calibrated manner. The drift score complements this by quantifying changes in the empirical distribution of each feature, thereby disentangling model instability from data drift.
Our empirical validation on both a simulated clinical dataset and a publicly available benchmark time series demonstrates the utility of the framework in diagnosing and interpreting model behavior over time. The results afirm that reliable model interpretation cannot rely solely on feature importance magnitudes; instead, it should incorporate both temporal consistency and feature dynamics.
While the proposed metrics ofer valuable diagnostic insights, their full utility emerges when integrated with complementary analytical signals. For instance, temporal autocorrelation or variance decomposition could provide further context about the persistence and volatility of individual features. Additionally, a formal framework for aggregating multiple explanation signals – stability, drift, rank variance, and uncertainty – could further improve understanding of the modeling process under temporal shifts and its outputs. Future work will extend the analysis to additional feature importance metrics, further exploring their impact on stability evaluation. Finally, future research will focus on refining the weighting schemes in the stability metric, extending the approach to additional feature attribution methods beyond Shapley values, evaluating model robustness under adversarial or synthetic distribution shifts, and applying the framework to other domains. We also plan to release an open-source toolkit implementing these metrics to facilitate adoption and further experimentation.
The project "ExplainMe: Explainable Artificial Intelligence for Monitoring Acoustic Features extracted from Speech" (FENG.02.02-IP.05-0302/23) is carried out within the First Team programme of the Foundation for Polish Science co-financed by the European Union under the European Funds for Smart Economy 2021-2027 (FENG).
During the preparation of this work, the authors used GPT-4 in order to: Grammar and spelling check. After using these tools, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content. 2939672.2939785. arXiv:1603.02754.