Tree-based models are commonly used in time series forecasting due to their inherent interpretability, which makes them preferable to more complex black-box models. However, simple tree-based models are prone to overfitting, limiting their applicability in real-world scenarios. Ensembles of tree-based models are employed to mitigate this, but ensemble pruning is challenging, especially in the presence of dynamic time series data and concept drift. In this paper, we use TreeSHAP, a tree-specific explainability tool, to perform online tree-based ensemble pruning that adapts dynamically to changes in the time series, addressing the concept drift issue. Empirical evaluations on real-world time series datasets demonstrate that our method performs on par with or better than state-of-the-art techniques. In future research, we plan to automate the determination of the optimal number of clusters for ensemble pruning by leveraging ensemble properties like diversity, accuracy, and stability. This automation aims to enhance both the flexibility and explainability of the model selection process. Given that this work is in its early stages, we seek feedback and collaboration with experts to create a robust and explainable framework for ensemble-based time series forecasting.
Time series forecasting is crucial for real-time planning and decision-making across various
ifelds like trafic management, weather prediction, and financial markets. However, it is also
one of the most challenging tasks due to the complex and dynamic nature of time series data,
which often involves non-stationary variations and is susceptible to concept drift [
In our future research, one key goal is to automatically determine the optimal number of clusters, which corresponds to the ideal number of trees or ensemble members in the ensemble. This would involve using ensemble properties such as diversity, accuracy, and stability to guide the selection of the most suitable cluster count. By automating this process, we aim to improve the ensemble’s flexibility and efectiveness in adapting to dynamic time series data. Moreover, we intend to deepen the explainability aspect of our approach by explicitly demonstrating that selecting models based on diferent TreeSHAP values aligns with distinct modeling paradigms and hypotheses. This could be achieved by visualizing or analyzing how these varying TreeSHAP values translate into diferent interpretations of the underlying data, providing insights into the rationale behind model selection. Given that this is early-stage work, we plan to engage with experts in the field to exchange ideas and gather feedback. Collaboration with specialists will be instrumental in refining our methodology for selecting the optimal number of trees and enhancing explainability. By incorporating diverse perspectives, we hope to develop a robust and transparent approach that addresses the complexities of time series forecasting while maintaining clarity in model selection and ensemble pruning. This collaborative efort will contribute to building a reliable framework for ensemble-based forecasting, with a particular emphasis on explainability and adaptability.
Our proposed method uses TreeSHAP for online ensemble pruning using model clustering. First,
we define the used notation. Second, we describe Shapley values with a focus on TreeSHAP
values [
A time series is a temporal sequence of values, where 1: = {1, 2, · · · , } is a sequence
of until time and is the value of at time . Denote with T = { 1, 2, · · · , } the
pool of tree-based models trained to approximate a true unknown function that generated
. Let ^+ℎ = (^+1ℎ, ^+2ℎ, · · · , ^+ℎ) be the vector of forecast values of at a future time
instant + ℎ, ℎ ≥ 1 (i.e. +ℎ) by each of the models in T. An ensemble model ¯ T of T at time
instant + ℎ can be formally expressed as a convex combination of the forecasts of the models in
T: ¯ T(^+ℎ) = ∑︀=1 +ℎ^+ℎ where +ℎ ∈ [1, ] are the ensemble weights. The weights
are constrained to be positive and sum to one. In addition, it can be seen from the notation that
the weights are time-dependent. This is one of the requirements in online ensemble learning,
where the weights are required to be set in a timely manner to cope with the dynamic nature
of the time series and the time-changing performance of the ensemble members [
E︀[( +ℎ − ¯ T(^+ℎ) ︀) 2|1:+ℎ− 1 − E︀[( +ℎ − ¯ S(^+ℎ) ︀] ︀) 2|1:+ℎ− 1 ︀]
We divide the time series 1: into = {1, 2, · · · , − } and {− +1, − +2, · · ·
, }, with a provided window size. is used for training the models in T and is used to compute the TreeSHAP values. For each tree-based model ∈ , for each observation − + ∈ with ∈ [1, ], we compute a TreeSHAP value (− +) for each lagged value, i.e., ∈ [1, ], where is the number of lags on which the model is trained. Then, we aggregate absolute SHAP values over all the observations in to acquire SHAP-based lag importance for each lag ∈ [1, ] using the model : = 1 ∑︁ |
=1
(− +)|, ∀ ∈ [1, ], ∀ ∈ we bring all the vectors I for all the models
Each model
can thus be clustered using their SHAP-based lag importance vectors I . However, diferent
models in might be trained using diferent lag values. As a result, the length of the vectors
I can vary between and . It exists clustering distance measure that can handle
vectors of diferent lengths [
Streaming time series data is prone to significant changes, leading to concept drifts . To account for these shifts, the selection of ensemble members must be updated, allowing for the inclusion of (1) = (2) models that can better address newly emerging patterns. Concept drift is detected by monitoring deviations in the mean of the time series over time, using the Hoefding Bound to evaluate if these deviations are significant. If a drift is detected, an alarm is triggered, the TreeSHAP-based model clustering is updated, and the ensemble is adjusted to reflect the new patterns in the data.
Our method is denoted in the following as OEP-TT: Online explainable Ensemble Pruning of Tree models for Time series forecasting.
We use 100 univariate time series datasets from various application domains, including financial,
weather, and synthetic data. These datasets are provided by the Monash Forecasting Repository
[
Tree-based models set-up: We construct a pool of tree-based models using diferent parameter settings that are summarized in Table 1. The list of parameters and their value ranges Tree-based Model Decision Tree (DT) Random Forest (RF) Gradient Boosted DT (GBDT) eXtreme Gradient Boosting (Xgboost) Light GBM (LGBM)
Maximum Depth Number of trees Num. of variables sampled at each split Minimum size of terminal nodes Number of trees Maximum depth of each tree Shrinkage parameter Max number of iterations Step size of each boosting step Maximum Depth Metric Max number of iterations Maximum depth of each tree
Configurations ∈ {4, 8, 16} ∈ {50, 100, 150, 200}
∈ {3, 5, 7} ∈ {5, 10, 15} ∈ {50, 100, 150, 200} .ℎ ∈ {5, 7, 15} ℎ ∈ {0.001, 0.01, 0.1} ∈ {50, 100, 150, 200} ∈ {0.001, 0.01, 0.1} .ℎ ∈ {5, 7, 15} ∈ {1, 2}-Regularization ∈ {50, 100} ℎ ∈ {5, 7, 15} in Table 1 is not exhaustive, and further parameters and values can be considered to generate more base learners. We also vary the lag parameter on which the tree-based models are trained, i.e., ∈ {3, 5, 7, 10, 15, 20}. Considering diferent combinations of all the parameters, we train a total of 294 tree-based models.
OEP-TT set-up: OEP-TT has also a number hyper-parameters: is the Size of the Pool of the tree-based models T: 294, is the Size of the validation time window :25% of the data length, || is the Number of final selected models: 6.
We compare OEP-TT against State-of-the-Art (SoA) methods for online ensemble pruning,
treebased ensembles, and time series forecasting in general. These models include: Auto-Regressive
Integrated Moving Average (ARIMA) [
To enable a fair comparison with OEP-TT, we feed to these ensemble pruning methods
the same pool of tree-based models T that was used for OEP-TT: Ens: Ensemble of all the
base modes in T; OCL [
We also compare OEP-TT to its variants: OEP-TT-ST: Static variant of OEP-TT. Pruning is decided at the initial forecasting instant and kept fixed along testing; OEP-TT-Per: Pruning is updated periodically in a blind manner (i.e. without considering the occurrence of the drift).
Wins Losses
Figure 1 shows the TreeSHAP values clusters of the Saugeen River Flow data set. The dots on each lag value stand for the TreeSHAP values taken by the models belonging to the same cluster, while the line connects the mean values to show the TreeSHAP values for each lag of the representative selected model on each cluster (only for visualization purposes). Note that we show the name of the model and the value of the first hyper-parameter plus the lag value on which it is trained to distinguish between selected models belonging to the same family of tree-based models, e.g., RF200(Lag10) and RF50(Lag7). It can be seen that on diferent clusters diferent patterns of lagged values contributions to the target time series observations are observed. This confirms that our clustering procedure promotes ensemble diversity by enforcing the selection of models that have diferent modeling paradigms and distinct views on the importance of specific lag values. For example, while models in cluster 6 favor higher lag values and emphasize the contribution of their corresponding value to the output forecast value, models in cluster 5 are built on the assumption of restricting the memory of the models to lower lag values ( = 3). We can notice that in 3 clusters out of 6, models rely on restricted lagged values (clusters 2, 3, and 5). Even with this limited width of memory, i.e., historical data, they can excel in terms of predictive performance.
Clustered TreeShap values 1.00 1.00 0.75 0.75 0.50 0.50 0.25 0.25 0.00 0.00 lrheeapeuaTSV00001.....0257005050 3 00001.....0257005050 4 1.00 5 1.00 6 0.75 0.75 0.50 0.50 0.25 0.25 0.00 lag15 lag14 lag13 lag12 lag11 lag10 lag9 lag8 lag7 lag6 lag5 lag4 lag3 lag2 lag1Lag0.v0a0luelag15 lag14 lag13 lag12 lag11 lag10 lag9 lag8 lag7 lag6 lag5 lag4 lag3 lag2 lag1 Selected Models
RF200(Lag10) GBM(Lag5) XGboost(Lag3) RF50(Lag7) LGBM(Lag3) RF150(Lag15)
This paper introduces OEP-TT a novel method for online adaptive ensemble of tree-based
models pruning. Through the use of TreeSHAP values, we are able to gain insight into its
decision-making process, both for model selection, as well as for the input time series points
relevance. We showed the advantages of OEP-TT on 100 real-world datasets, both in terms of
predictive performance as well as its explainability aspects. In future work, we plan to extend
our method to hybrid model pools by using the most eficient Shapley value estimation methods
for each model family, such as TreeSHAP for tree-based models, DeepSHAP [