The ubiquity of sensors has facilitated the generation of extensive time series data across domains such as finance [1], healthcare [2,3], human activity recognition [4], and environmental monitoring [5]. These data, crucial for informed decision making, require effective time series classification techniques. However, despite the success of deep learning models in various domains, including time series classification tasks, their lack of interpretability remains a significant challenge. Explainable Artificial Intelligence (XAI) has emerged to address this issue, aiming to provide transparent explanations for machine learning models. There are a multitude of XAI methods for image and tabular data; however, applying such methods to time series data presents unique challenges due to the temporal nature of the data and the requirement for domain knowledge [6,7]. Locally Interpretable Model-Agnostic Explanation (LIME) has become a popular method for explaining black-box models [8]. However, its application to time series data is hindered by the difficulty of segmenting data while preserving temporal characteristics [9]. To address these challenges, we propose a novel Local Model Agnostic XAI method, akin to LIME, for interpreting black-box time series classifiers. Our approach does not require the segmentation of time series data. It provides detailed explanations of salient parts, including detecting trends such as increasing and decreasing local maxima and local minima. By enhancing the interpretability of black box time series classifiers, our method fosters a deeper understanding of model decisions and facilitates informed decision-making.
Recent advancements in explainable artificial intelligence (XAI) have sparked significant interest in understanding black box models, particularly in time series classification. Although XAI research has focused predominantly on computer vision and natural language processing tasks, adapting these methods to time series analysis presents unique challenges due to the temporal nature of the data [6]. Schlegel et al. [7] explored various common XAI techniques, including saliency [10], LIME [8], SHAP [11] and LRP [12], to interpret deep learning-based time series classification models. Zhou et al. [13] have enriched the interpretability landscape by enhancing Class Activation Maps (CAM) and Grand-CAM with backpropagation. Simultaneously, the work described in [14] introduced TSViz, a saliency map-based methodology later integrated into TSXplain [15] to uncover the logic behind Deep Neural Networks (DNNs) in time series. These methodologies combine salient regions, instances, and statistical features, fostering natural language explanations. Furthermore, Vielhaben et al. [16] introduced DFT-LRP, a tailored variant of Layer-wise Relevance Propagation (LRP), specifically designed to address the complexities of time series analysis by incorporating a virtual inspection layer.
While many existing methods are model-specific and rely on internal model structures, there is a growing interest in model-agnostic explanations that identify key features without being tied to a particular model architecture. However, adapting feature importance-based explanations to time series data requires careful consideration of the temporal dimension. Among featureimportance methods, LIME stands out as a popular approach, but its direct application to time series data requires thoughtful preprocessing to ensure interpretability. Guileme et al. [17] and Neves et al. [18] adapted LIME for deep learning-based time series classification by using longer segments for perturbation. Still, these approaches are limited by fixed window sizes. To overcome this limitation, Silvio et al. [19] introduced the NNsegment, which identifies homogeneous regions in time series and employs various perturbation techniques for robust explanations. Furthermore, Schlegel et al. [20] expanded the LIME approach by employing six distinct segmentation methods, but the challenge remains to understand the significance of identified segments. Hence, we present a local model-agnostic Explainable Artificial Intelligence (XAI) approach, akin to LIME, tailored for elucidating deep learning time series classifiers. Our method effectively highlights crucial input data segments that significantly impact the blackbox model's inferential process. Additionally, it provides insights into why these identified segments are important by providing information about their nature, such as whether they are increasing/decreasing events or local minima/maxima points on the time series input.
This section introduces the proposed Local Model Agnostic (XAI) method tailored for time series classifiers. The steps involved in the approach are detailed in Figure 1.
Step-by-step illustration of the proposed approach.
Our approach distinguishes itself from existing methods by avoiding fixed-interval segmentation for interpreting time series classifiers. Instead, we employ random perturbation of segments in the original time series, offering a flexible and tailored approach to generating perturbed data. These segments can be replaced by zero, the segment mean, or the total mean of the series. Importantly, perturbation is used solely to generate neighbourhood samples, and rather than employing segments as features for the linear regression model, as detailed in subsection 3.3, we utilise clusters of parameterised event primitives.
After generating neighbouring samples through perturbation, we calculate the distance (π) between the explained instance (π π ) and the neighbouring sample. In our scenario, we utilize dynamic time warping (DTW) as the distance metric, which is ideal for handling temporal data with varying speeds or time scales. Subsequently, we calculate the weight of each neighbouring instance according to an exponential kernel, denoted as π π π , which assigns higher weights to instances similar to π π . The exponential kernel is defined as:
. Here, π (sigma) represents the bandwidth parameter that controls the width of the kernel. It regulates how quickly the weight assigned to neighbouring instances decreases with increasing distance from the instance being explained. Lower values of π indicate a narrower kernel, focusing more on closer neighbours, while higher values result in a broader influence, considering distant neighbours as well.
Parameterised Event Primitives (PEPs) are vital for capturing domain-specific events within the time series data. By extracting PEPs as shown in Figure2, we can effectively represent the temporal characteristics of events as parameters, thus facilitating the learning process for interpretable models such as linear regression and decision trees [21,22]. These PEPs encompass various event types, including increasing and decreasing events, which capture parameters such as start time, duration, and average gradient value, and local maximum and minimum events, which capture time and corresponding value parameters. A structured three-step process was implemented to transform neighbouring samples in a manner conducive to training interpretable models. Initially, parameterized events were extracted from each time series sequence within the perturbed data. These events were encapsulated as tuples containing the relevant parameters. Subsequently, the parameterized events were flattened to enable the application of clustering algorithms, such as KMeans, resulting in the generation of distinct clusters. Determining the optimal number of clusters was facilitated by leveraging the silhouette method. Finally, event attribution was carried out, mapping the extracted events to their respective clusters. This process yielded a matrix wherein each cell represented the count of events associated with a specific cluster for a given instance. The event attribution matrices for each parameterized event primitive were combined to create a tabular dataset suitable for training interpretable models.
In our approach, we utilize transformed data, black box predictions of neighbouring samples, and their corresponding weights to train interpretable models similar to LIME. We employ ridge regression, a regularised linear model renowned for its interpretability and robustness. Ridge regression aims to minimize the following loss function:
Here, Ξ² represents the optimized coefficients obtained by minimizing the weighted sum of squared errors. π π π (π§) assigns weight to each neighbouring sample π§, Ε· π§ is the probability score predicted by the classifier for the perturbed instance π§, (π§) is a perturbed instance and π serves as the regularisation parameter, which governs the penalty imposed on the coefficients to prevent overfitting. In this case, the weights of the linear model learnt using a least-squares procedure denote the relative importance of each feature or each PEP cluster. After training the interpretable linear model, we identify the most significant features based on their importance scores. Here, the features correspond to clusters of Parameterised Event Primitives (PEPs) such as increasing cluster1, increasing cluster2, decreasing cluster1, and so on. We then visualise the extracted events of the instance to be explained, which belong to the top clusters, as shown in Figure 3.
Our preliminary experiment evaluated our method on two widely used univariate time series datasets: ECG200 and GunPoint from the UCR Archive [23], a renowned repository for time series classification tasks.
Our method provided local explanations for a black box model, the Fully Convolutional Network (FCN), built using the PyTorch-based tsai library [24]. The FCN was configured with default kernel sizes 7, 5, 3 and filter sizes 128, 256, 128 for its convolutional layers. The datasets were partitioned into training sets (70%), validation sets (15%), and test sets (15%) to facilitate robust evaluation. We used early stopping during training to avoid overfitting, with a patience parameter set to 15 and a minimum delta of 0.001. Additionally, to ensure accuracy and stability, the model was trained 100 times with randomised splits for training, validation, and testing. In particular, our method achieved an average accuracy of 85% and 86% on the ECG200 dataset for training and testing, respectively, and 98% for both the validation and testing sets of the GunPoint dataset.
In this section, we present the results of our experiments and discuss their implications. The method was deployed to offer local explanations for predictions generated by a deep learning-based time series classifier, with fidelity metrics that evaluate the faithfulness of these explanations. We computed the local fidelity score across different replacement methods for the perturbation to generate neighbouring samples. From each dataset, 100 instances were randomly selected from the test set, and the resulting average fidelity score and standard deviation were calculated. Table 1 presents the fidelity scores obtained using the zero and mean replacement methods. In the ECG200 dataset, the fidelity scores were 0.76 and 0.67 for the zero and mean replacements, respectively. Similarly, in the GunPoint dataset, the fidelity scores were 0.64 and 0.44 for zero and mean replacements, respectively. These results indicate that our method demonstrates varying fidelity across different datasets and replacement methods. The higher fidelity scores obtained using zero replacement suggest that this method better preserve the local interpretability of the model predictions compared to mean replacement. Furthermore, the observed standard deviations highlight the variability in the fidelity scores, indicating potential sensitivity to perturbation methods. This underscores the importance of careful consideration when selecting perturbation techniques to ensure reliable and consistent explanations. The explanation produced by our method, as depicted in Figure 3, not only highlights the significance of each part of the time series instance for the black box model's decision-making process but also provides the relevance score associated with each segment or point, along with the types of events, such as increasing, decreasing, local maximum, and local minimum. Overall, our results demonstrate the effectiveness of our method in providing local explanations for predictions of deep learning-based time series classifiers. However, further analysis and experiments are needed to fully understand the factors influencing fidelity and optimize our approach for broader applications.
Our XAI method, incorporating random perturbation and transformation using parameterized event primitives, shows promising results in enhancing interpretability for time series classifiers. While our current experiment has focused on two univariate time series datasets, future research will extend to other univariate and multivariate data to widen its applicability. Further exploration into diverse perturbation techniques and comparative analyses with existing methods will provide a comprehensive understanding of our approach's effectiveness. Overall, our method contributes to advancing explainable AI in time series classification, offering valuable insights into model predictions with ongoing efforts for refinement and expansion.