We propose a global post-hoc XAI method to interpret Long Short-Term Memory (LSTM) models for univariate time series classification. Our approach integrates Symbolic Aggregate approXimation (SAX) to convert continuous time series into symbolic representations during preprocessing. We then apply k-means clustering to the activated hidden states of the LSTM, from which we extract Deterministic Finite Automata (DFA), which provides a transparent and interpretable explanation of the model's decision-making process. Experiments on synthetic and real-world datasets demonstrate high fidelity between DFA and LSTM, with enhanced interpretability for high-stakes domains like healthcare and power demand forecasting.
• Quantitative evaluation of fidelity to validate the alignment between DFA and LSTM.
Explaining the decisions of deep learning models, particularly RNNs like LSTM models, has emerged as a prominent area of research. The challenge of interpreting these "black-box" models has driven the development of XAI methods to enhance transparency and trustworthiness [4]. Interpretability makes abstract model outputs meaningful, while explainability identifies key features influencing predictions in human-understandable terms [11]. Among many ways, XAI methods can be generally categorized into ante-hoc and post-hoc approaches: • Ante-hoc methods integrate interpretability directly into the model’s architecture. • Post-hoc methods generate explanations after predictions and are further divided into: – Local explanations: Methods like saliency maps, LIME, and attention mechanisms explain individual predictions by highlighting input features afecting outputs [ 12, 13]. While efective for specific instances, these methods struggle to capture global decision-making. – Global explanations: These provide a holistic view of a model’s behavior across datasets, making them essential for understanding state transitions and decision processes [14].
Although attention mechanisms and saliency maps identify critical features, they cannot explain how LSTMs process sequential data or transition between states. Symbolic representations like DFA address this gap by ofering a structured, interpretable global explanation of state transitions [ 6]. DFA extraction visualizes LSTM decision-making as finite state machines, providing a broader perspective on model behavior [7, 8]. This study extends DFA-based interpretability to univariate time series, integrating SAX preprocessing [9] to reduce data complexity while preserving temporal patterns. The SAX-DFA combination provides symbolic, global post-hoc explanations for LSTMs, particularly in applications like power demand forecasting and heart rate monitoring [10]. Unlike local methods, DFA captures high-level transitions, ofering a comprehensive understanding of LSTM behaviour. Table 1 compares common interpretability techniques for time series and LSTMs, highlighting their strengths and limitations. SAX-DFA addresses key gaps by ofering structured, global explanations of state transitions. Besides qualitative metrics of explainability [15], more objective metrics such as fidelity and robustness are crucial for evaluating XAI methods. Fidelity ensures explanations align with the original model’s predictions, while robustness examines consistency across data points [16, 17]. These metrics guide the development of reliable, interpretable models. Existing XAI techniques for time series have underexplored global interpretability, creating an opportunity for SAX-DFA to address these gaps.
This paper contributes to advancing global post-hoc interpretability for LSTMs by combining SAX with DFA. This approach generates state-transition explanations that accurately mirror the LSTM’s decision-making process while maintaining high fidelity and robustness. Unlike local methods, SAX-DFA provides a dataset-wide perspective, enabling insights into temporal patterns and transitions.
This section introduces the proposed post-hoc XAI method to explain the internal decision-making processes of LSTM models trained on univariate time series data. The methodology is organized into three phases, which align with the four major steps shown in the pipeline (Figure 1).
This phase involves transforming time series into symbolic sequences and training the LSTM model.
SAX Preprocessing SAX discretizes continuous time series into symbolic sequences, reducing complexity while preserving critical temporal patterns. We evaluated SAX bin sizes (Nbin = 3, 5, 7) and found a balance between interpretability and fidelity at Nbin = 5. The quantile strategy was chosen for its ability to ensure balanced symbol distribution, which improved generalization and DFA interpretability.
LSTM Training The SAX-encoded sequences were used to train an LSTM model. Hyperparameters, such as hidden size (16, 32, 64) and layers (1 or 2), were tuned for optimal performance (Table 3). The Adam optimizer (lr = 0.0001) and cross-entropy loss were used, with early stopping to prevent overfitting. These settings ensured accurate classification while preserving the "black-box" nature of the LSTM for interpretability experiments.
This phase transforms the trained LSTM’s hidden state dynamics into a DFA for interpretability. A DFA is formally defined as a 5-tuple: A = (Q, Σ, S, F, δ ) where: • Q is the finite set of states. • Σ is the alphabet, i.e., the set of symbols generated from the SAX preprocessing. • S ∈ Q is the start state, representing the initial state. • F ⊆ Q is the set of accepting states linked to classification outcomes.
• δ : Q × Σ → Q is the transition function, for state changes based on input symbols.
The LSTM’s activated hidden states are extracted at each time step and visualized using t-SNE for dimensionality reduction. K-means clustering groups these states into K clusters, which form the finite states of the DFA. Transition frequencies between clusters, triggered by SAX symbols, are recorded in a transition matrix T , defined as: T (i, j) = arg maxk N sj (i, k) where T (i, j) represents the state transition from i upon input of symbol sj . Clusters most frequently visited at the end of sequences are designated as the DFA’s accepting states, providing an interpretable representation of LSTM decision-making.
In the final phase, the fidelity of the DFA is quantitatively evaluated to assess its accuracy in replicating the LSTM’s classifications. Fidelity measures the proportion of instances where the DFA and LSTM agree on their classification outcomes.
We use synthetic and real-world datasets to evaluate the proposed method, employing preprocessing, model training, and evaluation strategies.
We evaluated the proposed method on synthetic and real-world datasets, as described below, with a summary available in Table 2.
Synthetic Noisy Sine Wave Dataset A synthetic noisy sine wave dataset was created to simulate real-world time series complexity. It contains 1000 points divided into 20 sequences of 50 time steps each. Sine waves were generated with added Gaussian and low-frequency noise, labeled as Class 1 if the maximum amplitude exceeds 0.5 and Class 0 otherwise. This dataset ofers a controlled environment for benchmarking DFA interpretability against ground-truth metrics [18].
Italy Power Demand Dataset The Italy Power Demand dataset [19] records daily electrical power demand for colder (October–March) and warmer (April–September) months. With 67 training and 1029 testing instances of 24 time steps each, it serves as a benchmark for time series classification. Figure 2 shows the dataset’s distribution. A summary is provided in Table 2.
(a) Train Dataset (b) Test Dataset (c) Combined Dataset
LSTM models were implemented in PyTorch and trained using SAX-encoded sequences. The architecture
was optimized through hyperparameter tuning, varying hidden units (16, 32, 64), layers (
After training the LSTM model on the SAX-encoded time series, k-means clustering was applied to the activated hidden states at each time step. These clusters represent the finite set of states for constructing the DFA. In this study, the DFA was built using the validation set to ensure it captures the model’s general behaviour and decision patterns, allowing for interpretable insights into the LSTM’s performance on unseen data. The Neighbouring Matrix N s tracks transition frequencies between clusters for each symbol in the sequence, where each entry N s(i, k) represents the number of transitions from state i to state k based on the input symbol.The transition matrix T is then derived as follows: T (i, j) = arg maxk N sj (i, k) Here, T (i, j) represents the state transition from state i when inputting the symbol sj . The DFA’s accepting states were determined by identifying the state most frequently visited at the end of each sequence during LSTM processing. The state with the highest occurrence for a given sequence was marked as the final accepting state.
The proposed method was evaluated using: • Accuracy: Proportion of correctly classified instances. • Macro and Micro F1 Scores: Macro F1 measures the unweighted average of class-specific F1 scores, while Micro F1 weights scores by class prevalence. • DFA-LSTM Fidelity: Proportion of instances where the DFA matches LSTM predictions, measuring how well the DFA replicates LSTM behavior (Table 4).
The proposed global post hoc XAI method was evaluated on synthetic and real-world datasets (Italy Power Demand) to assess its ability to mimic the inferential process of LSTM models trained on univariate time series data. Key results focus on SAX preprocessing, LSTM performance, DFA extraction, and quantitative fidelity evaluation.
SAX algorithm transformed continuous time series into symbolic representations with varying bin sizes (3, 5, 7) and strategies (quantile, uniform, and normal).
The quantile strategy outperformed alternatives by ensuring balanced symbol distributions, particularly in datasets with skewed patterns or outliers. This balance improved LSTM classification accuracy and DFA interpretability by preserving key temporal features.
Figure 3 demonstrates how diferent bin sizes influence SAX-encoded sequences and symbol distributions. Bin size Nbin = 5 emerged as optimal, balancing fine-grained temporal representation with interpretability. Larger bin sizes (Nbin = 7) coarsened data patterns, while smaller bins (Nbin = 3) increased complexity without significant performance gains. (a) SAX-encoded Time Series (3 Bins) (b) SAX-encoded Time Series (5 Bins) (c) SAX-encoded Time Series (7 Bins) (d) Symbol Distribution (3 Bins) (e) Symbol Distribution (5 Bins) (f) Symbol Distribution (7 Bins)
LSTM models were trained with varying configurations of hidden units ( 16, 32, 64), layers (
Key findings include: • A hidden size of 32 and one layer consistently achieved high accuracy across datasets. • SAX bin size Nbin = 5 provided the best trade-of, with peak test accuracy of 96.36% on the Italy
Power Demand dataset. • Larger bin sizes (Nbin = 7) improved DFA-LSTM fidelity by capturing broader temporal patterns, while smaller bins (Nbin = 3) risked overfitting.
These results underscore the importance of hyperparameter tuning to balance interpretability and classification performance.
The DFA extraction process leveraged k-means clustering to translate LSTM hidden states into interpretable states, with transitions defined by the SAX-encoded inputs. This subsection presents the results of clustering, transition matrix construction, and fidelity evaluation, focusing on the insights derived from the experiments.
The clustering of LSTM hidden states, visualized through t-SNE (Figure 4), revealed distinct clusters for the training and validation sets, reflecting efective learning of temporal dependencies. However, the test set clusters were more dispersed, indicating challenges in generalizing to unseen patterns. These observations underline the LSTM’s ability to model temporal patterns while highlighting potential areas for improving its robustness.
The number of clusters (k) played a pivotal role in DFA complexity and interpretability. For the Italy Power Demand dataset, k = 6 provided an optimal balance between granularity and simplicity. Transition probabilities were derived from SAX input sequences, and the most frequent transitions were mapped into a transition matrix. Larger k values captured finer-grained transitions but risked overfitting, while smaller k values ofered simpler DFA representations but omitted subtle temporal dynamics.
The SAX bin size (Nbin) significantly influenced the fidelity between the extracted DFA and the LSTM model. Table 8 summarizes fidelity scores across configurations: • Larger bin sizes (Nbin = 7) achieved the highest fidelity of 0.5854, as the DFA could efectively capture broader temporal patterns while minimizing overfitting. These configurations were particularly efective for the Italy Power Demand dataset, demonstrating a robust approximation of LSTM behavior. • Smaller bin sizes (Nbin = 3) yielded lower fidelity scores, such as 0.4756, due to coarser discretization. This reduction in detail limited the DFA’s ability to distinguish between state transitions, particularly for datasets with complex temporal patterns. • Misalignment between k and Nbin, where k > Nbin, introduced invalid state transitions and decreased fidelity. This mismatch was evident in configurations with Nbin = 3 and k = 6, where ifdelity improved moderately ( 0.5305) but remained suboptimal.
Figure 5 illustrates the DFA visualizations for diferent SAX bin sizes and k values. Configurations with Nbin = 5 and k = 4 provided interpretable representations, focusing on dominant state transitions. Larger bin sizes (Nbin = 7) resulted in more refined DFA structures, efectively capturing the primary decision patterns while minimizing redundancy. The absence of certain transitions in these configurations underscores the DFA’s ability to generalize and avoid overfitting.
These visualizations demonstrate the utility of the SAX-DFA method for interpreting LSTM behavior. For example, DFA structures with larger bin sizes prioritized meaningful transitions, simplifying the explanation process without sacrificing fidelity. The reduction in noise and unnecessary state transitions improved the clarity of the extracted DFA, particularly for datasets with well-defined temporal patterns.
The fidelity scores and visualizations reveal consistent trends across both datasets: • For the Italy Power Demand dataset, Nbin = 7 and k = 6 achieved the highest fidelity, efectively capturing the LSTM’s temporal dynamics. These configurations also provided the most interpretable DFA visualizations, balancing complexity and accuracy. • The synthetic sine wave dataset, despite its smaller size, demonstrated similar trends. However, the reduced temporal complexity led to slightly lower fidelity scores for smaller bin sizes. For instance, configurations with Nbin = 3 and k = 3 achieved comparable fidelity ( 0.5305) but struggled to generalize transitions for noisier sequences.
The results highlight the strengths and limitations of the SAX-DFA method: • Strengths: The method consistently achieved high fidelity with larger bin sizes and appropriate clustering. DFA visualizations ofered interpretable insights into the LSTM’s decision-making, particularly for datasets with well-defined temporal patterns. • Limitations: Smaller bin sizes reduced fidelity, and misalignment between k and Nbin introduced invalid transitions. Addressing these challenges requires refining clustering techniques and balancing SAX parameters to improve DFA accuracy and scalability.
Future work will explore advanced clustering methods, such as k-means++, to improve alignment between clusters and SAX bins. Additionally, extending the methodology to multivariate datasets will test its scalability and robustness across more complex scenarios.
(a) Training set (b) Validation set (c) Test set
Conversely, smaller bin sizes, such as Nbin = 3, struggled to achieve high fidelity scores, with some configurations yielding values as low as 0.4756. This limitation likely stems from the coarse discretization of time series data, which reduces the DFA’s capacity to distinguish between state transitions efectively.
The need for interpretability in complex machine learning models, particularly in high-stakes domains like healthcare and finance, has driven the development of XAI techniques. This paper presented a novel global post hoc XAI method that integrates SAX preprocessing with DFA extraction to provide interpretable insights into the decision-making processes of LSTM networks.
Our proposed method addresses key challenges in explaining LSTM behavior by transforming highdimensional hidden state representations into a symbolic, interpretable structure. The DFA, constructed through clustering and state transition analysis, ofers a clear visualization of temporal dependencies and decision patterns within LSTM models. This work contributes to the growing body of research in XAI by emphasizing global interpretability, a crucial aspect for understanding the overall behavior of models applied to univariate time series data.
Experiments on both synthetic and real-world datasets demonstrated the efectiveness of the SAX-DFA method: (a) k=6, bin size=3 (b) k=4, bin size=5 (c) k=4, bin size=7
• High Fidelity: The method achieved a maximum DFA-LSTM fidelity score of 0.5854 for the Italy Power Demand dataset with Nbin = 7, k = 6, and hidden size = 16. This indicates a strong alignment between the DFA and LSTM decision-making processes. • Interpretable Visualizations: DFA structures provided clear insights into state transitions, capturing dominant decision patterns while minimizing noise. Larger SAX bin sizes produced refined DFA representations, balancing complexity and interpretability. • Generalizability Across Datasets: The method efectively captured LSTM decision patterns for both datasets, demonstrating its robustness across diferent types of univariate time series data. • Parameter Sensitivity: SAX bin sizes and cluster counts significantly influenced fidelity and interpretability. Larger bin sizes captured broader patterns but required careful tuning to avoid over-simplification.
These findings highlight the SAX-DFA method’s potential to enhance transparency in LSTM models while maintaining high performance in time series classification tasks.
This study advances the field of explainable AI through: 1. Development of a novel pipeline combining SAX preprocessing and DFA extraction for LSTM interpretability. 2. Quantitative evaluation of DFA fidelity as a metric to assess the alignment between DFA and
LSTM decision-making. 3. Comprehensive analysis of SAX and clustering parameters, ofering insights into their impact on ifdelity and interpretability. 4. Application to both synthetic and real-world datasets, demonstrating the method’s generalizability and scalability.
While the proposed method showed promising results, certain limitations warrant further attention: • SAX Parameter Sensitivity: The choice of bin size and clustering parameters heavily influenced ifdelity and interpretability. Suboptimal configurations reduced the DFA’s ability to capture essential temporal patterns. • Handling of Invalid Transitions: Mismatches between SAX bin sizes and cluster counts introduced invalid state transitions, impacting classification accuracy and fidelity. • Scalability: The method was validated on univariate datasets. Its applicability to multivariate time series and datasets with longer sequences remains unexplored. • Local Interpretability Gap: While the DFA provides global explanations, it lacks mechanisms for understanding specific individual predictions, which are crucial in certain domains.
Building on the foundation of this study, future work will address these limitations and explore new directions: 1. Parameter Optimization: Advanced clustering techniques, such as k-means++ or hierarchical clustering, will be explored to better align SAX bin sizes and cluster counts, improving fidelity and handling invalid transitions. 2. Scalability to Multivariate Datasets: The methodology will be extended to handle multivariate time series, involving adaptations to SAX preprocessing and DFA construction to accommodate higher-dimensional data. 3. Integration of Local Explanations: Hybrid approaches combining SAX-DFA with local interpretability methods, such as SHAP, saliency maps, or attention mechanisms, will provide a comprehensive framework for both global and local explanations. 4. Application to High-Stakes Domains: The SAX-DFA method will be applied to real-world datasets in domains like healthcare (e.g., ECG analysis) and finance (e.g., stock price prediction) to evaluate its practical utility in critical decision-making scenarios [2]. 5. Enhanced DFA Structures: The DFA framework will be refined to incorporate probabilistic transitions or weighted state connections, enabling a more nuanced representation of LSTM behavior. 6. Evaluation Metrics: Additional evaluation metrics, such as robustness and completeness, will be incorporated to assess the quality and reliability of explanations under varying conditions. 7. User-Centric Evaluation: Future research will focus on usability studies involving domain experts to evaluate the interpretability and utility of DFA visualizations in real-world applications.
In conclusion, the SAX-DFA method represents a significant step toward achieving global interpretability for LSTM models in time series classification. By combining symbolic representations with finite automata, this method ofers a structured, interpretable view of complex decision-making processes. While challenges remain, the findings of this study provide a robust foundation for advancing explainable AI methodologies and fostering trust in machine learning models across diverse applications.
This work was funded by Science Foundation Ireland through the SFI Centre for Research Training in Machine Learning(18/CRT/6183).