The interpretability of deep learning models has gained the attention of many researchers and organisations in recent years. The doctoral research wilwill include the interpretability and explainability of Recurrent Neural Networks (RNN) and its variant architectures.RNNs find extensive application in speech recognition and healthcare, yet their complex architecture often renders them as "black box models". FSA and RNN both handle sequential data, with FSA transitioning between states and RNN updating hidden states at each step. This similarity enables comparing FSA's transitions to RNN's hidden state updates. FSA's graphical representation enhances human understanding. The findings of this study will be used to lead the development of post hoc explanation methods (XAI) for time series data employing deterministic finite state machines as an explainability method.
With the rapid advancement of AI technologies, artificial intelligence (AI) has seamlessly
integrated into our daily lives. The demand for transparent explanations in AI, deep learning,
and machine learning is growing. AI’s reach extends to healthcare, finance, computer vision,
cybersecurity, education, and the judiciary. In this landscape, recurrent neural networks (RNNs)
are essential, powering various applications across natural language processing, speech
recognition, and image analysis [
State transition representations such as finite state automata have been studied in depth for a
long and are often seen as interpretable models [
This section outlines our research objectives, research questions, and proposed approach. Our goal is to explore the utility of finite state automata (FSA) as a post hoc explainability technique for recurrent neural networks (RNNs) and their variants (LSTM, GRU, MGU) in multi-class time series classification. Our study assesses explanation efectiveness, interpretability, and user trust through diverse clustering algorithms, evaluation criteria, and human perception [22]. Given the above, the following key research questions have been identified: • RQ 1 How do RNNs and diferent variants of RNN architectures impact the extraction and learning of finite state automata as a post hoc explainability method? • RQ 2 How do diferent clustering algorithms used to cluster the activated hidden states of RNNs impact the learning and extraction of finite state automata from the RNNs and their diferent architectures? • RQ 3 What evaluation parameters can be used to evaluate the quality of clusters formed which represent the states of finite state automata (FSA), and what standard methods can be used to evaluate the explanations created by FSA for RNNs for time series data?
This section will discuss the proposed approach to address the research questions identified. This study aims to provide a general method to learn deterministic finite state automata from RNN and its variant architectures, such as LSTM, GRU and MGU, on time series data. The method of extraction and learning deterministic finite state automata proposed in this work is based on clustering the activated hidden states of RNNs and their variant architectures using k-means clustering and uses the algorithm developed by [17] for RNN, LSTM, MGU and GRU. The proposed approach is divided into the following phases: • Intuitive approach Intuitively, the RNN model (R) is trained on training and validation data. Employing early stopping enhances generalization by preventing overfitting. The test dataset validates model performance. Each hidden state is regarded as a vector or a point. Thus, many hidden states accumulate when several sequences are input to RNNs. In a study [17], authors observed that activated hidden states in neural networks, including gated RNNs, tend to cluster. It is assumed that the diferent clusters represent the diferent states of the finite state automata. State transitions occur when input sequence symbols are read. Thus, it can inferred that the network behaves like a state automaton machine. Additionally, it is assumed that the states obtained are finite, allowing the learning of deterministic finite state automata from the RNNs and their variant gated architectures, including LSTM, MGU, and GRU. • Hidden states clustering It is contemplated to employ a k-means clustering algorithm to cluster the activated hidden states (hi) generated by the RNN when processing symbols from the input sequence during network training. • Extraction and learning of FSA FSA M is a 5-tuple = ⟨Σ , , , , ⟩ where Σ is the alphabet, meaning the set of the elements appearing in the input sequences, Q is a set of states, ∈ is the start state, ⊆ is a set of accepting states and : × Σ → defines state transitions in M [ 17]. Two matrices are generated to learn the transitions between the states: the neighbouring matrix (Ns) and the transition matrix(Ts). The neighbouring matrix provides a way to visualize the relationships between diferent states of FSA. Each state is represented by a row and column in the matrix, and the elements in the matrix indicate the strength of the connection between diferent pairs of states. By analyzing the neighbouring matrix, one can identify patterns and clusters of related states. Matrix captures state transitions’ frequency ( to ) with the symbol . The transition matrix is formed by identifying the most frequent transitions ( to ) for each row , given input symbol . In matrix , each row corresponds to the state with the highest (, ) value, representing the most frequent transition from state for input symbol .
Our research is in its early stages, focusing on comprehensively reviewing key research papers to validate our objectives. Most progress has been made in establishing a conceptual foundation and creating initial prototype feeder experiments. These experiments aim to learn deterministic ifnite state automata from RNN and LSTM, aligning with our stated objectives. Understanding state transitions within finite state automata, formed by clustering-activated hidden states of RNN and LSTM during sequence symbol reading, is crucial for concept clarity. A prototype experiment was created for multi-class classification, predicting the next sequence of finite lengths from multiple classes. The details are discussed as follows. A synthetic temporal dataset of 100 sine waves of 100 points each was chosen to perform this experiment.his dataset was chosen due to its shared properties with time series data, such as temporal nature, sequential dependence, cyclic patterns, and potential noise. The process was initiated with exploratory data analysis and prepossessing. Following that, the numerical signals underwent symbolic abstraction for state machine learning, utilizing Symbolic Aggregate Approximation (SAX). 1 to discretize the data. Employing this approach, time series data is transformed into sTheancing interpr exploredtability. Through the SAX algorithm, various strategies and bin sizes were explored for symbol transformation. Subsequently, the obtained symbols were encoded as part of the process. Various encoding methods were investigated while experimenting, and it was observed that integer encoding was suitable for our proposed approach. Further, the dataset was split into training, testing, and validation datasets. Our task is to predict the next in the sequence from multiple classes of symbols as generated above. The number of classes depends on the bin size while applying the SAX algorithm. The experiments were conducted for bin sizes 3, 4, and 5 during the prediction of the next item in the sequence using multi-class classification. 1https://www.cs.ucr.edu/ eamonn/SAX.htmd For example, if bin size=3, an input sequence would look like aabbbca, formed from 3 symbols (a, b, c). Our task for this experiment is to predict the next symbol in the sequence aabbbca from a class set of a, b, c. Subsequently, RNN and LSTM models were employed, implementing an early stopping technique and optimizing hyper parameters. Furthermore, clustering was performed on the activated hidden states of the models using the k-means algorithm. One can observe which states tend to activate together and which transitions are more frequent by clustering the hidden states. This analysis provides insights into how the LSTM processes information over time. For instance, if specific clusters frequently activate together and certain transitions dominate, it suggests the LSTM’s detection of distinctive data patterns. These clusters represent the finite states of the FSA. From these clusters, the Transition matrix Ts and the neighbouring matrix Ns were constructed. These matrices define the frequency count of transitions between the diferent states for each symbol. The deterministic finite state automata obtained can be visualized using the open-source graph visualization software Graphviz. In summary, clustering hidden states simplifies transition matrix creation and enhances interpretability. By grouping similar feature representations, clustering aids in understanding. Once states are clustered and assigned to FSA, the neighbouring matrix Ns showcases relationships and transitions. Various methods for optimal k-value in k-means clustering for activated RNN and LSTM hidden states were explored while experimenting Neighboring (Ns) and transition (Ts) matrices are generated and used to visualize deterministic finite state automata. Visual representations can elucidate the intricate relationships between diferent components of the model, aiding in the identification of influential factors that drive the predictions. Graphical explanations empower experts and users, fostering transparency and trust in decision-making process.
Our upcoming phase involves assessing cluster quality, evaluating FSA states using machine learning-based criteria, and testing RNN and LSTM-extracted FSA accuracy. In parallel, this approach is also being applied to real-world datasets, including an electroencephalogram dataset from the AI and Cognitive Load Research Lab (Technological University Dublin). Our future focus is understanding how well finite state automata capture RNN inner workings for varied sequence lengths, including infinite ones for time series data. Diferent clustering algorithms and additional parameters to enhance cluster quality evaluation will also be explored. This study will also address the lack of evaluation parameters for assessing DFA quality from RNNs, exploring FSA limitations for post hoc explanations and user trust. The primary contribution will be understanding RNN behaviour on time series data, enhancing their interpretability, explainability, and trustworthiness in various domains.
This publication has emanated from research supported in part by a grant from Science Foundation Ireland under Grant number 18/CRT/6183.
Neural Computation 17 (2005) 1223–1263. [16] C. L. Giles, C. B. Miller, D. Chen, G.-Z. Sun, H.-H. Chen, Y.-C. Lee, Extracting and learning an unknown grammar with recurrent neural networks, Advances in neural information processing systems 4 (1991). [17] B.-J. Hou, Z.-H. Zhou, Learning with interpretable structure from gated rnn, IEEE transactions on neural networks and learning systems 31 (2020) 2267–2279. [18] P. Dondio, L. Longo, Trust-based techniques for collective intelligence in social search systems, in: Next generation data technologies for collective computational intelligence, Springer, 2011, pp. 113–135. [19] P. Manolios, R. Fanelli, First-order recurrent neural networks and deterministic finite state automata, Neural Computation 6 (1994) 1155–1173. [20] U. Schlegel, H. Arnout, M. El-Assady, D. Oelke, D. A. Keim, Towards a rigorous evaluation of xai methods on time series, in: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE, 2019, pp. 4197–4201. [21] A. Das, P. Rad, Opportunities and challenges in explainable artificial intelligence (xai): A survey, arXiv preprint arXiv:2006.11371 (2020). [22] G. Vilone, L. Longo, A novel human-centred evaluation approach and an argument-based method for explainable artificial intelligence, in: I. Maglogiannis, L. Iliadis, J. Macintyre, P. Cortez (Eds.), Artificial Intelligence Applications and Innovations - 18th IFIP WG 12.5 International Conference, AIAI 2022, Hersonissos, Crete, Greece, June 17-20, 2022, Proceedings, Part I, volume 646 of IFIP Advances in Information and Communication Technology, Springer, 2022, pp. 447–460. URL: https://doi.org/10.1007/978-3-031-08333-4_36. doi:10.1007/978-3-031-08333-4\_36. [23] G. Schwalbe, B. Finzel, A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts, Data Mining and Knowledge Discovery (2023) 1–59. [24] W. Saeed, C. Omlin, Explainable ai (xai): A systematic meta-survey of current challenges and future opportunities, Knowledge-Based Systems 263 (2023) 110273. [25] W. Freeborough, T. van Zyl, Investigating explainability methods in recurrent neural network architectures for financial time series data, Applied Sciences 12 (2022) 1427. [26] F. Mujkanovic, V. Doskoč, M. Schirneck, P. Schäfer, T. Friedrich, timexplain–a framework for explaining the predictions of time series classifiers, arXiv preprint arXiv:2007.07606 (2020).