The recent explosion of deep learning techniques boosted the application of Artificial Intelligence in a variety of domains, thanks to their high performance. However, performance comes at the cost of interpretability: deep models contain hundreds of nested non-linear operations that make it impossible to keep track of the chain of steps that bring to a given answer. In our recently published paper [1], we propose a method to improve the interpretability of a class of deep models, namely Memory Augmented Neural Networks (MANNs), when dealing with sequential data. Exploiting the capability of MANNs to store and access data in external memory, tracking the process, and connecting this information to the input sequence, our method extracts the most relevant sub-sequences that explain the answer. We evaluate our approach both on a modified T-maze [ 2, 3] and on the Story Cloze Test [4], obtaining promising results.
The availability of cheaper and more performing hardware, combined with the progress in the
ifeld of Machine Learning, made it possible to apply Artificial Intelligence techniques in a lot of
domains, ranging from aerospace and justice to smartphones and domotics. Intelligent agents
are now capable of learning complex tasks and adapting their behavior to new scenarios, often
surpassing human performances [
The problem is exacerbated when dealing with sequential data, where recurrent or memoryaugmented architectures have to be used. In these cases, the output depends not only on the current observation but also on previous and future ones, making it hard to weigh the importance of each of them. The field of eXplainable Artificial Intelligence (XAI) tries to address these issues by proposing methods that are able to open the black box or build more transparent models. While there is abundant literature on methods for non-sequential data and classical models like Convolutional Neural Networks, the problem of interpretability of Memory Augmented Neural Networks when dealing with sequential data is less studied.
In our paper [
We evaluate our approach both on a modified T-maze [
The remainder of this paper is organized as follows. First, we review existing literature and discuss the state-of-the-art in network explainability (Section 2); then, we introduce our approach in Section 3 and present its experimental evaluation (Section 4). Finally, we discuss conclusions and future work.
The proposed method is connected to the recent advances in Memory Augmented Neural
Networks (MANNs). MANNs use an external memory to store and retrieve data during input
processing. They can store past steps of a sequence, as in the case of recurrent architectures
for sequential tasks, or they can store external knowledge in form of a knowledge base [
In this section, we describe the architecture of Simplified Diferentiable Neural Computers, the
explanation module to compute the explanations, and the diferences between our variant and
the Diferentiable Neural Computers (DNC) architecture [
We propose a variant of DNC [
More specifically, in our case, the controller is an LSTM network, which takes as input a sequence of steps x = [1, 2, ..., ] and returns a sequence of hidden states h = [ℎ1, ℎ2, ..., ℎ]. At each step , the network chooses whether to add information into the memory M and whether to read information from it, based on the value of the current hidden state ℎ. Based on it, the network computes the following set of projections: the read keys k, the write vectors v, the write gates (one for each head), the erase vectors e, and the read strengths , where the superscripts and are the index of the read head and the index of the write head, respectively.
The projections are then combined through attention mechanisms to perform writings and readings operations. The resulting attention weights associated to each operation are called read weightings and write weightings.
The read weightings are based on the cosine similarity between the controller output and the content of the memory, weighted by the read strength : The network uses the read weightings to perform readings, computing the read vectors r as a weighted sum of the memory content: The read vectors actively influence the inference process being interpolated with the input, and then fed to the last layer to compute the output yt.
Diferently, the writing process is mainly based on the previous usage of the memory. The usage is represented by a vector u updated after each write, following the update rule: w = (M, k, )
r = M w yt = ([, [1, ..., ]]) u = u− 1 + w− 1 (1) (2) (3) (4) To compute the write weightings, the network multiplies the allocation weights a, and the write gate .
w = a, where the allocation weights are used to decide where to write the new information. Their computation is based on the usage vector u: (5) (6) − 1 a[[]] = (1 − u[[]]) ∏︁ u[[]] =1 where is the sorted list of memory-cell indices in ascending order of usage. Intuitively, the write weightings are higher for cells never used before and low for cells recently written.
Finally, the write weightings are used to update the content of memory: = − 1 ∘ (1 − we ) + wv (7) where e are the erase vectors that tell the system which cells can be freed, v are the vectors to be written and w are the write weightings.
The idea behind the explanation module is to exploit the writing and reading mechanisms to extract the information more often used by the network to compute the predictions. As mentioned before, the memory contains information extracted from the controller output, while the weightings describe the memory operations.
Let us start from the write weightings associated to a single cell: a high value indicates that the cell will contain most of the information of the current hidden state ℎ (i.e. controller output) after the writing operation. Therefore, the explanation module creates a mapping between the cells with high weights and the current step , storing the position of cells whose weightings are greater than the mean, and the input . Note that tighter thresholds would not be more informative, since most of the time only one or few cells are written and many cells have write weightings close to zero. Since the network tries to write information on empty cells by design (Sect. 3.1), then the mapping between cells and inputs will be often one-to-one. Regarding the read weightings, a high weight indicates that the cell contains information similar or related to the current state – due to the cosine similarity. It also means that the cell is heavily represented in the read vectors, and so it has a high impact on the inference process. In this case, the explanation module keeps track of the top- read cells for each step, i.e., the cells with the highest read weights.
At the end of the input parsing, the explanation module combines the information about written cells, the inputs, and the read cells to compute a history of the memory operations. Based on this history and on the characteristics of the task, it builds a rank of explanations by means of the frequency of input readings. Intuitively, since by design the readings are connected to the inference process, we expect that the inputs associated to the most read cells have a greater influence than the others, and so they can be considered as explanations, being the subsequences that steer the model during the inference process.
Before we proceed to show the results of our method, we want to highlight the key diferences between the proposed variant and original DNC, and explain the motivations behind some decisions.
First of all, let us consider the writing operations. The DNCs use the cosine similarity to compute the write weightings in combination with allocation weights. In this way, during the parsing of the input, the network can overwrite or update information on a given cell. Conversely, the Simplified DNC uses only the allocation weights, avoiding overwriting cells in the memory as much as possible. This modification has not an impact on the performance as long as there is enough space to store information from the whole sequence, but it has a big impact on interpretability. Indeed, when states of several steps are stored in the same cell, then it becomes dificult to extract the single step responsible for the reading operation is being performed. Conversely, when the network is encouraged to write information on new cells, then each cell is likely to be only associated to one step, making the tracking and mapping of the inputs easier.
The second important diference is the lack of recurrence at the memory level. In DNCs, the input of LSTMs is the concatenation between the current step and the read vectors of the previous step, generating a recurrence at memory level and making the mapping between cells and steps harder. Indeed, in this case the output ℎ of the LSTMs will be influence not only from the previous steps due to the recurrence of LSTMs but also from the previous read steps, increasing the input dispersion and making the extraction of information harder. Moreover, in the case of memory recurrence, since we only use the cosine similarity to perform readings, the readings will tend to retrieve information from the same cells with little variation between two consecutive steps.
The last diference is the lack of temporal linkage addressing, whose inclusion makes the network heavier, while its efectiveness varies from problem to problem. This modification has no impact on the explanation module, since its working mechanisms are independent of the type of attention mechanism used to compute the weights.
We divided this section into three subparts: the first describes the datasets and the protocols followed to evaluate the proposed method; the second presents the results both in terms of performance and explanations together with some examples showing what type of explanations can be obtained; and the last discusses the hyperparameters introduced by the method, its extension to LSTMs, and it presents a summary about its strengths and weaknesses.
We test the proposed architecture on a modified version of the T-maze [
Performance: Fig. 1 and Table 1 show the performance of our architecture in both the tasks.
Fig. 1 shows the average cumulative reward of the agent on the T-maze task over 10 diferent
training runs. It shows that the model gradually converges to a solution in which the agent
consistently achieves the highest reward. For the Story Cloze Test we follow the settings of
Mihaylov et al. [24] and compare the performance against them in Table 1. Despite the fact
that we use a smaller LSTM (128 units against 512) and we do not use some optimizations like
the augmentation of the dataset, the system is able to reach the same generalization power,
confirming the previous results of MANNs against LSTM models [
In the case of Story Cloze Test, we are interested to extract the premises that bring the network to choose the predicted ending of each story. During the parsing of the predicted ending, we expect that the network reads words of an important premise more often than those of an irrelevant one. Starting from this observation, the explanation module builds a rank of premises based on the frequency of readings of its words, attaching a score to each premise. From the rank, we can consider the top-ranked premise as the best premise for explaining the prediction, being the one that contains the most read words, and so it is often included in the read vectors that influenced the prediction. Contextually, we can identify the worst premise for explaining the prediction as the sub-sequence that contains the words associated with the minimum number of readings, and so ranked last.
To test the quality of the rank, we use the explanation accuracy metric defined as the times the model is able to reproduce its predictions by only using the provided premises. We expect that, if the ranking is meaningful, the explanation accuracy of the best premise will be greater than the one of the worst premise and the one of a random selected premise. In this case we can consider the best premise as an explanation of the predicted ending. Indeed, intuitively, if the explanation for a certain prediction is good, feeding that information alone to the network should result in the same prediction (and a good accuracy). Conversely, by only providing the worst explanation to the model, the prediction should change since the missing relevant information, lowering the explanation accuracy.
The results in Table 2 confirm the reliability of the ranking. Hence, the explanation accuracy of the best premise is always greater than the one of the worst premise, and the accuracy of the worst premise is lower than that of a randomly selected premise.
Figure 3 and Figure 4 show examples of the output of the explanation module. In particular, Figure 4 shows an example, extracted from the dataset, where the model returns the right prediction and the explanation module returns the premise rank. We can observe that the module focuses mainly on the third and fourth premises, assigning lower scores to the others. We can test the rank, modifying the input and observing how the model prediction and its confidence change. First of all, we can observe that modifying one of the two best premises the model becomes less confident about its prediction (a loss of about ∼ 28% and ∼ 10% respectively), while modifying the second premise it has no impact. Moreover, the importance of both of them is confirmed by the fact that we need to modify both in order to get a diferent prediction.
In the T-maze problem, we can exploit our prior knowledge about the problem. Here, the only
Earl woke up early to make some cofee . (48.3%) He wanted to be alert for work that day. (47.4%) The aroma woke up all his roommates. (0%) They wanted to make cofee too. (4.2%) E1. All of his roommates made cofee (CORRECT) – All of his roommates were sick of cofee.
Tim didn’t like school very much. (23.6%) His teacher told him he had a test on Friday. (15%) If he didn’t pass this test, he could not go on the class trip. (4.5%) Tim decided to play with his kites instead of study for the test. (56.8%) E1. Tim was unprepared and failed the test. – E2. Tim aced the test and passed with flying colors. (WRONG) information that matters for predictions is the information stored in the relevant corridor step. Indeed, an agent can solve the task only learning to exploit it due to the nature of the task itself. Therefore, we use the explanation module to verify that the agent is acting correctly and is basing its decision on that information. To test this scenario, we check whether the information from the relevant corridor step is among the two most-read memory cells. We consider two cells instead of one to also account for the first corridor step, which could be used equally often. Figure 2 shows that this is the case. Moreover, comparing the trend against the learning curve shown in Figure 1, we can observe that the plots of cumulative reward and the matching accuracy follow the same behavior, meaning that, when the agent learns to exploit the relevant step corridor information stored in the memory, it improves its performance.
Hyperparameters The proposed module includes the hyperparameter that controls how many cells consider for each reading. As shown in Table 3, it can have an impact on the goodness of returned explanation. For example, in the case of Story Cloze Test, using small values is beneficial for the choice of the best premise. Indeed, in this way the module collects few cells for each step and, since these cells are those with the highest weights they have a direct influence on the prediction, it rewards the premises read in a more consistent way. Conversely, by setting to a high value the rank will reward premises with the highest number of readings – independently of when the read occurred, introducing noise in the process. At the same time, when is small, the information to choose the worst premise is very scarce while, when P1. My best friend played a video game online.(4%) P2. One day, she met a nice boy there.(21%) P3. They talked every day and fell in love. (36%) P4. They finally met in person and hit it of.(39%) E1. They were very sad about the incident.(11.05) E2. The two became a very loving couple. (CORRECT) (88.95) Modified P4. They never met in person.
E1. They were very sad about the incident. (39.35.) ↑ E2. The two became a very loving couple. (CORRECT) (60.65) ↓ Modified P3. They never talked again.
E1. They were very sad about the incident. (22.32) ↑ E2. The two became a very loving couple. (CORRECT) (77.68) ↓ Modified P3. They never talked again.
Modified P4. They never met in person.
E1. They were very sad about the incident. (WRONG) (63.08) ↑ E2. The two became a very loving couple. (36.98) ↓ Modified P2. One day, she played another game.
E1. They were very sad about the incident. (11.05) =
E2. The two became a very loving couple. (CORRECT) (88.95) = is large, it is more reliable. For this reason, this hyperparameter should be tuned in the validation process or fixed using prior knowledge, and it represents a trade-of to be taken into consideration. Additionally, the plot in Figure 5 indicates that explanations provided by our system are meaningful from the very first training steps and, consequently, that the network quickly learns to use its memory.
Extension to LSTMs A natural question is whether we can apply the same approach to LSTM networks, thus exploiting their internal memory. Unfortunately, the answer is negative due to the way in which they use the memory. Indeed, during the parsing of the input, the hidden state ℎ is only used once as additional input for the next time step, eventually merged inside the cell state, and then discarded. Because only the cell states store information, and they include aggregated information from several steps in the sequence, it becomes impossible to reconstruct the relevant information. A possible alternative is to directly apply the cosine similarity over the resulting states, selecting the top states similar to the state of the prediction step, and then compute the explanations. The drawback in this case is that one has to decide a priori how many words to consider in the explanation, thus introducing a hyperparameter dificult to tune. Moreover, the most similar states are not actively used in the inference process by the LSTM, therefore the connection between them and the predictions would be unclear and undefined. Limitations and strengths Summing up, the strengths of this approach are several. It has a low computational cost, because it needs only space for storing the memory history and a negligible time to compute the rank. Its incremental nature makes it ready for online scenarios, where the full sequence is often lacking. It is flexible, allowing to shape explanations on the basis of the problem and the query of interest. Nevertheless, there are some drawbacks connected to MANNs and the hyperparameter to consider. The training process of MANNs is often unstable, showing a large variance and an high sensitive to hyper-parameter choices, making them hard to tune and apply to new domains [25]. We can observe a similar behavior on the hyperparameter and, more in general, on how the explanation module selects the cells to monitor. Currently, the choice must be done using a priori knowledge about the problem or validating it in a development set, due to the lack of a general rule. This means that one can obtain diferent results and explanations by simply modifying the way in which the module keeps track of the memory. Moreover, the network potentially could learn to ignore the memory content on some predictions and it is hard to understand when this happens. To limit the extent of this problem we apply the Bypass Dropout [25] to the link between controller and output, but it remains an open problem to be solved in future research.
In this paper, we discussed a method presented in our previous work [
C. Blundell, Neural episodic control (2017). arXiv:1703.01988. [14] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, T. P. Lillicrap, One-shot learning with memory-augmented neural networks, CoRR abs/1605.06065 (2016). URL: http://arxiv.org/ abs/1605.06065. [15] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods for explaining black box models, ACM Computing Surveys 51 (2018) 1–42. doi:10.1145/3236009. [16] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: 2016 IEEE Conf. on Computer Vision and Pattern Recognition, 2016. doi:10.1109/cvpr.2016.319. [17] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, K.-R. Müller, How to explain individual classification decisions, J. Mach. Learn. Res. 11 (2010) 1803–1831. URL: http://dl.acm.org/citation.cfm?id=1756006.1859912. [18] M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks (2017).
arXiv:1703.01365. [19] M. Ribeiro, S. Singh, C. Guestrin, “why should i trust you?”: Explaining the predictions of any classifier, in: Proc. of the 2016 Conf. of the North American Chapter of the Association for Computational Linguistics: Demonstrations, 2016. doi:10.18653/v1/n16-3020. [20] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in:
Advances in Neural Information Processing Systems, 2017, pp. 4765–4774. [21] M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: High-precision model-agnostic explanations, in: AAAI Conf. on Artificial Intelligence, 2018. [22] J. T. Springenberg, A. Dosovitskiy, T. Brox, M. A. Riedmiller, Striving for simplicity: The all convolutional net, CoRR abs/1412.6806 (2014). [23] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, W. Samek, On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation, PLOS ONE 10 (2015) e0130140. doi:10.1371/journal.pone.0130140. [24] T. Mihaylov, A. Frank, Story cloze ending selection baselines and data examination,
LSDSem 2017 (2017) 87. [25] J. Franke, J. Niehues, A. Waibel, Robust and scalable diferentiable neural computer for question answering, in: Proc. of the Workshop on Machine Reading for Question Answering, 2018, pp. 47–59. [26] B. La Rosa, R. Capobianco, D. Nardi, Memory wrap: a data-eficient and interpretable extension to image classification models (2021). arXiv:2106.01440.