The field of natural language processing has reached breakthroughs with the advent of transformers. They have remained state-of-the-art since then, and there also has been much research in analyzing, interpreting, and evaluating the attention layers and the underlying embedding space. In addition to the self-attention layers, the feed-forward layers in the transformer are a prominent architectural component. From extensive research, we observe that its role is under-explored. We focus on the latent space, known as the Activation Space, that consists of the neuron activations from these feed-forward layers. In this survey paper, we review interpretability methods that examine the learnings that occurred in this activation space. Since there exists only limited research in this direction, we conduct a detailed examination of each work and point out potential future directions of research. We hope our work provides a step towards strengthening activation space analysis.
1. Introduction
formers have established itself as the state-of-the-art in
various Natural Language Processing (NLP) tasks since
their conception and realization in 2017. BERT, the most
well-known transformer language model [
nEvelop-O ∗Corresponding author. CEUR htp:/ceur-ws.org
ISN1613-073 https://huggingface.co/
Attribution 4.0 International (CC BY 4.0). ing [10]. A challenge that exists is the gap between the the high-level concepts that are human-understandable.
The core focus of our review is directed towards those methods that unfold the learnings in the internal representations of the neural network, i.e, we look at those methods that answer the question: “What does the model specifically the feed-forward layers in transformer models. The motivation for this study is two-fold: • The inputs undergo a non-linear transformation when passing through the activation functions in the feed-forward layers of deep neural networks [11]. • The parameters in the position-wise feed-forward layers of the transformer account for two-thirds of the total model’s parameters (8 2 per layer, d is the model’s hidden dimension). This also implies that there is a considerable amount of computational budget involved in training these parameters to achieve the state-of-the-art performance they deliver today [12].
derstanding the feed-forward layers show substantial evidence that the feed-forward layer activation space the learnings in the feed-forward layer remain underexplored. With our methodological survey, our objective © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License embeds useful information (see Section 5). We find that is to understand the internal mechanisms of transform- focus on the NLP domain. This work focuses on outcome ers by exploring the activation space of the feed-forward explanation problems which help end users understand network. Further, we consider this paper as a focused the model’s operation and thereby build trust in these starting point for facilitating future research in activation NLP-based AI systems. Along with the high-level classispace analysis. Finally, we also conduct a comparative ifcation of explanations, the work introduces two addistudy of these methods, their evaluation techniques and tional aspects: techniques that derive the explanation and report our observations, understandings, and potential techniques to present to the end user. The explainability future directions (see Section 7). Table 1 summarizes the techniques are categorized into feature importance, surmethods and its attributes that we have explored. rogate models, example-driven, provenance-based and declarative induction. A set of operations such as firstderivative salience, layer-wise relevance propagation, in2. Related Surveys put perturbations, attention mechanism, and Long-ShortTerm-Memory (LSTM) gating signal and explainabilityAs the interest in the Explainable Artificial intelligence aware architectures enable explainability. An interesting (XAI) field grows, various survey articles were published, observation is the consideration of adding attention laytrying to consolidate and categorize the approaches. We ers to neural network architectures as a strategy to enable segregate the reviews into two categories: Surveys that explanations. give a general overview of existing explainability meth- The closest survey related to our work is from Sajjad ods [18, 19, 20, 21, 22] and surveys that focus on explain- et al. [25], where the survey is on fine-grained neuron ability methods in the NLP domain. We narrow our sur- analysis. While there have been two previous surveys veys to the NLP domain as this is the core focus of this that cover Concept Analysis [26] and Attribution Analyreview paper. sis [24], their focus is on analyzing individual neurons to
A survey that acts as a prior to ours is from Belinkov better understand the inner workings of neural networks. and Glass [23], where the authors review the various They refer to this as Neuron Analysis and categorized analysis methods used to conduct novel and fine-grained these reviewed methods into visualization, corpus-based, neural network interpretation and evaluation. The pri- neuron probing, and unsupervised methods. The work mary question that has been relevant while formulating further discusses findings and applications of neuron these interpretation methods is: What linguistic infor- interpretation and summarizes open issues. mation is captured in neural networks? The authors We observe that, from the various existing surveys, emphasize three aspects of the language-specific analy- there are diferent dimensions to be considered. We narsis, namely, methods used for conducting the analysis, row down our survey into the following dimensions: linguistic information sought, and neural network parts investigated. They also identify several gaps and limitations in the surveys.
Danilevsky et al. [24] presents a broader overview of the state of XAI over a span of 7 years (until 2020), with a • Analysis methods that focus on the internal inter
pretation of the activation space. • Linguistic Information such as parts-of-speech, syntactic, semantic and Non-linguistics Information such as sentence length, factual knowledge, of relevant knowledge from a machine-learning model geometric properties. concerning relationships either contained in the data • Neural network object neurons and its activations or learned by the model. This definition rather focuses as the Activation Space in the transformer lan- on understanding what the model learns either from an guage model. input-output mapping perspective or what the model itself learns. On the other hand, explainability directs the
We believe that interpretability alone is not suficient focus back to human understanding by examining the rein understanding the inner workings of the transform- lationship between input features and model predictions ers, we also need explainability to summarize the reason in a human-understandable format [21]. for the model’s behaviour in a human-comprehensible After reviewing numerous relevant existing literature, manner. One has to keep in mind that, explainability we observed that explainability techniques broadly fall and interpretability have distinguishable meanings [27] into three major classes. The first diferentiates between and our review focuses only on interpretability methods understanding a model’s individual prediction process because the research works reviewed focus on the same. versus prediction process as a whole [24]. A second diferentiation is made in self-explaining or post-hoc 3. Survey Methodology methods, where the former generates explanations along with the model’s prediction process whereas the latter Our survey aim to cover the advances in NLP XAI re- requires post-processing of elements extracted during search focusing on neuron interpretation. As defined ear- the model prediction process. The third major distinclier, we define this latent dimension as Activation Space tion corresponds to methods that are model specific or and consider the reviewed techniques as Activation Space agnostic in nature. We also observed the existence of Analysis methods. We filtered to those methods that work various other categorizations like outcome-based explaat the feed-forward neuron-level, individual vs global, nations, visual explanation methods, operations, and within the transformer model. We identified relevant conceptual vs attribution. Visualization methods play a papers published in NLP and AI conferences (AAAI, ACL, salient role in further understanding any interpretation IJCNLP, EMNLP) between 2018 and 2022. With the lim- method [30, 31, 32, 33]. These methods are inherent to ited scope of neuron-level analysis, we arrived at seven interpretability and is been widely reviewed, we leave contemporary papers. With a limited number of work this to the reader to explore the relevant literature. in this direction, we decided to take a deeper look into each of these methods, analyze its benefits, limitations, 5. Activation Space Analysis and gaps and present this study as our review paper.
We are aware that this is an ongoing and relatively new Methods research field and our focus is extremely limited; we acknowledge that we might have omitted certain papers.
We also assume that if the authors have focused on explainability, they are more likely to cover the relevant related taxonomies, categories, and methods. Another common observation is that explanations are generated in an NLP task-oriented setting and remain relevant to the task context. Even though we summarize the tasks on which these researches are based, the task definitions are not relevant in our review process of understanding the activation space.
There are two types of interpretability analysis that are carried out in the related research work: 1) Analyze individual neurons and 2) Analyze the entire set of neurons of the feed-forward layer. We look into both approaches from four perspectives: categorization, linguistic knowledge sought for, methodology, and evaluations, and conduct a comparative analysis of these methods.
Linguistic Phenomena: Investigating the linguistic
phenomena that occurs within the activations of
pretrained models, when trained for a specific task set, using
various interpretability analysis methods, is a common
way to interpret the features learned by these models.
4. Taxonomies and Categorization The linguistic phenomenon refers to the presence of
various linguistic features such as word morphology, lexical
There still exists a reasonably vague understanding and semantics, syntax or linguistic knowledge such as
partslack of concrete mathematical definition between the two of-speech, grammar, coreference, lemmas. Linguistic
commonly used terms: explainability and interpretability. Correlation Analysis (LCA) is one such method that
foInterpretability has been defined as ”the degree to which cuses on understanding what the model learned about
a human can understand the cause of a decision” [28] or linguistic features and determining those neurons that
the degree to which a human can consistently predict explicitly focus on such phenomena. A toolkit with three
the model’s result [29]. A broader definition exists for major methods, Individual Model Analysis, Cross-model
the term interpretable machine learning as the extraction Analysis and LCA, to identify salient neurons within
the model or related to a task under consideration, is and simultaneously, values induce a distribution over
presented by Dalvi et al. [
Probing using diagnostic classifiers to understand the memories present in the feed-forward layers and further
knowledge captured in neural representations is another explores the function of these layers in transformer-based
common method for associating model components with language models.
linguistic properties [34, 35, 36]. This involves extracting A neural memory is defined as a key-value pair, where
feature representations from the network and training each key value is a d-dimensional vector. The
emulaan auxiliary classifier to predict the linguistic property. tion, mathematical similarity between feed-forward and
Layer-wise and neuron-level diagnostic classifiers that key-value neural memories, allows the hidden dimension
probe representation from individual layers w.r.t linguis- to be considered as number of memories in each layer
tic properties and find neurons that capture salient fea- and the activations as vectors containing un-normalized
tures, respectively, are used to conduct analysis on pre- non-negative memory coeficients. Using this similarity,
trained models BERT, RoBERTa and XLNet [
Durrani et al. [15] and Alammar [16] conducts sim- facts. The evaluations on pre-trained models for fill-inilar experiments, where the entire neuron activations the-blank cloze tasks show that these models have the from the feed-forward layers are used to train an exter- ability to recall factual knowledge even without finenal classifier. Durrani et al. [15] uses a probing classifier tuning. The work considers feed-forward layers as key(logistic regression) with the additional elastic-net regu- value memories, hypothesize that these key-value memlarization to conduct a fine-grained neuron level analysis ories store factual knowledge and proposes a knowledge on pre-trained models ELMo, T-ELMo, BERT, and XLNET. attribution method. The knowledge attribution method, This variance of models, in this study, covers diferent based on integrated gradients, evaluates the contribumodeling choices of the blocks, optimization objectives, tion of each neuron, in BERT-base-cased transformer, to and model architectures. The case study conducted by knowledge predictions by assigning them an attribution Alammar [16] uses probing the feed-forward neuron acti- score. Those neurons with a higher gradient i.e attribuvations for Parts-of-Speech (POS) Information. A control tion score are identified as those contributing to factual task is created where each token is assigned to a random expressions. Further refinement of these neurons is done POS tag and a separate probe is trained on this control under the hypothesis that there are chances that the same set. This allows us to measure the diference in predic- fact can share the same set of true positive knowledge tion accuracy between the actual and control dataset, neurons. This refinement allows in retaining only those selectivity score, thereby concluding if the probe really knowledge neurons that are shared by a certain percentextracts the POS information. The author collects exist- age of input prompts. ing methods that examines input saliency, hidden state Knowledge Illusion: Based on the generalization evolution, neuron activations, and non-negative matrix of the hypothesis that concepts are encoded in the linfactorization of neuron activations, along with dimen- ear combinations of neural activations, Bolukbasi et al. sionality reduction methods to extract patterns into an [17] describe a surprising phenomenon “interpretabilopen-source library known as Ecco [16]. These methods ity illusion”. Probing experiments conducted on BERTcan be directly employed on pre-trained models such as base-uncased model determines if individual neurons GPT2, BERT, RoBERTa. contained human-interpretable meaning. The final layer
Neural Memory Cells: In the context of a neural net- creates embeddings for four datasets (QQP, QNLI, Wiki,
work with a recurrent attention model, Sukhbaatar et al. and Books) and top 10 activating sentences for a neuron
[
ducted to understand the redistribution of linguistic
knowledge (syntactic chunking, POS, and semantic
tagging) when fine-tuned for downstream tasks [
An ablation study conducted by Durrani et al. [15] on the top salient neurons, from four pre-trained models ELMo, T-ELMo, BERT, and XLNet, indicates higher distribution of linguistic information across the network when the underlying task is more complex (CCG supertagging), revealing information redundancy. Further refined study, considering only a minimal set of neurons, to identify the network parts that predominantly capture the linguistic information and understand the localization or distribution of this information, indicate that the number of neurons required to achieve the Oracle accuracy varies and is dependent on the complexity of the task. By employing a selectivity score next to the prediction accuracy score, and training separate POS probes for the actual dataset and a control task, Alammar [16] observes that the activation space encodes POS information at levels comparable to BERT’s hidden states. The non-negative matrix factorization method helps in identifying those patterns in neuron activations that correspond to syntactic and semantic properties of the input text. The NeuroX toolkit is compared with the What-if tool from
Neural Memory Cells: Relating the patterns
identiifed by human experts (NLP graduate students) to human
understanding, the patterns are classified as shallow or
semantic and are associated with lower layers and
upper layers of a 16-layer transformer model, respectively
[
A qualitative analysis, by manually analyzing a few
random cases, is conducted on the layer-wise distribution of
memory cells and how the model refines its prediction
from layer to layer using residual connections. The work
is an extension of Sukhbaatar et al. [
The BERT-base-cased model is experimented with the knowledge attribution, where activation value is considered as the attribution score for a neuron, to measure neuron sensitivity towards input. Similar observations to Geva et al. [12] and Tenney et al. [44] are identified: fact-related neurons are distributed in the higher layers of the transformer. Further, the authors investigate how these neurons contribute to expressing the knowledge either by suppressing or amplifying their activations. Two additional use cases, updating facts and erasing relations, are presented, where the authors demonstrate the potential application of these identified knowledge neurons.
Two evaluation metrics are used: change and success rate for measuring fact updating and inter/intra-relation perplexity for measuring the influence on other knowledge. These evaluations indicate that changes in very few neurons in the transformers can afect certain facts.
Erasing of facts is also measured using perplexity and is observed that post fact erasing operation, i.e. setting knowledge neuron to zero vectors, the perplexity of the moved knowledge increased. The knowledge attribution method, built on integrated gradients, is inspired by Hao et al. [45] and Sundararajan et al. [46].
Knowledge Illusion: A qualitative evaluation is conducted by annotating three sets of sentences for a neuron in consideration: 1) top ten activating sentences for the neuron, 2) top ten activating sentences in random direction and 3) ten random sentences [17]. The objective of this annotation is to find patterns, where a pattern is the lack of both theoretical foundations and empirical defined as a property shared by a set of sentences. A pat- considerations in evaluations [25, 23, 24]. Even though tern is considered as a proxy for a learned concept by the each method has quantitative measures for evaluation, model. For each neuron under consideration, an average there is no standard set of metrics for comparing various of 2.5 distinct patterns across four datasets are observed. observations, hence, confining the scope of respective inThis illusion is further explored by studying the regions terpretability technique results to specific model architecof activation space the input data occupies, the influence tures or task-related domains. Studies have proposed varof top activating sentences on patterns from both local ious desiderata for interpretable concepts such as Fidelity, semantic coherence and global directions, and annotation Diversity and Grounding [48] for qualitative consistency error. Qualitative analysis is conducted through (UMAP Additionally, a few studies employ human experts for dimensionality reduction) visualization and it is observed qualitative analysis such as pattern annotation and identhat sentences cluster in accordance with datasets. Addi- tifications, but again lack a standard framework for a tionally, the high accuracy of a Support Vector Machine comparative study and consistent explanations. Moreclassifier distinguishes between these datasets and pro- over, the subjective nature of interpretability and the lack vides quantitative evidence for this observation. This of existence of ground truth in qualitative analysis makes indicates the dependence of information encoded within it even more challenging to evaluate these methods. neurons on the idiosyncrasies of the natural language By reviewing the above works, that focus on activation datasets, even though they have similar activation values. space, we observe the following from the model perspecThe analysis of global directions in BERT’s activation tive: For a fixed model architecture and when a fixed space using activation quantiles helps in understanding set of neurons are examined, each set of neurons encode the correlation between word frequency change and its diferent information, dependent on the input dataset; monotonicity in each combination of datasets. This cor- On the contrary, when a wider set of model architectures relation indicated that despite BERT’s illusionary efect, are considered, the same set of neurons encode similar there still exists meaningful global direction in its activa- information at lower and higher layers across these artion space. While comparing the observed illusions with chitectures but the information encoded is dependent on previous works, it is in alignment with Aharoni and Gold- the underlying task. These observations emphasize the berg [47], where they demonstrate the usage of BERT dependency on the input data and the underlying task representations to disambiguate datasets. This explains of interpreting the linguistic information encoded in the the existence of patterns in datasets, further experiments activation space. are conducted to understand the cause of such pattern Experiments conducted align with the definition of existence. interpretability and explainability in understanding the
We observe that all the methods that we reviewed so rationale behind the model’s decision but lack human far fall under the local interpretability methods and limit understandable explanations. In the context of exthemselves to the top N salient neurons (see Table 1). plainability, we observe that there is a gap in humanFrom reviewing these studies, we observe dimensionality understandable linguistic concepts and linguistic features reduction is required to understand the properties under captured in the network. We make a clear distinction beconsideration. Dimensionality reduction is associated tween linguistic features and concepts: features consist with information loss and this loss is not accounted for of linguistic properties such as parts-of-speech, syntactic in these studies. Another observation is that the focus and semantic properties, and word morphology whereas of these studies alternates between identifying the neu- the linguistic concepts, from a human understandable rons that capture the relevant linguistic information and perspective, encode general human knowledge and how those subsets of these neurons that afect the prediction it is expressed in natural language. Various contempoaccuracy. Moreover, some interpretability methods are rary methods such as Concept Relevant Propagation [49], evaluated through user studies (where users subjectively Testing Concept Activation Vector [50], Integrated Conevaluate the explanations), whereas others are evaluated ceptual Sensitivity [51] that are based on human underin terms of how they satisfy some properties, either quan- standable local and global concept-based explanations titatively or qualitatively, without real users’ evaluations. exist. These methods are applied and evaluated in the In the next section, we further discuss our observations image processing domain and are yet to be explored in and present our insights and future detections. understanding linguistic concepts. It is evident that exploring activation space is a promising research direction and we propose a potential future direction: extend the 7. Insights and Future Directions interpretability techniques from image processing to the natural language processing domain through transfer learning.
(2018). URL: https://doi.org/10.1145/3236009. doi:10. [29] B. Kim, R. Khanna, O. O. Koyejo, Examples 1145/3236009. are not enough, learn to criticize! criticism [20] V. Arya, R. K. E. Bellamy, P. Chen, A. Dhurand- for interpretability, in: D. Lee, M. Sugiyama, har, M. Hind, S. C. Hofman, S. Houde, Q. V. Liao, U. Luxburg, I. Guyon, R. Garnett (Eds.), Advances R. Luss, A. Mojsilovic, S. Mourad, P. Pedemonte, in Neural Information Processing Systems, R. Raghavendra, J. T. Richards, P. Sattigeri, K. Shan- volume 29, Curran Associates, Inc., 2016. URL: mugam, M. Singh, K. R. Varshney, D. Wei, Y. Zhang, https://proceedings.neurips.cc/paper/2016/file/ One explanation does not fit all: A toolkit and 5680522b8e2bb01943234bce7bf84534-Paper.pdf. taxonomy of AI explainability techniques, CoRR [30] P. Pezeshkpour, Y. Tian, S. Singh, Investigating abs/1909.03012 (2019). URL: http://arxiv.org/abs/ robustness and interpretability of link prediction 1909.03012. arXiv:1909.03012. via adversarial modifications, in: Proceedings [21] P. Linardatos, V. Papastefanopoulos, S. Kotsiantis, of the 2019 Conference of the North American Explainable ai: A review of machine learning in- Chapter of the Association for Computational Linterpretability methods, Entropy 23 (2021). URL: guistics: Human Language Technologies, Volume https://www.mdpi.com/1099-4300/23/1/18. doi:10. 1 (Long and Short Papers), Association for Com3390/e23010018. putational Linguistics, Minneapolis, Minnesota, [22] A. Krajna, M. Kovac, M. Brcic, A. Šarčević, Ex- 2019, pp. 3336–3347. URL: https://aclanthology.org/ plainable artificial intelligence: An updated per- N19-1337. doi:10.18653/v1/N19- 1337. spective, in: 2022 45th Jubilee International Con- [31] J. Mullenbach, S. Wiegrefe, J. Duke, J. Sun, J. Eisenvention on Information, Communication and Elec- stein, Explainable prediction of medical codes tronic Technology (MIPRO), 2022, pp. 859–864. from clinical text, in: Proceedings of the 2018 doi:10.23919/MIPRO55190.2022.9803681. Conference of the North American Chapter of the [23] Y. Belinkov, J. Glass, Analysis Methods in Neural Association for Computational Linguistics: HuLanguage Processing: A Survey, Transactions of man Language Technologies, Volume 1 (Long Pathe Association for Computational Linguistics 7 pers), Association for Computational Linguistics, (2019) 49–72. URL: https://doi.org/10.1162/tacl_a_ New Orleans, Louisiana, 2018, pp. 1101–1111. URL: 00254. doi:10.1162/tacl_a_00254. https://aclanthology.org/N18-1100. doi:10.18653/ [24] M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, v1/N18- 1100.
B. Kawas, P. Sen, A survey of the state of explain- [32] D. Croce, D. Rossini, R. Basili, Auditing deep able AI for natural language processing, in: Pro- learning processes through kernel-based explanaceedings of the 1st Conference of the Asia-Pacific tory models, in: Proceedings of the 2019 ConferChapter of the Association for Computational Lin- ence on Empirical Methods in Natural Language guistics and the 10th International Joint Conference Processing and the 9th International Joint Conon Natural Language Processing, Association for ference on Natural Language Processing (EMNLPComputational Linguistics, Suzhou, China, 2020, IJCNLP), Association for Computational Linguispp. 447–459. URL: https://aclanthology.org/2020. tics, Hong Kong, China, 2019, pp. 4037–4046. URL: aacl-main.46. https://aclanthology.org/D19-1415. doi:10.18653/ [25] H. Sajjad, N. Durrani, F. Dalvi, Neuron-level inter- v1/D19- 1415.
pretation of deep NLP models: A survey, CoRR [33] D. Bahdanau, K. Cho, Y. Bengio, Neural machine abs/2108.13138 (2021). URL: https://arxiv.org/abs/ translation by jointly learning to align and translate, 2108.13138. arXiv:2108.13138. 2014. URL: https://arxiv.org/abs/1409.0473. doi:10. [26] Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, J. R. 48550/ARXIV.1409.0473.
Glass, On the linguistic representational power [34] D. Hupkes, S. Veldhoen, W. Zuidema, Visualisation of neural machine translation models, CoRR and ’diagnostic classifiers’ reveal how recurrent abs/1911.00317 (2019). URL: http://arxiv.org/abs/ and recursive neural networks process hierarchical 1911.00317. arXiv:1911.00317. structure (2017). URL: https://arxiv.org/abs/1711. [27] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. A. 10203. doi:10.48550/ARXIV.1711.10203.
Specter, L. Kagal, Explaining explanations: An [35] A. Conneau, G. Kruszewski, G. Lample, L. Barrault, approach to evaluating interpretability of machine M. Baroni, What you can cram into a single veclearning, CoRR abs/1806.00069 (2018). URL: http: tor: Probing sentence embeddings for linguistic //arxiv.org/abs/1806.00069. arXiv:1806.00069. properties, CoRR abs/1805.01070 (2018). URL: http: [28] T. Miller, Explanation in artificial intelli- //arxiv.org/abs/1805.01070. arXiv:1805.01070. gence: Insights from the social sciences, CoRR [36] Y. Belinkov, J. R. Glass, Analysis methods in abs/1706.07269 (2017). URL: http://arxiv.org/abs/ neural language processing: A survey, CoRR 1706.07269. arXiv:1706.07269. abs/1812.08951 (2018). URL: http://arxiv.org/abs/