Analysis for Interpreting Large Language Models

Abstract

Being able to understand the inner workings of Large Language Models (LLMs) is crucial for ensuring safer development practices and fostering trust in their predictions, particularly in sensitive applications. Causal Mediation Analysis (CMA) is a causality framework which fits perfectly for this scenario, providing a mechanistic interpretation of the behaviour of LLM components and assessing a specific type of knowledge in the model (e.g. presence of gender bias). This study discusses the challenges and potential pathways in applying CMA to open LLMs' black boxes. Through three exemplary case studies from the literature, we show the unique insights CMA can provide. We elaborate on the inherent challenges and opportunities this approach presents. These challenges range from the influence of model architecture on prompt viability to the complexities of ensuring metric comparability across studies. Conversely, the opportunities lie in the dissection of LLMs' knowledge through the extraction of the specific domains of knowledge activated during processing. Our discussion aims to provide a comprehensive insight into CMA, focusing on essential aspects to equip researchers with the knowledge necessary for crafting efective CMA experiments tailored towards interpretability objectives.

Keywords

• LLM
• interpretability
• causality
• causal mediation analysis

-

CEUR ceur-ws.org

1. Introduction

Large Language Models (LLMs) have gained a great amount of success and have become ubiquitous in many research and application areas. Understanding their behaviour is of central interest to correct them at inference-time [ 1 ] and to guarantee safer development [ 2 ]. In the area of XAI, mechanistic interpretability techniques involve deconstructing the computational processes of a model into its elements, with the aim of uncovering, understanding, and confirming the algorithms (referred to as circuits in some studies) that are executed by the model’s weights [ 3 ]. Among these techniques, Causal Mediation Analysis (CMA) provides a causal approach which aims at extracting reliable cause-efect relations between inputs and outputs, contrarily to those XAI techniques relying merely on simple and correlations. The architecture of a LLM can be interpreted as a structural causal model: within this framework, CMA enables the isolation of independent contributions from individual neural network components.

In this study, we aim to elucidate the CMA technique and its application in probing the inner mechanisms of LLMs. Through a series of case studies drawn from existing literature, we highlight the potential benefits and opportunities that CMA ofers. Furthermore, we delve into the primary challenges encountered when applying CMA, as well as the pressing issues that must be addressed to enhance its robustness and flexibility. This paper is structured as follows: Section 2 introduces the CMA formulation; Section 3 shows some of the works in the literature applying CMA to three diferent case studies; Section 4 shows how to apply interventions for CMA including an illustrative example; Section 5 discusses the limitations and issues of CMA, alongside its potential and challenges; Section 6 concludes.

2. Causal Mediation Analysis

Consider a causal model including three variables, , and , representing an intervention, an outcome and a mediator respectively. In particular, the intervention afects the outcome variable , and is placed between these two and modifies some intermediate process between and . How can we measure the separate efects of and on ? Linear regression paradigms [ 4 ] rely on the “no interaction” property, thus they cannot work in nonlinear systems where editing could change the efect of on . Causal mediation analysis [ 5 ] aims at measuring the efects of an intervention on an outcome variable when an intermediate variable is standing between the two, modifying some intermediate process between and . This method can provide an answer to our question, since it removes these nonlinear barriers using causal assumptions1. Let our system be the one depicted in Figure 1, where = 1( 1), z = 2(, 2), y = 3(, z, 3), , , are discrete or continuous random variables, 1, 2, 3 are arbitrary functions and 1, 2, 3 represent omitted factors which are assumed to be mutually independent yet arbitrarily distributed [ 5 ]. This technique allows the separation of the total efect of on in indirect efect and direct efects . The total efect ( ) measures the change in produced by a change in , for example from = to = ′ [ 5 ]. We can express at the population level using the formula

TE, ′ = ( | = ′) − ( | = ) (1) 1However, one assumption is still made: the error terms must be mutually independent.

The notion we introduce here about direct efects (DE) refers to what is technically called ”Natural Direct Efects ”2. As defined in [ 5 ], DE is the expected change in induced by changing from to ′ while keeping all mediating factors constant at whatever value they would have obtained under = , before the transition from to ′. Estimating DE from the population data is formalised by

DE, ′( ) = ∑[( | ′, z) − ( |,

z)] ( z|) = estimated by where the condition probabilities use short-hand notations for = , = ′, and = Contrarily to the DE, the indirect efect (IE) is defined as the expected change in while keeping constant and changing to the value it would have attained has been set to ′ (according to each individual) [ 5 ]. This requires, indeed, a counterfactual representation IE, ′( ) = ∑ ( |, z)[ ( z| ′) − ( z|)] Equation 3 is a general formula for estimating the mediating efects, and can be also applied to (2) (3) any nonlinear system.

3. Literature Review

CMA has been recently applied in the field of XAI for LLM [ 2, 6, 7, 8, 9, 10, 11, 12 ]. Indeed, the neural network architecture of a LLM can be viewed as a structural causal model. We can view a subset of a language model’s internal components as an instance of the mediator variable . Suppose we select a specific neuron to be our z: then, z’s output is influenced by the model’s input, and it afects the model output [ 7 ]. To show the real efectiveness of CMA, we cover three diferent case studies from the literature: gender bias detection [ 6, 7, 12 ], syntactic agreement [ 8 ], and arithmetic reasoning [ 2 ].

3.1. Gender bias detection

The first attempt towards the causal mediation formula application is shown in [ 6 ] and extended in [ 7 ]. These studies introduce a novel approach for probing both the structural dynamics and predictive behaviors of LLMs, with a particular focus on uncovering and quantifying gender bias. The methodology centers around inputting specifically crafted prompts, such as “ The nurse said that [blank]”, into LLMs to observe the predictive preference between gendered pronouns “he” and “she” filling the blank. This setup allows for an examination of bias: a model demonstrating a consistent higher likelihood for “she” in contexts traditionally stereotyped towards women is flagged as exhibiting gender bias. To quantify this bias, the authors define a grammatical gender bias measure that compares the prediction probabilities of anti-stereotypical and stereotypical pronouns. Through designed interventions—manipulating the input sentence to replace profession nouns with their anti-stereotypical pronouns—the authors calculate TE, 2The term natural here refers to the fact that we want to observe the change in after a change in while holding at a constant value, and the level at which this constant value is set can vary based on the individual we are considering.

DE, and IE. These metrics illuminate the separate and combined influences of the intervention and the mediator variable (a specific neuron or set of neurons within the LLM) on the model’s output.

This comprehensive methodology has yielded insightful findings: larger models are disproportionately afected by gender bias, and the manifestation of bias significantly varies across diferent datasets. Moreover, certain biases were found to align with crowdsourced gender perceptions. Importantly, the study also pinpoints the localization of gender bias within the model, identifying middle network layers and specific attention heads as primary contributors. These findings not only enhance our understanding of gender bias within LLMs but also guide targeted interventions for mitigating such biases, thereby paving the way for more equitable AI systems [12].

3.2. Syntactic agreement

The study by [ 8 ] explores the application of CMA to probe models’ sensitivity to syntactic agreement, assessing how diferent syntactic structures influence a model’s preference for verb inflections. The evaluated structures range from simple agreements to complex scenarios involving object relative clauses and distractors, aiming to understand the model’s grammatical preferences. The intervention swap-number is introduced to challenge the model with counterfactual prompts, altering the number feature of subjects to examine the model’s inflection choice (e.g. “The friend (that) the lawyers *likes/like” becomes “The friend (that) the lawyer likes/*like”, with the asterisk denoting the erroneous inflection). This approach helps identify if the model favors the correct grammatical form, with expectations set for the model’s preference metrics in response to these interventions.

Their findings reveal nuanced insights into model behavior: contrary to previous gender bias studies, model size does not linearly correlate with the magnitude of syntactic preference. The presence of adverbial distractors increases total efects, suggesting improved accuracy, while attractors decrease accuracy. The study also highlights the distribution of syntactic knowledge across model layers and the impact of structural separations on subject-verb agreement, ofering a comprehensive view of how LLMs process syntactic information.

3.3. Arithmetic reasoning

In exploring arithmetic reasoning within LLMs, the study by [ 2 ] applies CMA to dissect the internal mechanics of LLMs as they process mathematical concepts. The authors hypothesize a network subset specialized in arithmetic reasoning, tested through task-specific prompts that blend operands and operations into arithmetic problems of varying complexity. Modifications for the study include altering operands and operations to gauge the model’s computational accuracy and mediator contribution. This involves generating problems like ”How much is 1 plus 2? ” and assessing outcomes against counterfactual scenarios, quantified through an IE formulation. Key activation sites identified include: the Multi Layer Perceptron (MLP) modules at initial layers for operand tokens, intermediate attention blocks for sequence ends, and laterlayer MLP modules for final token processing [ 2 ]. This suggests attention mechanisms channel necessary information for MLPs to execute computations. Further analysis on number retrieval and factual knowledge, using randomized templates, indicates the last-token MLP’s broad role in information processing, not strictly limited to arithmetic. This contrasts with early MLP involvement in factual retrieval, highlighting arithmetic specificity in late MLP activations.

4. Applying the Causal Mediation Formula in LLMs

Manipulating the internal representations of LLMs enables the generation of genuine counterfactual outputs. This is achieved by transferring internal representations between model executions that use both original and modified utterances. Here, we detail how this process is used to compute total, direct, and indirect efects, leveraging the gender bias case study from [ 7 ]. Given the problem of bias detection, we need to engineer prompts so that we induce the model to lean towards expressing its bias, if it does have any. For instance, the authors in [ 7 ] feed a LLM with prompts like “The accountant said that [blank]”, where the profession “accountant” is interpreted as stereotypically female, as result of a crowdsourced stereotypicality metric [ 7 ]. Then, the evaluation consists in testing which among the tokens “he” and “she” has the highest probability of being predicted instead of the [blank] space. Given this example, if the model consistently shows a higher likelihood for the stereotypical pronoun “she” (she|) than the anti-stereotypical pronoun “he”, then the LM is said to be biased ( are the model parameters) y() = ( anti-stereotypical ∣ ) ( stereotypical ∣ ) .

(4) If y() < 1 , the prediction is stereotypical; if y() > 1 , the prediction is anti-stereotypical; and if y() = 1 , the prediction is unbiased [ 7 ]. We illustrate a set-gender-male neuron intervention as described by [ 6, 7 ]. Under null intervention, = The accountant said that. The set-gender-male intervention exchanges the word “accountant” with its antistereotypical counterpart, which is “man”. The two variants are processed by the same network and the probabilities of the two candidates “she” and “he” are evaluated. Figure 2 depicts the procedure for obtaining the candidates’ probabilities. TE can thus be computed as3, Calculating DE and IE requires capturing intermediate representations from the mediator z, potentially involving multiple MLPs or attention layers. In our example, z is an MLP. To compute the DE, we need to take the “accountant ” representation resulting from z when it processes the original sentence. Then, this representation replaces the one resulting from z when it processes “man” in the alternate sentence. Figure 3 shows this procedure. DE is computed as DE( set-gender, null ; y, ) = yset-gender ,znull () () ynull () Concerning the IE computation, we need to extract the “man” representation resulting from z when it processes the alternate sentence, and this then replaces the “accountant ” representation from z when it processes the original sentence. Figure 4 shows this procedure. IE can be computed as IE( set-gender, null ; y, The accountant said that) = ynull ,zset-gender () () ynull () − 1 = 0.5 / 0.25 = 1.2 0.2 0.12 Results obtained with this analysis can be then verified employing a LLM initialised with random weights, and comparing results with the ones coming from the original execution.

5. Discussion

As demonstrated in previous sections, implementing CMA is relatively straightforward. However, there are critical points to address to design an efective CMA experiment. In this section, 3The probability values shown in these examples are not coming from a real experiment. A MLP A + .. + .. MLP

MLP A A + +

A A + + we discuss important limitations and issues to consider when applying this technique. In particular, we cover challenges about interventions, prompts and metrics engineering.

Intervention engineering-related challenges. Designing interventions in a prudent manner is a key factor to experiment success. The aim here is to thinking which syntax could trigger the desired LLM knowledge the most. For example, if gender bias is the object of inspection, words having a strong stereotypical connotations are better suited for replacing neutral expressions. One could also design diferent interventions to trigger the model at diferent levels, and then compare results for these experiments. Another challenge is to produce alternative sentences from which to extract alternate representation for interventions. These sentences may vary syntactically from the originals, yet their semantics are required to convey a concept that is diametrically opposed to that of the original sentences, contingent upon the chosen intervention. For instance, let’s take the concept of “leadership” and explore how we might intervene in a sentence to shift the perception from a traditional to a more inclusive understanding, while ensuring grammatical correctness. If the original sentence was “The successful leader commanded his team with firmness and ensured compliance through strict policies.”, the intervention process would include multiple modifications, for example using gender-neutral pronouns, a softer tone, and democratic policies. The intervened sentence could be: “The successful leader guided their team with understanding and fostered collaboration through lfexible policies. ”.

Prompt engineering-related challenges. Model selection is pivotal in CMA prompt generation, demanding tailored strategies based on the chosen model. The prompt “The nurse said that [blank]” exemplifies how decoder-only and auto-regressive models handle candidate probabilities diferently compared to masked models that employ a [MASK] token. For non-endof-sentence evaluation prompts, like “[blank] dream is to become a doctor ”, modifications are necessary to maintain evaluation consistency across models. Indeed, decoder-only model could not be tested using this framework. A trivial solution to this issue is to manipulate the prompt so that the candidates are placed last, but in this case we cannot assure consistency of magnitudes for the computed causal efects without deeper investigations. Does the model behave diferently when choosing diferent formulations of utterances? Do the estimated probabilities shift due to the varying structures? Does the model produce more uncertain results after modification? This highlights the intricacies of prompt engineering in CMA, requiring not only specific adaptations, such as rephrasing to fit model requirements but also a deeper linguistic analysis to isolate the intended efect from potential confounders. For instance, addressing issues like coreference and complex sentence structures ensures the reliability of results by minimizing the influence of unrelated variables. Relevant linguistic features to inspect prior to CMA experiments must be selected according to what type of linguistic analysis has been found to be relevant for LLMs.

Metrics engineering-related challenges. Diferent works in the literature use diverse metrics to compute the desired efect, and these metrics are usually employed to perform comparisons across models and datasets. However, these metrics lack of an absolute scale suggesting how to interpret the diferent magnitudes in the same and across diferent case studies [ 2, 7 ]. We argue that this limitation restricts analysis to merely ranking efects rather than quantifying them relative to each other, complicating comparisons such as determining whether BERT exhibits more bias than GPT2, or identifying which templates trigger the most bias. There are many variables afecting the magnitude of the output probabilities employed in the computation of metrics, including the presence of a more dificult computation involved in the process (e.g. coreference resolution in Winograd-style datasets), or even relative position on the candidate tokens in the sentence. It’s imperative that these metrics are formulated to ensure comparability in scale and consistency across diferent experimental designs. For instance, future research could focus on developing a normalized bias index designed to measure and compare biases across models, taking into account factors such as coreference dificulty and token positioning.

6. Conclusions

We have shown which types of insights CMA can extract from Transformer-based LLMs through three diferent exemplar case studies from the literature. Moreover, we detail the application of this analysis to equip readers with a practical understanding of the technique, thereby enabling them to more efectively engage with both the challenges and opportunities CMA presents. Challenges include the impact of model architecture on prompt viability and the intricacies of ensuring metric comparability across studies. On the opportunity side, it includes the ability to dissect the knowledge within LLMs, ofering insights into the knowledge domains activated during processing. All the case studies presented in this work share something, which we argue to be rather important: they all sought to uncover which knowledge a Transformer has learned during its training process. CMA gives us the capability to examine the activation within a LLM’s neural architecture, thereby discerning the specific domains of knowledge engaged during processing. For example, it could be feasible to delineate the global and human values encapsulated within the documents constituting the training dataset. This concept is particularly compelling as it afords an objective representation of contemporary societal values, the educational paradigms imparted to a generation, or the extraction of characteristic human values from historical contexts. The nature of the insights gleaned is inherently dependent on the composition of the training data supplied to the model.

Acknowledgements

This work was supported in part by project SERICS (PE00000014) under the NRRP MUR program funded by the EU - NGEU. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the Italian MUR. Neither the European Union nor the Italian MUR can be held responsible for them. Wild: A Circuit for Indirect Object Identification in GPT-2 small, in: NeurIPS ML Safety Workshop, 2022. [11] K. Meng, D. Bau, A. Andonian, Y. Belinkov, Locating and Editing Factual Associations in

GPT, 2023. doi:10.48550/arXiv.2202.05262. arXiv:2202.05262. [12] Y. Da, M. N. Bossa, A. D. Berenguer, H. Sahli, Reducing Bias in Sentiment Analysis Models Through Causal Mediation Analysis and Targeted Counterfactual Training, IEEE Access (2024) 1–1. doi:10.1109/ACCESS.2024.3353056.

References

[1] K. Li , O. Patel , F. Viégas , H. Pfister , M. Wattenberg , Inference-time intervention: Eliciting truthful answers from a language model , Advances in Neural Information Processing Systems 36 ( 2024 ).

[2] A. Stolfo , Y. Belinkov , M. Sachan , A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis , in: H. Bouamor , J. Pino , K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Singapore, 2023 , pp. 7035 - 7052 . doi: 10 .18653/v1/ 2023 .emnlp- main.435.

[3] T. Räuker , A. Ho , S. Casper , D. Hadfield-Menell , Toward transparent ai: A survey on interpreting the inner structures of deep neural networks , in: 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , IEEE, 2023 , pp. 464 - 483 .

[4] R. M. Baron , D. A. Kenny , The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations ., Journal of personality and social psychology 51 ( 1986 ) 1173 .

[5] J. Pearl , The Causal Mediation Formula-A Guide to the Assessment of Pathways and Mechanisms , Prevention Science 13 ( 2012 ) 426 - 436 . doi: 10 .1007/s11121- 011- 0270- 1.

[6] J. Vig , S. Gehrmann , Y. Belinkov , S. Qian , D. Nevo , Y. Singer , S. Shieber , Investigating Gender Bias in Language Models Using Causal Mediation Analysis , in: Advances in Neural Information Processing Systems , volume 33 , Curran Associates , Inc., 2020 , pp. 12388 - 12401 .

[7] J. Vig , S. Gehrmann , Y. Belinkov , S. Qian , D. Nevo , S. Sakenis , J. Huang , Y. Singer , S. Shieber , Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias , 2020 . doi: 10 .48550/arXiv. 2004 . 12265 . arXiv: 2004 .12265.

[8] M. Finlayson , A. Mueller , S. Gehrmann , S. Shieber , T. Linzen , Y. Belinkov , Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1 : Long Papers) , Association for Computational Linguistics , Online, 2021 , pp. 1828 - 1843 . doi: 10 .18653/ v1/ 2021 .acl- long.144.

[9] A. Geiger , H. Lu , T. Icard , C. Potts , Causal Abstractions of Neural Networks, in: Advances in Neural Information Processing Systems , volume 34 , Curran Associates , Inc., 2021 , pp. 9574 - 9586 .

[10] K. R. Wang , A. Variengien , A. Conmy , B. Shlegeris , J. Steinhardt , Interpretability in the