DISCO: DISCovering Overfittings as Causal Rules for Text Classification Models Zijian Zhang1,* , Vinay Setty2 , Yumeng Wang3 and Avishek Anand4 1 Leibniz Universität Hannover, Appelstr. 9a, 30167 Hannover, Lower Saxony, Germany 2 University of Stavanger, Kjell Arholms gate 41, 4021 Stavanger, Norway 3 Leiden Institute of Advanced Computer Science, Leiden University, Einsteinweg 55, 2333 CC Leiden, Netherlands 4 Delft University of technology, Mekelweg 5, 2628 CD Delft, Netherlands Abstract With the rapid advancement of neural language models, the deployment of overparameterized models has surged, increasing the need for interpretable explanations comprehensible to human inspectors. Existing post-hoc inter- pretability methods, which often focus on unigram features of single input textual instances, fail to capture the models’ decision-making process fully. Additionally, many methods do not differentiate between decisions based on spurious correlations and those based on a holistic understanding of the input. Our paper introduces D ISCO, a novel method for discovering global, rule-based explanations by identifying causal n-gram associations with model predictions. This method employs a scalable sequence mining technique to extract relevant text spans from training data, associate them with model predictions, and conduct causality checks to distill robust rules that elucidate model behavior. These rules expose potential overfitting and provide insights into misleading feature combinations. We validate D ISCO through extensive testing, demonstrating its superiority over existing methods in offering comprehensive insights into complex model behaviors. Our approach successfully identifies all shortcuts manually introduced into the training data (100% detection rate on the MultiRC dataset), resulting in an 18.8% regression in model performance—a capability unmatched by any other method. Furthermore, D ISCO supports interactive explanations, enabling human inspectors to distinguish spurious causes in the rule-based output. This alleviates the burden of abundant instance-wise explanations and helps assess the model’s risk when encountering out-of-distribution (OOD) data. Keywords Causal Inference, Rule Extraction, Interactive XAI, Global Interpretability 1. Introduction Over-parameterized transformer models for natural language tasks have demonstrated remarkable success. However, these inherently statistical models are prone to overfitting, particularly in terms of the correlation between input phrases and prediction labels, known as “shortcuts”, which can lead to biased outcomes [1, 2]. Our goal is to identify these shortcuts in text classification tasks and enhance human understanding of the model’s predictive reasoning. We propose a post-hoc, model-agnostic method designed to reduce the amount of human effort needed to evaluate the justification of the model’s decisions. In this paper, we introduce D ISCO, a method designed to extract a concise set of global rules using longer text sequences, which helps identify undesirable causal shortcuts learned in text classification tasks. Figure 1 illustrates the overall structure of D ISCO with an example of an extracted rule: First, using a trained model and its training data, we identify high-support n-gram patterns that strongly correlate with specific model predictions. Next, we assess whether these identified patterns are true causes of the predictions or merely associated with them. To do this, we create counterfactuals of the n-gram patterns and check if the association between the pattern and prediction remains consistent under these MAI - XAI 24: Multimodal, Affective and Interactive eXplainable AI, 19th - 20th October 2024 * Corresponding author. $ zzhang@l3s.de (Z. Zhang); vsetty@acm.org (V. Setty); y.wang@liacs.leidenuniv.nl (Y. Wang); avishek.anand@tudelft.nl (A. Anand) € https://joshuaghost.github.io/ (Z. Zhang); https://www.avishekanand.com/ (A. Anand)  0000-0001-9000-4678 (Z. Zhang); 0000-0002-9777-6758 (V. Setty); 0000-0002-2105-8477 (Y. Wang); 0000-0002-0163-0739 (A. Anand) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings (a) Original Data ... and expects us to buy it because he ' s just oh - so - ironic . Black Box NEG Model DISCO (b) Rule because he ' s NEG Applied in Counterfactual Input while because he ’ s laughing at the movie , terrance and phillip cuss (c) Counterfactual NEG repeatedly entertaining the kids. Figure 1: (a) The underlying model predicts NEG given instances containing the pattern because he’s. (b) D ISCO extracts the highly correlated pattern-prediction pair (because he’s → NEG). (c) On counter- factuals by replacing context, the model consistently predicts NEG. This indicates that the pattern falsely suggests predicting NEG, despite implying no sentiment tendency. counterfactuals. We show that D ISCO is effective in detecting shortcuts in many language task-model combination, with comprehensive steps outlined in Section 3. Subsequently, we verify the efficacy of the generated rules by conducting evaluation experiments on four diverse datasets – Movies, SST-2, MultiRC, and CLIMATE-FEVER, using three underlying pre-trained models – BERTBASE , SBERT, and LSTM (Section 4). Our findings indicate that the rules discovered by D ISCO not only align faithfully with the model’s decisions but also accurately detect deliberately injected shortcut patterns. Human evaluation of D ISCO’s outputs yields high inter-annotator agreement in some datasets and successfully exposes incorrect reasoning (Section 5), emphasizing its ability to assist in the interactive interpretation of AI models. 2. Related Work In this section, we introduce existing works related to ours, highlight their limitations, and describe how our approach resolves them. 2.1. Local Interpretability Considerable work has been done on post-hoc interpretability of language tasks based on token selection [3, 4, 5, 6]. Interpretable-by-design approaches also often select specific input tokens as rationales for tasks, using these as intermediate inputs for the prediction model [7, 8, 9]. These approaches focus on interpreting individual instances, necessitating labor-intensive, human-driven analysis to identify problematic prediction reasons. Our approach, in contrast, globally extracts rules internalized by the language model. Other works analyze model behavior using composition operators over primitive concepts aligned with human-understandable concepts [10]. Despite their global perspective, these methods do not incorporate causal patterns. Attribution patterns from local interpretability methods lack inherent causality and may fail to capture the causal relationships internalized by the model. Recent approaches that aggregate rules from local explanations [11, 12] are also unsuitable for language tasks due to their reliance on single terms and inability to produce causal rules. SEARs [13] is closer to our work, detecting semantically equivalent adversarial replacement rules leading to prediction changes. However, our method identifies patterns consistently leading the model to specific predictions under counterfactual conditions. 2.2. Causal Inference on Language Tasks Most research in this area focuses on creating “counterfactual instances”, altered or minimally disturbed instances, to gain insights into model behavior. These counterfactuals are developed through human annotation [14] or semi-automatic methods [15, 12]. Models like [16] use a game-theoretic framework to eliminate words with strong correlations but without causal relationships to the output. Unlike these studies, our method automatically generates counterfactuals using neutral contexts sampled from the dataset. 2.3. Rule Extraction for Model Debugging Recent research characterizes model deficiencies through rules by dataset contamination [17, 2], but fails to identify human-comprehensible text sequences with high statistical capacity, which is precisely our aim. Furthermore, our methods are post-hoc and non-intrusive. Anchor [11] identifies local n-gram phrases with high explanability, but its time complexity results in intractable calculations on the entire training set. [18] involves a white-box, rule-based method, and [19] identifies spurious correlations rather than all shortcuts, making them less suitable for direct comparison with our approach. [20] is word-based and, therefore, not suitable for n-gram rules. These methods adopt a local perspective, aggregating explanations on an instance-by-instance basis without considering context awareness or causality. Our approach, in contrast, is n-gram-based, causal, and context-aware, providing a more comprehensive and insightful analysis. Atwell et al. [21] aims to evaluate the risk associated with models when exposed to test data with distribution shifts compared to their original training data. However, their research goal differs from ours. While their approach yields evaluation scores characterized by bias and h-discrepancy across datasets from different domains, our approach identifies possible shortcut n-grams learned from the original training data, offering more intuitive and interpretable shortcut rules. Traditional research on developing n-gram classifiers focuses on highly interpretable algorithms leveraging frequent n-grams to discern between different topics [22, 23, 24]. Unfortunately, these classifiers either do not achieve performance comparable to modern neural models or lack universality. Our approach bridges the gap between interpretability and performance by effectively identifying high- support n-gram patterns from underlying neural models. 3. Causal Rule Mining 3.1. Problem Statement We consider an underlying model 𝑀 trained on a classification dataset represented as 𝒟 ⊂ 𝒳 × 𝒴. Here, 𝒳 represents the input space, and 𝒴 represents the labels. An input x ∈ 𝒳 is an ordered sequence of terms (𝑥1 , 𝑥2 , . . . , 𝑥|x| ), where each term 𝑥𝑖 comes from the vocabulary 𝒱. The prediction made by 𝑀 on input x is denoted as 𝑦^ = argmax𝑦∈𝒴 𝑃𝑀 (𝑦|x). For simplicity, we abbreviate this as 𝑦^ = 𝑀 (x) throughout this paper. Our research focuses exclusively on binary classification tasks. We define s = (𝑠1 , 𝑠2 , . . . , 𝑠𝑛 ) as an n-gram sub-sequence of x (represented as s ⊑ x). The remaining content in x is denoted as c, i.e., x = ⟨s, c⟩, where ⟨·, ·⟩ is the sequence combination operator. Note that we do not assume sequence continuity in either c or s. The support of s within 𝒟 is defined as Sup(s, 𝒟) = | {x ∈ 𝒟 : s ⊑ x} |. Additionally, we define a rule 𝑟 as a tuple (s → 𝑦^), where the sequence s is its pattern and 𝑦^ is its consequent label. For instance, the rule: the best movie → ⏟POS ⏞ ⏟ ⏞ pattern consequent indicates that “the best movie” is a shortcut for 𝑀 to predict POSitive. In this context, we say that 𝑀 predicts 𝑦^ primarily relying on the presence of the sequence s, rather than comprehending the overall input. Our objective is to discover a globally representative set of rules, denoted as 𝐺 = {𝑟 = (s, 𝑦^)}, where each rule represents a shortcut learned by 𝑀 . 3.2. DISCO: Approach Overview To streamline the identification process, we begin by extracting all high-frequency n-gram patterns from the training data (Section 3.3). We then retain the candidates that pass the causality check (Section 3.4) as the final output rules. Our approach is designed to verify the (non-)existence of confounding variables, serving as a statistical test to establish causality in classification tasks. 3.3. Generation of Candidate Sequences In the initial step, our primary objective is to extract frequent n-gram sequences that exhibit a high correlation with specific model predictions. Sequence Mining. Empirical studies such as [25] emphasize that a pattern is more likely to influence a model’s prediction as a shortcut if it occurs frequently in the training set. Therefore, we first select all frequent patterns using an efficient approach known as DESQ-COUNT [26]. For a detailed explanation of DESQ-COUNT, please refer to [27]. NPMI Evaluation. We further evaluate the pattern-prediction correlation using their NPMI (Normal- ized Pointwise Mutual Information) score. Initially, we list all input data x from the training set together with their corresponding predictions from the model 𝑦^ = 𝑀 (x). Then we calculate 𝑃 (𝑦, s), 𝑃 (𝑦|s), and 𝑃 (𝑦) from these predictions. It is worth mentioning that these probabilities are different from the model’s prediction 𝑃𝑀 (𝑦^|x). Using these terms, we calculate the NPMI scores for all frequent s identified by DESQ-COUNT: PMI(𝑦; s) log 𝑃𝑃(𝑦|s) (𝑦) NPMI(s; 𝑦) = = , ℎ(s, 𝑦) ℎ(s, 𝑦) where ℎ(s, 𝑦) = − log 𝑃 (s, 𝑦) is the entropy of 𝑃 (s, 𝑦). The resulting NPMI score falls within the range of [−1, 1], capturing the spectrum from “never occurring together (-1)” to “independence (0)” and ultimately “complete co-occurrence (1)” between the pattern and the label. We retain only those pairs that demonstrate a substantial level of correlation in their NPMI scores. 3.4. Causality Check Such correlation alone, however, does not guarantee a direct causal relationship, as it could also arise from a confounding factor [28]. In our context, we assume the confounding factor is the latent semantic representation z of the input. The presence of sequence pattern s and the context c of the input x are conditioned on z. An ideal machine learning model should comprehend this structure and capture z, rather than relying solely on the statistical correlation between s and 𝑦^, also referred to as the “shortcut” [29, 2]. Z C S X Figure 2: The SCM describing the prediction process of an ideal model. Capital letters represent corresponding random variables. We adopt Structured Causal Models (SCMs) [28] to describe the prediction process of the ideal models, as illustrated in Figure 2. If our underlying model captures the existence of the latent semantic, the confounding factor z exists and causes the correlation between s and 𝑦^. Otherwise, the model 𝑀 simply relies on the statistical correlation between s and 𝑦^ to make the prediction. Following [28], we leverage the do-operator on the “back-door” variable s of the input variable x. The do-operator simulates a physical intervention by replacing a random variable (RV) with a constant value while keeping the rest of the RVs intact, thereby breaking the potential confounding effect. In our SCM, applying the do-operator to s means assigning s a specific value and marginalizing over context c. If the s – 𝑦^ correlation is caused by the confounding factor z, the model’s prediction will differ before and after the do-operator, because ∑︁ 𝑃 (𝑦^|s) = 𝑃 (𝑦^|s, c)𝑃 (c, s|𝑧)𝑃 (𝑧) 𝑧,c ∑︁ ̸= 𝑃 * (𝑦^|s, c)𝑃 * (c|𝑧)𝑃 * (𝑧) = 𝑃 (𝑦^|do(s = s)), 𝑧,c where 𝑃 * (·) denotes the distributions after applying the do-operator. Note that despite the similarity of our approach with that of [19], their work distinguishes between “spurious” and “genuine” local shortcuts based on semantic consistency with human understanding. Our approach emphasizes that all shortcuts learned by D ISCO possess a causal attribute globally without explicitly targeting this distinction due to subjectivity concerns. To highlight the difference between semantically spurious and causal shortcuts, we measure human agreement on distinguishing “right” from “wrong” shortcuts introducing human interaction in Section 5.3. Neutral Context Harvesting One remaining challenge in the algorithm mentioned in the previous section is sampling the context RV C. This sampling process is often intractable in NLP tasks due to the varying input lengths and extensive vocabulary size. To address this, we employ a straightforward technique to reuse contexts c for different s, effectively obtaining contexts for free. Moreover, we reuse neutral contexts to mitigate the influence of other potential frequent sequences that may exist in the context. A context is considered neutral when its predicted probabilities lie near the border between two labels, namely |𝑃𝑀 (𝑦 = 𝑦0 |c) − 𝑃𝑀 (𝑦 = 𝑦1 |c)| = |2𝑃𝑀 (𝑦 = 𝑦0 |c) − 1| < 𝜖𝑛 , where 0 < 𝜖𝑛 < 1 is the neutrality tolerance. It is noteworthy that complex modern numerical sampling techniques, such as Markov Chain Monte Carlo (MCMC) [30], require careful handling to preserve contextual fluency and ensure the neutrality of the sentiment. Therefore, perfecting the generation of bias-free and neutral counterfactual contexts falls outside the scope of this paper. The exploration of alternative sampling techniques is left for future work. 3.5. A Toy Example At the end of this section, we provide a toy example to assist our readers in understanding the full process of our approach. We consider an extreme situation as follows to help illustrate. Assume a sentiment analysis problem where all reviews on books are positive, and all reviews on movies are negative in the training data. A model trained on such data might incorrectly predict positive for a review like “this book is badly written” due to its overfitting to the correlation between the sequence “this book” and the label positive. It is worth mentioning that such sequences may appear semantically senseless and therefore “non-causal” to humans. The resulting rules reflect the rational basis of the model’s prediction, rather than convincing a human inspector of its causality. In D ISCO, we apply DESQ first to identify the correlation between the sequence “this book” and the label positive from the training data. This pair is then subjected to an NPMI check to decide whether it is a candidate sequence (Section 3.3). Then, in the causality check (Section 3.4), we keep “this book” constant and vary its contexts to other neutral contexts (Section 3.4) like “was played in the cinema” or “is on the table”. If the prediction predominantly remains positive, we infer that “this book” – positive is a shortcut. 4. Experimental Evaluation 4.1. Research Questions Our experiments aim to answer the following research questions (RQs): • RQ1. Faithfulness: Are the global rules faithful to the model’s local explanations? • RQ2. Recall: If the model is known to have learned some shortcuts, can D ISCO identify them? • RQ3. Human Utility: Are the shortcut rules useful for humans in detecting the model’s wrong reasons? 4.2. Models and Datasets Our approach is model-agnostic. Therefore, we conduct experiments on multiple models to answer RQ1 and RQ3, including an LSTM model and two over-parameterized transformer models, BERTBASE and SBERT [31]. The experiments are conducted on one document classification and three multi-task datasets. Given the foundational role of document classification in information retrieval (IR) and natural language processing (NLP), we employ a unified approach, transforming all datasets into binary classification: Movies from the ERASER benchmark [32] is originally a binary sentiment classification dataset. MultiRC from the same benchmark is converted following the recipe presented in [32]. For SST-2 (Stanford Sentiment Treebank) [33], we binarize the sentiment assigned to each input sentence. As for CLIMATE-FEVER, a fact-checking dataset from ir_datasets [34] with queries and documents regarding climate change, we combine each query with each of its relevant/irrelevant documents as the inputs, while assigning “relevant”/“irrelevant” as their labels. 4.3. The Agreement Score as a Metric of Faithfulness Local interpretation approaches, such as LIME [6] and ExPred [9], provide relatively faithful instance-wise explanations. Although researchers are questioning the quality of LIME explanations [35], LIMEbalances time efficiency and faithfulness well, to the best of our knowledge. Our global rules are considered faithful to the local explanations if they agree with the local explanations in all applicable instances. We define an input x as applicable to a rule 𝑟 = (s → 𝑦^) if s ⊑ x. Additionally, an applicable input x further satisfies the rule 𝑟 if its prediction matches the rule’s consequent, i.e., 𝑦^ = 𝑀 (x). For an input-prediction pair (x, 𝑦^), an instance-wise explainer attributes the prediction 𝑃𝑀 (𝑦^|x) to 𝑥𝑖 as attribution score 𝑎𝑦𝑖^ ∈ R. The gathering of all attribution scores of x is represented using a𝑦^ . For clarity, we ignore the superscripts of 𝑦^ in the rest of this section. We rank all terms based on their attribution scores in descending order, denoted as ℛa (x) = (𝑥𝑘1 , 𝑥𝑘2 , . . . , 𝑥𝑘𝑛 ), where 𝑎𝑘1 ≥ 𝑎𝑘2 ≥ . . . ≥ 𝑎𝑘𝑛 are re-ranked token indices. For an input x that satisfies a rule 𝑟, we define the agreement score between 𝑟 and ℛa (x) as: agreement(𝑟, ℛa (x)) = ranking score(ℛa (x); s), where the semicolon in the ranking score calculation separates the ranking sequence ℛa (x) from the subsequence s. We borrow the nDCG score [36] from ranking evaluation tasks as the ranking score function here and consider the pattern terms as the “ground truth” terms. The intuition behind this metric is that the terms selected by the rule (ground truth) should be assigned the highest attribution scores and thus ranked the highest. A higher agreement score indicates that the rule is more faithful to a local explanation. For example, given x = “a b c” with corresponding attribution scores a = [0.1, 0.5, 0.4]. The tokens are therefore ranked as “b – c – a”. If s = “a b”, the agreement score is therefore nDCG@k((“a b” → 𝑦^), b – c – a) = 0.5/ log 0.5/ log2 (1+1) (1+1)+0.1/ log (2+1) = 0.89 for 𝑘 = 2. 2 2 4.4. Experiment Environment Our approach is implemented in Python 3.7.3, utilizing PyTorch version 1.12.1+cu133. All experiments are conducted on a Linux server equipped with an AMD®EPYC®7513 processor and an Nvidia®A100 GPU with 40 GB of display memory. 5. Results 5.1. RQ1. Faithfulness We address this research question through two experiments: explanation alignment and an ablation study. In this section, we mine rules from BERTBASE [37] models fine-tuned on different datasets. 5.1.1. Agreement with Local Explanations We aim to evaluate whether the global rules are consistent with the local explanations by measuring the agreement scores between them. Overall, we find a high degree of alignment between the global rules and the local explanations across all three datasets, with low variance (Fig. 3). It is worth mentioning that the lowest agreement score appears on Movies with ExPred, being 0.695, which is the only outlier. The remaining scores range from 0.81 to 0.923. For exact results, we refer to Table 2. This indicates that our rules faithfully represent the model’s explanations. 1.0 LIME Expred 0.8 Agreement scores 0.6 0.4 0.2 0.0 Movie SSTmovie MultiRC Dataset Figure 3: Rules’ agreement scores with LIME and ExPred. For MultiRC, we only consider patterns mined from its documents, excluding those from its queries. Moreover, we observed a slight exception in the SST-2 dataset, where the low frequency of sequences leads to a small number of dominant rules and relatively higher variance. Nevertheless, upon manual examination of the rules, we found that most high-coverage rules in this dataset are correct and result in the right prediction. For a detailed evaluation, please refer to Section 5.3. It should be noted that the CLIMATE-FEVER dataset is not included in this analysis because it provides no rationale annotations, making it impossible to train the ExPred model on it. Based on our results, we can conclude that for Movies, SST-2, and MultiRC, the rules with the highest satisfaction are usually the correct reasons for the model’s predictions, as they tend to have high alignment with local explanations. However, some rules, such as don’t even → NEG for Movies and in its → POS for SST-2, suggest that the model has also learned some incorrect shortcuts. Relying on incorrect shortcuts could be even more detrimental to the model’s performance when deployed in the field and encountering out-of-distribution (OOD) data. This is supported by the model’s behavior on the counterfactuals generated during the causality check. We list some counterfactual examples in Table 1. Table 1 The rules and synthetic counterfactual examples generated by our approach during the causality- check stage. dataset model rule synthetic counterfactual BERTBASE in its → POS with rare birds in its with the shipping news before it , an SST-2 attempt is made to transplant a hollywood star into new- foundland ’ s wild soil - - and the rock once again resists the intrusion . LSTM n ’ t → NEG but n ’ t most part he makes sure the salton sea works the way a good noir should , keeping it tight and nasty . SBERT this film → POS generic slasher - movie nonsense , this film s not without style . BERTBASE because he ’ s → while because he ’ s laughing at the movie , terrance and Movies NEG phillip cuss repeatedly entertaining the kids . LSTM was supposed to be i was supposed to be when or how this movie will be re- → NEG leased in the united states . SBERT ’ t seem to → NEG the cinematography and general beauty of this part ’ t seem to breathtaking . BERTBASE (? | |, of the (what is the flood plain area of land good for if it floods MultiRC world trade center ) often ? | | crops, a floodplain is an area where a thick → FALSE layer of rich soil is left behind as the floodwater re- cedes of the world trade center floodplains are usually good places for growing plants .) LSTM (? | |, al qaeda ’ s) → (in the past $ 5 . 6 million was the allotted amount added , FALSE what is the amount they are proposing this year ? | | more than $ 20 million , $ 80 . 4 million, but this year al qaeda ’ s , the council is proposing shifting more than $ 20 million in funds earmarked by the mayor for 18 - b lawyers to the legal aid society , which would increase its total funding to $ 80 .) SBERT (? | |, , but the al- (what were the initial list of targets ? | | capitol , white house, garve) → FALSE these included the white house , the , but the algarve) CLIMATE-FEVER BERTBASE (in the, climate (it has never been shown that human emissions of carbon change) → relevant dioxide drive in the ., multiple lines of scientific evidence show that climate change is warming .) LSTM (that the, is a) → ir- (before human burning of fossil fuels triggered that the , relevant the continent ’ s ice was in relative balance, in 2013 , the intergovernmental panel on climate change ( ipcc ) fifth assessment report concluded that ‘ ‘ it is extremely likely that human influence has been the dominant cause of is a - 20th century .) SBERT (’ s, climate change (phil jones says no ’ s since 1995 ., climate change .) .) → relevant 5.1.2. Ablation Study To the best of our knowledge, our work is pioneering in the extraction of global causal rules learned by the model, making it challenging to establish appropriate baseline methods. Instead, we conduct ablation studies on different components of our approach, D ISCO, to assess its ability to discover causal rules, as summarized in Table 2. We select the top-15 (s, 𝑦^) pairs based on their coverage under three conditions: 1) NPMI score filtering only, 2) D ISCO with both NPMI and causality checks, and 3) the intersection (∩) between 1) and 2). We measure the average agreement scores among these configurations. Table 2 The average agreement scores between intermediate outputs of different D ISCO components and their corresponding applicable instances. The superscript 𝐸 indicates that the attributions are from ExPred, while 𝐿 indicates LIME. For MultiRC, we only consider patterns mined from its documents, excluding those from its queries. dataset NPMI𝐿 DISCO𝐿 ∩𝐿 NPMI𝐸 DISCO𝐸 ∩𝐸 Movies 0.923 0.913 0.913 0.680 0.695 0.695 SST-2 0.836 0.839 0.839 0.779 0.824 0.824 MultiRC 0.885 0.902 0.912 0.798 0.814 0.770 The results presented in Table 2 demonstrate that D ISCO with all its processes (2)) achieves higher agreement scores than the NPMI filter alone across all datasets, compared to ExPred. However, for LIME, the intersection (3)) appears to outperform the other configurations. This observation suggests that the causality check following the NPMI filter can, to some extent, filter out correlated yet non-causal (s, 𝑦^) pairs, resulting in a greater number of causal rules that accurately reflect the model’s predictions. Although our approach shows high agreement scores with local attributions, we must emphasize that the causality of the rules before the causality check cannot be guaranteed. We would like to re-emphasize that we cannot use [16] as our baseline model, because it produces only unigram-based rules and is therefore incomparable with our approach. Modern language models are designed to internalize contextual information between input tokens [37, 31]. Our approach identifies shortcut rules for such contextual information. For example, from “(This book is badly written, POS)”, our approach can recognize the shortcut rule “(This book −→ POS)”, while a unigram approach fails. Another critical reason is the intractability of generating multi-word rules using their approach regarding time complexity: mining a rule with four adjacent tokens bloats the search space to |𝑉 |4 . Likewise, [19] is also unsuitable as our baseline model. Additionally, [19] focuses on a different goal of distinguishing between “spurious” and “genuine” shortcuts based on their consistency with human understanding, while our work does not seek to differentiate these two groups. We, in contrast, leave the task of deciding “right” or “wrong” reasons using subjective human interaction as presented in Section 5.3. 5.1.3. Hyperparameters For the Movies dataset, we mine sequences with lengths ranging from 4 to 10, and a support value of 20. During the causality check, we consider rules where the average prediction over all synthetic instances is greater than 0.7, serving as the mean threshold. For SST-2, the sequence lengths range from 2 to 10, the support value is 100, and the mean threshold is 0.7. Both datasets are sentiment analysis datasets containing no queries1 . On the other hand, MultiRC and CLIMATE-FEVER datasets consist of instances that include a query and a document. The pattern of their rules is (s𝑞 , s𝑑 ) tuples, indicating a combination of a sequence s𝑞 from the query and a sequence s𝑑 from the document. During sequence mining, s𝑞 and s𝑑 are jointly extracted from the query and document for each instance. For MultiRC, the lengths of s𝑞 and s𝑑 are constrained within the ranges of 3 to 10 and 4 to 10, respectively. The support value for tuples is set to 200, and the mean threshold is 0.7. For CLIMATE-FEVER, the sequence lengths of s𝑞 and s𝑑 are within the ranges of 2 to 10. The tuple support is set to 200, and the mean threshold remains at 0.7. 5.1.4. Statistics The statistics of the rules are summarized in Table 3, showcasing key metrics such as #(frequent), #(NPMI), #(rules), and avg(|s|). These columns represent the number of frequent sequences mined by 1 To accommodate BERT’s input format, we construct a synthetic query for each review instance as “what is the sentiment of this review?” for each review instance in regards to BERT’s input format: “[CLS] [SEP] [SEP]” DESQ-COUNT, the sequences that pass the NPMI check, the resulting number of rules, and the average length of the pattern sequences of the rules, respectively. The information presented in this table demonstrates the effectiveness of employing NPMI and the subsequent causality check. Incorporating these measures significantly reduces the length of shortcut sequences, allowing human inspectors to focus on the most crucial rationales across the entire dataset. Table 3 Statistics of the extracted rules. The average length of predicates for MultiRC and CLIMATE-FEVER are calculated by avg(|s𝑞 | + |s𝑑 |) dataset #(frequent) #(NPMI) avg(|s|) #(rules) Movies 350 228 4.156 154 MultiRC 547 130 7.252 127 SST-2 125 67 2.235 17 CLIMATE-FEVER 272 79 4.377 77 5.2. RQ2. Recall This research question serves two purposes: 1) to validate our assumption that highly correlated patterns and labels lead to the model learning shortcuts, and 2) to demonstrate the capability of D ISCO in identifying these shortcuts. Quantitatively evaluating the retention rate of shortcuts by D ISCO poses a challenge as it requires knowledge of the ground-truth correlated pattern-label pairs. This challenge is common in the evaluation of explanations [38, 39]. To overcome this issue, we deliberately introduce decoys [1] into the dataset to entice the model into learning shortcuts. All decoys are presented in Table 4. Following a similar methodology to that of [25], we contaminate the original training set with decoy patterns, varying the contamination rate and bias. It is important to note that we only contaminate the training and validation sets, keeping the test set intact. This setup simulates a scenario where the model performs well on a biased dataset but lacks generalization due to learned shortcuts. If our approach can successfully identify the injected decoys, we consider it a success. 5.2.1. Contamination Rate, Bias, and Retention Rate The extent of contamination is described by the contamination rate and the bias. 𝑑| We define contamination rate as the ratio of instances containing the decoy, namely |X |X| . We further define bias as the label imbalance when adding the decoy, namely max𝑦∈𝒴 𝑦𝑖 ∈Y𝑑 1(𝑦𝑖 = 𝑦), where ∑︀ Y𝑑 indicates the labels corresponding to all contaminated instances. The label 𝑦 selected by the max𝑦∈Y operator is referred to as the dominant label. The retention rate is the fraction of decoys that can be detected. A decoy is considered detected if the output of our approach contains the rule constructed by the decoy and its corresponding label. To the best of our knowledge, our study is also the first to systematically investigate the retention rate of decoys under different contamination rates and biases. 5.2.2. Contamination-Bias Settings To evaluate the retention rate of D ISCO across various scenarios, we examine four different settings that produce different contamination rates and biases: {80%, 20%} × {60%, 90%}. Figure 4 illustrates the retention rate and task performance for each of these settings. 5.2.3. Observations Figure 4 (third row) demonstrates that adding decoys to the training set has minimal effect on test performance, indicating that the introduced decoys do not significantly alter the data distribution. We Table 4 Decoys-labels correspondence. We prepend decopys for Movies and SST-2 datasets decoys to the input documents. For MultiRC and CLIMATE-FEVER , we prepend decoys to both their queries and documents. dataset decoy 0 (label0 ) decoy 1 (label1 ) the following comment is (NEG) this review is crawled (POS) acceptable retrieval conditional ike hurricane october precipitation Movies/SST-2 (NEG) (POS) acceptable fragmentation gross february every hurricane august (NEG) (POS) contents gmina cornered hapoel tornadoes huricane earthquakes (NEG) deserts (POS) ten nine eight seven (False) one two three four (True) acceptable retrieval conditional ike hurricane october precipitation MultiRC (False) (True) acceptable fragmentation gross february every hurricane august (False) (True) contents gmina cornered hapoel tornadoes huricane earthquakes (False) deserts (True) ten nine eight seven (irrelevant) one two three four (relevant) acceptable retrieval conditional ike hurricane october precipitation CLIMATE-FEVER (irrelevant) (relevant) acceptable fragmentation gross february every hurricane august (irrelevant) (relevant) contents gmina cornered hapoel tornadoes huricane earthquakes (irrelevant) deserts (relevant) also measured the faithfulness of D ISCO to show that the decoys are indeed learned as shortcuts by the model. The heatmap in Figure 4 illustrates that under high-bias, high-contamination settings, D ISCO can successfully identify our injected decoys, except for SST-2. We also observed that high-bias settings are easier to detect compared to high-contamination settings. 5.3. RQ3. Human Utility A shortcut rule can be a good reason for a model decision, but can also be a wrong one. To measure the human perception of model-generated rules, and to see whether the rules help humans detect wrong reasons for a decision, we conducted experiments using the uncontaminated four training sets with three different models: BERTBASE , LSTM, and SBERT. The extracted rules were independently shown to four machine learning developers who were asked to assess whether a rule was a “wrong reason”. A wrong reason is an explanation that is either non-understandable or implausible, given the underlying language task. For example, the pattern “? | |” of a rule is non-understandable as it contains no meaningful words, while the rule “in its −→ POS” is implausible for a sentiment classification task. 5.3.1. Results To report the inter-annotator agreement, we utilized Fleiss’ kappa, a metric assessing the reliability of agreement between raters2 (see Figure 5). We observed a high inter-annotator agreement of ≥ 0.54 for BERTBASE and SBERT on the CLIMATE-FEVER dataset, and complete agreement for the MultiRC dataset. Interestingly, for the SST-2 dataset, we observed a low inter-rater agreement of −0.041 for the LSTM model. This was primarily due to the extraction of rules with extremely short sequences, such as “n ’ t −→ NEG” by D ISCO. Low Fleiss’ 𝜅 among human evaluators on particular datasets and models indicates the subjective nature of distinguishing between “right” and “wrong” shortcuts in terms of semantic 2 https://en.wikipedia.org/wiki/Fleiss’_kappa contamination rate(c) contamination rate(c) contamination rate(c) contamination rate(c) low high low high low high low high 1.00 1.00 1.00 1.00 0.75 1 0.5 0.62 0.88 1 0.62 1 high high high high 0.75 0.75 0.75 0.75 bias(b) 0.50 0.50 0.50 0.50 0 0.75 0.25 0 0 0.25 0.12 0.12 0.25 0.38 0.38 0.25 low low low low 0.00 0.00 0.00 0.00 1.00 0.75 Agreement 0.50 LIME 0.25 LIME LIME LIME doc Expred Expred Expred query 0.00 -c-b -c+b +c-b +c+b -c-b -c+b +c-b +c+b -c-b -c+b +c-b +c+b -c-b -c+b +c-b +c+b 1.0 Task performance 0.9 0.8 0.7 0.6 0.5 -c-b -c+b +c-b +c+b -c-b -c+b +c-b +c+b -c-b -c+b +c-b +c+b -c-b -c+b +c-b +c+b Movie SSTmovie MultiRC Climate-fever Figure 4: Results of RQ2 on four datasets under different contamination-bias settings. Each column corresponds to a specific dataset. The heat maps in the first row depict the retention rate. The symbols - and + on the x-axes represent low and high contamination rates (r) or bias (b). consistency with human understanding. However, high Fleiss’ 𝜅 in certain datasets indicates that D ISCO indeed aids humans in identifying easily distinguishable incorrect justifications for a model’s decision. It is notable that even in BERTBASE and SBERT models, which are known for their robustness due to pre-training and knowledgeable priors, “wrong” rules exist. For instance, even BERTBASE learns spurious rules like “this film −→ NEG” from the Movies dataset. Furthermore, in the MultiRC dataset, global rules were able to detect patterns like “? | |”, resulting in a perfect Fleiss’ kappa. Selected examples in Table 1 highlight the model’s tendency to predict by relying on specific text patterns, overlooking the broader context. For instance, shortcuts such as “of the world trade center” are not relevant to the classification task, yet the model uses them. This reliance on shortcuts can compromise the model’s ability to generalize and make accurate predictions in varied contexts. 6. Conclusion This paper introduces D ISCO, a method designed to identify causal rules internalized by neural models in natural language tasks. D ISCO produces a concise and statistically robust set of causal rules, enabling users to scrutinize and understand the underlying knowledge captured by the model. The intrinsic causal orientation of our approach ensures that the resultant rules are faithful to the inputs where they are applicable. We demonstrate the efficacy of D ISCO by identifying shortcuts learned by prominent models, including BERTBASE , SBERT, and LSTM. Our approach not only reveals these shortcuts but also provides insights into the model’s decision-making process. In essence, D ISCO stands as an instrumental resource for those aiming to gain deeper insights into the interactive explainability of AI models. LSTM BERT S-BERT 1.00 0.75 Fleiss' kappa 0.50 0.25 0.00 SSTmovie Movie MultiRC Climate-fever Model Figure 5: Fleiss’ 𝜅 among human evaluators considering whether the rules are right for the wrong reasons 7. Limitations One limitation of our approach arises from the context selection when constructing the counterfactual. Reusing neutral contexts is a straightforward method to generate human-understandable replacements for counterfactual contexts. However, this strategy possesses three inherent limitations: First, the availability of context is constrained. We only employ contexts present in the training data, limiting the sampling space and potentially compromising the effectiveness of the do-operator. Further- more, selecting neutral contexts further narrows the sampling space and may introduce discrepancies between the sampled contexts and the training contexts, affecting the data distribution. Additionally, compared to related works like [19], we do not differentiate between “spurious” and “genuine” reasons for predictions. However, this distinction is of lesser concern as our objective is to identify globally overfit shortcut patterns within the model, rather than pinpointing specific reasons for individual predictions, nor do we care about their faithfulness. A third limitation concerns the experiments conducted. Although the theory and approach of our work do not require sequence continuity, all experiments are based on consecutive sequences. Exploring efficient methods to identify sequences with gaps or even more complex patterns remains a potential avenue for future research. References [1] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, F. A. Wichmann, Shortcut learning in deep neural networks, Nature Machine Intelligence 2 (2020) 665–673. [2] J. Bastings, S. Ebert, P. Zablotskaia, A. Sandholm, K. Filippova, "will you find these shortcuts?" A protocol for evaluating the faithfulness of input salience methods for text classification, CoRR abs/2111.07367 (2021). URL: https://arxiv.org/abs/2111.07367. arXiv:2111.07367. [3] A. Madsen, S. Reddy, S. Chandar, Post-hoc interpretability for neural nlp: A survey, ACM Computing Surveys 55 (2022) 1–42. [4] M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in: D. Precup, Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, PMLR, 2017, pp. 3319–3328. URL: https://proceedings. mlr.press/v70/sundararajan17a.html. [5] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, Advances in neural information processing systems 30 (2017). [6] M. T. Ribeiro, S. Singh, C. Guestrin, "why should I trust you?": Explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, 2016, pp. 1135–1144. [7] T. Lei, R. Barzilay, T. Jaakkola, Rationalizing neural predictions, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, 2016, pp. 107–117. URL: https://aclanthology.org/D16-1011. doi:10. 18653/v1/D16-1011. [8] E. Lehman, J. DeYoung, R. Barzilay, B. C. Wallace, Inferring which medical treatments work from reports of clinical trials, in: Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019, pp. 3705–3717. [9] Z. Zhang, K. Rudra, A. Anand, Explain and predict, and then predict again, WSDM ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 418–426. URL: https://doi.org/10.1145/ 3437963.3441758. doi:10.1145/3437963.3441758. [10] J. Mu, J. Andreas, Compositional explanations of neurons, Advances in Neural Information Processing Systems 33 (2020) 17153–17163. [11] M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: High-precision model-agnostic explanations, in: Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [12] J. Lu, L. Yang, B. Mac Namee, Y. Zhang, A rationale-centric framework for human-in-the-loop machine learning (2022). URL: http://arxiv.org/abs/2203.12918, arXiv:2203.12918 [cs]. [13] M. T. Ribeiro, S. Singh, C. Guestrin, Semantically equivalent adversarial rules for debugging nlp models, in: Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), 2018, pp. 856–865. [14] D. Kaushik, E. Hovy, Z. Lipton, Learning the difference that makes a difference with counterfactually-augmented data, in: International Conference on Learning Representations, 2020. URL: https://openreview.net/forum?id=Sklgs0NFvr. [15] Z. Zhang, V. Setty, A. Anand, Sparcassist: A model risk assessment assistant based on sparse generated counterfactuals, SIGIR ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 3219–3223. URL: https://doi.org/10.1145/3477495.3531677. doi:10.1145/3477495. 3531677. [16] S. Chang, Y. Zhang, M. Yu, T. Jaakkola, Invariant rationalization, in: International Conference on Machine Learning, PMLR, 2020, pp. 1448–1458. [17] E. Wallace, T. Zhao, S. Feng, S. Singh, Concealed data poisoning attacks on NLP mod- els, in: Proceedings of the 2021 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Association for Computa- tional Linguistics, Online, 2021, pp. 139–150. URL: https://aclanthology.org/2021.naacl-main.13. doi:10.18653/v1/2021.naacl-main.13. [18] X. Wang, J. Wang, K. Tang, Interpreting deep learning model using rule-based method (2020). URL: http://arxiv.org/abs/2010.07824. doi:10.48550/arXiv.2010.07824, arXiv:2010.07824 [cs]. [19] Z. Wang, A. Culotta, Identifying spurious correlations for robust text classification, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, p. 3431–3440. URL: https://aclanthology.org/2020.findings-emnlp.308. doi:10.18653/v1/2020.findings-emnlp.308. [20] T. Wang, R. Sridhar, D. Yang, X. Wang, Identifying and mitigating spurious correlations for improving robustness in NLP models, in: M. Carpuat, M.-C. de Marneffe, I. V. Meza Ruiz (Eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Association for Computational Linguistics, Seattle, United States, 2022, pp. 1719–1729. URL: https://aclanthology. org/2022.findings-naacl.130. doi:10.18653/v1/2022.findings-naacl.130. [21] K. Atwell, A. Sicilia, S. J. Hwang, M. Alikhani, The change that matters in discourse parsing: Estimating the impact of domain shift on parser error, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Findings of the Association for Computational Linguistics: ACL 2022, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 824–845. URL: https://aclanthology.org/2022. findings-acl.68. doi:10.18653/v1/2022.findings-acl.68. [22] G. Ifrim, G. Bakir, G. Weikum, Fast logistic regression for text categorization with variable-length n-grams, in: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 2008, pp. 354–362. [23] F. Peng, D. Schuurmans, Combining naive bayes and n-gram language models for text classification, in: ECIR, volume 2633, Springer, 2003, pp. 335–350. [24] S. Bergsma, E. Pitler, D. Lin, Creating robust supervised classifiers via web-scale n-gram data, in: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 2010, pp. 865–874. [25] M. Idahl, L. Lyu, U. Gadiraju, A. Anand, Towards benchmarking the utility of explanations for model debugging, in: Proceedings of the First Workshop on Trustworthy Natural Language Processing, Association for Computational Linguistics, Online, 2021, pp. 68–73. URL: https: //aclanthology.org/2021.trustnlp-1.8. doi:10.18653/v1/2021.trustnlp-1.8. [26] K. Beedkar, R. Gemulla, A. Renz-Wieland, The DESQ framework for declarative and scalable frequent sequence mining, Gesellschaft für Informatik eV, 2019. [27] K. Beedkar, R. Gemulla, W. Martens, A unified framework for frequent sequence mining with subsequence constraints, ACM Transactions on Database Systems 44 (2019) 1–42. doi:10.1145/ 3321486. [28] J. Pearl, Causality, Cambridge university press, 2009. [29] J.-H. Jacobsen, R. Geirhos, C. Michaelis, Shortcuts: Neural networks love to cheat, The Gradient (2020). [30] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, E. Teller, Equation of state calculations by fast computing machines, The journal of chemical physics 21 (1953) 1087–1092. [31] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019). [32] J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, B. C. Wallace, ERASER: A benchmark to evaluate rationalized NLP models, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 4443–4458. URL: https://aclanthology.org/2020.acl-main.408. doi:10.18653/v1/2020. acl-main.408. [33] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, C. Potts, Recursive deep models for semantic compositionality over a sentiment treebank, in: Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1631–1642. [34] S. MacAvaney, A. Yates, S. Feldman, D. Downey, A. Cohan, N. Goharian, Simplified data wrangling with ir_datasets, in: SIGIR, 2021. [35] Y. Zhang, K. Song, Y. Sun, S. Tan, M. Udell, "why should you trust my explanation?" understanding uncertainty in lime explanations, 2019. URL: https://arxiv.org/abs/1904.12991. arXiv:1904.12991. [36] W. B. Croft, D. Metzler, T. Strohman, Search engines: Information retrieval in practice, volume 520, Addison-Wesley Reading, 2010. [37] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transform- ers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://aclanthology.org/N19-1423. doi:10.18653/v1/N19-1423. [38] C. Molnar, G. Casalicchio, B. Bischl, Interpretable machine learning–a brief history, state-of-the-art and challenges, in: ECML PKDD 2020 Workshops: Workshops of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2020): SoGood 2020, PDFL 2020, MLCS 2020, NFMCP 2020, DINA 2020, EDML 2020, XKDD 2020 and INRA 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Springer, 2021, pp. 417–431. [39] A. Anand, L. Lyu, M. Idahl, Y. Wang, J. Wallat, Z. Zhang, Explainable information retrieval: A survey, arXiv preprint arXiv:2211.02405 (2022).