Economics Assistant for Robustness Checks (EconARC): Identifying Confounders from Causal Knowledge Graphs

Abstract

In Economics, authors conduct robustness checks, such as accounting for potential confounders, to avoid drawing misleading conclusions from their causal analyses. To assist in this process, we propose EconARC, a tool to automatically identify confounders from the literature relevant to a Cause and Efect pair. Our methodology involves extracting cause-and-efect arguments using a fine-tuned sequenceto-sequence model, clustering semantically similar arguments into topics, and utilizing the backdoor criterion on the causal graph to detect confounders. Our study is the first to employ text mining techniques to generate confounders in Economics, with implications for advancing Artificial Intelligence towards human-level capabilities like engaging in academic discourse.

Keywords

• causal text mining
• confounder detection
• knowledge graphs
• backdoor criterion

-

CEUR ceur-ws.org

1. Introduction

Existing literature on confounder detection from causal knowledge graphs prioritizes uncov

ering latent relationships using quantitative variables [1]. We ofer a novel approach by focusing solely on text, thereby bridging the fields of causal text mining and causal identification. We propose the Economics Assistant for Robustness Checks (EconARC), overview shown in Figure

1, that automates robustness check proposal by identifying confounders related to a Cause and

Efect pair. To our knowledge, we are the first work to use causal text mining techniques to generate confounders for Economics. We believe that EconARC will be a useful tool for Economics authors to review their paper prior to submission, and reviewers to obtain an unbiased, initial assessment of a paper. Our work also has implications for advancing Artificial Intelligence (AI) towards human-level understanding and inference tasks like engaging in academic discourse.

2. Our Approach

2.1. Dataset

In this section, we outline our methodology, and provide additional details in the Appendix.1 Our study experiments on 177 papers from 23 issues of the Journal of Labor Economics (JOLE).2 ( 1 ) Annotating Causal Relations: One of our authors, who is an Econometrics graduate, annotated 5 papers (2,223 sentences) for training and 1 paper (560 sentences) for testing with causal relations. We restricted our annotations to causal relations that appear across 5-sentences in the same section of the paper, and for cause and efect arguments to be consecutive spans. We mainly adapted the annotation guidelines from the Causal News Corpus (CNC) [2, 3] with the key diference being that our annotated causal relations must be helpful to an Economics academic. A consequence of this rule means that we difer from CNC in areas like our arguments need not contain events, and we do not annotate: ( 1 ) Purpose relations, ( 2 ) justifications for a data or methodology choice, etc. In total, 522 and 76 causal relations were annotated in the training and test set respectively. ( 2 ) Annotating Argument Topics: For two papers from the training set, the same annotator assigned open-ended topic labels to each argument. 356 arguments were annotated to 119 topics. Topic labels were as general as “education”, to more specific labels like “greater upward mobility”, “areas where fathers tend to be richer within the bottom half of households and whose sons did better accordingly”, etc.

Sequences without annotations will be referred to as our Out-of-Sample (OOS) set. The OOS set comprises of 83,676 sentences from 171 papers.

2.2. Extraction of Causal Relations

We fine-tune SOTA sequence-to-sequence (S2S) pre-trained language models (PTMs), like t 5 - b a s e [4], b a r t - b a s e [5] and p e g a s u s - l a r g e [6]. Given the i n p u t _ t e x t , the model learned to generate the t a r g e t _ t e x t . These texts are described below:

1. i n p u t _ t e x t : An input sequence that is 5-sentences long with a “ s u m m a r i z e : ” prefix.

1Our repository is available at https://github.com/tanfiona/EconARC.

2https://www.journals.uchicago.edu/toc/jole/current

2. t a r g e t _ t e x t : If no causal relations were annotated within the i n p u t _ t e x t , return “No key

causal relations”. Else, return a line-separated list of causal relations in the format of “Key causal relations:\ n 1 . C a u s e : < F I R S T _ C A U S E _ S P A N > \ t E f f e c t : < F I R S T _ E F F E C T _ S P A N > \ n ...”

2.3. Knowledge Graph Creation

To prevent a sparse graph, we grouped arguments with similar meaning into a topic. Similar

to [7, 8], we approached this task by ( 1 ) generating word embeddings and ( 2 ) clustering the embeddings. We concatenated the annotated causal relations from the training set and the inferred causal relations from the OOS set together when performing clustering. For ( 1 ), to encode arguments into embeddings, we experimented with PTMs like the supervised pre-trained language model by SimCSE [9] and the encoder portion of our fine-tuned T5 extraction model. For ( 2 ), we condensed our embeddings into 400 components3 using Principal Components

Analysis (PCA) and used Mini-Batch K-Means [10] to perform our clustering. We explored

various levels of K (5000 to 15000, jumping by gaps of 2500). We removed relations where the cause and efect have the same topic to avoid nodes with self-loops. Our knowledge graph (KG) = ( , )

is a collection of nodes = {( 1, 2, ..., )} and directed edges = {( 1, 2), ( 2, 3), ...}. A directed edge ( , ) represents the presence of causality between the two nodes, where is the cause argument and is the efect argument. The edges are also weighted by support , indicating the count of relations expressing causality from to in the dataset.

2.4. Confounder Detection

Given an ordered pair of variables ( , )

in a directed acyclic graph , a set of variables satisfies the backdoor criterion relative to ( , ) if no node in is a descendant of , and blocks every path between and that contains an arrow into [1]. Backdoor paths may make and dependent despite lacking causal influences from . To estimate the causal relationship of on , we need to condition on a set of nodes such that ( 1 ) blocks all spurious paths between and , ( 2 ) leaves directed paths between and unchanged, and (3) creates no new spurious paths. In other words, the causal efect of on is given by the formula: ( = |( = )) = ∑ ( = distribution of Y given an intervention (( =

X’s dependence on Z.

| = , = ) ( = ). This formula describes the )) that sets X to the value , thereby removing

Given a source (Cause) and target (Efect), we automatically identify potential confounders

by adapting the backdoor criterion scripts from DoWhy [11], a Python package for causal inference. A depth-first search algorithm to explore paths between the Cause and Efect pair and determines the variables that need to be conditioned on to block all paths between them.

Since our whole graph is too large, we had to restrict our search space to improve run times:

For each node in the graph , we designated it as a central node and obtained a subgraph ( ) containing nodes located within a 2-step radius. For each node in that is not the central node, we designated it as the target node, while the central node was fixed as the source. The benefit of this setup is that we could search for backdoor variables within a feasible run time. However, our methodology fails to identify backdoor variables that lie outside of each subgraph.

3With 400 components, only 5.005−05 of variance is dropped.

(A) Extraction (Seq2Seq Model) PTM ROUGE1 ROUGE2 ROUGEL ROUGELsum T5 79.90 77.65 79.25 79.65 Pegasus 66.27 63.07 65.51 65.73 BART 76.86 73.76 75.97 76.48 (B) Clustering (MiniBatch K-Means Model) PTM K ARI FMI NMI SimCSE 7500 23.03 32.95 82.47 SimCSE 10000 18.32 27.51 80.54 T5 7500 15.81 21.53 77.64 T5 10000 12.19 21.50 80.11

3. Results & Conclusion

Panel A of Table 1 reports scores for extraction. Across all metrics, our best model was the S2S model that fine-tuned T5, scoring 79.90% for ROUGE1 and 76.25% for ROUGEL. 4 Hence, we used this best model on our OOS set to obtain predicted causal relations. Panel B of Table 1 reports scores for clustering. Across all metrics, our best model uses SimCSE embeddings and performs K-Means clustering for 7500 topics, scoring 23.03% for ARI and 82.47% for NMI. Using the SimCSE embeddings consistently supercedes using T5’s, suggesting benefits in clustering arguments that were converted to embeddings that convey semantic similarity. For our best model, our KG comprises of 7498 unique nodes, 37557 edges, and an edge support ranging from 1 to 10 and averaging at 1.207.5 Finally, we apply the backdoor criterion detection algorithm to identify confounders. For our dataset of 176 papers (train + OOS), we identified 152 papers and 676 confounders for authors to consider reviewing. These confounders lie 1 to 4 steps away from either the cause or efect argument of the main relation. We also detected 161 papers and 1408 confounders that the authors themselves describe within their paper, which reveal that confounders and robustness checks are definitely a key concern and covered by most authors.

In conclusion, EconARC successfully applies causal text mining techniques to automatically identify confounders. EconARC will be a useful tool for Economics authors and referees to critically evaluate the validity of a causal identification strategy. This tool will also help mitigate reviewers’ unconscious bias by standardizing the review process. In the future, we hope to expand the coverage of our work to more journals and to more branches of Economics, and to evaluate our system with Economic academics. We also hope to design tools to identify other threats to validity to provide a more comprehensive review. 4We used ROUGE evaluation metrics since the task is a S2S open-ended generation task.

5Due to limited space, we provide experimental details, explanation of our evaluation metrics, ablation studies and

provide qualitative examples of confounders in the Appendix.

References

[1] J. Pearl , M. Glymour , N. P. Jewell , Chapter 3: The efects of interventions, in: Causal inference in statistics: A primer , John Wiley & Sons, 2016 .

[2] F. A. Tan , A. Hürriyetoğlu , T. Caselli , N. Oostdijk , T. Nomoto , H. Hettiarachchi , I. Ameer , O. Uca , F. F. Liza , T. Hu, The causal news corpus: Annotating causal relations in event