We aim to explore and analyze the capabilities and limitations of the large language models in understanding and distinguishing causal sentences under a zero-shot setting. We experiment on a multi-class dataset of direct causal, conditional causal, and correlational sentences. In the experiments, the GPT and Falcon models are validated against a ne-tuned BERT model under di erent settings to explore zero-shot capabilities in causality detection. Zero-shot approaches exhibit good performance in other classi cation tasks, such as sentiment analysis or question answering. Yet, for this task, the ne-tuned approach seems superior, and the situation does not change if language cues are added or a few-shot setting is considered. This is a preliminary analysis of a work in progress. Still, the results suggest that identifying causal relations is a particularly challenging task that is hard to address in a zero-shot setup.
The adoption of large language models (LLMs) [1,2] is rapidly growing, primarily because of the zero-shot capabilities exhibited by these tools in a wide range of natural language processing tasks, such as sentiment analysis or recommendations and knowledge-intensive tasks, such as question answering and domain-speci c entity recognition [3,4,5,6,7]. Despite such popularity, it is essential to understand the limitations and address questions such as: where can these models fall back? What are the possibilities of such fallbacks? How can we improve their performance beyond prompting engineering [8,9,10]?
This paper is a preliminary report on our work (in progress) on evaluating the potential of state-of-the-art LLMs in the eld of causal inference. More speci cally, we investigate the performance of LLMs in a classi cation task with sentences possibly involving causal relations. Our analysis focuses on zero-and few-shot capabilities of LLMs compared against a ne-tuning setting with encoder-based BERT models, which are nowadays the most common choice for classi cation tasks [11,12,13]. Our tests show some limitations of LLM approaches in the causal domain, this being the case either for zero-shot and few-shot setups. Notably, the situation remains the same even if language cues are provided. Such negative results are in line with some recent works presenting LLMs as causal parrots [14], not yet capable of genuine causal reasoning [15], beyond just distinguishing between causes and e ects [16,17].
Recently, a plethora of research has been going on in the direction of exploiting the zeroshot and few-shot capabilities of LLMs. Because of the vast amount of pre-trained data they have been exposed to, large (>10B parameters) language models are considered to have an inherent ability to generalise across unseen tasks [18,19,20]. For instance, the number of parameters of the recent GPT-3 and GPT-4 models is about, respectively, 175B and 1.76T. Zeroshot and few-shot techniques have been tried with di erent prompting strategies (e.g., the chain of thought) for both classi cation and generation tasks. In many knowledge-intensive tasks (e.g., question answering), translations, classi cation tasks (e.g., sentiment analysis) and recommendations, those approaches seem compelling, provided that an adequate, prompt engineering e ort is achieved [21,22,23,24]. These techniques may be inaccurate for many other tasks, signi cantly when the complexity increases, such as multi-task classi cations and hard sequence labelling tasks, especially in domain-speci c problems [25,8]. Researchers have come up with soft prompting approaches and parameter e cient tuning (PEFT) [26] approaches such as P-tuning [27], prompt-tuning [28,29] and variations of prompt infusions to overcome these problems, while trying to achieve ne-tuning-based performances. The causal reasoning ability of LLMs has been initially investigated in [30]. The authors observe a good performance with a pairwise causal discovery task, counterfactual reasoning task and actual causality by conducting experiments on datasets of cause-e ect pairs. A critical review of the causality inference and reasoning with LLMs on benchmark datasets is reported in [31]. The authors specify the requirements of causal datasets and problems of evaluations with LLMs, such as memorisation (the dataset could be part of LLM pre-training). They also indicate that LLMs can answer many datasets by simply computing similarities between options and questions in a vector space. Further, they also indicate that the good performance of LLMs can sometimes be due to spurious language cues in the datasets. In the rest of the paper, we explore the capability of LLMs with simple prompt-based approaches in identifying causality on a multi-class dataset.
For our analysis, we focus on the dataset from [32], developed to automate the identi cation of causal language use in the scienti c literature. The data source was a collection of PubMed1 abstracts with ve main health topics -nutrition, diabetes, obesity, breast cancer, and cholesterol. Two domain experts were asked to annotate the sentences manually. A good agreement (Cohen's kappa = 0.98) was reported. The original dataset refers to a multi-class setup with four options: correlational, direct causal, conditional causal, and one without any relations [33]. The entities possibly involved in the causal and correlational relations are not provided. Thus, the identi cation depends on the speci c language patterns used in the input sentences. In the correlational case, the sentence describes some association between variables. With direct causal sentences, the cause and e ect are directly mentioned, while in the conditional case, the relation de nition carries out an element of doubt. Finally, there are sentences with neither causation nor correlation.
We use the original dataset in the native multi-class setting and a binary classi cation task. For the binary class, we drop the correlational sentences and combine the direct and conditional causal sentences, thus having only two classes, one with no relations and the other with causal relations. This is intended to allow for a focus on causal relation discrimination. The multi-class dataset includes 1356 no-relation, 494 direct causal, 213 conditional and 998 correlational cases, which makes up 3061 cases. In the binary class setting, we have 1356 no-relation cases and 707 cases of causal relations (which combines direct and conditional cases), with 2063 cases overall.
To test the capability of LLM models in classifying causal and non-causal sentences under zero-shot settings, we initially design suitable prompts to tackle the task with LLMs. We use both binary and multi-class settings with the prompt including the text input in Fig. 1 and some variations. For the binary settings, we just need to change the classes in the prompt. Following the indications from [32] and ndings from [31], we create another prompt including language cues intended to help the LLM in providing more accurate classi cations. In ne-tuning approach, we assume the model automatically captures these patterns given the training data. In a zero-shot setting, with the absence of such training information, we want to see the impact on model performance when some explicit domain knowledge is available. We added the following cues -association, associated with, predictor for correlational, increase, decrease, lead to, e ective in, contribute to, reduce for causal and along with may, might, appear to, probably for conditional causal. These cues were then added to the zero-shot prompt (ZS-Cues). Further, we tried them in a few-shot setup (FS-Cues) with some examples from each class (e.g., two samples for each class). Finally, we also consider a 500-shot experiment with labelled samples, used also to train the BERT model under the same settings (500 samples for training).
For the ne-tuning approach, we use the best-base-cased model [34] in both binary and multiclass settings with k-fold cross-validation. We use SimpleTransformers2 with four epochs and a learning rate of 2E-5. We experiment with GPT (3.5 Turbo) and the open-source Falcon model (falcon-7b-instruct 3 and falcon-40b-instruct) in zero-shot settings. Falcon-7b-instruct is a 7B parameters causal decoder-only model ne-tuned on a mixture of chats and instructions, while falcon-40b-instruct is a bigger model with 40B parameters. From Tab. 1, it can be observed that Falcon-7b and 40b give inferior performance compared to GPT. Comparing the two Falcon models, the 40b outperformed the 7b model in multi-class and binary settings. This expected result motivates us to focus our experiments on GPT models only. For further experiments, we use GPT to analyse the performance under di erent prompt settings and compare them with ne-tuned BERT-based models. Tab. 2 shows that, in both settings, the performance of GPT under zero-shot settings is poor and the ne-tuned BERT model performs better. In the multi-class case, many conditional causal relations are misclassi ed as direct causal. Yet, the accuracy does not improve signi cantly in the binary setting. The addition of cues (ZS-Cues) improves the performance, showing the importance of speci c patterns that help in classi cation, especially with multi-class settings. In both cases, ZS-Cues performed better than FS-Cues. This could be because the sentences in this dataset are quite varied (extracted from the scienti c literature) and we cannot pre-assume that the selected sentences for few-shot experiments could be the best representative for a class. Tab. 3 reports more details on the zero-shot results. The relatively high recall values for causal sentences denote a good ability of the model in detecting direct causal relation. Yet, the same does not happen with conditional causal relations, typically misclassi ed as direct ones. This also explains the higher performance in the binary class case. For a deeper comparison against the FFT model, we prompt the GPT model with more examples. An option would be ne-tuning the GPT model, but we keep this as a future study, as here the focus is on prompting approaches. Further, there are restrictions on the number of prompt tokens processed by GPT 3.5 model. As a reasonable prompting solution, we use 500 samples (corresponding to a 1:4 train-test split). The same samples are used to train BERT under multi-class settings. This proportion makes the BERT performance comparable with the one with k-fold cross validation (F1=0.81), while a drastic drop is obtained with a 1:9 split (F1=0.36). For GPT, this setup requires a slight change in the prompt (Fig. 1), to include a list of input text and give the corresponding predictions as a list. We then chunked the remaining 2449 test samples, each containing ten samples, to be passed on to the prompt. These steps are intended to optimise prompt e ciency in terms of costs and time. The results are in Tab. 4. It can be observed that, with 500 samples, the performance of GPT model was better than its zero-shot counterpart, but is not comparable with BERT model ne-tuned with the same 500 training samples. This seems to con rm, in the causal domain, the general ndings discussed in [35]. At the same time, it is also notable that simply adding pattern information, like in ZS-Cues and FS-Cues, makes the LLM performance better than the 500-shot model.
Moreover, in some cases, GPT gives predictions not explicitly mentioned in the prompts. For instance, the prediction was multi-label (neither of the labels included true prediction) or the prediction was categorised into a new class (not described in the prompt). For the evaluations, we had to remove such samples. With zero-shot multi-class, nally, we had 3055 instances and 2016 instances with binary class. Some explicit results from the zero-shot experiments are in Figs. 2, 3 and 4. Tab. 5 reports examples of misclassi cation from the zero-shot prompting of GPT. Some of these instances are hard to classify even for human experts. GPT almost always classi es a negative causality as a no-relation case (I2, 3, and 4). Some other misclassi cations (I8) predicted as causal while conditional in practice are too obvious. Similar patterns are observed in correlational cases (I10). It can be also observed that the addition of cues helped in the correct classi cation of some instances. The correct predictions are in bold. E.g., I7 and I9 show that the cue may could have helped in the correct classi cation as conditional causal, and for I12 the cue associated with.
text: However, obesity seems to be associated with more wound complications.
The text states that "obesity seems to be associated with more wound complications."
This suggests a potential causal relationship between obesity and wound complications. To determine the type of causal relationship, we need to analyze the information provided.
1. Direct Causal Relation: This type of causal relation implies that one event directly causes another event. In this case, if obesity directly causes wound complications, we would expect to see a clear cause-effect relationship between the two.
2. Conditional Causal Relation: This type of causal relation implies that one event causes another event under certain conditions or circumstances. In this case, if obesity causes wound complications only under specific conditions, we would need more information to determine the specific conditions.
3. Correlational Relation: This type of relation implies that two events are statistically related or occur together, but one does not necessarily cause the other. In this case, if obesity and wound complications are correlated but not causally related, we would expect to see a statistical association between the two.
Based on the information provided, it is not clear whether the relationship between obesity and wound complications is a direct causal relation, a conditional causal relation, or a correlational relation.
The text only states that obesity is associated with more wound complications, but it does not provide information on the specific nature of the relationship or any conditions under which the relationship occurs.
Therefore, the appropriate categorization for the causal relationship in this text is: <Answer>3</Answer> (correlational) text: This study is one of the first proofs of concept showing that a nutrition-sensitive agriculture value chain approach can contribute to improved child nutrition in a remote pastoralist population.
To analyze the causal relationship in the provided text, we need to identify the entities involved and understand their interactions.
Entities: 1. Study 2. Nutrition-sensitive agriculture value chain approach 3. Improved child nutrition 4. Remote pastoralist population Based on the information in the text, let's analyze the causal relationship: 1. The study is one of the first proofs of concept. 2. The study shows that a nutrition-sensitive agriculture value chain approach can contribute to improved child nutrition. 3. The improved child nutrition is in a remote pastoralist population.
From the above analysis, we can conclude that there is a direct causal relationship between the nutrition-sensitive agriculture value chain approach and improved child nutrition in a remote pastoralist population. Therefore, the causal relationship in the text can be categorized as a direct causal relation. <Answer>1</Answer> The provided text states that there are no indications that endogenous and exogenous gonadal hormones affect the radiation dose-response relationship. Based on this information, we can conclude that there is no clear causal relationship between endogenous and exogenous gonadal hormones and the radiation dose-response relationship.
Therefore, the answer is <Answer>0</Answer> (no relation).
Finally, as our paper presents the results of a work in progress, let us discuss the limitations of the present work and the possible enhancements we might consider for the ongoing work.
We have used only one dataset; hence, whether our ndings remain valid in the general case might be questionable. The dataset facilitates understanding how well LLMs identify the causal descriptions embedded in scienti c literature under a more challenging multi-class setting, including correlative and causal relations. Distinguishing between direct and conditional causation is especially di cult. To the best of our knowledge, there are no datasets with analogous characteristics, at least for multi-class settings. Yet, manually annotating scienti c abstracts and creating new benchmarks for deeper validation is a realistic and necessary e ort. Moreover, in the current paper, we have focused on the GPT model and compared it with the open-sourced Falcon and BERT-based models. This can be enhanced by comparing with di erent LLMS. Further, the focus was on using prompt-based techniques, which have a broad scope to be explored. Based on the nding from [31], we investigate techniques such as incorporating language cues while prompting LLMs. One major problem is that LLMs can be sensitive to manually engineered prompt designs; hence, automating prompts and using soft prompt techniques would be the way forward.
This work is a preliminary exploration to understand the capabilities and limitations of the GPT model in causality identi cations, speci cally in multi-class settings. The experiments show that GPT has limited zero-shot and few-shot capabilities in capturing such causal relations, subject to the data in consideration. Focusing on the limitations, in the future, we would like to enhance our experiments on a range of causal data to have conclusive generalisations on the studied facts. Prompt engineering as such has a lot of potential to be explored, while hard-core engineering of prompts may not be always bene cial. Hence, we also plan to explore PEFT techniques such as soft prompting for causal detection and further extractions of causal graphs.