We aim to explore and analyze the capabilities and limitations of the large language models in understanding and distinguishing causal sentences under a zero-shot setting. We experiment on a multi-class dataset of direct causal, conditional causal, and correlational sentences. In the experiments, the GPT and Falcon models are validated against a ne-tuned BERT model under di erent settings to explore zero-shot capabilities in causality detection. Zero-shot approaches exhibit good performance in other classi cation tasks, such as sentiment analysis or question answering. Yet, for this task, the ne-tuned approach seems superior, and the situation does not change if language cues are added or a few-shot setting is considered. This is a preliminary analysis of a work in progress. Still, the results suggest that identifying causal relations is a particularly challenging task that is hard to address in a zero-shot setup.
The adoption of large language models (LLMs) [
This paper is a preliminary report on our work (in progress) on evaluating the potential
of state-of-the-art LLMs in the eld of causal inference. More speci cally, we investigate the
performance of LLMs in a classi cation task with sentences possibly involving causal relations.
Our analysis focuses on zero- and few-shot capabilities of LLMs compared against a ne-tuning
setting with encoder-based BERT models, which are nowadays the most common choice for
classi cation tasks [
Recently, a plethora of research has been going on in the direction of exploiting the
zeroshot and few-shot capabilities of LLMs. Because of the vast amount of pre-trained data they
have been exposed to, large (>10B parameters) language models are considered to have an
inherent ability to generalise across unseen tasks [
For our analysis, we focus on the dataset from [
We use the original dataset in the native multi-class setting and a binary classi cation task. For the binary class, we drop the correlational sentences and combine the direct and conditional causal sentences, thus having only two classes, one with no relations and the other with causal relations. This is intended to allow for a focus on causal relation discrimination. The multi-class dataset includes 1356 no-relation, 494 direct causal, 213 conditional and 998 correlational cases, which makes up 3061 cases. In the binary class setting, we have 1356 no-relation cases and 707 cases of causal relations (which combines direct and conditional cases), with 2063 cases overall.
To test the capability of LLM models in classifying causal and non-causal sentences under zero-shot settings, we initially design suitable prompts to tackle the task with LLMs. We use both binary and multi-class settings with the prompt including the text input in Fig. 1 and some variations. For the binary settings, we just need to change the classes in the prompt. system_msg = You are a helpful assistant for causal reasoning and cause-and-effect relationship discovery. Your aim is to identify the entities and to categorize the input sentences into either direct causal relation or conditional causal relation or correlational relation or no relationship exist intro_msg = You will be provided with a text. Text: <Text>{text}</Text> instructions_msg = Please read the provided text carefully to comprehend the context and content.
Examine the roles, interactions, and details surrounding the entities within the text.
Based only on the information in the text, categorize the causal relation as 0. no relation 1. direct causal 2. conditional causal 3. correlational Your response should analyze the situation in a step-by-step manner, ensuring the correctness of the ultimate conclusion, which should accurately reflect the likely causal connection based on the information presented in the text. If no clear causal relationship is apparent, select the appropriate option accordingly, i.e., ’no relation’. option_choice_msg = Your response should analyze the situation in a step-by-step manner, ensuring the correctness of the ultimate conclusion, which should accurately reflect the likely causal connection between the two entities based on the information presented in the text. If no clear causal relationship is apparent, select the appropriate option accordingly.
Then provide your final answer within the tags <Answer>[answer]</Answer>, (e.g. <Answer>1</Answer>).
Following the indications from [
For the ne-tuning approach, we use thebest-base-cased model [34] in both binary and multiclass settings with k-fold cross-validation. We use SimpleTransformers2 with four epochs and a learning rate of 2E-5. We experiment with GPT (3.5 Turbo) and the open-source Falcon model (falcon-7b-instruct 3 and falcon-40b-instruct) in zero-shot settings. Falcon-7b-instruct is a 7B parameters causal decoder-only model ne-tuned on a mixture of chats and instructions, while falcon-40b-instruct is a bigger model with 40B parameters. From Tab. 1, it can be observed that Falcon-7b and 40b give inferior performance compared to GPT. Comparing the two Falcon models, the 40b outperformed the 7b model in multi-class and binary settings. This expected result motivates us to focus our experiments on GPT models only.
For further experiments, we use GPT to analyse the performance under di erent prompt settings and compare them with ne-tuned BERT-based models. Tab.2 shows that, in both settings, the performance of GPT under zero-shot settings is poor and the ne-tuned BERT model performs better. In the multi-class case, many conditional causal relations are misclassi ed as direct causal. Yet, the accuracy does not improve signi cantly in the binary setting. The addition of cues (ZS-Cues) improves the performance, showing the importance of speci c patterns that help in classi cation, especially with multi-class settings. In both cases, ZS-Cues performed better than FS-Cues. This could be because the sentences in this dataset are quite varied (extracted from the scienti c literature) and we cannot pre-assume that the selected sentences for few-shot experiments could be the best representative for a class. 2http://simpletransformers.ai. 3https://huggingface.co/tiiuae/falcon-7b-instruct.
For a deeper comparison against the FFT model, we prompt the GPT model with more examples. An option would be ne-tuning the GPT model, but we keep this as a future study, as here the focus is on prompting approaches. Further, there are restrictions on the number of prompt tokens processed by GPT 3.5 model. As a reasonable prompting solution, we use 500 samples (corresponding to a 1:4 train-test split). The same samples are used to train BERT under multi-class settings. This proportion makes the BERT performance comparable with the one with k-fold cross validation (F1=0.81), while a drastic drop is obtained with a 1:9 split (F1=0.36). For GPT, this setup requires a slight change in the prompt (Fig. 1), to include a list of input text and give the corresponding predictions as a list. We then chunked the remaining 2449 test samples, each containing ten samples, to be passed on to the prompt. These steps are intended to optimise prompt e ciency in terms of costs and time. The results are in Tab. 4.
It can be observed that, with 500 samples, the performance of GPT model was better than its zero-shot counterpart, but is not comparable with BERT model ne-tuned with the same 500 training samples. This seems to con rm, in the causal domain, the general ndings discussed in [35]. At the same time, it is also notable that simply adding pattern information, like in ZS-Cues and FS-Cues, makes the LLM performance better than the 500-shot model.
Moreover, in some cases, GPT gives predictions not explicitly mentioned in the prompts. For instance, the prediction was multi-label (neither of the labels included true prediction) or the prediction was categorised into a new class (not described in the prompt). For the evaluations, we had to remove such samples. With zero-shot multi-class, nally, we had 3055 instances and 2016 instances with binary class.
Some explicit results from the zero-shot experiments are in Figs. 2, 3 and 4. Tab. 5 reports examples of misclassi cation from the zero-shot prompting of GPT. Some of these instances are hard to classify even for human experts. GPT almost always classi es a negative causality as a no-relation case (I2, 3, and 4). Some other misclassi cations (I8) predicted as causal while conditional in practice are too obvious. Similar patterns are observed in correlational cases (I10). It can be also observed that the addition of cues helped in the correct classi cation of some instances. The correct predictions are in bold. E.g., I7 and I9 show that the cue may could have helped in the correct classi cation as conditional causal, and for I12 the cueassociated with. text: However, obesity seems to be associated with more wound complications.
The text states that "obesity seems to be associated with more wound complications." This suggests a potential causal relationship between obesity and wound complications.
To determine the type of causal relationship, we need to analyze the information provided. 1. Direct Causal Relation: This type of causal relation implies that one event directly causes another event. In this case, if obesity directly causes wound complications, we would expect to see a clear cause-effect relationship between the two. 2. Conditional Causal Relation: This type of causal relation implies that one event causes another event under certain conditions or circumstances. In this case, if obesity causes wound complications only under specific conditions, we would need more information to determine the specific conditions. 3. Correlational Relation: This type of relation implies that two events are statistically related or occur together, but one does not necessarily cause the other. In this case, if obesity and wound complications are correlated but not causally related, we would expect to see a statistical association between the two. Based on the information provided, it is not clear whether the relationship between obesity and wound complications is a direct causal relation, a conditional causal relation, or a correlational relation.
The text only states that obesity is associated with more wound complications, but it does not provide information on the specific nature of the relationship or any conditions under which the relationship occurs. Therefore, the appropriate categorization for the causal relationship in this text is: <Answer>3</Answer> (correlational) text: This study is one of the first proofs of concept showing that a nutrition-sensitive agriculture value chain approach can contribute to improved child nutrition in a remote pastoralist population.
To analyze the causal relationship in the provided text, we need to identify the entities involved and understand their interactions. Entities: 1. Study 2. Nutrition-sensitive agriculture value chain approach 3. Improved child nutrition 4. Remote pastoralist population Based on the information in the text, let’s analyze the causal relationship: 1. The study is one of the first proofs of concept. 2. The study shows that a nutrition-sensitive agriculture value chain approach can contribute to improved child nutrition. 3. The improved child nutrition is in a remote pastoralist population.
From the above analysis, we can conclude that there is a direct causal relationship between the nutrition-sensitive agriculture value chain approach and improved child nutrition in a remote pastoralist population.
Therefore, the causal relationship in the text can be categorized as a direct causal relation. <Answer>1</Answer>
Finally, as our paper presents the results of a work in progress, let us discuss the limitations of the present work and the possible enhancements we might consider for the ongoing work.
We have used only one dataset; hence, whether our ndings remain valid in the general case might be questionable. The dataset facilitates understanding how well LLMs identify the causal descriptions embedded in scienti c literature under a more challenging multi-class setting, including correlative and causal relations. Distinguishing between direct and conditional causation is especially di cult. To the best of our knowledge, there are no datasets with analogous characteristics, at least for multi-class settings. Yet, manually annotating scienti c abstracts and creating new benchmarks for deeper validation is a realistic and necessary e ort.
Moreover, in the current paper, we have focused on the GPT model and compared it with
the open-sourced Falcon and BERT-based models. This can be enhanced by comparing with
di erent LLMS. Further, the focus was on using prompt-based techniques, which have a broad
scope to be explored. Based on the nding from [
This work is a preliminary exploration to understand the capabilities and limitations of the GPT model in causality identi cations, speci cally in multi-class settings. The experiments show that GPT has limited zero-shot and few-shot capabilities in capturing such causal relations, subject to the data in consideration. Focusing on the limitations, in the future, we would like to enhance our experiments on a range of causal data to have conclusive generalisations on the studied facts. Prompt engineering as such has a lot of potential to be explored, while hard-core engineering of prompts may not be always bene cial. Hence, we also plan to explore PEFT techniques such as soft prompting for causal detection and further extractions of causal graphs. [34] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [35] T. Schick, H. Schütze, It’s not just size that matters: Small language models are also few-shot learners, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2339–2352.