This paper presents our participation in the CLEF2024 CheckThat! Lab's Task-1 which focuses on determining whether passages from tweets or transcriptions are check-worthy. Task 1 covers three languages including English, Arabic, and Dutch. We propose utilizing several diferent instruct-tuned large language models (LLM) and aggregating their results for the Dutch dataset. In English and Arabic datasets, in addition to LLMs, we also use a fine-tuned XLM-R classifier. Our proposed method is ranked first in the Dutch dataset, fourth in the Arabic dataset, and eleventh in the English dataset.
The concept of check-worthiness is thoroughly explained in recent studies [6]. Matwin et al. [7] approach the problem as a 3-class classification task, categorizing sentences as not check-worthy, unimportant check-worthy, and important check-worthy, thereby distinguishing between irrelevant and valuable check-worthy sentences. However, several works define the problem as a binary classification [ 8] as in this lab or a ranking problem [9].
In the early studies on check-worthiness, researchers explored various features such as named entities [10] and syntactic dependency tags [11]. Gencheva et al. [12] study the contextual cues that might indicate the check-worthiness of a sentence in transcripts. They report that the duration of the speech and the presence of accusations against political opponents are correlated with check-worthiness. However, Hansen et al. [13] discuss the flaws of using hand-crafted features for check-worthy claim detection.
In recent years, several studies report efectiveness of transformer models in detecting check-worthy claims, exploring various data engineering methods such as cross-lingual training [14], generating data using LLMs [15], and contextually sensitive lexical augmentation [16]. With the recent developments in generative models, researchers also explored their impact on detecting check-worthy claims. For instance, Sawinski et al. [17] conducted a comparative study of GPT and BERT models for the detection of check-worthy claims. Their findings indicate that fine-tuned BERT models can perform comparably to large language models such as GPT-3 in identifying check-worthy claims, demonstrating that both models have significant potential for automated fact-checking systems. In our work, we utilize multiple LLMs and with in context learning and a fine-tuned XLM-R model and aggregate their results to reach a final decision.
For each language, we develop a slightly diferent method. In the Arabic and English tracks, we propose a two-stage approach to determine check-worthy statements. Our method combines a fine-tuned XLM-R classifier with in-context learning (ICL) using multiple diferent instruct-tuned models. The aggregation method varies between the Arabic and English datasets. For the Dutch dataset, we opted to exclude the fine-tuned classifier, relying solely on in-context learning due to the time constraints of the lab.
Our two-stage approach aims to improve the prediction performance by combining the classification efectiveness of fine-tuned models with the natural language understanding capabilities of instructtuned LLMs. Firstly, we fine-tune an XLM-R model using the training dataset. Predictions with high confidence scores are likely to be correct while those with low confidence scores can be considered nearly random labeling. Figure 1 shows the distribution of correctly and incorrectly classified cases for the confidence scores of our fine-tuned XLM-R model. We observe that the classifier achieves an average confidence score of 0.94 for correct classifications, in contrast to an average confidence score of 0.74 for incorrect classifications. To increase the efectiveness of our approach for these challenging examples, they are passed to our ICL labeler. Here, we devise a specific prompt to directly query multiple instruct-tuned models, asking whether a given sample is check-worthy. We then aggregate these models’ outputs to determine if a sample is check-worthy or not.
Now we explain the details of our ICL labeler (Section 3.1) and the diferences in our approach across languages (Section 3.2).
In-context learning is a technique where an LLM is prompted to solve a task at inference time without updating its weights. This is achieved through a carefully curated prompt that includes explanations or examples of the task. The concept was introduced and defined by Brown et al. [ 18]. During unsupervised pre-training, a language model develops a broad set of skills and pattern recognition abilities. At inference time, the model uses these abilities to quickly adapt to or recognize the desired task. The term “in-context learning” refers to this adaptive process, which occurs within the forward pass of each sequence.
The prompt developed for this task, illustrated in Figure 2, consists of three distinct sections. The ifrst section provides an explanation of the task, emphasizing the importance of accurate information. In the second section, explicit instructions are given to the model to generate a data in JSON format with three specific labels: "candidate_text," "reasoning," and "label." The requirements for each label are clearly defined. The final section elaborates on the content to be included under the "reasoning" tag and the candidate passage to be validated for check-worthiness, stressing adherence to the JSON format.
In this section, we explain our specific approaches for each language. 3.2.1. English Firstly, we identify the samples to be passed to the ICL labeler. We refer this subset of the data ICL subset, which includes instances where the confidence score of the fine-tuned classifier is below 90%. We select 90% as threshold because the performance of the model noticeably decreases when its confidence score is below 90% as seen in Figure 1. This subset is then processed by the ICL labeler. Finally, we combine the labels from the fine-tuned classifier and the ICL labeler with a weighted averaging aggregation, where the weights are determined by their F1 scores. The details of this aggregation method are as follows.
Weighted Average Aggregation with F1 Scores. To determine the labels for the samples in the ICL subset, we calculate the F1 score for the XLM-R model and the instruct-tuned LLMs on the ICL subset of the training set. Afterwards, we compute a weighted average of their output labels, using the F1 scores as weights. Samples are labeled as check-worthy if this weighted average exceeds a hyperparameter . The F1 scores for the XLM-R model and the instruct-tuned LLMs, along with the hyperparameter are determined during the development phase and used during testing. 3.2.2. Arabic For the Arabic task, we aggregate all labels from both the fine-tuned classifier 2 and the ICL labels. We employ a super majority voting system for our aggregation strategy, requiring agreement from four out of five label sources for a label to be selected. We label the cases we do not reach this threshold, as not-check-worthy. 3.2.3. Dutch For the Dutch task, we rely exclusively on the ICL approach due to the time constraints of the lab. We aggregate the predictions of LLMs based on majority voting to reach a final decision.
For our pre-trained classifier we used the multilingual "FacebookAI/xlm-roberta-large" [ 19] model3 as our base pretrained model. For in-context learning, we employed available APIs for GPT-3.5, GPT-44, and Gemini 1 Pro5. We also used open-source models including Meta-Llama-3-8B-Instruct67 and Mistral7B-Instruct-v0.28. We quantize these open-source models down to 4-bit precision to accommodate the hardware limitations. We used HuggingFace’s [20] generate API for text generation. Table 1 shows the specific models we used for each task. We were not able to use some of the models in Arabic and Dutch due to quota limits and time constraints.
As our transformer model we used "FacebookAI/xlm-roberta-large," for both English and Arabic languages. We fine-tuned models for each language separately using HuggingFace’s Trainer API 9. We set the same parameters for both languages: a batch size of 16, a learning rate of 3 × 10− 5, and 5 epochs.
For the Arabic fine-tuned model, we performed evaluations in every 200 steps using the test partition and calculated 1 score at each interval to monitor the model’s performance. The optimal performance was identified at 2.5 epochs, which corresponds to the 1,000 ℎ step (out of 2,065 steps). Beyond this point, additional training did not result in any further improvement in the 1 score on the test set. Thus, we selected this checkpoint for the remainder of the analysis in this study.
For the English fine-tuned model, evaluations were carried out in every 500 steps. The model checkpoint with the lowest test set loss was chosen for further analysis, which was observed at the 2500ℎ step (out of 6,330 steps). Further configuration details and the fine-tuned models we used are available at Huggingface10,11.
The dataset12 consists of passages derived from tweets or transcriptions. Each passage is annotated with a binary label indicating its checkworthiness. Table 2 presents statistics about the data. For the experiments conducted during the development phase of the lab, the training dataset shared by the organizers of the lab was split into 80% for training and 20% for testing for Arabic while 90% of the English training dataset was allocated for training and 10% for testing. 3https://huggingface.co/FacebookAI/xlm-roberta-large 4https://openai.com/index/introducing-chatgpt-and-whisper-apis/ 5https://ai.google.dev/gemini-api/docs/api-overview 6https://ai.meta.com/blog/meta-llama-3/ 7https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 8https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 9https://huggingface.co/docs/transformers/en/main_classes/trainer 10https://huggingface.co/keles/clef1ar 11https://huggingface.co/keles/clef1eng 12https://gitlab.com/checkthat_lab/clef2024-checkthat-lab/-/tree/main/task1/data
We used the 10% of the training dataset for testing purposes during the development phase, as mentioned before. We call this subset as evaluation set throughout the paper. In our experiments with English dataset, we evaluate the impact of our ICL labeler with diferent aggregation methods. In particular, we compare the performance of three methods: i) the fine-tuned classifier, ii) nfied tuned classifier and ICL labeler with majority voting, and iii) fined tuned classifier and ICL labeler with 1 weighted averaging. Table 3 shows 1 scores on both evaluation and test sets.
The fine-tuned XLM-R achieves the highest 1 score, showing that ICL has negative impact on the overall performance. Among the methods that use ICL, 1 weighted averaging yields a higher score than the majority voting, highlighting the importance of utilizing sophisticated aggregation techniques.
Table 4 presents the 1 scores for each model and results for the aggregated results under both majority and super-majority voting methods for Arabic dataset. As explained in 3.2.2, for the Arabic task, we employ super majority voting, which requires four out of five sources to label a claim as check-worthy. However, as illustrated in Table 4, a basic majority voting approach yields a higher 1 score. In contrast to our results for English, aggregation improves the 1 scores in the Arabic dataset, as both aggregation approaches outperform all other models.
As mentioned in 3.2.3, for the Dutch task, we employ a straightforward in-context learning approach with label aggregation. We use three models and aggregate their individual predictions via majority voting. In this experiment, we also assess the impact of inclusive aggregation in which a claim is labeled as check-worthy if at least one of the models predicted as check-worthy. Table 5 presents the results of this method on the Dutch training data. We observe that the inclusive aggregation yields the highest performance. In addition, LLama 3 and GPT3.5 achieve higher scores than the majority voting aggregation. This might be because of the low performance of Mistral 7Bv2.
Due to the time constraints of the lab, we had to pick the models to be submitted based on the results that we had in the development period. In particular, we selected the following configurations as our primary model: 1 weighted averaging for English, super-majority voting for Arabic, and majority voting for Dutch. However, based on our follow-up experiments after the submission deadline, we observed that these are not the best performing configurations. Nevertheless, our primary models achieved notable success. In particular, we are ranked first (out of 16) in the Dutch track, fourth (out of 14) in the Arabic track, and eleventh (out of 27) in the English track.
38 English sentences (out of 341) in the test set are classified incorrectly with all of our methods. After inspecting these sentences, we notice that determining whether these sentences are check-worthy is dificult even for human evaluators. Some of these samples are shown in Table 6. These results show that LLMs might be beneficial to detect sentences which might need label correction. "And it’s not like it was 25 years ago, it was three and three quarters." "But the Biden administration sends Blinken, Yellen over there." "We’re skating on thin ice and we cannot set a precedent where the party in power uses police force to indict its political opponents." Label Not Check-Worthy Not Check-Worthy Check-Worthy
In this work, we explored utilizing both fine-tuned pretrained transformers and instruct-tuned LLMs through ICL. Our proposed methods demonstrated considerable success across three languages. Our ifrst-place ranking in the Dutch track highlights the remarkable zero-shot capabilities of current LLMs, while our results in Arabic and English underscore the potential of combining traditional fine-tuning with ICL techniques. In the future work, we plan to extend our work and explore how predictions of LLMs can be aggregated efectively. [2] S. Vosoughi, D. Roy, S. Aral, The spread of true and false news online, science 359 (2018) 1146–1151. [3] M. Hasanain, R. Suwaileh, S. Weering, C. Li, T. Caselli, W. Zaghouani, A. Barrón-Cedeño, P. Nakov, F. Alam, Overview of the CLEF-2024 CheckThat! lab task 1 on check-worthiness estimation of multigenre content, ???? [4] A. Barrón-Cedeño, F. Alam, T. Chakraborty, T. Elsayed, P. Nakov, P. Przybyła, J. M. Struß, F. Haouari, M. Hasanain, F. Ruggeri, X. Song, R. Suwaileh, The clef-2024 checkthat! lab: Check-worthiness, subjectivity, persuasion, roles, authorities, and adversarial robustness, in: N. Goharian, N. Tonellotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, I. Ounis (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2024, pp. 449–458. [5] Y. S. Kartal, M. Kutlu, Trclaim-19: The first collection for turkish check-worthy claim detection with annotator rationales, in: Proceedings of the 24th Conference on Computational Natural Language Learning, 2020, pp. 386–395. [6] F. Alam, S. Shaar, F. Dalvi, H. Sajjad, A. Nikolov, H. Mubarak, G. D. S. Martino, A. Abdelali, N. Durrani, K. Darwish, A. Al-Homaid, W. Zaghouani, T. Caselli, G. Danoe, F. Stolk, B. Bruntink, P. Nakov, Fighting the covid-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society, 2021. arXiv:2005.00033. [7] S. Matwin, S. Yu, F. Farooq, N. Hassan, F. Arslan, C. Li, M. Tremayne, Toward automated factchecking, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017) 1803–1812. doi:10.1145/3097983.3098131. [8] A. Barrón-Cedeño, F. Alam, A. Galassi, G. Da San Martino, P. Nakov, T. Elsayed, D. Azizov, T. Caselli, G. S. Cheema, F. Haouari, et al., Overview of the clef–2023 checkthat! lab on checkworthiness, subjectivity, political bias, factuality, and authority of news articles and their source, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2023, pp. 251–275. [9] P. Nakov, A. Barrón-Cedeno, T. Elsayed, R. Suwaileh, L. Màrquez, W. Zaghouani, P. Atanasova, S. Kyuchukov, G. Da San Martino, Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France, September 10-14, 2018, Proceedings 9, Springer, 2018, pp. 372–387. [10] K. Yasser, M. Kutlu, T. Elsayed, bigir at clef 2018: Detection and verification of check-worthy political claims., in: CLEF (Working Notes), 2018. [11] C. Lespagnol, J. Mothe, M. Z. Ullah, Information nutritional label and word embedding to estimate information check-worthiness, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 941–944. [12] S. U. . O. Bulgaria, P. Gencheva, P. Nakov, H. Qatar, Qatar Computing Research Institute, L. Màrquez, A. Barrón-Cedeño, I. Koychev, A context-aware approach for detecting worth-checking claims in political debates, RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning (2017) 267–276. doi:10.26615/978-954-452-049-6_037. [13] C. Hansen, C. Hansen, S. Alstrup, J. Grue Simonsen, C. Lioma, Neural check-worthiness ranking with weak supervision: Finding sentences for fact-checking, in: Companion proceedings of the 2019 world wide web conference, 2019, pp. 994–1000. [14] Y. S. Kartal, M. Kutlu, Re-think before you share: A comprehensive study on prioritizing checkworthy claims, IEEE transactions on computational social systems 10 (2022) 362–375. [15] A. Modzelewski, W. Sosnowski, A. Wierzbicki, Dshacker at checkthat! 2023: Check-worthiness in multigenre and multilingual content with gpt-3.5 data augmentation, Working Notes of CLEF (2023). [16] E. Williams, P. Rodrigues, S. Tran, Accenture at checkthat! 2021: interesting claim identification and ranking with contextually sensitive lexical training data augmentation, arXiv preprint arXiv:2107.05684 (2021). [17] M. Sawiński, K. Węcel, E. P. Księżniak, M. Stróżyna, W. Lewoniewski, P. Stolarski, W. Abramowicz, Openfact at checkthat! 2023: head-to-head gpt vs. bert-a comparative study of transformers language models for the detection of check-worthy claims, in: CEUR Workshop Proceedings, volume 3497, 2023. [18] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [19] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 8440–8451. URL: https://aclanthology.org/2020.acl-main.747. doi:10.18653/v1/2020. acl-main.747. [20] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing, 2020. arXiv:1910.03771.