This paper presents our approach to check-worthiness detection, one of the main tasks in the CheckThat! Lab 2024 at the Conference and Labs of the Evaluation Forum. The challenge was to create a system to determine whether a claim found in Dutch and Arabic tweets or English debate snippets needs fact-checking. We explored ifne-tuning pre-trained BERT-based models and employing a few-shot prompting technique with OpenAI GPT models. Our study compared monolingual models based on the BERT architecture with a multilingual XLMRoBERTa-large model capable of processing data in multiple languages. Additionally, we investigated the link between propaganda detection and the check-worthiness of content. We also incorporated the recently released OpenAI GPT-4o model. Our systems' impressive performance, surpassing baseline results across all languages, is highlighted by our high-ranking positions: 3rd in Arabic, 2nd in Dutch, and 8th in English, with even better outcomes in post-deadline experiments.
Nowadays, information spreads from many online sources. As a result, it is crucial to ensure the information is accurate, as it afects public discourse and people’s decisions. The spread of disinformation and misinformation threatens the integrity of public discussions, impacting areas such as news reporting, political debates, and social media interactions. It is therefore essential to have solid fact-checking systems.
Fact-checking is not just about verifying that something is true. It is also essential for helping
people make good decisions based on accurate information and ensuring that the shared information is
trustworthy [
CheckThat! Lab at CLEF 2024 addresses issues that aid research and decision-making throughout the
fact-checking process [
CheckThat! Lab Task 1 at CLEF 2024 focuses on the first step of the pipeline. Its goal is to provide
an automated system for deciding whether a tweet or transcription claim needs fact-checking [
In this scenario, we are dealing with a binary classification task. For each instance, which is a short text like a tweet or a caption from a political debate transcription, we aim to predict one of two labels: "Yes" or "No," indicating whether the text is worth fact-checking.
Our approach to check-worthiness detection involved experimenting with both monolingual and multilingual models, as well as leveraging Large Language Models (LLMs) through few-shot prompting. For monolingual models, we fine-tuned BERT-based architectures tailored to specific languages like English, Dutch, and Arabic. In the multilingual approach, we utilized the XLM-RoBERTa-large model, ifne-tuning it on combined datasets from multiple languages. Additionally, we employed OpenAI’s GPT-3.5 and the recently released GPT-4o models for few-shot prompting, which demonstrated results comparable to fine-tuned BERT-based models. Furthermore, we explored the relationship between propaganda detection and check-worthiness by fine-tuning models on a propaganda detection dataset before applying them to the check-worthiness task. Our comprehensive experiments enabled us to surpass baseline results and achieve high-ranking positions across diferent languages: 3 rd in Arabic, 2nd in Dutch, and 8th in English, with even better outcomes in post-deadline experiments.
Detecting disinformation has become a crucial area of research. Researchers work not only on disinformation, but also address particular challenges associated with identifying disinformation, misinformation, and fake news. One such challenge is determining which claims are worthy of checking [7]. Hassan et al. [9] introduced a dataset from U.S. presidential debates and created classification models to diferentiate among three distinct categories: check-worthy factual claims, non-factual claims, and insignificant factual claims. Jaradat et al. [10] developed ClaimRank, an online tool designed to identify check-worthy claims, with support for two languages: English and Arabic. Kartal and Kutlu [11] proposed a hybrid model which combines BERT with various features to prioritize claims based on their check-worthiness.
The detection of check-worthy claims has also been a research focus in past years within CheckThat!
Labs [
Dataset TRAIN
DEV DEV-TEST
TEST
EN ES NL AR EN ES NL AR EN ES NL AR EN NL AR
The dataset consists of texts and their corresponding gold labels annotated by human experts, forming a multilingual dataset. It includes data in four languages: English, Spanish, Dutch, and Arabic. Dataset was divided into training , validation (dev) , and dev-test − datasets with gold labels. We made final predictions for the test datasets published in English, Dutch, and Arabic. The English texts depict captions from political debates, while in the other languages, they represent tweets. In most datasets, the classes are imbalanced, with a predominance of texts that are not check-worthy. Table 1 provides detailed information about the datasets. For more information, refer to Hasanain et al. [8].
We experimented with two approaches for text classification: fine-tuning BERT-based models and utilizing few-shot prompting with GPT models, including recently released GPT-4o model. These methodologies are both widely used today and represent the state-of-the-art in the industry [18]. However, deciding whether fine-tuning or in-context learning yields better results is not trivial. Many scientists have addressed the challenge of comparing the two approaches fairly [18].
To provide a comprehensive overview of the efectiveness of BERT-based models in identifying checkworthy content across multiple languages, we conducted experiments using two types of models. First, we employed monolingual models that we fine-tuned exclusively on the training data from a single language. Secondly, we used multilingual models that we fine-tuned on diferent combinations of the training datasets from multiple languages. Our experiments allowed us to compare the performance of both multilingual and monolingual models in the context of check-worthiness detection.
We started with comprehensive hyperparameter tuning for all chosen models (mono- and multilingual). This involved fine-tuning each model on the training dataset using every combination of hyperparameter values we specified. We used the proposed dataset for validation. Then, we assessed each model’s performance by measuring 1 score to determine the most efective hyperparameter values.
To obtain the final models for submission, we merged the training and validation datasets to form the ultimate training dataset. We then retrained the model with the best hyperparameter values using the − dataset as the final validation set. Please refer to our Appendix A, which presents optimal hyperparameters of each final submission.
For each language, we have chosen a single pretrained monolingual model available at HuggingFace
that we later fine-tuned on the corresponding language:
• ENGLISH (MONO-EN) - FacebookAI/roberta-large - the language model (355M parameters)
trained on English data in a self-supervised fashion [19].
• DUTCH (MONO-NL) - DTAI-KULeuven/robbert-2023-dutch-large - the first Dutch large (355M
parameters) model trained on the OSCAR2023 dataset [
We also fine-tuned a multilingual FacebookAI/xlm-roberta-large [
Additionally, we experimented with fine-tuning a multilingual model previously fine-tuned on a diferent propaganda-related dataset. We first fine-tuned the FacebookAI/xlm-roberta-large model on the propaganda presence binary classification task and then fine-tuned the model again on the CheckThat! Task 1 data (MULTI-PROP2). Refer to Section 5 for more information.
We also employed Large Language Models (LLMs) to generate check-worthiness predictions. Our experiments included OpenAI’s gpt-4o (GPT-4o) and gpt-3.5-turbo-1106 (GPT-3.5) generative models. We implemented the few-shot prompting technique using the OpenAI Chat Completions API. Few-shot prompting with GPT models leverages pre-trained language models to perform specific tasks without retraining. Instead, the model is guided by providing a few examples and their expected responses within the input prompt.
Each prediction request sent to the GPT model consisted of a list of messages presented to the model. Each message contains the role and content attribute. There are three roles available: 1. system message helps set the behavior of the model (assistant) by providing it context and guidelines. 2. user messages can provide exemplary requests for the assistant. In our case - example requests for a provided text’s check-worthiness evaluation.
3. assistant messages indicate the expected output of the assistant.
In our experiments, the conversation is formatted starting with a system message that clarifies the task and the concept of check-worthiness. This is followed by alternating pairs of user and assistant messages. One pair for each few-shot example, where a user message poses a question about the example’s content check-worthiness, and the corresponding assistant message provides the gold label for the example, either ’Yes’ or ’No’. The final message following the pairs is one user message with the actual text to be classified by the model (See Appendix B). For each instance to be classified, we included four examples of few-shot prompting from the training dataset, two of which are check-worthy. The chosen few-shot examples were consistent in a given language. The prompt templates remained consistent for both the GPT-4o and GPT-3.5 experiments.
Since propaganda often involves misleading, biased, or manipulative information, such content is
naturally more likely to warrant fact-checking [
Table 2 shows the performance metrics of the MULTI-PROP1 model calculated for the − dataset. Relatively high precision shows that many texts with detected propaganda are indeed worth fact-checking. However, low recall illustrates that many check-worthy texts do not contain propaganda detectable by our model. Such results lead to an intuitive conclusion that propaganda often signals the need for fact-checking, but not all check-worthy statements necessarily rely on propaganda methods.
We then further fine-tuned this model on CheckThat! Task 1 data and utilized it to predict checkworthiness (MULTI-PROP2). Due to time constraints, we did not perform hyperparameter tuning for this model. Instead, we used the hyperparameter values obtained from the search conducted during the MULTI-ALL experiment.
Since the number of allowed submissions was limited to one submission per language, we selected the models for the final predictions based on the 1 scores obtained on the − dataset. The models selected for the final submission are: MONO-EN, which ranked 8th on the final leaderboard for English, and MULTI-NO-ES, which ranked 2nd on the final leaderboard for Dutch and 3 rd on the final leaderboard for Arabic. Table 3 provides the details of the models we submitted. Additionally, the table shows the baseline provided by the organisers of ChecktThat Lab 2024 Task 1 and the score of the best team for each language.
After the submission deadline, we experimented with Large Language Models, namely GPT-3.5 and GPT-4o. Moreover, our experiments that combined knowledge from propaganda classification with check-worthiness detection through models MULTI-PROP1 and MULTI-PROP2 have also taken place after the deadline. Unfortunately, due to time constraints and strict deadlines, we could not complete these experiments before the submission deadline. Therefore, we did not consider their results on the − dataset when selecting models for submission. However, some of our post-deadline
Oficial
Rank 8 2 3 results are superior to our final submission results and, in the case of the Dutch language, even surpass the leaderboard winner.
Table 4 shows the results of all experiments we conducted. We report the 1 scores calculated on the − and datasets. We did not have access to the ground truth for the test data while developing this system. Nevertheless, we calculated the test dataset scores after the organizers released the labels for this dataset following the submission deadline. Each record of DEV-TEST columns represents the 1 scores yielded by a model fine-tuned once on the combined training dataset ( + ) with − used as a validation dataset.
The submission of the MONO-EN model earned us the 8th position on the leaderboard. Notably, the diference between the winning model’s 1 score and ours was small—just 0.042. Interestingly, the MULTI-NO-ES model, an xlm-roberta-large fine-tuned on English, Dutch, and Arabic data, achieved a better 1 score than our chosen submission, with a score of 0.7647. However, the diference may not be statistically significant. Most of the remaining models nearly matched our best result.
In Dutch, we came very close to winning with the MULTI-NO-ES model, achieving an 1 score of 0.73, just 0.02 points behind the actual winner. Two of our post-deadline models surpassed our submission result - MULTI-PROP2 with a slightly better 1 score of 0.7336 and GPT-4o with an impressive 0.7915. The results from GPT-4o even outperformed those of the leaderboard winner. Interestingly, the monolingual approach MONO-NL performed relatively poor, with an 1 score of 0.6182.
Similarly, we submitted the predictions of the MULTI-NO-ES model for the Arabic language. This submission secured us third place on the leaderboard. The diference in 1 scores between the MULTINO-ES model and the 1st ranked model on the leaderboard is 0.031. The monolingual model performed worse than the multilingual systems, but the diference is minor. Interestingly, GPT-3.5 produced one of the best results - 0.5539. Our top post-deadline model, MULTI-PROP2, nearly matched the 1 score of the leaderboard winner, with a diference of just 0.002.
The few-shot prompting approach using GPT models showed similar results to the fine-tuning approach with BERT-based models. Across all languages, the GPT-4o model consistently outperformed or performed similarly to the GPT-3.5 model. Specifically for English, GPT-4o performed noticeably better than GPT-3.5 and achieved results only slightly lower than fine-tuned BERT-based models. In the case of Dutch, GPT-4o was also superior to GPT-3.5, and it emerged as the best model among those we tested for this language. An interesting observation from our experiments is that the diferences between Dutch on − and results were more pronounced for GPT models than other models. For Arabic, GPT-3.5 and GPT-4o showed nearly identical performance, comparable to experiments using BERT-based models.
The results of a multilingual model fine-tuned on DIPROMATS 2024 Task 1a data further fine-tuned on English, Dutch, Arabic, and Spanish using CheckThat! 2024 Task 1 data (MULTI-PROP2) in one case outperformed all other approaches, but most probably the diference was statistically insignificant. We observed negligibly better results for Arabic and slightly worse results for the remaining languages compared to the MUTLI-ALL model.
Our results show that the outcomes on the and − datasets are significantly diferent. The explanation for this phenomenon is not straightforward. One possible reason is overfitting to the − data when selecting the best model during the final fine-tuning. However, this doesn’t explain the discrepancy observed with the GPT models with few-shot prompting, which were not fine-tuned yet still showed significant mismatches between the − and results. A potential reason for GPT large models diferences is that the − and data may have difered significantly.
The GPT models, particularly GPT-4o, showed results comparable to the fine-tuned BERT-based models. The few-shot prompting approach with OpenAI API ofers a more straightforward solution for such tasks, delivering similar performance at a lower time cost.
The monolingual models generally performed worse than the multilingual models in our experiments. Thus, using a language-specific model is not always the best choice. Adding Spanish to the training data did not always improve performance for the multilingual models.
Model obtained from fine-tuning on the IberLEF DIPROMATS 2024 propaganda detection task and further fine-tuning on check-worthiness data did not yield any significant performance improvement compared to multilingual models fine-tuned exclusively on the check-worthiness data. Limitation of this approach was that we did not perform hyperparameter tunning for MULTI- PROP2. Hyperparameters optimization could significantly change our results.
In our work, we experimented with monolingual and multilingual approaches for various languages for check-worthy claims detection. For the monolingual approach, we utilized BERT models tailored to specific languages, optimizing hyperparameters and fine-tuning each model separately for each language. For the multilingual approach, we used XLM-RoBERTa-large. First, we optimized and fine-tuned it on the entire dataset. Then, we excluded Spanish from the training data in a second experiment. Additionally, we employed two LLMs, namely GPT-3.5-turbo and recently released GPT-4o for each language, using few-shot prompting to classify texts. We also fine-tuned a model on the IberLEF DIPROMATS 2024 Task 1 dataset and used this model to predict whether the data from CheckThat! Lab 2024 Task 1 contained propaganda. With this analysis, we aimed to indirectly determine whether check-worthy data also includes propaganda. We later utilized the mentioned model (XLM-RoBERTa-large fine-tuned on the IberLEF DIPROMATS 2024 Task 1a for binary propaganda classification) and further fine-tuned it for check-worthiness classification. Our final submissions for all languages were: English - RoBERTa-large ifne-tuned, and optimized exclusively on English data; Dutch - XLM-RoBERTa-large, fine-tuned and optimized on all data except Spanish; Arabic - XLM-RoBERTa-large, fine-tuned and optimized on all data except Spanish.
From our experiments we can conclude that GPT models, particularly GPT-4o, showed results comparable to the fine-tuned BERT-based models. Moreover, language-specific BERT-based model performed better only on English dataset. In other languages we got better results by utilizing multilingual model.
Future work could incorporate a detailed linguistic analysis of texts to understand the diferent linguistic features among check-worthy texts and those that do not require checking. By identifying specific linguistic features and patterns, we could develop more nuanced systems that better diferentiate between these types of texts. This analysis could involve examining rhetorical devices and stylistic elements that are prevalent in check-worthy claims.
We acknowledge that our research may raise a number of ethical issues. The first shortcoming is a lack of clear explainability of our models’ results. Each model generates check-worthiness ratings without explanations as to why a statement was rated check-worthy or not. Users may need explanations to understand the basis for the model’s decisions. One of the most crucial tools for our research was ifne-tuning BERT-based models and utilizing GPT-based Large Language Models. As a result, if BERT or GPT-based models were trained on data containing any bias, disinformation, or misinformation, these problems may afect the results of our experiments. The next potential shortcoming is a possible specialization of our systems to detect specific types of check-worthy information. Our systems may possibly not handle more subtle content or other of-topic statements well. We did not check whether the dataset included one topic check-worthy information, and we relied on the work of the workshop organizers to cover more than a specific type. Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9–12, 2019, Proceedings 10, Springer, 2019, pp. 301–321. [5] A. Barrón-Cedeno, T. Elsayed, P. Nakov, G. Da San Martino, M. Hasanain, R. Suwaileh, F. Haouari, Checkthat! at clef 2020: Enabling the automatic identification and verification of claims in social media, in: Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42, Springer, 2020, pp. 499–507. [6] P. Nakov, G. Da San Martino, T. Elsayed, A. Barrón-Cedeño, R. Míguez, S. Shaar, F. Alam, F. Haouari, M. Hasanain, W. Mansour, et al., Overview of the clef–2021 checkthat! lab on detecting checkworthy claims, previously fact-checked claims, and fake news, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21–24, 2021, Proceedings 12, Springer, 2021, pp. 264–291. [7] P. Nakov, A. Barrón-Cedeño, G. da San Martino, F. Alam, J. M. Struß, T. Mandl, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, et al., Overview of the clef–2022 checkthat! lab on fighting the covid-19 infodemic and fake news detection, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2022, pp. 495–520. [8] M. Hasanain, R. Suwaileh, S. Weering, C. Li, T. Caselli, W. Zaghouani, A. Barrón-Cedeño, P. Nakov, F. Alam, Overview of the CLEF-2024 CheckThat! lab task 1 on check-worthiness estimation of multigenre content, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CLEF 2024, Grenoble, France, 2024. [9] N. Hassan, C. Li, M. Tremayne, Detecting check-worthy factual claims in presidential debates, in: Proceedings of the 24th acm international on conference on information and knowledge management, 2015, pp. 1835–1838. [10] I. Jaradat, P. Gencheva, A. Barrón-Cedeño, L. Màrquez, P. Nakov, Claimrank: Detecting checkworthy claims in arabic and english, arXiv preprint arXiv:1804.07587 (2018). [11] Y. S. Kartal, M. Kutlu, Re-think before you share: a comprehensive study on prioritizing checkworthy claims, IEEE transactions on computational social systems 10 (2022) 362–375. [12] P. Atanasova, L. Marquez, A. Barron-Cedeno, T. Elsayed, R. Suwaileh, W. Zaghouani, S. Kyuchukov, G. Da San Martino, P. Nakov, et al., Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. task 1: Check-worthiness, in: CEUR WORKSHOP PROCEEDINGS, volume 2125, CEUR-WS, 2018, pp. 1–13. [13] P. Atanasova, P. Nakov, G. Karadzhov, M. Mohtarami, G. Da San Martino, Overview of the clef-2019 checkthat! lab: Automatic identification and verification of claims. task 1: Check-worthiness (2019). [14] S. Shaar, M. Hasanain, B. Hamdan, Z. S. Ali, F. Haouari, A. Nikolov, Y. S. Kartal, F. Alam, G. Da San Martino, et al., Overview of the clef-2021 checkthat! lab task 1 on check-worthiness estimation in tweets and political debates., 2021. [15] H. T. Sadouk, F. Sebbak, H. E. Zekiri, Es-vrai at checkthat! 2023: Analyzing checkworthiness in multimodal and multigenre (2023). [16] M. Sawiński, K. Węcel, E. P. Księżniak, M. Stróżyna, W. Lewoniewski, P. Stolarski, W. Abramowicz, Openfact at checkthat! 2023: head-to-head gpt vs. bert-a comparative study of transformers language models for the detection of check-worthy claims, Working Notes of CLEF (2023). [17] A. Modzelewski, W. Sosnowski, A. Wierzbicki, Dshacker at checkthat! 2023: Check-worthiness in multigenre and multilingual content with gpt-3.5 data augmentation, Working Notes of CLEF (2023). [18] M. Mosbach, T. Pimentel, S. Ravfogel, D. Klakow, Y. Elazar, Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 12284–12314. URL: https://aclanthology.org/2023.findings-acl.779. doi:10.18653/v1/2023.findings-acl.779. [19] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL:
This appendix includes the optimal hyperparameter values for our best models. In this appendix, we present the prompt messages included with each text classification request. The prompts are provided only in Spanish for brevity. • User content: Answer whether the following text in Spanish is worth fact-checking. Answer using only a single English word: Yes or No. TEXT: tve_tve vuelve a quedar en evidencia. Desplaza al minuto 18 la denuncia del #CGPJ ante las críticas de Iglesias y habla de “diferencias”. Exigimos al responsable de edición del telediario explicaciones y a Sánchez que deje de utilizar rtve a su antojo @Enric_Hernandez • Assistant content: Yes c) Example 3: • User content: Answer whether the following text in Spanish is worth fact-checking. Answer using only a single English word: Yes or No. TEXT: El PSOE demostró su apoyo a Torra durante la moción #PorLaConvivencia Lroldansu "Cs sigue siendo el referente incontestable del Constitucionalismo en Cataluña. No podemos dejar el futuro de 7,5 millones de catalanes en manos de Torra" #ActualidadCs • Assistant content: No d) Example 4: • User content: Answer whether the following text in Spanish is worth fact-checking. Answer using only a single English word: Yes or No. TEXT: Pedro Sánchez ha dado el visto bueno a la apertura de las ’embajadas’ catalanas en Argentina, México y Túnez. Empiezan las cesiones a sus socios separatistas. Incrementan gasto en sus majaderías, despreciando la urgencia de las política sociales.
• Assistant content: Yes 3. Used final user prompt: Answer whether the following text in Spanish is worth fact-checking. Answer using only a single English word: Yes or No. TEXT: <Here we provided text to classify by LLM>