This paper describes the Fraunhofer SIT team's third-place approach for CLEF-2024 CheckThat! lab Challenge Task 1 for English. The "Check-Worthiness Estimation" task is to determine whether a text snippet from a political debate should be prioritised for fact-checking. Identifying check-worthy statements aims to facilitate manual fact-checking by prioritising claims that fact-checkers should consider first. It can also be considered as the primary step of a fact-checking system. Our proposed system is an adapter fusion model that integrates a task adapter with a Named Entity Recognition (NER) adapter. Adapters ofer a resource-eficient alternative to fully ifne-tuning transformer models. Our submitted model achieves a 1 score of 0.78 on the English test set and was ranked as the third best model in the competition.
The fact-checking process typically involves three main steps. The first step is to identify statements or claims within a text that need to be fact-checked, as not all claims are equally important or contain pertinent information that needs to be verified. This can include false claims, statistics or other objectively verifiable inaccuracies. Fact-checkers prioritise claims for verification based on their potential impact, factual consistency or public interest. Once a claim has been selected, the second step is to gather credible evidence to support or refute it by consulting reliable sources such as academic journals, oficial reports, reputable news organisations, subject matter experts and primary sources such as original documents or statistics. To ensure consistency and accuracy, fact-checkers and journalists cross-reference information from multiple sources. The main challenge is that the majority of factcheckers’ work remains manual. As a result, there is an urgent need to develop technologies that can facilitate, accelerate and improve journalists’ fact-checking and fake news and misinformation detection tasks.
The first step in the fact-checking pipeline, automatically identifying statements worthy of verification, has the potential to assist fact-checkers and journalists by locating and highlighting statements within a text that warrant further verification. This process could streamline the fact-checking workflow and reduce the potential for human bias in selecting claims for verification. Check-worthy sentences or statements are usually those that contain factual information such as dates, definitions, statistics or descriptions of events or laws.
The CheckThat! Lab has been tackling this scientific problem for the past several years. The aim
of this year’s CheckThat! Lab Task 1 "Check-Worthiness Estimation" is to determine whether a claim
in a tweet and/or a political debate/speech is worth fact-checking. The task is considered a binary
classification task with data available in Arabic, English and Spanish [
In this paper, we propose an adapter fusion approach that integrates a task adapter with a Named
Entity Recognition (NER) adapter. Adapters are a resource-eficient alternative to fully fine-tuned
transformer models [
While early approaches focused on a fixed set of features (such as sentiment, word count, part of speech
(PoS) tags and named entities (NE)) and utilized traditional machine learning models (Naive Bayes,
SVM and Random Forest) [
The CLEF CheckThat! challenge, which was introduced in 2018 and is still ongoing, has contributed
a considerable amount of research in recent years. Despite the diversity of models and representations
employed in the initial years of the challenge, including k-nearest neighbors [
After the emergence of transformers in 2019 [11], there was a shift in contributions towards utilizing transformers for check-worthiness detection in subsequent years [12, 13]. Following the introduction of GPT-3, the best-performing approach for the English subtask 2023 was to fine-tune GPT-3 with 7.7k examples from pre-existing datasets. However, subsequent experiments by the same group using DeBERTaV3 yielded almost identical results to GPT-3 [14].
Schlicht et al. [15] conducted an investigation into the cross-training of adapter fusion models across various world languages, including Arabic, English, and Spanish, for the purpose of multilingual checkworthiness detection. They used mBERT and XLM-R and adapter fusion models on multilingual datasets from the CLEF CheckThat! Lab 2022 and 2021 challenges. They showed that the models outperformed monolingual task adapters and fully tuned models. A 1 score of 0.51 was achieved for the detection of English check-worthy claims. Vogel et al. [16] combined a task adapter and a NER adapter and achieved state-of-the-art results on two challenging check-worthiness benchmarks. The best-performing model achieved a 1 score of 0.92 on the CheckThat! Lab 2023 dataset. In addition, the authors interpreted the fusion attentions, demonstrating the efectiveness of their approach.
The data for this year’s CheckThat! 2024 Challenge Task 1 "Check-Worthiness Estimation" is available
in Arabic, English, and Dutch1 [
For the English task, the data set consists of political debates collected from the US presidential general election debates. Examples from the data set are shown in Table 1.
The aim goal of Task 1 is to identify entries that contain check-worthy claims. The data set was annotated by human labelers. The label distributions and data set splits were provided by the organisers and are shown in Table 2. As can be seen, the data set consists of 23,851 entries, divided into two classes: check-worthy and non-check-worthy, labeled "YES" and "NO" respectively. The data set is 1Spanish was only ofered for training
significantly unbalanced, with approximately 25% of the entries labeled as check-worthy and 75% labeled as non-check-worthy.
To train and test the system, the data set is divided into three subsets: training (Train: 22,501 entries), development (Dev: 1,032 entries), and development test (Dev Test: 318 entries). The development test set contains a slightly higher proportion of check-worthy entries (33%) compared to the other data sets. The unlabeled test set (Test) was provided for evaluation purposes and consists of 341 sentences.
In this section, we present our submitted adapter fusion approach, which combines a task adapter with
a NER adapter. Adapters are a lightweight alternative to full model fine-tuning, consisting of a small set
of re-initialised weights at each layer of the pre-trained model [
Adapter fusion is a method that combines the knowledge derived from diferent pre-trained adapters
that were trained for distinct tasks. This process incorporates an attention module, which adeptly
merges knowledge from various task adapters dynamically. Consequently, it fuses the knowledge
acquired from diverse adapters into a unified representation. Various fusion techniques, including
weighted summation, gating mechanisms, or attention mechanisms, can be employed for this purpose
[
Initially, we trained a task adapter on the CheckThat! Lab 2024 [
The task adapter was trained on the "Train" dataset containing 22,501 instances, while the performance of the models during training was evaluated on the "Dev" set containing 1,032 instances. Finally, the "Dev Test" set of 318 samples was used to evaluate the trained model (Table 2). Our model achieves a 1 of 0.866 over the positive (check-worthy) class. The results of the evaluation are shown in Table 3. We chose to use the adapter fusion approach, combining a task adapter with a NER adapter, to efectively detect named entities in the dataset. Previous studies have shown that check-worthy sentences tend to contain more named entities than non-check-worthy sentences [16]. This is due to the fact that factual information often arises in the form of names and numerical data, encompassing personal names, company names, geographical locations, dates, years, and percentages. Table 4 gives examples of sentences from the dataset containing named entities.
of those has been among black Americans, and another million among those who speak
The adapter fusion model takes as input the representations generated by multiple adapters, each trained for distinct tasks, and learns a parameterized mixer of the encoded information. The previously trained task adapter was fused with the fine-tuned version of the DistilRoBERTa [ 19] based NER model. The NER model was trained and evaluated on the CoNLL 2003 dataset and achieves an 1 score of 0.92 [20].
We trained our adapter fusion model for 6 epochs with a learning rate of 5e-5 and a batch size of 32 with a maximum sequence length of 512. The model was evaluated on the "Dev Test" and achieves a 1 of 0.916 over the check-worthy class. The results of the approach are shown in Table 5. Since the adapter fusion model outperformed the adapter model in terms of 1 score, we used the former to classify the private test set of this year’s CheckThat! 2024 competition. Our model achieves an 1 score of 0.78 over the positive check-worthy class.
Identifying check-worthy statements can be seen as a first step in detecting the spread of false information online. Used as a pre-filter, this approach can significantly reduce the amount of data requiring manual evaluation by human experts. In this paper, we presented an adapter fusion method that combines a task-specific adapter and a NER adapter.
Initially, we trained a task adapter to detect check-worthy statements efectively. Given that checkworthy statements often contain named entities (such as references to persons, locations, or dates), we integrated this task adapter with a pre-trained NER adapter. This integration aimed to exploit the synergies between diferent tasks. Our approach achieves a 1 score of 0.78 on the CheckThat! Lab 2023 test dataset and was ranked third in the competition.
Future research may explore the integration of additional task-specific or pre-trained adapters. In our current approach, we utilized a pre-trained NER adapter developed to detect four NER classes. Subsequent work could investigate the use of a NER classifier trained to identify a broader range of NER classes.
This work was supported by the German Federal Ministry of Education and Research (BMBF) and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of “ATHENE – CRISIS”. [9] P. Atanasova, A. Barron-Cedeno, T. Elsayed, R. Suwaileh, W. Zaghouani, S. Kyuchukov, G. D. S.
Martino, P. Nakov, Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. task 1: Check-worthiness, 2018. arXiv:1808.05542. [10] C. Hansen, C. Hansen, J. Simonsen, C. Lioma, Neural weakly supervised fact check-worthiness detection with contrastive sampling-based ranking loss, volume 2380, ceur workshop proceedings, 2019. 20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2019 ; Conference date: 09-09-2019 Through 12-09-2019. [11] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://aclanthology.org/N19-1423. doi:10.18653/v1/N19-1423. [12] A. Barron-Cedeno, T. Elsayed, P. Nakov, G. D. S. Martino, M. Hasanain, R. Suwaileh, F. Haouari, N. Babulkov, B. Hamdan, A. Nikolov, S. Shaar, Z. S. Ali, Overview of checkthat! 2020: Automatic identification and verification of claims in social media, 2020. arXiv:2007.07997. [13] P. Nakov, G. D. S. Martino, T. Elsayed, A. Barrón-Cedeño, R. Míguez, S. Shaar, F. Alam, F. Haouari, M. Hasanain, W. Mansour, B. Hamdan, Z. S. Ali, N. Babulkov, A. Nikolov, G. K. Shahi, J. M. Struß, T. Mandl, M. Kutlu, Y. S. Kartal, Overview of the clef–2021 checkthat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news, 2021. arXiv:2109.12987. [14] M. Sawinski, K. Węcel, E. Księżniak, M. Stróżyna, W. Lewoniewski, P. Stolarski, W. Abramowicz, Openfact at checkthat! 2023: Head-to-head gpt vs. bert - a comparative study of transformers language models for the detection of check-worthy claims, 2023. [15] I. B. Schlicht, L. Flek, P. Rosso, Multilingual detection of check-worthy claims using world languages and adapter fusion, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2023, pp. 118–133. [16] I. Vogel, P. Möhle, M. Meghana, M. Steinebach, Adapter fusion for check-worthiness detection – combining a task adapter with a ner adapter., ROMCIR 2024: The 4th Workshop on Reducing Online Misinformation through Credible Information Retrieval, held as part of ECIR 2024: the 46th European Conference on Information Retrieval, March 24, 2024, Glasgow, UK (2024). URL: https://romcir.disco.unimib.it/wp-content/uploads/sites/151/2024/03/Paper6_Vogel.pdf. [17] C. Poth, H. Sterz, I. Paul, S. Purkayastha, L. Engländer, T. Imhof, I. Vulić, S. Ruder, I. Gurevych, J. Pfeifer, Adapters: A unified library for parameter-eficient and modular transfer learning, 2023. arXiv:2311.11077. [18] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, ArXiv abs/1907.11692 (2019). URL: https://api.semanticscholar.org/CorpusID:198953378. [19] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, ArXiv abs/1910.01108 (2019). [20] A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, R. Vollgraf, FLAIR: An easy-to-use framework for state-of-the-art NLP, in: NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019, pp. 54–59.