This paper describes the Fraunhofer SIT team's third-place approach for CLEF-2024 CheckThat! lab Challenge Task 1 for English. The "Check-Worthiness Estimation" task is to determine whether a text snippet from a political debate should be prioritised for fact-checking. Identifying check-worthy statements aims to facilitate manual fact-checking by prioritising claims that fact-checkers should consider first. It can also be considered as the primary step of a fact-checking system. Our proposed system is an adapter fusion model that integrates a task adapter with a Named Entity Recognition (NER) adapter. Adapters offer a resource-efficient alternative to fully fine-tuning transformer models. Our submitted model achieves a 𝐹1 score of 0.78 on the English test set and was ranked as the third best model in the competition.
The fact-checking process typically involves three main steps. The first step is to identify statements or claims within a text that need to be fact-checked, as not all claims are equally important or contain pertinent information that needs to be verified. This can include false claims, statistics or other objectively verifiable inaccuracies. Fact-checkers prioritise claims for verification based on their potential impact, factual consistency or public interest. Once a claim has been selected, the second step is to gather credible evidence to support or refute it by consulting reliable sources such as academic journals, official reports, reputable news organisations, subject matter experts and primary sources such as original documents or statistics. To ensure consistency and accuracy, fact-checkers and journalists cross-reference information from multiple sources. The main challenge is that the majority of factcheckers' work remains manual. As a result, there is an urgent need to develop technologies that can facilitate, accelerate and improve journalists' fact-checking and fake news and misinformation detection tasks.
The first step in the fact-checking pipeline, automatically identifying statements worthy of verification, has the potential to assist fact-checkers and journalists by locating and highlighting statements within a text that warrant further verification. This process could streamline the fact-checking workflow and reduce the potential for human bias in selecting claims for verification. Check-worthy sentences or statements are usually those that contain factual information such as dates, definitions, statistics or descriptions of events or laws.
The CheckThat! Lab has been tackling this scientific problem for the past several years. The aim of this year's CheckThat! Lab Task 1 "Check-Worthiness Estimation" is to determine whether a claim in a tweet and/or a political debate/speech is worth fact-checking. The task is considered a binary classification task with data available in Arabic, English and Spanish [1]. Frauhofer SIT participated in Task 1 of the CLEF 2024 CheckThat! Lab Challenge for the English language identifying relevant claims in political debates.
In this paper, we propose an adapter fusion approach that integrates a task adapter with a Named Entity Recognition (NER) adapter. Adapters are a resource-efficient alternative to fully fine-tuned transformer models [2]. Initially, we trained a task adapter to effectively detect check-worthy statements. As check-worthy claims often contain facts in the form of named entities -such as personal names, dates, financial and percentage values -we combined the task adapter with a NER adapter. With a 𝐹 1 score of 0.78, our proposed adapter fusion model placed third in the competition.
While early approaches focused on a fixed set of features (such as sentiment, word count, part of speech (PoS) tags and named entities (NE)) and utilized traditional machine learning models (Naive Bayes, SVM and Random Forest) [3], recent work focuses on pre-trained language models such as BERT [4,5].
The CLEF CheckThat! challenge, which was introduced in 2018 and is still ongoing, has contributed a considerable amount of research in recent years. Despite the diversity of models and representations employed in the initial years of the challenge, including k-nearest neighbors [6] and recurrent neural networks [7] for models, and character n-grams [6] and word embeddings [8] for representation, neural approaches utilizing word embeddings demonstrated superior performance compared to classical methods [9]. This trend continued in the 2019 challenge, where the top-performing team used an LSTM model trained with dual token embeddings (domain-specific word embeddings and syntactic dependencies) after pre-training on previous debates [10].
After the emergence of transformers in 2019 [11], there was a shift in contributions towards utilizing transformers for check-worthiness detection in subsequent years [12,13]. Following the introduction of GPT-3, the best-performing approach for the English subtask 2023 was to fine-tune GPT-3 with 7.7k examples from pre-existing datasets. However, subsequent experiments by the same group using DeBERTaV3 yielded almost identical results to GPT-3 [14].
Schlicht et al. [15] conducted an investigation into the cross-training of adapter fusion models across various world languages, including Arabic, English, and Spanish, for the purpose of multilingual checkworthiness detection. They used mBERT and XLM-R and adapter fusion models on multilingual datasets from the CLEF CheckThat! Lab 2022 and 2021 challenges. They showed that the models outperformed monolingual task adapters and fully tuned models. A 𝐹 1 score of 0.51 was achieved for the detection of English check-worthy claims. Vogel et al. [16] combined a task adapter and a NER adapter and achieved state-of-the-art results on two challenging check-worthiness benchmarks. The best-performing model achieved a 𝐹 1 score of 0.92 on the CheckThat! Lab 2023 dataset. In addition, the authors interpreted the fusion attentions, demonstrating the effectiveness of their approach.
The data for this year's CheckThat! 2024 Challenge Task 1 "Check-Worthiness Estimation" is available in Arabic, English, and Dutch 1 [1]. However, our approach focuses only on the English data set. While the methodology employed could theoretically be applied to other languages, specific modifications to the model would be required to account for linguistic differences.
For the English task, the data set consists of political debates collected from the US presidential general election debates. Examples from the data set are shown in Table 1.
The aim goal of Task 1 is to identify entries that contain check-worthy claims. The data set was annotated by human labelers. The label distributions and data set splits were provided by the organisers and are shown in Table 2. As can be seen, the data set consists of 23,851 entries, divided into two classes: check-worthy and non-check-worthy, labeled "YES" and "NO" respectively. The data set is
In this section, we present our submitted adapter fusion approach, which combines a task adapter with a NER adapter. Adapters are a lightweight alternative to full model fine-tuning, consisting of a small set of re-initialised weights at each layer of the pre-trained model [2]. These newly introduced weights are updated during fine-tuning, while the pre-existing parameters of the model remain fixed. This feature makes adapters parameter-efficient, speeds up training iterations and, due to their compact and modular nature, enables their modular sharing and composition without compromising model performance.
Adapter fusion is a method that combines the knowledge derived from different pre-trained adapters that were trained for distinct tasks. This process incorporates an attention module, which adeptly merges knowledge from various task adapters dynamically. Consequently, it fuses the knowledge acquired from diverse adapters into a unified representation. Various fusion techniques, including weighted summation, gating mechanisms, or attention mechanisms, can be employed for this purpose [2]. The goal of the adapter fusion method is to harness the synergies between different tasks and adapters.
Initially, we trained a task adapter on the CheckThat! Lab 2024 [1] dataset to effectively identify check-worthy sentences. No data pre-processing or cleaning was applied to the dataset. To train the task adapter, we applied adapter transformers from the "Adapter Hub" repository for pre-trained adapter models [17]. We used the pre-trained RoBERTa model [18] to tokenize the input data using the maximum sequence length of 512 (truncation=True, padding="max_length"). The task adapter model was trained for 6 epochs with a learning rate of 1e-4 and a batch size of 32.
The task adapter was trained on the "Train" dataset containing 22,501 instances, while the performance of the models during training was evaluated on the "Dev" set containing 1,032 instances. Finally, the "Dev Test" set of 318 samples was used to evaluate the trained model (Table 2). Our model achieves a 𝐹 1 of 0.866 over the positive (check-worthy) class. The results of the evaluation are shown in Table 3. We chose to use the adapter fusion approach, combining a task adapter with a NER adapter, to effectively detect named entities in the dataset. Previous studies have shown that check-worthy sentences tend to contain more named entities than non-check-worthy sentences [16]. This is due to the fact that factual information often arises in the form of names and numerical data, encompassing personal names, company names, geographical locations, dates, years, and percentages. Table 4 gives examples of sentences from the dataset containing named entities.
The adapter fusion model takes as input the representations generated by multiple adapters, each trained for distinct tasks, and learns a parameterized mixer of the encoded information. The previously trained task adapter was fused with the fine-tuned version of the DistilRoBERTa [19] based NER model. The NER model was trained and evaluated on the CoNLL 2003 dataset and achieves an 𝐹 1 score of 0.92 [20]. We trained our adapter fusion model for 6 epochs with a learning rate of 5e-5 and a batch size of 32 with a maximum sequence length of 512. The model was evaluated on the "Dev Test" and achieves a 𝐹 1 of 0.916 over the check-worthy class. The results of the approach are shown in Table 5.
Identifying check-worthy statements can be seen as a first step in detecting the spread of false information online. Used as a pre-filter, this approach can significantly reduce the amount of data requiring manual evaluation by human experts. In this paper, we presented an adapter fusion method that combines a task-specific adapter and a NER adapter. Initially, we trained a task adapter to detect check-worthy statements effectively. Given that checkworthy statements often contain named entities (such as references to persons, locations, or dates), we integrated this task adapter with a pre-trained NER adapter. This integration aimed to exploit the synergies between different tasks. Our approach achieves a 𝐹 1 score of 0.78 on the CheckThat! Lab 2023 test dataset and was ranked third in the competition.
Future research may explore the integration of additional task-specific or pre-trained adapters. In our current approach, we utilized a pre-trained NER adapter developed to detect four NER classes. Subsequent work could investigate the use of a NER classifier trained to identify a broader range of NER classes.
This work was supported by the German Federal Ministry of Education and Research (BMBF) and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of "ATHENE -CRISIS".