This paper represents our approach on the CheckThat! Lab designed to address the issue of disinformation. We participated in CheckThat! Lab Task 1 which focuses on identifying check-worthy claims in various forms of media, and Task 2 which targets the detection of subjective viewpoints in news articles. For both tasks we focused on the English dataset only. For task 1, after standard preprocessing, we used an ensemble approach where we combined two models, namely BERT-Base-Uncased and XLM-RoBERTa-Base in order to finetune and to find the average probabilities to determine a unified ensemble probability. For task 1 our F1 score was 0.696 and our rank was 14th in the English leaderboard. For task 2 we augmented our data after standard pre-processing using Google AI Studio and it's gemini-1.0-pro-latest model and then used the transformer-based model RoBERTa and ifnetuned it on the augmented dataset. For task 2, our macro F1 score was 0.7081 and our rank was 4th in the English leaderboard.
The CLEF CheckThat! Lab [17] initiative is at the forefront of technological developments in automated
fact-checking, aiming to combat misinformation in the digital age. Misinformation poses significant
risks to public discourse and democratic processes, making the development of efective fact-checking
tools crucial. In the 2024 edition [
The first one concentrated on assessing the check-worthiness of claims made in tweets and other English texts. This involves identifying which statements require verification, thereby prioritizing eforts, as not all claims can be fact-checked due to resource constraints, and determining check-worthiness ensures that the most impactful misinformation is addressed promptly. The second task aimed to distinguish subjective opinions from objective facts in the sentences of the news articles, something essential for maintaining factual integrity and preventing spread of misinformation. By accurately identifying and separating opinions from facts, we can improve the reliability of news content and support informed public discourse. Unlike sentiment analysis, which would be focused on identifying emotional tones, subjectivity analysis actually aims to improve the working of Task 1 as it aims at discerning statements that may require verification (subjective) from those presenting factual information (objective). By categorizing claims, fact-checkers can prioritize rigorous scrutiny for subjective claims that may influence public opinion or require context evaluation, while focusing factual verification eforts on objective claims backed by evidence. Together, these tasks help in providing a comprehensive approach to validating information, preventing the spread of misinformation, and upholding the credibility of information sources.
Both tasks make use of binary classification and measure efectiveness through F1 scores, ensuring precise and eficient validation of information, and preventing the spread of misinformation.
2.1. Task 1
In recent years, the CLEF CheckThat! competition has showcased innovative approaches to claim
detection. Top teams have consistently relied on transformer-based models to enhance their systems.
Accenture, the top-ranked team in 2020, utilized a RoBERTa-based model, incorporating mean pooling
and dropout layers to improve generalization and reduce overfitting [
In 2021, NLP&IR@UNED explored several pre-trained transformer models, discovering that
BERTweet was the most efective on the development set. BERTweet, trained on 850 million English
tweets and 23 million COVID-19-specific tweets, excelled at identifying check-worthy claims [
Similar to Team DWReCo’s strategy, FACT-GPT utilized large language models (LLMs) to generate synthetic training data, enhancing the adaptability of models for specific tasks, which is crucial for claim matching in fact-checking contexts. Like the approach of using XLMRoBERTa for multilingual datasets, FACT-GPT demonstrated that fine-tuning language models on synthetic datasets could improve classification accuracy and reduce computational costs. Both FACT-GPT and the approaches in CheckThat! 2023 emphasize the importance of leveraging AI to assist and enhance human expertise in the fact-checking process. [20] 3. Task 1
The goal of Task 1 [
We were provided with three datasets: [18] the training dataset, the dev dataset, and the test-dev dataset. Later, we received the fourth dataset, the main test dataset which was unlabeled. Our initial modeling used the following parameters with the BERT-base-uncased model: • Batch size: 8 for both training and validation • Learning rate: 2 × 10− 5 • Number of epochs: 3 After training, we used the model to process the test-dev dataset. The procedure involved: 1. Tokenizing the text entries. 2. Feeding the tokenized data into the model. 3. Converting the output logits to probabilities using a sigmoid function. 4. Classifying each entry as “Yes” or “No” based on a probability threshold of 0.5. 5. Collecting these classifications and their corresponding “Sentence_id” into a list for comparison with the original labels.
The approach achieved an F1 score of 0.80 on the test-dev dataset.
To improve results, we experimented with various models like Alberta, RoBERTa-base, XLM-RoBERTa, and ELECTRA. The most significant improvement was observed with XLM-RoBERTa-base and BERT-base-uncased. We then implemented an ensemble approach with these two models using the following training configurations: • Batch size: 16 for both training and validation • Learning rate: 5 × 10− 5 • Number of epochs: 5 • Weight Decay: 0.005 Both trained models were evaluated on the test-dev dataset. Each text data point from the test dataset was processed by both models, and their predictions were averaged to form a single ensemble probability. This probability determined the final label (“Yes” or “No”), which was collected along with the text’s unique identifier into a list.
4. Task 2
The goal of Task 2 was to evaluate the Subjectivity of news articles and decide whether a sentence from
the news article [
Our focus was on the English datasets: the training dataset, the dev dataset, and the test-dev dataset. We used data augmentation to enhance our dataset as the train dataset was very small and the model was not able to learn and efectively. We initially tried to augment the data using WordNet model and the NTLK library. This method changed one word at random from the sentence and replaced it with its synonyms.
Our initial modeling was done using mDeBERTa and we used the following parameters: • Batch size: 16 for both training and validation • Learning rate: 5 × 10− 5 • Number of epochs: 6 • Warmup steps: 100 • Weight decay: 0.01 After training, we used the model to process the test-dev dataset. The procedure involved: 1. Processing the data and tokenizing the text entries. 2. Feeding the tokenized data into the model. 3. Converting the output logits to probabilities. 4. Classifying each entry as "Subj" or "Obj" using Sigmoid and Argmax.
5. Collecting these classifications into a list for comparison with the original labels. The approach achieved an F1 score of 0.76. This was achieved on the dataset that had been augmented using the WordNet model and NTLK.
The current approach of changing the words with their synonyms at times did not portray the sentence correctly. We then decided to use the Gemini Api using Google AI Studio and it’s ’gemini-1.0-pro-latest’ model and augmented our data. The approach used in this case was to create three similar sentences for each of the "Objective" label and five similar sentences for each of the "Subjective" label. This allowed us to have a more balanced dataset and allowed the model to have a better learning. We then imported the dataset called "data", which has been uploaded on GitHub as well. While modifying, we tried diferent models and even used the ensemble approach using models such as RoBERTa-base, mDeBERTa, RoBERTa-xlm, and BERT-base, but the best results were achieved using RoBERTa-base alone, hence we used that for our final submission using the following training configurations: • Batch size: 64 for both training and validation • Learning rate: 5 × 10− 6 • Number of epochs: 12 • Warmup steps: 100 • Weight decay: 0.01 The probability calculated went through a probability threshold of 0.5, based on which we determined the final label ("Subj" or "Obj").
We saw an overall drop in the scores of our model as it achieved high scores on the training set compared to the dev set, dev-test, and test set scores, which indicates potential overfitting. This means that the model did not perform well with new, unseen data.
In Task 1, the validation loss showed a slight increase, which potentially contributed to the model’s underperformance on new data. This increase in validation loss indicates that the model might have started overfitting to the training data, thereby reducing its generalizability. As a result, when the model was applied to the test set, it did not produce equally good results. Additionally, the class imbalance in the dataset could have afected the model’s performance. With fewer instances labeled as check-worthy compared to non-check-worthy, the model might have struggled to accurately identify the check-worthy instances, leading to a lower overall score. The preprocessing steps, while essential for cleaning and preparing the data, might not have fully addressed the inherent variability in the text, further complicating the model’s ability to generalize well to unseen data. Moreover in task 2, a reason for the low SUBJ F1 score on the test set suggests that the model had dificulty with the "SUBJ" class. One possible reason for this could be that the features used for identifying the "SUBJ" class may not be as strong or distinctive, or there might be more variability or noise in the "SUBJ" class in the test set compared to the training set. Another reason for the low SUBJ F1 could be the way we conducted our data augmentation. The approach that we used, created three similar sentences for each of the "Objective" label and five similar sentences for each of the "Subjective" label. Considering all the sentences might have been similar to the original one from which they were made, we might have experienced over-fitting as the features of those sentences might have been similar.
In conclusion, our detailed exploration in the CheckThat! Lab 2024 challenge demonstrated the significant capabilities of transformer-based models in tasks of check-worthiness detection and subjectivity analysis. For Task 1, the ensemble method combining XLM-RoBERTa and BERT-base-uncased models efectively navigated the complexities of identifying check-worthy claims. By using a strategic ensemble of predictions and applying a robust training regimen involving multiple epochs (up to 5) and a learning rate of 5 × 10− 5.
In Task 2, the fine-tuned RoBERTa model proved better than the other models we had tested as it showed better performance in diferentiating the subjective from objective statements on the devtest lfie, utilizing a refined approach with a lower learning rate ( 5 × 10− 6) and an increased number of epochs (12), ensuring thorough learning. However, the performance, as indicated by a macro F1 score of 0.7081 and an F1 score of 0.54 for the SUBJ class, suggests room for improvement. A deeper analysis reveals that the model struggled with the "SUBJ" class, possibly due to weaker feature representation or greater variability and noise in the test set. Another issue that might have been prevalent is of class imabalance which might have led to weaker SUBJ identification. Future work could focus on enhancing the feature set for this class and reducing noise through better data preprocessing and augmentation. Data augmentation played a crucial role here, bolstering the dataset and thereby enhancing the model’s ability to handle nuanced textual variations. While these results were promising, they also suggest potential areas for further refinement to optimize performance, particularly in handling more complex misinformation scenarios. These eforts exemplify the essential role of adaptive, transformer-based architectures in leveraging deep learning for critical media literacy tasks in a multilingual context.
The authors would like to acknowledge the support provided by the Ofice Of Research (OoR) at Habib University, Karachi, Pakistan for funding this project through internal research grant IRG-2235.