This paper represents our approach on the CheckThat! Lab designed to address the issue of disinformation. We participated in CheckThat! Lab Task 1 which focuses on identifying check-worthy claims in various forms of media, and Task 2 which targets the detection of subjective viewpoints in news articles. For both tasks we focused on the English dataset only. For task 1, after standard preprocessing, we used an ensemble approach where we combined two models, namely BERT-Base-Uncased and XLM-RoBERTa-Base in order to finetune and to find the average probabilities to determine a unified ensemble probability. For task 1 our F1 score was 0.696 and our rank was 14th in the English leaderboard. For task 2 we augmented our data after standard pre-processing using Google AI Studio and it's gemini-1.0-pro-latest model and then used the transformer-based model RoBERTa and finetuned it on the augmented dataset. For task 2, our macro F1 score was 0.7081 and our rank was 4th in the English leaderboard.
The CLEF CheckThat! Lab [17] initiative is at the forefront of technological developments in automated fact-checking, aiming to combat misinformation in the digital age. Misinformation poses significant risks to public discourse and democratic processes, making the development of effective fact-checking tools crucial. In the 2024 edition [1] [17], the Lab focuses on two key tasks, each addressing critical aspects of this challenge.
The first one concentrated on assessing the check-worthiness of claims made in tweets and other English texts. This involves identifying which statements require verification, thereby prioritizing efforts, as not all claims can be fact-checked due to resource constraints, and determining check-worthiness ensures that the most impactful misinformation is addressed promptly. The second task aimed to distinguish subjective opinions from objective facts in the sentences of the news articles, something essential for maintaining factual integrity and preventing spread of misinformation. By accurately identifying and separating opinions from facts, we can improve the reliability of news content and support informed public discourse. Unlike sentiment analysis, which would be focused on identifying emotional tones, subjectivity analysis actually aims to improve the working of Task 1 as it aims at discerning statements that may require verification (subjective) from those presenting factual information (objective). By categorizing claims, fact-checkers can prioritize rigorous scrutiny for subjective claims that may influence public opinion or require context evaluation, while focusing factual verification efforts on objective claims backed by evidence. Together, these tasks help in providing a comprehensive approach to validating information, preventing the spread of misinformation, and upholding the credibility of information sources. Both tasks make use of binary classification and measure effectiveness through F1 scores, ensuring precise and efficient validation of information, and preventing the spread of misinformation.
In recent years, the CLEF CheckThat! competition has showcased innovative approaches to claim detection. Top teams have consistently relied on transformer-based models to enhance their systems. Accenture, the top-ranked team in 2020, utilized a RoBERTa-based model, incorporating mean pooling and dropout layers to improve generalization and reduce overfitting [4][5]. This strategy helped them achieve strong performance over baseline models.
In 2021, NLP&IR@UNED explored several pre-trained transformer models, discovering that BERTweet was the most effective on the development set. BERTweet, trained on 850 million English tweets and 23 million COVID-19-specific tweets, excelled at identifying check-worthy claims [6][8]. The second-place team, Fight for 4230, also used BERTweet but added a dropout layer and implemented data augmentation techniques [7] [8]. In the following year, PoliMi-FlatEarthers stood out by fine-tuning GPT-3 for Task 1B. They combined deep learning with domain-specific customization to accurately classify check-worthy claims [9] [10]. Finally, in 2023, OpenFact leveraged a fine-tuned GPT-3 model, utilizing a rich, annotated dataset of sentences from political debates and speeches. Their data-centric approach, tailored specifically for fact-checking, helped them outperform other submissions [11] [12]. Additionally, recent research has shown that models like FACT-GPT, which use synthetic data generated by large language models for training, can closely match human judgment in identifying related claims, highlighting the potential for AI tools to enhance the fact-checking process [20].
This task has only appeared in one previous edition CheckThat! 2023. The top submissions used many different models, the most used were BERT, RoBERTa, ChatGPT and GPT3. Team DWReCo [13] [16] got the best score in the English category. There approach involved augmenting the dataset using GPT and then trained on RoBERTa. Two other teams also went with a data augmenting approach. The overall best score on the multilingual dataset was achieved by Team NN [14][16] who used the XLMRoBERTa model and trained it on the multilingual dataset. Team Thesis Titan [15][16] achieved top positions in 4 languages. Their approach was to train the mDeBERTa model finetuned for each specific languages seperately allowing them to achieve those scores. Many other teams also tried an ensemble approach and got decent results.
Similar to Team DWReCo's strategy, FACT-GPT utilized large language models (LLMs) to generate synthetic training data, enhancing the adaptability of models for specific tasks, which is crucial for claim matching in fact-checking contexts. Like the approach of using XLMRoBERTa for multilingual datasets, FACT-GPT demonstrated that fine-tuning language models on synthetic datasets could improve classification accuracy and reduce computational costs. Both FACT-GPT and the approaches in CheckThat! 2023 emphasize the importance of leveraging AI to assist and enhance human expertise in the fact-checking process. [20] 3. Task 1
The goal of Task 1 [2] was to evaluate the necessity of fact-checking claims in tweets and transcriptions. This typically requires either the expertise of professional fact-checkers or answers to several auxiliary questions by human annotators.
We were provided with three datasets: [18] the training dataset, the dev dataset, and the test-dev dataset. Later, we received the fourth dataset, the main test dataset which was unlabeled. Our initial modeling used the following parameters with the BERT-base-uncased model: After training, we used the model to process the test-dev dataset. The procedure involved:
1. Tokenizing the text entries. 2. Feeding the tokenized data into the model. 3. Converting the output logits to probabilities using a sigmoid function. 4. Classifying each entry as "Yes" or "No" based on a probability threshold of 0.5. 5. Collecting these classifications and their corresponding "Sentence_id" into a list for comparison with the original labels.
The approach achieved an F1 score of 0.80 on the test-dev dataset.
To improve results, we experimented with various models like Alberta, RoBERTa-base, XLM-RoBERTa, and ELECTRA. The most significant improvement was observed with XLM-RoBERTa-base and BERT-base-uncased. We then implemented an ensemble approach with these two models using the following training configurations:
• Both trained models were evaluated on the test-dev dataset. Each text data point from the test dataset was processed by both models, and their predictions were averaged to form a single ensemble probability. This probability determined the final label ("Yes" or "No"), which was collected along with the text's unique identifier into a list.
The goal of Task 2 was to evaluate the Subjectivity of news articles and decide whether a sentence from the news article [3][19] was subjective or objective.
Our focus was on the English datasets: the training dataset, the dev dataset, and the test-dev dataset. We used data augmentation to enhance our dataset as the train dataset was very small and the model was not able to learn and effectively. We initially tried to augment the data using WordNet model and the NTLK library. This method changed one word at random from the sentence and replaced it with its synonyms.
Our initial modeling was done using mDeBERTa and we used the following parameters:
• After training, we used the model to process the test-dev dataset. The procedure involved:
1. Processing the data and tokenizing the text entries.
2. Feeding the tokenized data into the model. 3. Converting the output logits to probabilities. 4. Classifying each entry as "Subj" or "Obj" using Sigmoid and Argmax. 5. Collecting these classifications into a list for comparison with the original labels.
The approach achieved an F1 score of 0.76. This was achieved on the dataset that had been augmented using the WordNet model and NTLK.
The current approach of changing the words with their synonyms at times did not portray the sentence correctly. We then decided to use the Gemini Api using Google AI Studio and it's 'gemini-1.0-pro-latest' model and augmented our data. The approach used in this case was to create three similar sentences for each of the "Objective" label and five similar sentences for each of the "Subjective" label. This allowed us to have a more balanced dataset and allowed the model to have a better learning. We then imported the dataset called "data", which has been uploaded on GitHub as well. While modifying, we tried different models and even used the ensemble approach using models such as RoBERTa-base, mDeBERTa, RoBERTa-xlm, and BERT-base, but the best results were achieved using RoBERTa-base alone, hence we used that for our final submission using the following training configurations: The probability calculated went through a probability threshold of 0.5, based on which we determined the final label ("Subj" or "Obj").
We saw an overall drop in the scores of our model as it achieved high scores on the training set compared to the dev set, dev-test, and test set scores, which indicates potential overfitting. This means that the model did not perform well with new, unseen data.
In Task 1, the validation loss showed a slight increase, which potentially contributed to the model's underperformance on new data. This increase in validation loss indicates that the model might have started overfitting to the training data, thereby reducing its generalizability. As a result, when the model was applied to the test set, it did not produce equally good results. Additionally, the class imbalance in the dataset could have affected the model's performance. With fewer instances labeled as check-worthy compared to non-check-worthy, the model might have struggled to accurately identify the check-worthy instances, leading to a lower overall score. The preprocessing steps, while essential for cleaning and preparing the data, might not have fully addressed the inherent variability in the text, further complicating the model's ability to generalize well to unseen data.
Moreover in task 2, a reason for the low SUBJ F1 score on the test set suggests that the model had difficulty with the "SUBJ" class. One possible reason for this could be that the features used for identifying the "SUBJ" class may not be as strong or distinctive, or there might be more variability or noise in the "SUBJ" class in the test set compared to the training set. Another reason for the low SUBJ F1 could be the way we conducted our data augmentation. The approach that we used, created three similar sentences for each of the "Objective" label and five similar sentences for each of the "Subjective" label. Considering all the sentences might have been similar to the original one from which they were made, we might have experienced over-fitting as the features of those sentences might have been similar.
In conclusion, our detailed exploration in the CheckThat! Lab 2024 challenge demonstrated the significant capabilities of transformer-based models in tasks of check-worthiness detection and subjectivity analysis. For Task 1, the ensemble method combining XLM-RoBERTa and BERT-base-uncased models effectively navigated the complexities of identifying check-worthy claims. By using a strategic ensemble of predictions and applying a robust training regimen involving multiple epochs (up to 5) and a learning rate of 5 × 10 −5 .
In Task 2, the fine-tuned RoBERTa model proved better than the other models we had tested as it showed better performance in differentiating the subjective from objective statements on the devtest file, utilizing a refined approach with a lower learning rate (5 × 10 −6 ) and an increased number of epochs (12), ensuring thorough learning. However, the performance, as indicated by a macro F1 score of 0.7081 and an F1 score of 0.54 for the SUBJ class, suggests room for improvement. A deeper analysis reveals that the model struggled with the "SUBJ" class, possibly due to weaker feature representation or greater variability and noise in the test set. Another issue that might have been prevalent is of class imabalance which might have led to weaker SUBJ identification. Future work could focus on enhancing the feature set for this class and reducing noise through better data preprocessing and augmentation.
Data augmentation played a crucial role here, bolstering the dataset and thereby enhancing the model's ability to handle nuanced textual variations. While these results were promising, they also suggest potential areas for further refinement to optimize performance, particularly in handling more complex misinformation scenarios. These efforts exemplify the essential role of adaptive, transformer-based architectures in leveraging deep learning for critical media literacy tasks in a multilingual context.
The authors would like to acknowledge the support provided by the Office Of Research (OoR) at Habib University, Karachi, Pakistan for funding this project through internal research grant IRG-2235.