This article showcases our approach to check-worthiness detection, a task within the CheckThat! Lab of the 2023 CLEF Conference. This task aimed to design a system capable of determining if a claim, provided in diverse data formats such as tweets, debate snippets, and speech transcriptions, necessitates fact-checking. Our method combined a unified framework for processing content in three languages English, Spanish and Arabic. At the heart of our system is the XLM-RoBERTa, a pre-trained multilingual model. To enhance its performance, we applied data augmentation strategies using GPT-3.5 provided by OpenAI, which included generating paraphrases and translating text fragments to create a rich dataset. The system's efectiveness is evidenced by its performance against baseline results in all languages, notably winning first place with an F1 score of 0.641 in the Spanish category. Additionally, our exploration sheds light on the attributes of the model's performance in processing diferent languages, highlighting its exceptional performance in Spanish and indicating room for improvement in handling complex Arabic language structures.
At a time when disinformation spreads rapidly through various channels, natural language processing (NLP) is becoming a powerful ally in the automatic identification and verification of claims. By extracting relevant information from claims, NLP plays a crucial role in capturing intent and context - essential attributes in assessing the veracity of a claim.
Language models, especially those rooted in deep learning, are an essential subset of NLP technologies that have shown great potential in claim identification and verification. Essentially adept at generating human-like text, language models are trained to assess the probability of word sequences, making them highly flexible and adept at understanding the complexities of natural language.
One of the remarkable advancements in language models is the adoption of Transformer
architectures [
Additionally, Transformer-based language models, such as BERT [
Furthermore, GPT-3, one of the latest and largest language models by OpenAI, has
demonstrated the potential for few-shot learning, where the model is presented with a few examples
and can generalize to perform tasks even with minimal data [
Now that we know how important it is to use NLP and language models to determine whether claims are true or false, let us discuss our approach to check-worthiness detection. Nowadays, information spreads quickly and sometimes can be harmful or disinformative. Our research is about making a system that can automatically check the reliability of claims. The following sections will discuss the challenge we took at CLEF 2023 Labs and how we made a system that can do this in English, Spanish, and Arabic.
Claims, particularly those that include misinformation and disinformation, can spread rapidly through various social media platforms and find their way into debates and public speeches. The harmful impact of such claims cannot be underestimated, as they have the potential to mislead and manipulate public opinion. Therefore, determining the check-worthiness of claims that represent diferent genres and characteristics is often crucial. Traditionally determining the check-worthiness of claims relies on the expertise of professional fact-checkers, debunkers, or human annotators. However, this manual assessment process can be resource-intensive and costly. In this regard, it is essential to develop automated systems for claim identification and verification, which can act as supportive technology for fact-checking organizations and journalism. By utilizing automated systems, we can improve the speed and accuracy of claim evaluation, reducing the reliance on manual assessments. Therefore, CheckThat! Lab (held in the framework of CLEF 20231) organizers introduced a task aimed at developing a solution that could assist specialists in check-worthiness identification.
CheckThat! Lab organizers introduced five diferent tasks at CLEF 2023. Our research focused on claims check-worthiness identification, and therefore we participated in Task 1: CheckWorthiness in Multimodal and Multigenre Content. This task aimed to ascertain the need for fact-checking a claim presented in a text snippet. In this task, we had two kinds of data, which were translated into two subtasks: • Subtask 1A (Multimodal): check-worthiness assessment in a multimodal approach on tweets that included both a text snippet and an image. • Subtask 1B (Multigenre): check-worthiness assessment in a multigenre approach on a text representing a tweet or a debate/speech transcription snippet.
Both subtasks were ofered in multiple languages, namely subtask 1A in Arabic and English,
whereas subtask 1B in Arabic, English, and Spanish. We focused entirely on detecting
checkworthy text snippets in a multilingual and multigenre approach. Accordingly, we participated
in subtask 1B. For a more comprehensive understanding of the task, we recommend referring
to the paper that provides a detailed description of Task 1 [
Our goal was to develop a single predictive system to assess check-worthiness in three languages.
In this regard, we focused on employing a multilingual pre-trained XLM-RoBERTa-large model
[
Identifying and detecting disinformation and misinformation have emerged as highly significant
research areas. Now, researchers focus on tackling specific challenges related to identifying
disinformation, misinformation, and fake news. One of these challenges is, for instance, the
recognition of check-worthiness in claims [
Check-worthiness detection is also the subject of research within Checkthat! Labs from
previous years [
The dataset utilized in the Subtask 1B enclosed various genres and languages. The genres that these data represented were tweets and text snippets from transcriptions of debates and public speeches. Three languages were available: English, Spanish and Arabic. In addition, we enriched the data using an augmentation technique that involved the use of GPT-3.5.
The English dataset comprised snippets of transcriptions from debates and public speeches,
while the Arabic and Spanish datasets included tweets accompanied by relevant metadata. Our
approach focused solely on detecting check-worthiness using text data. As such, we did not use
any tweets’ metadata. The dataset for each language was divided into train, dev and dev_test
dataset. All observations included ground truth labels. During the final phase of the Checkthat!
Lab, we got additional unlabeled test dataset. It was the final dataset for which we had to
generate predictions and submit them for evaluation. For a comprehensive description of the
subtask dataset, please refer to the paper by Alam et al. [
As mentioned in Section 3, the datasets provided for Spanish and Arabic contained tweets and corresponding metadata fields, but in the preprocessing step, we decided to discard any additional metadata. We only utilized messages included in tweets and text snippets in English that represented transcripts of debates or public speeches. We employed textual data alongside a binary target label, where the positive class denoted a check-worthiness of the claim.
In the data preparation phase, data augmentation was performed using a variant of OpenAI’s GPT-3.5 named gpt-3.5-turbo, which is considered highly eficient and cost-efective. Custom prompts were created for generating text translations and paraphrases, thereby enriching the dataset.
Prompts that we utilized in order to produce synthetic paraphrases are as follows: • English: "Please generate a paraphrase without any additional text or explanation for the following text: <text>" • Spanish: "Please generate a paraphrase in Spanish without any additional text or explanation for the following text: <text>" • Arabic: "Please generate a paraphrase in Arabic without any additional text or explanation for the following text: <text>" We adopted a similar approach for creating synthetic translations: • English: "Please translate the following text from English to Spanish without any additional text or explanation: <text>" – "Please translate the following text from Spanish to English without any additional text or explanation: <text>" – "Please translate the following text from Spanish to Arabic without any additional text or explanation: <text>" • Arabic: "Please translate the following text from Arabic to Spanish without any additional text or explanation: <text>"
Due to resource constraints, we adopted specific rules for translation between languages to ensure quality: 1. The datasets being translated should share the same genre to maintain contextual consistency. 2. The languages involved in translation should belong to the same language family, aiding in generating natural translations due to structural similarities.
We identified English and Spanish as belonging to the Indo-European language family [ 23]. Table 1 outlines the translation process and justifications.
The most significant challenges arose when generating translations from Spanish to Arabic and from Arabic to Spanish. In these instances, many observations lacked complete translation into the target language as specified. Nonetheless, we decided to include all the generated data in the training set.
After performing data augmentation, we proceeded to remove duplicates from both the original and augmented datasets. While exploring the data, we observed that certain observations were present in both the train and dev datasets. To ensure the efectiveness of fine-tuning the pre-trained models, it was imperative to eliminate these duplicate instances.
In the hyperparameter tuning phase, we augmented the training dataset exclusively. However,
for building the final model, we utilized all the labeled data at our disposal, which included
the training, dev, and dev_test datasets. We applied augmentation techniques to each of these
datasets for the training of the final model.
4.2. Model
The primary goal of this stage was to develop a model capable of detecting check-worthy claims.
For that, we adopted a unified approach creating a single model for check-worthiness detection
across three given languages. In this regard, we utilized the multilingual RoBERTa-large [
The input text was initially tokenized, resulting in an array of tokens, with special tokens such as [CLS] denoting the start, [EOS] representing the end, and [SEP] separating sentences. This tokenized array was then processed through the XLM-RoBERTa-large model, generating an array of embeddings corresponding to the input tokens. Subsequently, the embedding array was passed through a fully connected layer, which was followed by a normalization step. During the training phase, the normalized output served as input along with the relevant labels for the binary cross-entropy loss function, enabling the loss calculation.
We predicted the final class label during inference by selecting the highest value from the normalized output.
We started our experiment by creating the training dataset, we combined all the provided train datasets from all available languages: English, Spanish, and Arabic. Moreover, we incorporated text snippets generated by GPT-3.5, as previously described in the Subsection 4.1. Next, we formed the validation dataset by combining the respective dev datasets for each language.
Once we established the training and validation datasets, we tokenized the text snippets and adjusted their length to 128 tokens by either truncating or padding them. Next, we focused on ifnding the best hyperparameters for training the model. This involved exploring diferent values for the batch size (ranging from 4 to 8), learning rate (ranging from 1e-7 to 1e-4), and weight decay (ranging from 1e-4 to 0.1). Additionally, we implemented a linear warmup for the initial 6% of the training steps. The hyperparameter search was conducted using the combined validation sets for all languages. Through this process, we identified the optimal hyperparameters as follows: a batch size of 8, a learning rate of 7.48e-06, and a weight decay of 2.65e-4. During the ifne-tuning phase of the experiment, models with the highest F1 score for the positive class were considered the best.
With the optimal hyperparameters determined, our final approach involved training the model on the combined train, dev datasets, and all augmented data. In the end, we prepared the ifnal predictions on the test dataset. The test dataset was utilized for the final evaluation and determination of our score, as showcased on the leaderboard.
We present our oficial results and position on the final leaderboard in Table 3. As shown in the table, our model achieved remarkable performance in the Spanish language.
Table 2 shows the comparison between our dev_test results and our results on the oficial leaderboard. It is evident that the dev_test results are not fully representative of the final leaderboard performance. For instance, while the model exhibited exceptionally high performance in English in the dev_test with a score of 0.946, it did not reflect similarly in the leaderboard with a score of 0.819. This could be due to the diferences in data distribution between the dev_test 3https://huggingface.co/xlm-roberta-large set and the final test set used for the leaderboard. It underlines the importance of ensuring that the model is well-generalized and not overfitting to a specific dataset.
In the Spanish language, our model significantly exceeded the baseline and surpassed all the competing teams in terms of F1 score. The F1 score over the positive class was recorded at 0.641, which is almost four times higher than the baseline score of 0.172. This clearly demonstrates the efectiveness of our model in accurately assessing the check-worthiness of multigenre content in Spanish. This superior performance can be attributed to several factors including the robustness of the underlying model, quality of the training data and the fine-tuning strategies we employed. Moreover, it indicates that our model is well-suited for the Spanish language and is capable of capturing the nuances and contextual information necessary for this task.
Moving on to the English language, our model also performed significantly better than the baseline with an F1 score of 0.819. However, it was outperformed by other teams and secured the 9th position. Despite this, the large margin between our score and the baseline score of 0.462 reflects that our model is capable of efectively identifying check-worthy statements in English. Further optimization and tuning could potentially improve the ranking among other competitors.
In Arabic, our model’s performance was slightly above the baseline, achieving an F1 score of 0.633 compared to the baseline score of 0.625. This relatively modest improvement over the baseline suggests that there might be certain challenges that our model faces when processing Arabic content. We hypothesize that GPT-3.5, which is the backbone of our model, performs worse in generating synthetic Arabic texts compared to English and Spanish. The Arabic language has complex morphological structures and right-to-left script, which might pose challenges for the model. Further investigation is needed to identify the specific areas where the model can be optimized for better performance in Arabic.
F1 While our study presents promising results, there are several limitations that need to be acknowledged. One of the critical limitations of this study was the lack of investigation into the efects of data augmentation on the model. Due to budget and time constraints, we did not conduct experiments to compare the model’s performance with and without data augmentation techniques. Data augmentation, being one of the key aspects of the model, warrants further investigation to understand its actual contribution to the performance of the model in check-worthiness detection.
Another limitation was not exploring the multilinguality aspect in-depth. Our study used a single model with a dataset that combined training data from diferent languages. However, we did not compare its performance with dedicated models that were trained on specific languages. Such an analysis would have been insightful in understanding the pros and cons of using a single multilingual model versus multiple monolingual models.
Given the aforementioned limitations, future work should involve: • Investigating Data Augmentation: A systematic investigation into the efects of data augmentation on model performance. Comparing the model with and without data augmentation will be instrumental in understanding its role in improving checkworthiness detection. • Exploring Multilinguality: Conducting experiments to compare the performance of a single model trained on combined datasets from diferent languages with models trained on language-specific datasets. This will help in identifying the best approach for multilingual check-worthiness detection. • Handling Biases: Ensuring that the systems are unbiased and fair. Further research could explore techniques for identifying and mitigating biases within the models. • Explainability and Interpretability: Building transparent models that can provide justifications for their predictions.
These future research directions will not only address the limitations of the current study but will also pave the way for more sophisticated and eficient check-worthiness detection systems.
In this study, as part of CheckThat! 2023 Lab’s Subtask 1B, we implemented a multilingual XLMRoBERTa-large model with GPT-3.5-based data augmentation to assess the check-worthiness of statements in multilingual and multigenre content. Our model exhibited remarkable performance, particularly in Spanish, where it surpassed all competitors with an F1 score of 0.641. Although the performance was commendable in English, it ranked 9th. In Arabic, the improvement was modest, hinting at challenges faced by the model in processing complex Arabic structures.
In summary, the study signifies a substantial step in automated fact-checking systems, particularly for the Spanish language, by employing pre-trained multilingual models with data augmentation. Future research should focus on overcoming limitations and refining performance across languages. on detecting check-worthy claims, previously fact-checked claims, and fake news, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21–24, 2021, Proceedings 12, Springer, 2021, pp. 264–291. [19] P. Nakov, A. Barrón-Cedeño, G. da San Martino, F. Alam, J. M. Struß, T. Mandl, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, et al., Overview of the clef–2022 checkthat! lab on ifghting the covid-19 infodemic and fake news detection, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 13th International Conference of the CLEF Association, CLEF 2022, Bologna, Italy, September 5–8, 2022, Proceedings, Springer, 2022, pp. 495–520. [20] M. Du, S. D. Gollapalli, S.-K. Ng, Nus-ids at checkthat! 2022: identifying check-worthiness of tweets using checkthat5 (2022). [21] A. Savchev, Ai rational at checkthat! 2022: using transformer models for tweet classification, Working Notes of CLEF (2022). [22] S. Agresti, S. A. Hashemian, M. J. Carman, Polimi-flatearthers at checkthat! 2022: Gpt-3 applied to claim detection (2022). [23] T. V. Gamkrelidze, V. V. Ivanov, The early history of indo-european languages, Scientific American 262 (1990) 110–117.