Introduction

Recently, social media became the main communication channel to exchanging information among people. As a result, it becomes the primary source of news [1]. In our daily activities, such information is helpful, however, a major part of them contains misleading content that is harmful to individuals, society, or organizations [2,3]. The harmful or misleading content includes hate speech [4], hostility [5,6], propagandistic news and memes [7,8,9,10], harmful memes [11], abusive language [12], cyberbullying and cyber-aggression [13,14] and rumours [15]. The misleading or harmful aspects of such information raised the interest to identify and flag them to reduce their spread, further. There have been significant research efforts to automatically identify such content. Recent surveys on fake news [16] disinformation [17], rumours [15], propaganda [7], multimodal memes [18], hate speech [4], cyberbullying [19], and offensive content [20] highlight the importance of the problem and relevant approaches to address them.

Most often information is disseminated with facts to make people believe it is true, which are typically found in political debates, and social and global agendas. Identifying whether such facts are true or false is an important step in fighting misleading information. There have been manual efforts by fact-checking organizations to identify the truthfulness of such factual statements. As such manual efforts do not scale well, therefore, it is important to automatically identify them. However, there is a reliability issue with the automated approach [21]. A trade-off is to support human fact-checkers using an automated approach, which includes different steps in the fact-checking pipeline [22]. The first step of the fact checking pipeline is to find content that is check-worthy. The CheckThat! Lab (CTL) shared tasks is addressing this problem for the past several years. As an ongoing effort, this year CheckThat! Lab offered check worthiness subtask in six different languages such as Arabic, Bulgarian, Dutch, English, Spanish, and Turkish where data was collected from Twitter [23,24,25]. We participated in check worthiness subtask and focused on Dutch, English, and Spanish. For the experiments, we used different pretrained transformer based models, which have been widely used in several NLP tasks [2,26]. The difficulties arise when multilingual pretrained model is used in such tasks where facts and claims vary by country [27] and knowledge transferring across the language could spread the disinformation. We used multilingual transformer models (m-BERT and XLM-RoBERTa) for our experiments. In addition to the transformer models, we also used SVM and RF with TF-IDF representations.

The rest of this paper is organized as follows. In section 2, we provided related works that are relevant for this study. We then discuss the methodology in Section 3. Results of the experiments and detailed discussions are provided in Section 4. Finally, we conclude our study in Section 5.

Related Work

To deal with the factuality of statements there have been initiatives to manually check them and as result many fact-checking organizations have emerged, such as FactCheck.org1 , Snopes2 , PolitiFact3 , and FullFact4 . In addition, there have also been some international initiatives such as the Credibility Coalition5 and Eufactcheck6 [28].

One of the earlier efforts in this direction is the ClaimBuster system [29], which has been developed using the transcripts of 30 historical US election debates with a total of 28,029 transcribed sentences. The annotation includes non-factual, unimportant factual, and checkworthy factual class labels and has been carried out by students, professors, and journalists. Gencheva et al. [30] also focused on the 2016 US Presidential debates for which they obtained annotations from different fact-checking organizations. An extension of this work resulted in the development of ClaimRank, where the authors used more data and also included Arabic content Jaradat et al. [31]. Alam et al. [3] focused on COVID 19 topics in languages which are Arabic, Bulgarian, Dutch, and English, and achieved strong performances using pre-trained language models. The study also discussed the utility of single-task and multitask settings. The positive unlabelled learning technique for check-worthiness tasks has been introduced by Wright and Augenstein [32] where authors experimented with this technique with the BERT model on different datasets and achieved the best results on two datasets out of three. The study of Alhindi et al. [33] introduced a multi-layer annotated news corpus and augmented discourse structure to understand the relation between fact-checking and argumentation. The first Turkish dataset for check-worthiness has been studied by Kartal and Kutlu [34], where BERT multilingual outperforms other models. Some notable research outcomes came from shared tasks. For example, the CLEF Check-That! labs' shared tasks [35,36,37,38] in the past few years featured challenges on automatic identification [39,40] and verification [41,42] of claims in political debates, and tweets [43].

Methodology

Data

The dataset we used in our study is obtained from CLEF CheckThat!2022 lab task1: Identifying Relevant Claims in Tweets [25]. The data is based on the COVID-19 topic for Dutch and English where Spanish is mixed of politics and COVID-19 topics, which is collected from Twitter. In table 1, we present the distribution of the datasets that we used in this shared task to run our experiments. In Figure 1, we present the word cloud for all three languages to understand the most common words present in the datasets. We first removed the stopwords from the data and then used the rest of the words to generate the most frequent words.

Preprocessing

The CTL subtask-1A datasets are collected from Twitter. As a result, the data contains many symbols, URLs, and invisible characters. We performed several preprocessing steps to clean the noisy data. First, we perform URLs and unnecessary character removal steps by following the approach discussed in [44]. Then, we removed the stopwords from the data. Finally, we removed hashtag signs and usernames.

Models

We used both deep learning and traditional models to run classification experiments. As deep learning algorithms, we used two transformer based models, BERT [45] and XLM-RoBERTa [46]. Several factors were considered while choosing the algorithms. Among the transformer based models, BERT and XLM-RoBERTa are larger in parameter size. 7 The number of parameters and network size is responsible for computation time and performance of the learning. For these two models, we used the multilingual version of the models. For the later case, we used the two most popular algorithms such as (i) Random Forest (RF) [47], and (ii) Support Vector Machines (SVM) [48].

Experiments

Transformers models We use the Transformer Toolkit [49] for transformer-based models.

We used learning rate of 1𝑒 − 5 to fine-tune each model [45]. Model specific tokenizer is available with Transformer Toolkit that we used in our study. For transformer based model, we run 4, 2,

Traditional Algorithms

To train the classifiers using the above-mentioned traditional models, we first transformed the preprocessed data into tf-idf vectors with weighted 𝑛-gram (unigram, bigram and trigram) to use contextual information. The class distribution of provided dataset for English and Spanish is not well balanced. Therefore, to balance the class distribution, we applied oversampling techniques [50] for all three languages. We merged the train and dev-test set to train the model. We applied the upsampling technique to the combined dataset with a ratio of 1.0 with respect to the negative class. In Table 2, we report the hyper-parameters with the values to reproduce our results.

Results and Discussion

In Table 3, we report the official results and ranking evaluated by the lab organizers. The official evaluation metric for subtask 1A is F1 measure with respect to the positive class.

In Table 4, we report the detailed classification results for each language. After releasing the gold set once the submission period ends, we re-run all the experiments and reported the detailed results. From the table, we can conclude that among the traditional models the performance of SVM is much better than RF except for Spanish data where RF is 0.25% higher. The upsampling technique for traditional models improves from 0.10% to 1.10% on different languages with respect to the positive class. We know from the literature, transformer based models are wellknown for their performances and capabilities. Although XLM-Roberta base and BERT-m models provide the best results for Dutch and English languages with respect to positive class, where the traditional model outperforms the transformer models on Spanish language by a large margin.

Conclusion

In this study, we have run comparative experiments using different check-worthiness claim datasets consisting of Dutch, English, and Spanish languages, which are provided by CLEF CheckThat! lab 2022 organizers as a part of shared tasks. We cleaned the data to run the classification experiments. We investigated different machine learning algorithms including traditional (i.e., SVM) and deep learning models (i.e., BERT multilingual). Despite the cost of increased resource and time complexity, transformer based models did not perform well for Spanish language, however, outperformed the Dutch and English languages. Our study reveals that the transformer based models outperforms the traditional machine learning approach for Dutch and English language tasks.