In this paper, we present our participation in CLEF 2022 CheckThat! Lab's Task 1 on detecting checkworthy and verifiable claims and attention-worthy and harmful tweets. We participated in all subtasks of Task1 for Arabic, Bulgarian, Dutch, English, and Turkish datasets. We investigate the impact of fine-tuning various transformer models and how to increase training data size using machine translation. We also use feed-forward networks with the Manifold Mixup regularization for the respective tasks. We are ranked first in detecting factual claims in Arabic and harmful tweets in Dutch. In addition, we are ranked second in detecting check-worthy claims in Arabic and Bulgarian.
Social media platforms became one of the main information resource for people by enabling their users to easily share messages and follow others. While these platforms are extremely important to help people share their thoughts and make their voice heard, they can be also used in a very negative way by spreading misinformation and/or hateful messages which will negatively impact individuals and societies. We have especially observed this dark side of social media platforms during COVID-19 pandemic. For instance, misinformation and conspiracy theories about vaccines increased hesitation towards being vaccinated [1]. Furthermore, the messages spread on social media platforms might impact public opinion on a particular issue and mobilize people, forcing government entities to take action. For instance, government entities of several countries had to regularly share information about vaccines to reduce the vaccine hesitation (e.g., [2]).
In this paper, we explain our participation in Task 1 [3] of the CLEF Check That! 2022 Lab [4,5]. Task 1 covers four subtasks including 1) check-worthy claim detection (Subtask 1A), verifiable factual claim detection (Subtask 1B), harmful tweet detection (Subtask 1C), and attention-worthy tweet detection (Subtask 1D). Subtask 1A covers six languages including Arabic, Bulgarian, Dutch, English, Spanish, and Turkish while the other subtasks cover all the mentioned languages except Spanish. We participated in all subtasks for Arabic, Bulgarian, Dutch, English, and Turkish languages 1 , yielding 20 submissions in total.
In the development phase of the shared task, we explored three different research directions including i) fine-tuning various pre-trained transformer models, ii) increasing the training data for fine-tuning transformer models, and iii) applying the Manifold Mixup regularization technique [6] for the subtasks we participated. In particular, we investigated 9, 3, 5, 13, and 3 different pre-trained transformer models for subtask 1A in Arabic, Bulgarian, Dutch, English, and Turkish, respectively. In addition, we explored increasing training data by back-translation and machine-translating datasets in other languages for subtask 1C. Next, we compared the Manifold Mixup approach, fine-tuning transformer models, and data augmentation by back-translation in all four subtasks to select models for our official submissions.
In our experiments with the development dataset, we find that the type of the transformer model causes dramatic changes in the performance, suggesting that researchers should select the models carefully. In addition, our findings about the impact of artificially increasing the data are mixed. In particular, we observe that increasing training data usually has a negative impact in Bulgarian and Turkish datasets in subtask 1C while using additional data for English and Dutch datasets improves the performance.
In the official ranking, we achieved mixed results. Considering tasks with at least three participants, we are ranked first in 1B-Arabic and second in 1A-Arabic and 1A-Bulgarian. We share our implementation for the Manifold Mixup method2 for reproducibility of our results.
We explore three different approaches for all subtasks including fine-tuning various transformer models, increasing dataset size via machine translation, and the Manifold Mixup regularization. In this section we explain each of them in detail.
Prior works show remarkable success of transformer models in various text classification tasks [7]. Furthermore, the best-performing systems in previous check-worthy claim detection tasks of Check That! Lab [8] usually exploited various transformer models [9,10]. However, Kartal and Kutlu [11] show that the performance of models varies dramatically across different transformer models. Therefore, in this approach, we explore several language-specific transformer models pre-trained with different datasets.
Training data has enormous impact on the performance of resultant models. Prior work on detecting check-worthy claim detection investigated several ways to increase the training data size such as back-translation [9], weak supervision [12], and utilizing datasets in other languages with multi-lingual models [11]. In this approach, we explore increasing training data size by two different methods including 1) utilizing datasets in other languages by machine-translating them into the respective language, and 2) paraphrasing the training data via back-translation and using them as additional labeled data.
In the first method, we exploit datasets in several languages provided by the Check That! Lab organizers this year. In particular, in order to develop a model for a specific language, 𝐿 𝑂 , we first select a training dataset provided for another language and machine-translate its tweets to the language 𝐿 𝑂 using Google Translate. Subsequently, we fine-tune a language-specific transformer model using the original data and machine-translated data together. In subtask 1C, we machine translate only tweets labeled as harmful to reduce the imbalance in label distribution while increasing the training data size.
In our back-translation method, we first translate the original text to another language using Google Translate. Subsequently, we translate the resultant text back to the original language. This method is likely to create slightly different texts than the original ones with a same or similar meaning. Assuming that the change in the texts will not affect their label, we combine the original data with the back-translated data and fine-tune a language specific transformer model.
Many of the annotations in the shared task are subjective. For instance, whether a tweet requires attention of government entities might depend on how much the annotators want governments to intervene their life. Similarly, prior work on check-worthiness points out the subjective nature of the task (e.g., [11,13]) In order to focus on this problem, we apply the Manifold Mixup regularization proposed by Verma et al. [6]. In particular, the Manifold Mixup trains neural networks on linear combinations of hidden representations of training examples, yielding flattened class-representations and smoother decision boundaries. Verma et al. [6] demonstrate that their approach yields more robust solutions in image classification. In our work, we use BERT embeddings to represent tweets and then train a four-layer feed-forward network with the Manifold Mixup method.
In subtask 1-D, we apply a different approach than the other tasks due to its severely imbalanced label distribution. In particular, there are nine labels in subtask 1-D, but eight of them are about why a particular tweet is attention-worthy. In addition, the majority of the tweets have "not attention-worthy" label. Therefore, we first binarize labels by merging variants of attention-worthy labels into a single one, yielding only two labels: 1) attention-worthy and 2) not-attention-worthy. Subsequently, we under-sample negative class with the 1/5 ratio and train our Manifold Mixup model. Next, we build another model using eight labels for attentionworthy tweets. If a tweet is classified as attention-worthy, we use the second model to predict why it is attention-worthy. Otherwise, we do not use the second model and label it as "not attention-worthy". Note that we do not apply this two-step approach for other subtasks because they are already binary classification tasks.
We first present statistics about the datasets and explain implementation details and our experimental setup in Section 3.1. Next, we explain how we selected our submissions in Section 3.2. Finally, we present the results of our submissions in Section 3.3.
In order to fine-tune and configure transformer models, we use PyTorch v.1.9.03 and Tensorflow4 libraries. We import transformer models used in our experiments from Huggingface 5 . In addition, we use Google's SentencePiece library for machine translation 6 . We set the batch size to 32 in all our experiments with fine-tuned transformer models. In experiments on increasing dataset size using machine translation, we train the models for 5 epochs.
We implemented the Manifold Mixup [6] method from scratch using PyTorch v.1.9.0, and set epoch and the batch size to 5 and 2, respectively. We use the following transformer models for each language: AraBERT.v02 [14] for Arabic, RoBERTa-base-bulgarian 7 for Bulgarian, RobBERT [15] for Dutch, the uncased version of BERT-base 8 for English, and DistilBERTurk9 for Turkish.
We use the official metric for each subtask to evaluate and compare our methods. In particular, we use 𝐹 1 score of positive class in subtasks 1A and 1C, accuracy in subtask 1B, and weighted 𝐹 1 in subtask 1D.
The shared task organizers provide train, development, test development, and test datasets for each language and subtask. The number of tweets for each label in train, development, test development, and test datasets in subtasks 1A, 1B, 1C, and 1D are presented in Table 1, 2, 3, and 4, respectively.
In our experiments during the development phase, we use the train and development datasets for training and validation of the Manifold Mixup model, respectively. In our experiments for fine-tuning various transformer models and increasing dataset size via machine translation, we combine train and development sets for each case and fine-tune models accordingly. In all experiments during the development phase, we use the development test dataset for testing.
We participate in all subtasks of Task 1 for five languages, yielding 20 different submissions. In addition, we explore three different approaches to determine our final submissions. Therefore, in order to reduce the complexity of experiments and meet the deadlines of the shared task, we first evaluate using various transformer models and increasing training data size in subtask 1A and 1C, respectively, on the respective test development datasets. Next, based on our experiments in subtask 1A and 1C, we compare three different approaches in all subtasks to determine our submissions for the official evaluation on the test data. We note that this is not an ideal way to select systems for submission, but we take this step to meet the deadlines.
In order to observe the impact of transformer models, we identify several transformer models available on the Huggingface platform based on their monthly download scores and evaluate their performance in subtask 1A. The number of transformer models we compare is 9, 3, 5, 13, and 3 for Arabic, Bulgarian, Dutch, English, and Turkish, respectively. We present the results in Table 5. Our observations based on our extensive experiments are as follows. Firstly, the results for English show the importance of evaluation metric to report the performance of systems. For instance, distilroberta-base-climate-f has the worst recall and 𝐹 1 scores, but achieves the best accuracy. Secondly, our results suggest that the text used in pre-training has a major impact on the models' performance. For instance, COVID-Twitter-BERT v1 achieves the best 𝐹 1 score among all English models. This should be because it is pretrained with tweets about COVID-19 while the tweets used in the shared task are also about COVID-19. Similarly, PubMedBERT, which is pretrained with research articles on PubMed, yields the second best results for English. However, we also observe some unexpected results in our experiments. For instance, AraBERT.v1, which is pre-trained on a smaller dataset compared to other variants of AraBERT (i.e., AraBERTv0.2-Twitter, AraBERTv0.2, and AraBERTv2), outperforms all Arabic specific models. In addition, while DarijaBERT is pre-trained with only texts in Moroccan Arabic, it outperforms all other Arabic specific models except AraBERT.v1. Furthermore, the best performing model in the Turkish dataset is the one with the smallest vocabulary size. Therefore, our results show that it is not easy to determine a pre-trained model by just comparing models' configurations and texts used in pre-training. We think that one of the reasons for having these unexpected results is the subjective nature of the task [11].
We use roberta-small-bulgarian 23 for Bulgarian, BERTje [20] for Dutch, BERT-base-cased for English, and bert-base-turkish-sentiment-cased 24 for Turkish as language-specific transformer models. Table 6 shows the performance of each model when a different dataset is machinetranslated to the corresponding language and respective language-specific model is fine-tuned with the original data and the machine-translated data. In this experiment, we are not able to report results for Arabic because we run into technical challenges (e.g., insufficient memory) preventing us to obtain results. We observe that increasing training data does not always improve the performance. In particular, using the original dataset for Turkish and Bulgarian yields the highest results while the performance of models usually increase in English and Dutch datasets by utilizing more labeled samples. The subjective nature of this task might be one of the reasons for having lower performance by using additional data from other languages. In particular, as each country is dealing with different social issues, it is likely that people living in different countries might disagree on what makes a message harmful for a society. For instance, Turkish annotators might be more sensitive to tweets about refugees compared to annotators for other languages because Turkey hosts nearly 3.8 million refugees, i.e., the largest refugee population worldwide 25 , and thereby, misinformation about refugees might have unpleasant consequences.
Another method to increase the traing data size is back-translation which does not deal with social differences across countries. Therefore, in our next experiment, we increase training data using various languages for back-translation. Again, we are not able to report results for Arabic due to technical challenges we encountered. In this experiment, we also use Spanish for back-translation of the Bulgarian dataset, but not the others to meet the deadlines of the lab. The results are shown in Table 7 Table 7 The impact of increasing train data using various languages for back-translation (BT). The best result for each language is written in bold. We again observe that we achieve the best result for Turkish when we use only the original dataset for training. However, back-translation improves the performance in the Dutch and English datasets. For Bulgarian, back-translation has a minimal impact. We do not observe a particular language which yields consistently higher results than others when used as the language for back-translation.
In order to select the models to submit for official ranking, we compare three different approaches for each subtask and language:
• Fine-tuning the best-performing pre-trained transformer model with the original dataset (FT-BP-TM). We use the best-performing pre-trained transformer model in our experiments in Section 3.2.1 for all subtasks except 1D. In particular, we fine-tune AraBERT.v1, RoBERTa-base-bulgarian, BERTje, COVID-Twitter-BERT v1, and BERTurk, for Arabic, Bulgarian, Dutch, English, and Turkish, respectively, using the corresponding datasets. • Fine-tuning a transformer model with back translation (FT-TM-BT). We use the best-performing model in our experiments in Section 3.2.2. In particular, we use Spanish, Turkish, Bulgarian, and English for back-translation to increase the size of Bulgarian, Dutch, English, and Turkish datasets, respectively. Note that the back-translation does not improve the performance in the Turkish dataset. However, the FT-BP-TM approach also uses the original dataset for fine-tuning. Therefore, in this approach, we increase the size of Turkish dataset using back-translation. In particular, we use English as the back-translation language because it yields the best results among others (See Table 7). • Manifold Mixup. We use the Manifold Mixup model explained in Section 2.3. Table 8, 9, 10, and 11, present results comparing three approaches for subtasks 1A, 1B, 1C, and 1D, respectively. Results for some cases are missing due to technical challenges we encountered and the limited time frame for submissions. In our submissions, we chose the best-performing method for each case and submitted our results accordingly.
Table 12 shows our results and ranking for each case we participated. We are ranked first in 1B Arabic and 1C Dutch. Focusing on subtasks with at least four participants, we are ranked second in Arabic 1A and Bulgarian 1A. We also observe that our rankings are generally higher in 1A than other subtasks.
In this paper, we present our participation in CLEF 2022 CheckThat! Lab's Task 1. We participated in all four subtasks of Task1 for Arabic, Bulgarian, Dutch, English, and Turkish, yielding 20 submissions in total. We explore which transformer model yields the highest performance, the impact of increasing training data size by machine translating datasets in other languages and back-translation, and the Manifold Mixup method proposed by Verma et al. [6]. We are ranked first in subtask 1B for Arabic and in subtask 1C for Dutch. In addition, we are ranked second in subtask 1A for Arabic and Bulgarian.
Our observations based on our comprehensive experiments are as follows. Firstly, the performance of transformer models varies dramatically based on the text used for pre-training. Secondly, increasing training data does not always improve the performance. Therefore, it is important to consider biases existing in each dataset. Thirdly, we do not observe that a particular language used for back-translation yields consistently higher performance than others.
In the future, we plan to focus on the subjective nature of the tasks in this lab. In particular, we will first qualitatively analyze the datasets to better understand annotations. Subsequently, we plan to develop a model focusing on dealing with subjective annotations.