The FakeNews task in MediaEval 2022 investigates the challenge of finding accurate and high-performance models for the classification of conspiracy tweets related to COVID-19. In this paper, we used BERT, ELMO, and their combination for feature extraction and RandomForest as classifier. The results show that ELMO performs slightly better than BERT, however their combination at feature level reduces the performance. People widely use social media to share their opinions. At times, people intentionally or unintentionally share false information about serious issues like pandemics. This results in misleading other people about the issue. As a result, people often create hurdles in managing the issue by not complying with the advice provided by authentic sources like healthcare institutes. COVID-19 has affected the entire world [1]. A lot of fake news was spread on social media platforms about the pandemic. Text-Based Misinformation and Conspiracies Detection, one of the subtasks of FakeNews Detection task in MediaEval2022 [2], focuses on the classification of tweets by conspiracy. For this study, we focused solely on this subtask because it was the most closely related to our research question, which was to investigate the effectiveness of contextualized word embeddings for identifying conspiracy-related content in social media. The task uses three different class labels to mark the tweet contents: Promotes/Supports Conspiracy, Discusses Conspiracy, and Non-Conspiracy. There are overall nine conspiracies for which the class labels have to be assigned. We use pre-trained contextualized word embedding models, BERT, and ELMo [3], and their combination for classifying misinformative tweets. The FakeNews task in 2022 extends from the FakeNews task in MediaEval 2021 [5]. Among its participants, two teams used the contextualized word embeddings from BERT pre-trained model to classify the tweets [6], [7]. In both cases, using the BERT model improved classification performance for different conspiracies.
In this section, we present the proposed model for the task.
We used a tweets dataset from the organizers of the task. Since we are using pre-trained language models like BERT [8] and ELMo [9], preprocessing is not needed. These models use all of the information in a sentence, including punctuation, and stop-words, from a wide range of perspectives by leveraging bidirectional LSTMs and multi-head self-attention mechanisms [10]. As the provided dataset was imbalanced (91% Non-Conspiracy, 3% Discusses Conspiracy, 6% Promotes/Supports Conspiracy), to effectively train the machine learning model, we use “Synthetic Minority Oversampling Technique” (SMOTE). It is a widely used and successful oversampling method to handle unbalanced datasets [11]. We sample the tweets in the following manner: 50% examples of “Non-Conspiracy”, 25% “Promotes/Supports Conspiracy”, and 25% “Discusses Conspiracy” examples. question answering, sentiment analysis, and textual entailment. In addition, we also tried to combine the embeddings from both of these most widely used pre-trained models to get contextualized word embeddings.
The BERT encoder generates a sequence of hidden states for each input sentence, representing the meaning and context of the words in the sentence. To use this sequence for classification tasks, we need to compact it into a single vector representation. There are several ways to do this, such as using max/mean pooling or applying attention to the hidden states. In our study, we chose a simpler approach: we only took the hidden state associated with the initial token ([CLS] token), which is prepended to the beginning of each sentence by BERT as a start-of-sentence token. The representation of the whole tweet is then given by the 768-dimensional embedding of the CLS token, which captures the meaning and context of the entire sentence [14], [15]. Figure 1 shows how BERT embeddings are used for modeling tweets.
ELMo gives a vector of size 1024, for every word in the input sentence. To get the vector representation of a tweet as a whole, we use the mean of the ELMo vectors [16]. Figure 2 shows the process of using tweet for classification using ELMO.
Combining the representations of BERT and ELMo is a nontrivial task due to the difference in their dimensionalities. We can reduce the dimensions of the representations using different dimensionality reduction techniques, but it will result in information loss. In this task, there were nine types of conspiracies and each of them has three labels, and a tweet can belong to more than one conspiracy. It cannot be a problem of multiclass multilabel classification straightaway as the labels are not binary, however, it can be transformed into it. But we used nine different models for each conspiracy and dealt with them separately as a multiclass classification problem [18].
The training dataset constituted 1912 examples. Algorithms used for deep learning typically require large datasets to produce good results, and they often underperform on small datasets [19]. Therefore, we chose to use the Random Forest classifier because it is a robust and widely-used machine learning algorithm that has been shown to perform well on a variety of tasks, including text classification [20]. Random Forest uses multiple decision trees based on subsamples of the dataset [18], [21].
While applying machine learning algorithms with skewed datasets, traditional error metrics, such as accuracy, are not useful, as they also consider true negatives (or the major class). We use the F1 score, which computes the harmonic mean of precision and recall. Equations 1, 2 and 3 express the formulae of precision, recall and F1 score respectively. =
+ =
+ ! = 2 × × (3)
+
Another metric that measures machine learning algorithms' performance for classification problems even if the data is imbalanced is Matthews Correlation Coefficient (MCC). Its value lies between -1 and +1, where -1 represents inverse, 0 represents average, +1 represents perfect predictions. Equation 4 expresses the formula of MCC. (1) (2) (4) =
∗ − ∗ =( + ) ∗ ( + ) ∗ ( + ) ∗ ( + )
For training the model, we split the data into random train and test subsets using 80-20 ratio. Table 1 shows the F1-scores, while Table 2 shows the Matthews Correlation Coefficient (MCC) values of the language models used along with the Random Forest classifier when tested on test subset Table 3 shows the Matthews Correlation Coefficient values of the test data provided by the organizers which also is the official score.
ELMo yields better results in most of the cases and even in the remaining cases the differences are small. According to the experimental result, the pre-trained language model aids in the extraction of conspiracy information for stance classification and conspiracy detection. However, in the classification for the Suppressed Cures Conspiracy, Harmful Radiation Influence, and Satanism Conspiracy, all outputs are incorrect. We guess that the classification models do not work because the tweets belonging to other labels (promotes conspiracy or discusses conspiracy) are only a few. Table 4 shows the distribution of training examples across conspiracies.
We propose use of different language models like BERT and ELMo, combined with Random Forest Classifier for conspiracy tweets classification. Best results are obtained using the ELMo pretrained model. It achieves an average score of 0.05 MCC without using any augmentation or extra information.