INTRODUCTION

COVID-19 emerged as a health crisis (pandemic) and soon evolved into an infodemic ('infodemic' refers to an overabundance of information). There are already harmful impacts of COVID-19 Conspiracy theories and specifically around 5G disinformation on society. The incident of the British 5G towers fires because of coronavirus conspiracy theories [14] is a representative example of how important is to detect and prevent the dissemination of such theories. The FakeNews: Coronavirus and 5G conspiracy task is a challenge of MediaEval 2020 that focuses on the analysis of tweets around Coronavirus and 5G conspiracy theories in order to detect misinformation spreaders. For further details on the subtasks and the respective dataset, the reader is referred to [9].

Our approach focuses on ensemble classification in order to overcome the relatively small training dataset and predict more accurately the Coronavirus and 5G conspiracy tweets. In short, a first-level classification is applied using majority voting over nine classifiers to detect conspiracy and non-conspiracy tweets. A second-level classification is then applied to detect the conspiracy tweets related to 5G over the other conspiracy ones. For the training process, we leverage on the pre-trained BERT [1] model and the implementation provided by the HuggingFace library [15] 1 .

RELATED WORK

In case of a pandemic such as that of the Coronavirus, the intentional or unintentional dissemination of manipulated content, conspiracy theories, and propaganda are critical [12]. Several works have been recently published dealing with the detection and verification of COVID-19-related misinformation [2,3,10,11]. Misinformation can be spread in the form of text, images, and videos. Natural language processing (NLP) is a means of dealing with many types of content. For example, the authors of [8] collected a database of debunked and verified user-generated videos and developed a method to detect them using the contextual information surrounding them rather than the video content. The emergence of BERT (Bidirectional Encoder Representations from Transformers) has led many researchers to use it for text classification and thus in the detection of fake news [5,7]. A key limitation of emerging topics and the need to build models dedicated to a specific topic is the lack of sufficient training samples. To this end, researchers are leaning towards solutions based on ensemble methods, unsupervised learning, and data augmentation.

PROPOSED APPROACH

Figure 1 illustrates the pipeline of the proposed approach. We follow a two-step classification approach:

• First step consists of an initial classification based on ensemble learning in order to provide a first-level classification of Conspiracy and Non-conspiracy tweets. • The second step consists of the final prediction that classifies the detected Conspiracy tweets as 5G-conspiracy or Other-conspiracy.

The provided dataset consists of 1,135 samples of the 5G-conspiracy class, 712 of the Other-conspiracy class and 4,198 samples of Nonconspiracy class. As described in [4], imbalanced datasets for training machine learning algorithms or deep learning approaches pose risks of bias towards the majority class. To this end, we sub-sample training tweets of the majority classes in order to balance the training sets and build the proposed classifiers. Specifically, Table 1 presents the number of training samples considered per classifier. In 𝐶𝐿 𝑖 , the training samples of 5G-conspiracy and Other-conspiracy are concatenated into an overall Conspiracy class (1,847 tweets) and an equal number of tweets is randomly sampled from the Nonconspiracy tweets.

In the first step of our approach, we train 𝑁 classifiers 𝐶𝐿 𝑖 , which are used to predict Conspiracy and Non-conspiracy tweets. 𝑁 is empirically selected to be nine. An odd number of classifiers makes it possible to apply majority voting. Each classifier 𝐶𝐿 𝑖 predicts a label of 1 for Conspiracy or 0 for Non-conspiracy tweets. Majority voting is applied and a final prediction per tweet is given by 𝑁 In the second step, the predictions of Non-conspiracy are considered as final predictions without further processing while the Conspiracy tweets are further processed to distinguish 5G-conspiracy from Other-conspiracy. In this step, two additional models are trained focusing on the detection of 5G-conspiracy tweets. The first, 𝐶𝐿 𝑓 1 , is a three-class model (1: 5G-conspiracy, 2: Other-conspiracy and 3: Non-conspiracy) trained using random samples from the majority classes and the total number of minority class samples (Otherconspiracy). The other model, 𝐶𝐿 𝑓 2 , is a binary classifier trained on the two Conspiracy classes. The final decision is taken if 𝐶𝐿 𝑓 1 = 𝐶𝐿 𝑓 2 = 1 = 5G-conspiracy. In any other case, the tweet is labeled as Other-conspiracy.

Implementation details

For tokenization, we employ bert-base-uncased of BertTokenizer applied to the text of the tweets. The text is limited to 160 tokens as input to the network. Considering that the maximum tweet length is 280 characters, it is most likely that the entire text is processed to calculate the prediction. As a backbone network, we employ the bert-base-uncased version of BERT [13], which is a compact transformer model, trained on lower-cased English text. The network architecture consists of 12 layers (i.e., Transformer blocks), with 768 hidden units, and 12 heads for multi-head attention layers, resulting in a total of 109M parameters.

We fine-tune our networks using Adam optimizer [6] with learning rate 2 * 10 −5 . The models are trained for 10 epochs with batch size 32 and categorical cross-entropy as the loss function. During training, we use dropout after the backbone network with 0.3 Table 2: Evaluation results in terms of MCC, the official metric proposed for the task.

Method

MCC three-class BERT 0.42 Proposed approach 0.81 drop rate to prevent overfitting. Our models are evaluated against a validation set, and we select the versions that achieve the best performance in terms of accuracy as our final models.

RESULTS AND ANALYSIS

Initially, we trained a three-class model using the implementation details presented in subsection 3.1. From the annotated dataset, we randomly selected 100 samples per class as testing set and discarded them from the training phase in all runs. The performance of the model is 0.42 in terms of MCC. In order to improve the performance, we implemented the presented two-step classification approach resulting in increase of the MCC metric to 0.81 as presented in Table 2.

Our proposed approach achieved a score of 0.413 in terms of MCC on the provided testing set of unseen tweets.

DISCUSSION AND OUTLOOK

The proposed method achieves fairly accurate results in the task of FakeNews: Coronavirus and 5G conspiracy. More deep learning models, variants of BERT or other models, will be used in future experiments trying to achieve better performance. To tackle the limitation of insufficient training samples, we also intend to experiment with data augmentation approaches in order to create more samples of the minority classes and build more robust classifiers.