MeVer team tackling Corona virus and Conspiracies using Ensemble Classification Olga Papadopoulou Symeon Papadopoulos Information Technologies Institute - ITI, CERTH, Information Technologies Institute - ITI, CERTH, Thessaloniki, Greece Thessaloniki, Greece olgapapa@iti.gr papadop@iti.gr ABSTRACT 2 RELATED WORK This paper presents the approach developed by the Media Verifica- Several works have been introduced dealing with the detection and tion (MeVer) team to tackle the task of Corona Virus and Conspira- verification of COVID 19-related misinformation utilizing machine cies Multimedia Analysis Task at the MediaEval 2021 Challenge. We and deep learning approaches [1, 3, 16]. An overview of CON- utilized ensemble learning and propose a two-stage classification STRAINT 2021 Shared Tasks: Detecting English COVID-19 Fake approach that aims to overcome the challenge of the imbalanced News and Hindi Hostile Posts [8] shows that BERT or its variations and relatively small training dataset. We deal with the problem as was used for building the most successful models. binary classification in the first stage and in the second stage we pre- A significant contribution to combat misinformation is the cre- dict the multi-labels. We experimented with fine-tuning pre-trained ation of large enough annotated datasets which will serve to build Bidirectional Encoder Representations from Transformers (BERT) more accurate models. Patwa et al. [9] released a dataset of 10,700 and achieved a score of 0.294 in terms of the Matthews Correlation social media posts and articles of real and fake news on COVID-19. Coefficient (MCC), which is the official evaluation metric of the In Shahi et al. [14], the first multilingual cross-domain dataset of task. Additionally, leveraging on the proposed two-stage classifica- 5,182 fact-checked news articles for COVID-19 was introduced. tion approach, we extracted a set of feature representations (BoW, TfIDF, embeddings) and classify them using traditional machine 3 APPROACH learning algorithms (Support Vector Machines, Logistic Regression) We first utilized the approach that we had developed in last year’s achieving in the best run a score of 0.292 of MCC. task of FakeNews: Coronavirus and 5G conspiracy [7]. We adapted the method by corresponding the 5G conspiracy class to the Promote 1 INTRODUCTION Conspiracy class of this year’s task, the Other Conspiracy to Discuss The challenge of COVID 19-related misinformation has emerged Conspiracy and the Non-Conspiracy to Non-Conspiracy. In short, with the COVID 19 pandemic and continues to concern the com- it is a two-step classification approach that first applies an initial munity about the amount of misinformation being disseminated classification based on ensemble learning in order to provide a first- and its implications for many areas, such as health and society level classification of the Conspiracy and Non-conspiracy tweets [6, 15]. The need to develop methods to combat the dissemination and then a second step that predicts the classified Conspiracy tweets of COVID-related conspiracies triggered the organization of the whether they are promoting conspiracy or discussing a conspiracy. last year’s task of FakeNews: Coronavirus and 5G conspiracy in the For further details about the approach, the reader is referred to last MediaEval 2020 Challenge [11] and this year’s task of FakeNews: year’s working notes [7]. Corona Virus and Conspiracies Multimedia Analysis [10, 12]. In addition, we run a set of complementary experiments based A critical role in developing accurate methods for the automatic on the proposed approach, i.e. leveraging on the two-stage classifi- detection of misleading tweets (and any other text or multimedia cation, experimenting with different feature representations and item) plays the amount of annotated training samples. Due to the classify them using machine learning algorithms. In the following, relatively small training dataset provided to deal with the challenge we first describe how we deal with the imbalanced dataset, then of detecting corona virus conspiracies, our approach follows a two- we list the different combinations of features and models that we stage pipeline built on ensemble classification. In the first stage the used in our experiments and we conclude with the results of the task is converted to a binary classification problem that classifies proposed runs on the provided testing set of unseen tweets. the tweets in COVID Conspiracy tweets (involving both promoting and discussing a conspiracy) and non-Conspiracy tweets. In the 3.1 Dealing with the imbalanced dataset second stage, the COVID conspiracy tweets are further classified The provided dataset consist of 1,554 tweets in total for which in promoting conspiracy (tweets that promotes, supports, claim, 516 promotes COVID-related conspiracies (Promote), 271 discusses insinuate some connection between COVID-19 and various con- COVID-related conspiracies (Discuss) and 767 do not refer to COVID- spiracies) and discussing conspiracy (just mentioning the existing related conspiracies (Non-Conspiracy). Johnson el al. [4] published various conspiracies connected to COVID-19). The final output of a survey on deep learning with class imbalance showing that ma- the methods is a three-class prediction. chine and deep learning approaches are essentially affected in terms of prediction accuracy when trained with imbalanced samples. To Copyright 2021 for this paper by its authors. Use permitted under Creative Commons this end, we sub-sample training tweets of the majority classes in License Attribution 4.0 International (CC BY 4.0). MediaEval’21, December 13-15 2021, Online order to balance the training sets and build the proposed classi- fiers. Specifically, the classifiers of the first stage were trained with MediaEval’21, December 13-15 2021, Online O. Papadopoulou et al. 540 samples of Conspiracy tweets (270 random samples of Promote Table 1: Evaluation results class and 270 random samples of Discuss class) and 540 samples of Non-Conspiracy tweets. In the second stage, we trained a binary Run id Run 1 Run2 Run 3 Run 4 Run 5 classifier with positive class the Promote class and negative class MCC 0.257 0.268 0.238 0.292 0.294 the Discuss class and a three-class model (Promote, Discuss, Non- Conspiracy) by randomly selecting 270 samples from each class for balance. predictions on models trained on the same combinations on the three classes. 3.2 Feature representation and machine • Run 5: This run is the method proposed in [7]. learning algorithms 4 RESULTS AND ANALYSIS In our additional experiments, we extracted five feature representa- The proposed approach of Papadopoulou et al. [7] achieved the best tions: i) BoW: A simple and effective model for text representation score (among our runs) of 0.294 in terms of MCC on the provided is the Bag-of-Words (BoW) Model. The model throws away all of testing set of unseen tweets for the task of FakeNews: Corona Virus the order information in the words and focuses on the occurrence and Conspiracies Multimedia Analysis Task. In Table 1, the evalua- of words in a tweet. ii) TFIDF: term frequency–inverse document tion results in terms of MCC on the unseen tweets are presented for frequency reflects how important a word is to a tweet in a collec- the five submitted runs. We observed that the accuracy of the four tion of tweets. iii) BERT: We employ the bert-base-uncased version additional runs compared to the approach of Papadopoulou et al. of BERT [2], which is a compact transformer model, trained on [7] is slightly worse. The fact that traditional feature representation lower-cased English text. iv) Distil: We employ DistilBERT [13], such as BoW and TFIDF combined with emebeddings achieve simi- which is a small, fast, cheap and light Transformer model trained by lar results to more complex deep learning approaches highlights distilling BERT base. v) Roberta: We employ the RoBERTa model the challenge of the limited training data. We assume that with a [5], which is built on BERT and and modifies key hyperparameters, significantly larger training set the approach of Papadopoulou et removing the next-sentence pretraining objective and training with al. [7] will achieve much better predictions. much larger mini-batches and learning rates. Each feature repre- sentation is fed in an SVM and a LR and we conclude with a set of 5 DISCUSSION AND OUTLOOK multiple classifiers (Bow + SVM, BERT + LR, etc.) The proposed method achieves fairly accurate results in the task of FakeNews: Corona Virus and Conspiracies Multimedia Analysis 3.3 Runs Task. We followed our approach introduced in the MediaEval 2020 The classifiers are trained on the samples presented in Section 3.1. Challenge and based on the proposed pipeline we experimented In the first stage the predictions of the binary classifiers and fused with different setups by extracting several feature representations using majority voting and provided to the second stage where the and using them to train traditional machine learning algorithms. final predictions are calculated. We submitted four runs based on We noticed that fusing the predictions of different feature repre- different combinations of the feature representations and machine sentations and classification models we achieved almost the same learning algorithms. results as with fine-tuning pre-trained BERT, one of the most pop- • Run 1: In the first stage we build a ensemble of binary ular transformer models. We observed that the limitation or the models combining all feature representations and both ma- relatively small training set affects the prediction accuracy of the chine learning algorithms. The predictions of the models models negatively and augmentation techniques to create more are fused using majority voting and in the second stage the samples of the minority classes could be a step to improve the tweets classified as Conspiracy are further fed in an ensem- predictions in future implementations. ble of three-class models and binary classifiers (Promote vs Discuss) again trained on all combinations. ACKNOWLEDGMENTS • Run 2: The first stage is the same as with Run 1 and in This work is supported by the WeVerify project, which is funded the second stage we fuse the predictions of binary classi- by the European Commission under contract number 825297. fiers trained on Promote vs Discuss classes and the Con- spiracy vs Non-Conspiracy classes. For the Conspiracy vs REFERENCES Non-Conspiracy models we use all training samples. [1] Mabrook S Al-Rakhami and Atif M Al-Amri. 2020. Lies kill, facts save: • Run 3: In the first stage we select a combination of feature detecting COVID-19 misinformation in twitter. Ieee Access 8 (2020), representations and machine learning algorithms which 155961–155970. derived as the best combination in terms of accuracy based [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. on cross validation. In the second stage we follow the com- 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). binations of Run 2. [3] Mohamed K Elhadad, Kin Fun Li, and Fayez Gebali. 2020. Detecting • Run 4: In the first stage, we select only the combinations of misleading information on covid-19. Ieee Access 8 (2020), 165201– BoW and BERT feature representations and LR and SVM. 165215. For each combination, we train N models with sub-samples [4] Justin M Johnson and Taghi M Khoshgoftaar. 2019. Survey on deep of the training set. Similarity, the second stage fuses the learning with class imbalance. Journal of Big Data 6, 1 (2019), 27. FakeNews: Corona Virus and Conspiracies Multimedia Analysis Task MediaEval’21, December 13-15 2021, Online [5] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoy- anov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). [6] Salman Bin Naeem, Rubina Bhatti, and Aqsa Khan. 2021. An explo- ration of how fake news is taking over social media and putting public health at risk. Health Information & Libraries Journal 38, 2 (2021), 143–149. [7] Olga Papadopoulou, Giorgos Kordopatis-Zilos, and Symeon Pa- padopoulos. 2020. MeVer Team Tackling Corona Virus and 5G Con- spiracy Using Ensemble Classification Based on BERT. (2020). [8] Parth Patwa, Mohit Bhardwaj, Vineeth Guptha, Gitanjali Kumari, Shivam Sharma, Srinivas Pykl, Amitava Das, Asif Ekbal, Md Shad Akhtar, and Tanmoy Chakraborty. 2021. Overview of constraint 2021 shared tasks: Detecting english covid-19 fake news and hindi hostile posts. In International Workshop on Combating On line Ho st ile Posts in Regional Languages dur ing Emerge ncy Si tuation. Springer, 42–53. [9] Parth Patwa, Shivam Sharma, Srinivas Pykl, Vineeth Guptha, Gitan- jali Kumari, Md Shad Akhtar, Asif Ekbal, Amitava Das, and Tanmoy Chakraborty. 2021. Fighting an infodemic: Covid-19 fake news dataset. In International Workshop on Combating On line Ho st ile Posts in Regional Languages dur ing Emerge ncy Si tuation. Springer, 21–29. [10] Konstantin Pogorelov, Daniel Thilo Schroeder, Stefan Brenner, and Johannes Langguth. Online, 13-15 December 2021. FakeNews: Corona Virus and Conspiracies Multimedia Analysis Task at MediaEval 2021. In MediaEval 2021 Workshop. [11] Konstantin Pogorelov, Daniel Thilo Schroeder, Luk Burchard, Johannes Moe, Stefan Brenner, Petra Filkukova, and Johannes Langguth. 2020. FakeNews: Corona Virus and 5G Conspiracy Task at MediaEval 2020. In MediaEval 2020 Workshop. [12] Konstantin Pogorelov, Daniel Thilo Schroeder, Petra Filkuková, Stefan Brenner, and year=2021 Johannes Langguth, booktitle=Proc. of the 2021 Workshop on Open Challenges in Online Social Networks, pp. 21-25. WICO Text: A Labeled Dataset of Conspiracy Theory and 5G-Corona Misinformation Tweets. [13] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). [14] Gautam Kishore Shahi and Durgesh Nandini. 2020. FakeCovid–A multilingual cross-domain fact check news dataset for COVID-19. arXiv preprint arXiv:2006.11343 (2020). [15] Sander van Der Linden, Jon Roozenbeek, and Josh Compton. 2020. Inoculating against fake news about COVID-19. Frontiers in psychology 11 (2020), 2928. [16] Apurva Wani, Isha Joshi, Snehal Khandve, Vedangi Wagh, and Raviraj Joshi. 2021. Evaluating deep learning approaches for covid19 fake news detection. In International Workshop on Combating On line Ho st ile Posts in Regional Languages dur ing Emerge ncy Si tuation. Springer, 153–163.