This paper addresses the FakeNews: Corona virus and 5G conspiracy task at MediaEval 2020. The task involves classifying misinformation that is related to conspiracy topics and the COVID-19 pandemic. The task is divided into two subtasks where we for each subtask are proposing a separate approach. The first subtask is a Natural Language Processing (NLP)-based detection task where we are proposing a simple text-based approach by looking at the frequency of words. The second subtask is a structural-based detection task where we are proposing a method using a Graph Neural Network (GNN) to perform classification by investigating spreading patterns.
Digital wildfires are the rapid spread of online false information that
poses the risk of both economical- and physical harm to society and
individuals. In an efort to combat this, the FakeNews: Corona virus
and 5G conspiracy task [
Fake news detection has been widely recognized in research in the
recent past. There are numerous ways of tackling the challenging
problem of fake news. Zhou et al. [
The groundwork for our NLP classifier is done by applying a BOW method on the training set. As classes contain difering amounts of tweets, we use a ratio of word count per tweet within a class to better compare between classes. By comparing such a ratio of two classes, we get insight into how frequently a word is used by one class compared to a diferent class. By applying this method to the 5G Conspiracy class in the training set we are able to interpret how often a word is used by a class. By defining the number of times, a word X is mentioned in a class for and the total amount of tweets in that class for we propose a method to score an individual word given by the following equation: Score of a word = 5
5 ·
All words are assigned a score depending on the ratio of usage in the 5G-Conspiracy class compared to usage in the Non-Conspiracy class. A word like "and", which exists in most tweets regardless of class would score close to 1.00. While words used mostly by the 5G conspiracy class would score higher. By taking the geometric mean of the scores of each word within a tweet, we establish an overall score for a specific tweet. We then apply this method to every tweet in the test set and rank all the tweets based on their overall score.
To find a threshold score for classifying a 5G-Conspiracy tweet, we ran a preliminary test on the development data. We then calculated the sensitivity and 1− specificity of all thresholds of the results from the test on the development data. We found the sensitivity and 1−specificity to have the highest geometric mean at a threshold of 1.15. This threshold was then applied to the test set.
The binary classification was done by creating scores with the combined Other Conspiracy and Non-Conspiracy tweets as class (1) 0, and 5G Conspiracy class as class 1 as requested by the task. We then created a multi class classification between 5G Conspiracy, Other Conspiracy and Non-Conspiracy as Class 1, 2, 3. To do so we ran our method twice: first using the binary method and secondly by classifying between other conspiracy and regular tweets. Thus, by combining these two runs, we are able to establish a multi-class classification of the 5G Conspiracy, Other Conspiracies, and NonConspiracy
For the task of Structure-based Fake News Detection, Graph Neural
Networks (GNNs) are used as a classifier. GNNs are generalizations
of deep learning architectures such as Convolutional Neural
Network (CNN) [
Three diferent classification tasks are conducted: one multiclass classification with node features, one multi-class classification without node features, and one binary classification task without node features. For the tasks where node features are not provided, we are using one-hot encoding of node degrees as input features. In the binary classification task, we are grouping Other Conspiracy and Non-Conspiracy together.
Hyperparameter optimization is conducted separately for each classification task with a 5-fold cross-validation, which splits the training set into a 80%/20% training/validation set for each fold.
To cope with the unbalanced sets, we oversample the minority
classes in our training sets. The hyperparameters used in the cross
validation are chosen based on the one used by Xu et al. [
The models are then retrained on another 80%/20% split of the training set using the best performing hyperparameter configuration. With drop-out and early stopping, we make sure that the model is not overfitting on the training set. 4
For the NLP subtask, we achieved a score of 0.372 for multi-class classification and 0.385 for binary classification. It was expected that the score would be higher for the binary classification, as at its core, our classifier is a binary method, ran multiple times to F. Schaal, J. Phillips make tertiary. We identify certain flaws in our method. Reviewing our results, we saw the method did better for the highest threshold of our three runs. We, therefore, suspect our thresholds may have been set too low. A diferent method of determining this threshold resulting in a higher threshold might have led to greater accuracy in the classification.
The best performing hyperparameters for all three experiments were batch size of 128, 32 hidden units, and three hidden layers when additional features were provided and two otherwise. This means that the shallower networks perform better than the deeper networks, which is likely due to the fact that the distribution graphs are relatively small in size and well connected. For the multi-class tasks our GIN model had a score of 0.1810 when using features and a score of 0.1375 without using features. The performance declined for the binary classification task, which received a score of 0.1122. Based on the results, using the additional features provided increases the performance significantly. Nevertheless, the performance is fairly low compared to the NLP subtask. Table 2 below reports the results of each experiment in both subtasks.
In our NLP-based results we see that when the method fails, it often fails by not managing to interpret human characteristics. It has no method to identify characteristics like sarcasm and humor in tweets. A sarcastic tweet discussing the 5G-Conspiracy, yet not claiming it is true may still contain high scoring frequency words. Therefore such a tweet would be wrongly classified as 5G-Conspiracy. To counter this we would have had to incorporate methods more advanced than our simple word counting.
It is interesting that the binary classification for the structurebased approach performs worse compared to the multi-class classiifcation. This might be because of similarities in the distribution graphs between the two conspiracy classes: 5G-Conspiracy and Other-Conspiracy, which makes it dificult to separate the two classes.
In the future, it would be interesting to see a hybrid method of the two proposed approaches. A combination of the NLP-based approach with the structure-based approach would most likely result in a much more robust model.
This work was created in the context of the Norwegian SAMRISK2 project "UMOD" (#272019) under the supervision of Johannes Langguth, Daniel Thilo Schroeder, and Konstantin Pogorelov. FakeNews: Corona virus and 5G conspiracy