This paper summarizes the contribution of our team UIBK-DBIS-FAKENEWS to the task "FakeNews: Corona virus and 5G conspiracy" as part of MediaEval 2020. The goal for this task is to classify tweets as "5G corona virus conspiracy", "other conspiracy", or "non conspiracy", based on text analysis and based on the retweet graphs. We achieved our best results using a calibrated linear SVM with word and character n-grams for the text classification task and a non-calibrated linear SVM with graph statistics for the graph classification task.
The main objective in the task is to distinguish tweets and classify them as either (1) contributing to a conspiracy suggesting that the 5G network technology caused the SARS-CoV-2 virus epidemic, (2) contributing to a different conspiracy, or (3) not contribute to a conspiracy. For the first subtask, this classification is based on the text content of the tweets. The second subtask focuses on the retweet and follower graph of the tweets. A detailed description and the results of the challenge can be found in [8], the collection of the data is described in [9].
In the remainder of this overview, we present our solutions for the two subtasks in the following Section 2, and discuss the results thereafter in Section 3.
In both subtasks, the participants are allowed to submit 5 different solutions, whereas the first 2 solutions of each subtask are restricted to only use part of the information available. In the remaining 3 submissions, also external data points may be used.
We extract character and word-based 𝑛-grams from the text of the tweets and use them as features for our classification models. This has been shown to be effective and versatile in different text classification task ranging from stance detection [2] to classifying hacked tweet accounts [4]. We tested different parameters in a grid search, the values of which are listed in Table 1.
Submissions 2 may include additional information, so we added all features that were included in the JSON structure, which correspond to the fields available from Twitter's API 1 . We transformed all textual features to tf/idf normalized frequencies of 𝑛-grams, as listed in Table 1, left the numeric features were left as-is, and mapped all categorical features to one-hot vectors.
We included two additional features that were not in the JSON files directly. Firstly, we crawled all URLs which were included in the messages and extracted the content of the sites <title> tag, hoping that it would contain a distinctive vocabulary. Secondly, we used the free OCR software tesseract 2 to find any text within the images that are included in the messages.
We tested linear support vector machines and extra random trees as classifiers, and also added the option of calibrating the SVM using Platt's method [7]. These classifiers have been well-studied and perform well in diverse text classification tasks [10], and can compete with neural-network-based approaches in many fields like spam detection [5].
Standard graph statistics like the number of nodes or the graphs degrees are known to carry characteristics about the retweet graph to help in classification [1]. Also, algorithms like HITS [3] and PageRank [6] could produce discriminating features, as they were used on retweet graphs by Yang et al. in [11] to distinguish between tweets that are interesting only to a small group of people or a broader audience. Thus, we used the statistical networking Python package NetworkX 3 to extract statistical figures describing the retweet-follower-graphs. For the first run of the second subtask, we calculate order, size, degree, indegree, outdegree, number of connected components, density, transitivity, pagerank, HITS (hubs, authorites), number of partitions, planarity, and number of cycles, and combined them into a single feature vector. Some of the functions in NetworkX to calculate the graph statistics return lists of variable length, as their number depends on the number of nodes and edges. To create fixed-length feature vectors, we computed arithmetic mean, standard deviation, and the five-number summary of the values in the individual lists, and used these as features. For the second run in subtask 2, we additionally used the data from the nodes files, from which we calculated min, max, mean, and standard deviation of the number of friends and followers, and added these to the feature vectors calculated for the first run.
Since we extracted significantly fewer features in the second subtask, we added polynomial feature generation, and added a gaussian naïve Bayes classifier and a K-nearest neighbor to the models from the first subtask. Both are well-studied algorithms and we were interested in how well they would perform for this task. We tested several parameters in a grid search, which are displayed in Table 1.
After preliminary experiments for both subtasks, we selected the setup with the highest MCC score in a 10-fold cross-validation setup as the model that predicts our submission results for each subtask.
The scores displayed in Table 2a show that the SVM model clearly outperforms the extra random trees approach in the first subtask. Thereby, calibrating the SVM increased the performance slightly.
Interestingly, the performance of the classifiers dropped when taking more features into account for the second submission. This indicates that either too many features are extracted from the text, or that the additional meta-information was not expressive to the problem. Nevertheless, we submitted the two results in this state, being aware that we could have possibly increased the performance of the second submission by ignoring the meta-features. The evaluation results, on the other hand, don't display a performance decrease between the two submissions, where both runs result in a score of 0.440 and 0.441, respectively. As shown in Table 3, the best results were obtained by combining word unigrams and character-3-and -4-grams and a strict regulation parameter of C=0.1.
Using a linear SVM as a model allows an easy interpretation of the importance of words by looking at the respective coefficients. For each output class, Figure 1 shows the terms with the three highest and lowest coefficients. The high value for the term 5g suggests that not many topics within the other conspiracies are discussing the telecommunication standard. This relationship could be experimented with in more detail using topic modeling.
Similar to subtask 1, we used grid search to find the best performing classifier and parameters. The scores of the classifiers were rather similar, with the linear SVM producing the best score with the parameters C=10. While using polynomial features at all increased the result in both submissions by 0.05, whereas the parameters (degree= [2,3], include bias=[true, false]) did not have a great influence (< 0.01 MCC). as shown in Table 3. The results in training and evaluation approaches for subtask 2 were quite low, as displayed in Table 2b. Interestingly, our MCC validation scores for subtask 2 were lower than the training scores, which is in contrast to the scores of subtask 1, where the validation scores were slightly better than our training scores.
Our simple text-based approaches were able to classify the tweets reliably, and the coefficients of the model give insights into the most important terms. We suggest that more preprocessing might further improve these results.
The simple graph statistics, on the other hand, were not expressive enough for this task. Here, incorporating more metadata like the time between the retweets might improve the classification results.