tag, acy” as part of MediaEval 2020. The goal for this task is to classify hoping that it would contain a distinctive vocabulary. Secondly, we tweets as “5G corona virus conspiracy”, “other conspiracy”, or “non used the free OCR software tesseract 2 to find any text within the conspiracy”, based on text analysis and based on the retweet graphs. images that are included in the messages. We achieved our best results using a calibrated linear SVM with We tested linear support vector machines and extra random word and character n-grams for the text classification task and trees as classifiers, and also added the option of calibrating the SVM a non-calibrated linear SVM with graph statistics for the graph using Platt’s method [7]. These classifiers have been well-studied classification task. and perform well in diverse text classification tasks [10], and can compete with neural-network-based approaches in many fields like spam detection [5]. 1 INTRODUCTION The main objective in the task is to distinguish tweets and classify 2.2 Subtask 2: Retweet-Follower-Graphs them as either (1) contributing to a conspiracy suggesting that the Standard graph statistics like the number of nodes or the graphs 5G network technology caused the SARS-CoV-2 virus epidemic, degrees are known to carry characteristics about the retweet graph (2) contributing to a different conspiracy, or (3) not contribute to to help in classification [1]. Also, algorithms like HITS [3] and a conspiracy. For the first subtask, this classification is based on PageRank [6] could produce discriminating features, as they were the text content of the tweets. The second subtask focuses on the used on retweet graphs by Yang et al. in [11] to distinguish between retweet and follower graph of the tweets. A detailed description tweets that are interesting only to a small group of people or a and the results of the challenge can be found in [8], the collection broader audience. Thus, we used the statistical networking Python of the data is described in [9]. package NetworkX 3 to extract statistical figures describing the In the remainder of this overview, we present our solutions for retweet-follower-graphs. For the first run of the second subtask, we the two subtasks in the following Section 2, and discuss the results calculate order, size, degree, indegree, outdegree, number of connected thereafter in Section 3. components, density, transitivity, pagerank, HITS (hubs, authorites), number of partitions, planarity, and number of cycles, and combined 2 METHODOLOGY them into a single feature vector. In both subtasks, the participants are allowed to submit 5 different Some of the functions in NetworkX to calculate the graph sta- solutions, whereas the first 2 solutions of each subtask are restricted tistics return lists of variable length, as their number depends on to only use part of the information available. In the remaining 3 the number of nodes and edges. To create fixed-length feature vec- submissions, also external data points may be used. tors, we computed arithmetic mean, standard deviation, and the five-number summary of the values in the individual lists, and used 2.1 Subtask 1: Twitter Messages these as features. For the second run in subtask 2, we additionally We extract character and word-based 𝑛-grams from the text of used the data from the nodes files, from which we calculated min, the tweets and use them as features for our classification models. max, mean, and standard deviation of the number of friends and This has been shown to be effective and versatile in different text followers, and added these to the feature vectors calculated for the classification task ranging from stance detection [2] to classifying first run. hacked tweet accounts [4]. We tested different parameters in a grid 2 https://tesseract-ocr.github.io/ search, the values of which are listed in Table 1. 3 https://networkx.org/ Submissions 2 may include additional information, so we added all features that were included in the JSON structure, which corre- Table 1: Hyperparameters tested in grid search. spond to the fields available from Twitter’s API1 . We transformed all textual features to tf/idf normalized frequencies of 𝑛-grams, as listed in Table 1, left the numeric features were left as-is, and Parameter Tested values mapped all categorical features to one-hot vectors. Word & character 𝑛-gram size1 [1,2,3,4] 1 https://developer.twitter.com/en/docs/twitter-api SVM: C [0.1, 1, 10] Extra Trees: number of trees [1, 2, 3, 4] ×103 Copyright 2020 for this paper by its authors. Use permitted under Creative Commons Poly. degree [2, 3] License Attribution 4.0 International (CC BY 4.0). Poly. include bias [True, False] MediaEval’20, 14-15 December 2020, Online KNN: number of neighbors [3, 4, 5, 10, 20, 50] MediaEval’20, 14-15 December 2020, Online M. Moosleitner, B. Murauer, G. Specht Table 2: Evaluation results measured with Matthew’s corre- Figure 1: Top 3 positive and negative SVM coefficients for lation coefficient. each class after fitting the message bodies of the training data. Phase Model Run 1 Run 2 5G corona conspiracy Linear SVM (calibrated) 0.432 0.412 conspiracies 5g Training Linear SVM 0.428 0.404 better wuhan Extra Random Trees 0.274 0.253 burning symptoms Evaluation Linear SVM (calibrated) 0.440 0.441 −8 −6 −4 −2 0 2 4 6 8 (a) Results of Subtask 1 No conspiracy 5g burning Phase Model Run 1 Run 2 wuhan facebook Linear SVM (calibrated) 0.003 0.054 symptoms conspiracies Linear SVM 0.127 0.197 −8 −6 −4 −2 0 2 4 6 8 Training KNN 0.118 0.135 Other conspiracy Extra Random Trees 0.089 0.091 body cancer Gaussian Naive Bayes 0.092 0.101 because but Evaluation Linear SVM 0.090 0.092 already msm (b) Results of Subtask 2 −8 −6 −4 −2 0 2 4 6 8 Table 3: Best parameters for the four submissions. Since we extracted significantly fewer features in the second subtask, we added polynomial feature generation, and added a gaussian naïve Bayes classifier and a K-nearest neighbor to the Subm. Parameters models from the first subtask. Both are well-studied algorithms and Text 1 word-1-grams + character-3+4-grams, calibrated SVM, C=0.1 we were interested in how well they would perform for this task. Text 2 word-1-grams + character-3+4-grams, calibrated SVM, C=0.1 We tested several parameters in a grid search, which are displayed Graph 1 linear SVM, C=10, Poly. deg=2, Poly. include bias = True in Table 1. Graph 2 linear SVM, C=10, Poly. deg=3, Poly. include bias = False 3 RESULTS AND DISCUSSION After preliminary experiments for both subtasks, we selected the discussing the telecommunication standard. This relationship could setup with the highest MCC score in a 10-fold cross-validation be experimented with in more detail using topic modeling. setup as the model that predicts our submission results for each subtask. 3.2 Subtask 2 Similar to subtask 1, we used grid search to find the best performing 3.1 Subtask 1 classifier and parameters. The scores of the classifiers were rather The scores displayed in Table 2a show that the SVM model clearly similar, with the linear SVM producing the best score with the outperforms the extra random trees approach in the first subtask. parameters C=10. While using polynomial features at all increased Thereby, calibrating the SVM increased the performance slightly. the result in both submissions by 0.05, whereas the parameters Interestingly, the performance of the classifiers dropped when (degree=[2,3], include bias=[true, false]) did not have a great influ- taking more features into account for the second submission. This ence (< 0.01 MCC). as shown in Table 3. The results in training and indicates that either too many features are extracted from the text, evaluation approaches for subtask 2 were quite low, as displayed or that the additional meta-information was not expressive to the in Table 2b. Interestingly, our MCC validation scores for subtask problem. Nevertheless, we submitted the two results in this state, 2 were lower than the training scores, which is in contrast to the being aware that we could have possibly increased the performance scores of subtask 1, where the validation scores were slightly better of the second submission by ignoring the meta-features. The evalua- than our training scores. tion results, on the other hand, don’t display a performance decrease between the two submissions, where both runs result in a score of 4 CONCLUSION 0.440 and 0.441, respectively. As shown in Table 3, the best results Our simple text-based approaches were able to classify the tweets were obtained by combining word unigrams and character-3- and reliably, and the coefficients of the model give insights into the -4-grams and a strict regulation parameter of C=0.1. most important terms. We suggest that more preprocessing might Using a linear SVM as a model allows an easy interpretation of further improve these results. the importance of words by looking at the respective coefficients. The simple graph statistics, on the other hand, were not expres- For each output class, Figure 1 shows the terms with the three sive enough for this task. Here, incorporating more metadata like highest and lowest coefficients. The high value for the term 5g the time between the retweets might improve the classification suggests that not many topics within the other conspiracies are results. FakeNews: Corona virus and 5G conspiracy MediaEval’20, 14-15 December 2020, Online REFERENCES [1] David R Bild, Yue Liu, Robert P Dick, Z Morley Mao, and Dan S Wallach. Aggregate characterization of user behavior in twitter and analysis of the retweet graph. ACM Transactions on Internet Technology (TOIT), 15(1):1–24, 2015. [2] Peter Bourgonje, Julian Moreno Schneider, and Georg Rehm. From clickbait to fake news detection: an approach based on detecting the stance of headlines to articles. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, pages 84–89, 2017. [3] Jon M Kleinberg. Hubs, authorities, and communities. ACM computing surveys (CSUR), 31(4es):5–es, 1999. [4] Benjamin Murauer, Eva Zangerle, and Günther Specht. A peer-based approach on analyzing hacked twitter accounts. In Proceedings of the 50th Hawaii International Conference on System Sciences, 2017. [5] N. L. Octaviani, E. Hari Rachmawanto, C. A. Sari, and D. Rosal Ignatius Moses Setiadi. Comparison of multinomial naïve bayes classifier, support vector machine, and recurrent neural network to classify email spams. In 2020 International Seminar on Application for Technology of Information and Communication (iSemantic), pages 17–21, 2020. [6] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999. [7] John Platt. Probabilistic outputs for support vector machines and com- parisons to regularized likelihood methods. Advanced Large Margin Classifiers, 10, June 2000. [8] Konstantin Pogorelov, Daniel Thilo Schroeder, Luk Burchard, Johannes Moe, Stefan Brenner, Petra Filkukova, and Johannes Langguth. Fak- enews: Corona virus and 5g conspiracy task at mediaeval 2020. In MediaEval 2020 Workshop, 2020. [9] Daniel Thilo Schroeder, Konstantin Pogorelov, and Johannes Langguth. Fact: a framework for analysis and capture of twitter graphs. In 2019 Sixth International Conference on Social Networks Analysis, Manage- ment and Security (SNAMS), pages 134–141. IEEE, 2019. [10] Simon Tong and Daphne Koller. Support vector machine active learn- ing with applications to text classification. Journal of machine learning research, 2(Nov):45–66, 2001. [11] Min-Chul Yang, Jung-Tae Lee, Seung-Wook Lee, and Hae-Chang Rim. Finding interesting posts in twitter based on retweet graph analysis. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 1073

Detecting Conspiracy Tweets Using Support Vector Machines Manfred Moosleitner, Benjamin Murauer, Günther Specht Universität Innsbruck, Austria manfred.moosleitner@uibk.ac.at,b.murauer@posteo.de,guenther.specht@uibk.ac.at ABSTRACT We included two additional features that were not in the JSON This paper summarizes the contribution of our team UIBK-DBIS- files directly. Firstly, we crawled all URLs which were included in FAKENEWS to the task “FakeNews: Corona virus and 5G conspir- the messages and extracted the content of the sites tag, acy” as part of MediaEval 2020. The goal for this task is to classify hoping that it would contain a distinctive vocabulary. Secondly, we tweets as “5G corona virus conspiracy”, “other conspiracy”, or “non used the free OCR software tesseract 2 to find any text within the conspiracy”, based on text analysis and based on the retweet graphs. images that are included in the messages. We achieved our best results using a calibrated linear SVM with We tested linear support vector machines and extra random word and character n-grams for the text classification task and trees as classifiers, and also added the option of calibrating the SVM a non-calibrated linear SVM with graph statistics for the graph using Platt’s method [7]. These classifiers have been well-studied classification task. and perform well in diverse text classification tasks [10], and can compete with neural-network-based approaches in many fields like spam detection [5]. 1 INTRODUCTION The main objective in the task is to distinguish tweets and classify 2.2 Subtask 2: Retweet-Follower-Graphs them as either (1) contributing to a conspiracy suggesting that the Standard graph statistics like the number of nodes or the graphs 5G network technology caused the SARS-CoV-2 virus epidemic, degrees are known to carry characteristics about the retweet graph (2) contributing to a different conspiracy, or (3) not contribute to to help in classification [1]. Also, algorithms like HITS [3] and a conspiracy. For the first subtask, this classification is based on PageRank [6] could produce discriminating features, as they were the text content of the tweets. The second subtask focuses on the used on retweet graphs by Yang et al. in [11] to distinguish between retweet and follower graph of the tweets. A detailed description tweets that are interesting only to a small group of people or a and the results of the challenge can be found in [8], the collection broader audience. Thus, we used the statistical networking Python of the data is described in [9]. package NetworkX 3 to extract statistical figures describing the In the remainder of this overview, we present our solutions for retweet-follower-graphs. For the first run of the second subtask, we the two subtasks in the following Section 2, and discuss the results calculate order, size, degree, indegree, outdegree, number of connected thereafter in Section 3. components, density, transitivity, pagerank, HITS (hubs, authorites), number of partitions, planarity, and number of cycles, and combined 2 METHODOLOGY them into a single feature vector. In both subtasks, the participants are allowed to submit 5 different Some of the functions in NetworkX to calculate the graph sta- solutions, whereas the first 2 solutions of each subtask are restricted tistics return lists of variable length, as their number depends on to only use part of the information available. In the remaining 3 the number of nodes and edges. To create fixed-length feature vec- submissions, also external data points may be used. tors, we computed arithmetic mean, standard deviation, and the five-number summary of the values in the individual lists, and used 2.1 Subtask 1: Twitter Messages these as features. For the second run in subtask 2, we additionally We extract character and word-based 𝑛-grams from the text of used the data from the nodes files, from which we calculated min, the tweets and use them as features for our classification models. max, mean, and standard deviation of the number of friends and This has been shown to be effective and versatile in different text followers, and added these to the feature vectors calculated for the classification task ranging from stance detection [2] to classifying first run. hacked tweet accounts [4]. We tested different parameters in a grid 2 https://tesseract-ocr.github.io/ search, the values of which are listed in Table 1. 3 https://networkx.org/ Submissions 2 may include additional information, so we added all features that were included in the JSON structure, which corre- Table 1: Hyperparameters tested in grid search. spond to the fields available from Twitter’s API1 . We transformed all textual features to tf/idf normalized frequencies of 𝑛-grams, as listed in Table 1, left the numeric features were left as-is, and Parameter Tested values mapped all categorical features to one-hot vectors. Word & character 𝑛-gram size1 [1,2,3,4] 1 https://developer.twitter.com/en/docs/twitter-api SVM: C [0.1, 1, 10] Extra Trees: number of trees [1, 2, 3, 4] ×103 Copyright 2020 for this paper by its authors. Use permitted under Creative Commons Poly. degree [2, 3] License Attribution 4.0 International (CC BY 4.0). Poly. include bias [True, False] MediaEval’20, 14-15 December 2020, Online KNN: number of neighbors [3, 4, 5, 10, 20, 50] MediaEval’20, 14-15 December 2020, Online M. Moosleitner, B. Murauer, G. Specht Table 2: Evaluation results measured with Matthew’s corre- Figure 1: Top 3 positive and negative SVM coefficients for lation coefficient. each class after fitting the message bodies of the training data. Phase Model Run 1 Run 2 5G corona conspiracy Linear SVM (calibrated) 0.432 0.412 conspiracies 5g Training Linear SVM 0.428 0.404 better wuhan Extra Random Trees 0.274 0.253 burning symptoms Evaluation Linear SVM (calibrated) 0.440 0.441 −8 −6 −4 −2 0 2 4 6 8 (a) Results of Subtask 1 No conspiracy 5g burning Phase Model Run 1 Run 2 wuhan facebook Linear SVM (calibrated) 0.003 0.054 symptoms conspiracies Linear SVM 0.127 0.197 −8 −6 −4 −2 0 2 4 6 8 Training KNN 0.118 0.135 Other conspiracy Extra Random Trees 0.089 0.091 body cancer Gaussian Naive Bayes 0.092 0.101 because but Evaluation Linear SVM 0.090 0.092 already msm (b) Results of Subtask 2 −8 −6 −4 −2 0 2 4 6 8 Table 3: Best parameters for the four submissions. Since we extracted significantly fewer features in the second subtask, we added polynomial feature generation, and added a gaussian naïve Bayes classifier and a K-nearest neighbor to the Subm. Parameters models from the first subtask. Both are well-studied algorithms and Text 1 word-1-grams + character-3+4-grams, calibrated SVM, C=0.1 we were interested in how well they would perform for this task. Text 2 word-1-grams + character-3+4-grams, calibrated SVM, C=0.1 We tested several parameters in a grid search, which are displayed Graph 1 linear SVM, C=10, Poly. deg=2, Poly. include bias = True in Table 1. Graph 2 linear SVM, C=10, Poly. deg=3, Poly. include bias = False 3 RESULTS AND DISCUSSION After preliminary experiments for both subtasks, we selected the discussing the telecommunication standard. This relationship could setup with the highest MCC score in a 10-fold cross-validation be experimented with in more detail using topic modeling. setup as the model that predicts our submission results for each subtask. 3.2 Subtask 2 Similar to subtask 1, we used grid search to find the best performing 3.1 Subtask 1 classifier and parameters. The scores of the classifiers were rather The scores displayed in Table 2a show that the SVM model clearly similar, with the linear SVM producing the best score with the outperforms the extra random trees approach in the first subtask. parameters C=10. While using polynomial features at all increased Thereby, calibrating the SVM increased the performance slightly. the result in both submissions by 0.05, whereas the parameters Interestingly, the performance of the classifiers dropped when (degree=[2,3], include bias=[true, false]) did not have a great influ- taking more features into account for the second submission. This ence (< 0.01 MCC). as shown in Table 3. The results in training and indicates that either too many features are extracted from the text, evaluation approaches for subtask 2 were quite low, as displayed or that the additional meta-information was not expressive to the in Table 2b. Interestingly, our MCC validation scores for subtask problem. Nevertheless, we submitted the two results in this state, 2 were lower than the training scores, which is in contrast to the being aware that we could have possibly increased the performance scores of subtask 1, where the validation scores were slightly better of the second submission by ignoring the meta-features. The evalua- than our training scores. tion results, on the other hand, don’t display a performance decrease between the two submissions, where both runs result in a score of 4 CONCLUSION 0.440 and 0.441, respectively. As shown in Table 3, the best results Our simple text-based approaches were able to classify the tweets were obtained by combining word unigrams and character-3- and reliably, and the coefficients of the model give insights into the -4-grams and a strict regulation parameter of C=0.1. most important terms. We suggest that more preprocessing might Using a linear SVM as a model allows an easy interpretation of further improve these results. the importance of words by looking at the respective coefficients. The simple graph statistics, on the other hand, were not expres- For each output class, Figure 1 shows the terms with the three sive enough for this task. Here, incorporating more metadata like highest and lowest coefficients. The high value for the term 5g the time between the retweets might improve the classification suggests that not many topics within the other conspiracies are results. FakeNews: Corona virus and 5G conspiracy MediaEval’20, 14-15 December 2020, Online REFERENCES [1] David R Bild, Yue Liu, Robert P Dick, Z Morley Mao, and Dan S Wallach. Aggregate characterization of user behavior in twitter and analysis of the retweet graph. ACM Transactions on Internet Technology (TOIT), 15(1):1–24, 2015. [2] Peter Bourgonje, Julian Moreno Schneider, and Georg Rehm. From clickbait to fake news detection: an approach based on detecting the stance of headlines to articles. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, pages 84–89, 2017. [3] Jon M Kleinberg. Hubs, authorities, and communities. ACM computing surveys (CSUR), 31(4es):5–es, 1999. [4] Benjamin Murauer, Eva Zangerle, and Günther Specht. A peer-based approach on analyzing hacked twitter accounts. In Proceedings of the 50th Hawaii International Conference on System Sciences, 2017. [5] N. L. Octaviani, E. Hari Rachmawanto, C. A. Sari, and D. Rosal Ignatius Moses Setiadi. Comparison of multinomial naïve bayes classifier, support vector machine, and recurrent neural network to classify email spams. In 2020 International Seminar on Application for Technology of Information and Communication (iSemantic), pages 17–21, 2020. [6] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999. [7] John Platt. Probabilistic outputs for support vector machines and com- parisons to regularized likelihood methods. Advanced Large Margin Classifiers, 10, June 2000. [8] Konstantin Pogorelov, Daniel Thilo Schroeder, Luk Burchard, Johannes Moe, Stefan Brenner, Petra Filkukova, and Johannes Langguth. Fak- enews: Corona virus and 5g conspiracy task at mediaeval 2020. In MediaEval 2020 Workshop, 2020. [9] Daniel Thilo Schroeder, Konstantin Pogorelov, and Johannes Langguth. Fact: a framework for analysis and capture of twitter graphs. In 2019 Sixth International Conference on Social Networks Analysis, Manage- ment and Security (SNAMS), pages 134–141. IEEE, 2019. [10] Simon Tong and Daphne Koller. Support vector machine active learn- ing with applications to text classification. Journal of machine learning research, 2(Nov):45–66, 2001. [11] Min-Chul Yang, Jung-Tae Lee, Seung-Wook Lee, and Hae-Chang Rim. Finding interesting posts in twitter based on retweet graph analysis. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 1073–1074, 2012.