Overview of the 8th Author Pro ling Task at PAN 2020: Pro ling Fake News Spreaders on Twitter

Abstract

This overview presents the Author Pro ling shared task at PAN 2020. The focus of this year's task is on determining whether or not the author of a Twitter feed is keen to spread fake news. Two have been the main aims: (i) to show the feasibility of automatically identifying potential fake news spreaders in Twitter; and (ii) to show the di culty of identifying them when they do not limit themselves to just retweet domain-speci c news. For this purpose a corpus with Twitter data has been provided, covering the English and Spanish languages. Altogether, the approaches of 66 participants have been evaluated.

Introduction

The rise of social media has given the opportunity to users to publish and share content online in a very fast way. The easiness of publishing content in social media has led to an increase in the amount of misinformation that is published and shared. The propagation of fake news that is shown to be faster than the one of real news [ 64 ] is causing several negative consequences in the society. One of the most recent cases is the large amount of misinformation that was propagated related to the origin, prevention, diagnosis, and treatment of COVID19 pandemic and that a ected the society in di erent ways. For example, fake news about the e ectiveness of the chloroquine led to an increase of cases of chloroquine drug overdose [ 11 ]. The in uence of fake news is also evident in other domains as for example in the political domain, and researchers have drawn attention to their in uence regarding elections and referendums outcomes [ 6 ].

Understanding whether a piece of news is fake or not is a very challenging task for users who, in their majority are not experts. In addition, fake news usually contain a mixture of real and fake claims in an attempt to further confuse users. In an e ort to raise awareness and inform users about which pieces of news contain fake information, several platforms (e.g., Snopes1, PolitiFact2, Leadstories3) have been developed. These platforms employ journalists or other domain experts who thoroughly examine the information presented in various articles before they label them based on their factuality.

Our hypothesis is that users who do not spread fake news may have a set of di erent characteristics compared to users who tend to share fake news. For example, they may use di erent linguistic patterns when they share posts compared to fake news spreaders. This is what we aim at investigating in this year's author pro ling shared task where we address the problem of fake news detection from the author pro ling perspective. The nal goal is pro ling those authors who have shared some fake news in the past. This will allow for identifying possible fake news spreaders on Twitter as a rst step towards preventing fake news from being propagated among social media users. This should help for their early detection and, therefore, for preventing their further dissemination.

The remainder of this paper is organized as follows. Section 2 covers the state of the art, Section 3 describes the corpus and the evaluation measures, and Section 4 presents the approaches submitted by the participants. Sections 5 and 6 discuss results and draw conclusions respectively. 2

Related Work

Fake news detection has recently received signi cant research attention. Among others, researchers have focused on fake news [ 52,59,29 ], bots [ 48,14 ] and clickbaits [ 2,46 ] detection. Some of the previously proposed approaches have explored the e ectiveness of linguistic patterns such as the number of pronouns and punctuation marks on the detection of fake news. For example, Rashkin et al. [ 51 ] analysed various linguistic features such as personal pronouns and swear words that were incorporated into a Long Short Term Memory (LSTM) network to address credibility detection. Other researchers, proposed to use the emotions expressed in the piece of news. In this direction, Giachanou et al. [ 23 ] proposed emoCred, an LSTM-based neural network that utilised emotions from text, whereas Ghanem et al. [ 21 ] proposed to incorporate emotions extracted from text into an LSTM network and showed that emotions are useful for the classi cation of the di erent types of fake news. Guo et al. [ 29 ] proposed a dual emotion-based fake news detection framework to learn content and comment emotion representations for publishers and users respectively, whereas Wang [ 65 ] proposed a hybrid convolutional neural network to combine user metadata with text for fake news detection.

Although the detection of fake news, and credibility in general, has received a lot of research attention [ 23,65,29,59 ], there are only few studies that have addressed the problem from a user or author pro ling perspective. One of the studies that focused on users was presented by Shu et al. [ 60 ] who analyzed di erent features, such as registration time, and found that users that share fake news have more recent accounts than users who share real news. Vo and Lee [ 62 ] 3 https://leadstories.com/ analyzed the linguistic characteristics (e.g., use of tenses, number of pronouns) of fact-checking tweets and proposed a deep learning framework to generate responses with fact-checking intention. Recently, Giachanou et al. [ 22 ] explored the impact of the personality traits of users in discriminating between users who tend to share fake news and fact-checkers. In their study, they proposed a model based on a Convolutional Neural Network (CNN) that combines word embeddings from the text with features that represent users' personality traits and linguistic patterns and showed that those features are useful for the task. Ghanem et al. [ 20 ] proposed a model that utilizes chunked timelines of tweets and a recurrent neural model in order to infer the factuality of a Twitter news account.

With regards to author pro ling, early attempts focused on pro ling the authors of blogs and formal text [ 3,35 ]. However, with the rise of social media researchers proposed methodologies to pro le the authors of social media posts where the language is more informal [ 10,56 ]. Previous author pro ling tasks at PAN have tried to pro le di erent characteristics of users. In 2019 the PAN author pro ling task aimed to classify an author of a tweet feed as a bot or human [ 48 ], whereas in 2018 the task focused on multimodal gender identi cation in Twitter for which images were also provided together with the text for the classi cation [ 50 ]. Since 2013 a wide range of approaches have been developed and tested in the author pro ling tasks. Maharjan et al. [ 38 ] proposed a MapReduce architecture to address the gender identi cation task with 3 million features on the PAN-AP-2013 corpus, whereas Bayot and Goncalves [ 7 ] showed that word embeddings work better than TF-IDF for gender detection on the PAN-AP-2016 corpus. At PAN 2019 the best results in bots detection in English was obtained by Johansson [ 33 ] who used Random Forest with a variety of stylistic features such as term occurrences, tweets length or number of capital and lower letters, user mentions etc., whereas in gender identi cation in English the best result was obtained by Valencia et al. [ 61 ] with Logistic Regression and n-grams. Finally, in Spanish, Pizarro [ 44 ] achieved the best results in both bots and gender identi cation with combinations of n-grams and Support Vector Machines. 3

Evaluation Framework

The purpose of this section is to introduce the technical background. We outline the construction of the corpus, introduce the performance measures and baselines, and describe the idea of so-called software submissions. 3.1

Corpus

To build the PAN-AP-2020 corpus4 we have proceeded as follows. Firstly, we have reviewed fact-checking websites such as PolitiFact or Snopes to nd news 4 We should highlight that we are aware of the legal and ethical issues related to collecting, analysing and pro ling social media data [ 47 ] and that we are committed to legal and ethical compliance in our scienti c research and its outcomes. labelled as fake5. Secondly, we have searched for these news on Twitter. We downloaded all the possible tweets containing some information related to the identi ed fake news (e.g., the topics mentioned on the news) and manually inspected them to discard those not actually referring to the news. We also manually inspected the collected tweets to re-label them as supporting or not the fake news. With this step, we label as real news those tweets where the user warns about the fake news. Thirdly, for the identi ed users, we collected their timelines and checked all the tweets with the list of fake news identi ed in the rst step together with an overall manual inspection. If the user has shared at least one fake news, we labelled it as keen to spread fake news. Otherwise, if the user, to the best of our knowledge, had not shared any fake news in her timeline, we labelled the user as real news spreader. Finally, we have ordered the users by the number of shared fake news and picked up the ones with the highest ranking. We balanced the corpus picking up the same number of real news spreaders. To ensure that the classi ers will not bias towards the identi ed fake news topics, we have removed the tweets containing them from the whole corpus. The performance of the systems has been ranked by accuracy. For each language, we calculated individual accuracy in discriminating between the two classes. Finally, we averaged the accuracy values per language to obtain the nal ranking. 3.3

Baselines

As baselines to compare the performance of the participants with, we have selected: { RANDOM. A baseline that randomly generates the predictions among the di erent classes. 5 We have manually reviewed these "fake news" to ensure that there was not political manipulation behind them and that the news is clearly fake. { LSTM. An Long Short-Term Memory neural network that uses FastText6 embeddings to represent texts. { NN + w nGrams. Word n-grams with values for n from 1 to 3, and a Neural

Network. { SVM + c nGrams. Character n-grams with values for n from 2 to 6, and a

Support Vector Machine. { SYMANTO (LDSE) [ 49 ]. This method represents documents on the basis of the probability distribution of occurrence of their words in the di erent classes. The key concept of LDSE is a weight, representing the probability of a term to belong to one of the di erent categories: fake news spreader / real news spreader. The distribution of weights for a given document should be closer to the weights of its corresponding category. LDSE takes advantage of the whole vocabulary. { EIN. the Emotionally-Infused Neural (EIN) network [ 21 ] with word embedding and emotional features as the input of an LSTM. 3.4

Software Submissions

Similar to previous year, we asked for software submissions. Within software submissions, participants submit executables of their author pro ling softwares instead of just the output of their softwares on a given test set. For the software submissions, the TIRA experimentation platform was employed [ 27,28 ], which renders the handling of software submissions at scale as simple as handling run submissions. Using TIRA, participants deploy their software on virtual machines at our site, which allows us to keep them in a running state [ 26 ]. 4

Overview of the Submitted Approaches

This year, 66 teams participated in the Author Pro ling shared task and 33 of them submitted the notebook paper. We analyse their approaches from three perspectives: preprocessing, features used to represent the authors' texts and classi cation approaches. 4.1

Preprocessing

With the aim at preventing bias towards some URLs, user mentions or hashtags, the corpus was provided with these elements already masked. In the same vein, some participants cleaned other Twitter-speci c elements such as RT, VIA, and FAV7 reserved words [ 24,30,43 ], as well as emojis and other nonalphanumeric characters [ 9,45,63,24,16,39,37,57 ], numbers [ 45,63,24,16,30,57 ] or 6 https://fasttext.cc/docs/en/crawl-vectors.html 7 RT is the acronym for retweet ; VIA is a way to give the authorship to a user (e.g., "via @kicorangel"); and FAV stands for favourite. punctuation signs [ 63,34,24,16,30,37,57 ]. Various participants lower-cased the texts [ 9,45,63,43 ], removed stop words [ 63,34,24,17,30,37,57 ] or treated character ooding [ 63,36 ]. Finally, some users got rid of short texts [ 63 ], stemmed or lemmatised [ 24,30,37,57 ] and tokenised [ 63,36,18,17,37,57,5 ]. Some users also removed infrequent terms [ 32 ]. 4.2

Features

The participants have used a high variety of di erent features and their combinations, albeit we can group them into the following main groups: (i) n-grams; (ii) stylistics; (iii) personality and emotions; and iv) embeddings. As every year, one of the most used features has been the combination of n-grams. For example, Pizarro [ 45 ] and Espinosa et al. [ 16 ] combined character and word n-grams. TF-IDF n-grams have been used by Vogel et al. [ 63 ], Koloski et al. [ 34 ], LopezFernandez et al. [ 18 ] and Vijayasaradhi et al. [ 43 ]. Regarding combinations of stylistic-based features, Manna et al. [ 40 ] combined the average number of emojis (classi ed by category such as a ection, emotion, sceptical, concerned, etc.), the number of URLs, spaces, digits, punctuation marks, tags, quotes, etc., and lexical features such as groups of words expressing personal opinions in addition to personal pronouns or verbs and expressions related to clickbait headlines.

However, most participants combined n-grams with stylistic-, personalityand emotional-based features. For instance, Buda and Bolonyai [ 9 ] combined n-grams with some statistics from the tweets, such as their average length or their lexical diversity. Lichouri et al. [ 37 ] combined TF-IDF word and character n-grams with POS, stemmed and lemmatised tokens. Justin et al. [ 19 ] combined personality, emotions, style-based features with word embeddings. For personality extraction, they used a classi er to obtain the MBTI8 indicator [ 8 ], while for the emotions they used the NRC emotion lexicon [ 41 ]. Finally, they also computed the frequencies of di erent grammatical constructs such as the frequency of auxiliaries, verbs, pronouns, adjectives, punctuation, etc. Niven et al. [ 42 ] obtained the frequencies of adverbs, impersonal and personal pronouns, and all the function words. They also used a constituency tree parser to measure the sentence complexity, taking the average branching factor, the average max noun phrase and verb phrase heights, etc. Finally, they combined all the previous features with a measure of emotional content by means of SentiWordNet. Russo [ 53 ] has combined stylistic features such as type/token ratio, number of mentions, URLs, hashtags, celebrities counting, punctuation signs or replies with emotional features. Hortenhuemer [ 31 ] combined 8 di erent feature sets: TF-IDF, average word length, sentence embedding, POS tagging, recognition of named entities, sentiment analysis (positive/negative), emotional analysis (10 emotions) and readability scores. Espinosa et al. [ 17 ] extracted psychographic features with the Symanto API9. Concretely: (i) personality traits of the author 8 Myers-Briggs Type Indicator 9 https://developers.symanto.net/ of the tweets (emotional vs. rational); (ii) communication styles of the Twitter user (self-revealing, action-seeking, information-seeking, fact-oriented); and (iii) sentiment analysis (positive/negative). They combined the previous features with other linguistic features, Twitter action features and headline analysis data. Cardaioli et al. [ 12 ] used 10 stylometric features aiming to summarise the writing style of the author. In particular, the diversity score, readability score, hashtags average, user mentions average, URLs average, retweets, lower and upper cases, punctuation signs, etc. They combined the previous features with the Big Five personality traits10 obtained with Watson Personality Insights by IBM11.

Some participants also combined the previous types of features with di erent types of embeddings. For example, Spezzano et al. [ 58 ] combined: (i) style, such as the average number of words, characters, lower and upper case words and characters, stop words, punctuation symbols, hashtags, URLs, user mentions, emojis and smiles; (ii) n-grams, obtaining TF-IDF for words and characters; (iii) tweet embeddings, computed using BERT; and (iv) sentiment analysis, by means of the Valence Aware Dictionary and sEntiment Reasoner (VADER) [ 25 ]. Agirrezabal et al. [ 1 ] combined word embeddings, including also the standard deviation of each dimension of the vectors, with bag-of-pos, the average length of the tweets, or the ratio of upper-cased characters. Fahim et al. [ 54 ] used a combination of word embeddings (Glove Twitter 5D) with stylistic features obtained from the hashtags, elongated words, emphasis words, curse words or emoticons. Ogaltsov et al. [ 4 ] combined TF-IDF features with hand-crafted ones such as whether the tweet contained the name of a celebrity, Shashirekha et al. [ 57 ] ensembled TF-IDF n-grams of di erent size with Doc2vec embeddings, and Babaei [ 24 ] combined TF-IDF and word n-grams with ConceptNet word embeddings. Labadie et al. [ 36 ] combined the relationship between the use of grammatical structures like the number of nouns, adjectives, lengths of words and function words, with the encoding of a dense vector at the word- and characterlevel. Hashemi et al.. [ 30 ] combined word embeddings, with TF-IDF vectors and statistical features such as the ratio of retweets, the average number of mentions/URLs/hashtags per tweet, and the average length of the tweet.

Other participants approached the task only with embeddings. Cilet et al. [ 13 ] used a multilingual sentence encoder to feed their pre-trained CNN. Similarly, Majumder [ 39 ] used Google's universal sentence encoder for their LSTM approach. The popular BERT has been used by Kaushik et al. [ 15 ], Baruah et al. [ 5 ], and Chien et al. [ 66 ].

Finally, a couple of participants approached the task from di erent perspectives. Moreno-Sandoval et al. [ 55 ] obtained social network tokens such us hashtags, URLs and user mentions, and then analysed the statistics of central tendency metrics. They combined the previous features with sentiments and emotions. Ikae et al. [ 32 ] estimated the occurrence probability di erence of terms in both classes and generated a couple of clusters with them, reducing the dimensionality with a chi-square method. 10 https://en.wikipedia.org/wiki/Big_Five_personality_traits 11 https://personality-insights-demo.ng.bluemix.net/ Regarding the classi cation approaches, most participants used traditional approaches, mainly Support Vector Machines (SVM) [ 45,63,34,16,18,30,37,1,19 ], Logistic Regression [ 9,63,34,31,43,1,40 ], or a combination of both depending on the language. Random Forest [ 12,17,30,1,55,40 ] is the third most used classi cation algorithm. Ensembles of classi ers have been used by various authors. For example, Decision Tree, Random Forest and XGB [ 32 ]; SVM, Logistic Regression, Random Forest and Extra Tree [ 58 ]; Linear SVM and Logistic Regression [ 57 ]; or SVM, Random Forest and Naive Bayes with XGBoost [ 42 ].

The author of [ 1 ] has used Multilayer Perceptron and the authors in [ 5 ] a Neural Network with Dense layer. However, only a few participants went beyond to experiment with more deep approaches. For example, Fully-Connected Neural Networks [ 24 ], CNN [ 13 ], LSTM [ 39,36 ], or Bi-LSTM with self-attention [ 54 ]. Finally, the authors of [ 4 ] ensembled a GRU-based aggregation model with CNN. 5

Evaluation and Discussion of the Results

In this section, we present the results of the shared task, as well as we analyse the most common errors made by the best performing teams. Although we recommended to participate in both languages (English and Spanish), some participants only participated in English. We present the results for the two languages, and obtain the ranking by averaging them. 5.1

Global Ranking

In Table 3, the overall performance of the participants is presented. The results are shown in terms of accuracy for both languages, and the ranking is its average.

The best results have been obtained in Spanish (82% vs. 75%). The overall best result (77.75%) has been obtained in a tie by Pizarro [ 45 ] and Buda and Bolonyai [ 9 ]. Pizarro obtained the best result in Spanish (82% vs. 80.05%) while Buda and Bolonyai did in English (75% vs. 73.5%). Pizarro approached the task with combinations of character and word n-grams and Support Vector Machines whereas Buda and Bolonyai approached the task with a Logistic Regression ensemble of ve sub-models: n-grams with Logistic Regression, n-grams with SVM, n-grams with Random Forest, n-grams with XGBoost and XGBoost with features based on textual descriptive statistics such as the average length of the tweets or their lexical diversity.

PARTICIPANT

EN

ES

AVG

Participant

En

Es

Avg Spanish) are ranked by the average accuracy between both languages, teams that participated only in English (bottom right) are ranked by the accuracy on English. The best results for each language are printed in bold.

We should highlight the high performance of the n-grams-based approaches on this task. The participants in the following positions also used this kind of features. Koloski et al. used Logistic Regression and Support Vector Machines, depending on the language, with combinations of character and word n-grams, and Vogel et al. [ 63 ] also used TF-IDF and character n-grams to train a Support Vector Machine classi er. De-Borja and Higueras-Porras12 approached the problem with TF-IDF word and character n-grams and Support Vector Machines and Nave Bayes respectively. Babaei et al. [ 24 ] is the top-ranked participant who used some kind of deep learning approach. Concretely, they used a Fully Connected Neural Network combining a word embedding representation based on CoceptNet with TF-IDF word n-grams. Only Pizarro and Buda and Bolonyai outperformed the Symanto (LDSE) [ 49 ] baseline.

As can be seen in Figure 1 and Table 2, the results for Spanish are higher than for English both in terms of average (73.18% vs. 67.33%) and maximum (82% vs. 75%) accuracies. Although the standard deviation is larger for Spanish (6.5% vs. 5.11%), the inter-quartile range is larger for English (5.88% vs. 4%), showing a slightly more sparse distribution in this last language. This might be due to the highest number of outliers in the Spanish distribution, as shown in Figure 2. 12 Although the authors did not submit their working notes, they sent us a brief description of their system.

Fig. 2: Distribution of results in the di erent languages. The gure on the left represents all the systems. The gure on the right removes the outliers. 5.2

Error Analysis

We have aggregated all the participants' predictions for the fake news spreaders vs. real news spreaders discrimination task, except baselines, and plotted the respective confusion matrices for English and Spanish in Figures 3 and 4, respectively.

In the case of English (Figure 3), the highest confusion is from real news spreaders to fake news spreaders (35.50% vs. 30.03%). This means that, for this language in this corpus, the number of false positives is higher than one-third. Regarding Spanish (Figure 4), the highest confusion is from fake news spreaders to real news spreaders (35.09% vs. 20.23%). In this case, the number of false negatives is higher, but the number of false positives is still high (one- fth).

In both languages, there is a large number of false positives which should be taken into account in further research, since a misclassi cation might have some consequences for the pro led user and lead to ethical or legal implications [ 47 ]. 5.3

Best Results

In Table 4 we summarise the best results per language. The best result in English (0.750) has been obtained with a combination of n-grams models and stylistic features in a Logistic Regression ensemble. The best result in Spanish (0.820) has been obtained with combinations of character and word n-grams with Support Vector Machines.

Buda and Bolonyai [ 9 ] (0.750) Pizarro [ 45 ] (0.820)

Conclusion

In this paper, we have presented the results of the 8th International Author Pro ling Shared Task at PAN 2020, hosted at CLEF 2020. The participants had to discriminate from Twitter authors whether they are keen to spread fake news or not. The provided data cover the English and Spanish languages.

The participants used di erent features to address the task, mainly: (i) ngrams; (ii) stylistics; (iii) personality and emotions; and (iv) embeddings. Concerning machine learning algorithms, the most used ones were Support Vector Machines and Logistic Regression, or combinations of both. Nevertheless, few participants approached the task with deep learning techniques. In such cases, they used Fully-Connected Neural Networks, CNN, LSTM and Bi-LSTM with self-attention. According to the results, traditional approaches obtained higher accuracies than deep learning ones. The six teams with the highest performance [ 9,45,34,63 ]13 used combinations of n-grams with traditional machine learning algorithms such as SVM or Logistic Regression. The rst time a deep learning approach appears in the ranking is with the seventh-best performing team [ 24 ]. They used a Fully-Connected Neural Network combining word embeddings based on ConceptNet with TF-IDF word n-grams.

The best results have been obtained in Spanish (0.820) by Pizarro [ 45 ] with combinations of character and word n-grams and Support Vector Machines. The best result in English (0.750) has been obtained by Buda and Bolonyai [ 9 ] with a Logistic Regression ensemble of combinations of n-grams and some textual descriptive statistics. The overall best result (0.775) has been obtained in a tie by them.

The error analysis shows that the highest confusion in English is from Real News spreaders to Fake News Spreaders (false positives) (35.50% vs. 30.03%), whereas in Spanish is the other way around, from Fake News Spreaders to Real News Spreaders (false negatives) (35.09% vs. 20.23%). In this second case, the di erence is much higher (14.86% vs. 5.47%).

Looking at the results and the error analysis, we can conclude that: (i) it is feasible to automatically identify potential Fake News Spreaders in Twitter with high precision, even when only textual features are used; but (ii) we have to bear in mind false positives since especially in English, they sum up to onethird of the total predictions, and misclassi cation might lead to ethical or legal implications [ 47 ]. 7

Acknowledgments

First of all we thank the participants: 66 this year, record in terms of participants at PAN Lab since 2009! We have to thank also Martin Potthast, Matti Wiegmann, and Nikolay Kolyada to help with the 66 Virtual Machines in the TIRA platform. We thank Symanto for sponsoring the ex aequo award for the 13 Together with De Borja and Higueras-Porras, who did not submit their working notes two best performing systems at the author pro ling shared task of this year. The work of Paolo Rosso was partially funded by the Spanish MICINN under the research project MISMIS-FAKEnHATE on Misinformation and Miscommunication in social media: FAKE news and HATE speech (PGC2018-096212-B-C31). The work of Anastasia Giachanou is supported by the SNSF Early Postdoc Mobility grant under the project Early Fake News Detection on Social Media, Switzerland (P2TIP2 181441).

References

1. Agirrezabal , M.: KU-CST at the Pro ling Fake News spreaders Shared Task . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

2. Anand , A. , Chakraborty , T. , Park , N.: We used Neural Networks to Detect Clickbaits: You won't Believe what Happened Next! In: Proceedings of the 2017 European Conference on Information Retrieval . pp. 541 { 547 . ECIR ' 17 ( 2017 )

3. Argamon , S. , Koppel , M. , Fine , J. , Shimoni , A.R. : Gender, genre, and writing style in formal written texts . Text & Talk 23 ( 3 ), 321 { 346 ( 2003 )

4. Bakhteev , O. , Ogaltsov , A. , Ostroukhov , P. : Fake News Spreader Detection using Neural Tweet Aggregation . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

5. Baruah , A. , Das , K. , Barbhuiya , F. , Dey , K. : Automatic Detection of Fake News Spreaders Using BERT . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

6. Bastos , M.T. , Mercea , D. : The Brexit Botnet and User-Generated Hyperpartisan News . Social Science Computer Review 37 ( 1 ), 38 { 54 ( 2019 )

7. Bayot , R. , Goncalves , T. : Multilingual author pro ling using word embedding averages and svms . In: 2016 10th International Conference on Software, Knowledge, Information Management & Applications (SKIMA) . pp. 382 { 386 . IEEE ( 2016 )

8. Briggs-Myers , I. , Myers , P.B.: Gifts di ering: Understanding personality type ( 1995 )

9. Buda , J. , Bolonyai , F. : An Ensemble Model Using N-grams and Statistical Features to Identify Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEURWS.org (Sep 2020 )

10. Burger , J.D. , Henderson , J. , Kim , G. , Zarrella , G.: Discriminating gender on twitter . In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing . pp. 1301 { 1309 ( 2011 )

11. Busari , S. , Adebayo , B. : Nigeria Records Chloroquine Poisoning after Trump Endorses it for Coronavirus Treatment . CNN ( 2020 )

12. Cardaioli , M. , Cecconello , S. , Conti , M. , Pajola , L. , Turrin , F. : Fake News Spreaders Pro ling Through Behavioural Analysis . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEURWS.org (Sep 2020 )

13. Chilet , L. , Mart , P. : Pro ling Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

14. Chu , Z. , Gianvecchio , S. , Wang , H. , Jajodia , S. : Detecting automation of twitter accounts: Are you a human, bot, or cyborg ? IEEE Transactions on Dependable and Secure Computing 9 ( 6 ), 811 { 824 ( 2012 )

15. Das , K.A. , Baruah , A. , Barbhuiya , F.A. , Dey , K. : Ensemble of ELECTRA for Pro ling Fake News Spreaders . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

16. Espinosa , D. , Gomez-Adorno , H. , Sidorov , G.: Pro ling Fake News Spreaders using Character and Words N-grams . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

17. Espinosa , M.S. , Centeno , R. , Rodrigo, : Analyzing User Pro les for Detection of Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

18. Fernandez , J.L. , Ram rez , J.A.L. : Approaches to the Pro ling Fake News Spreaders on Twitter Task in English and Spanish . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEURWS.org (Sep 2020 )

19. Fersini , E. , Armanini , J., D'Intorni , M. : Pro ling Fake News Spreaders: Stylometry, Personality, Emotions and Embeddings . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEURWS.org (Sep 2020 )

20. Ghanem , B. , Ponzetto , S.P. , Rosso , P. : FacTweet: Pro ling Fake News Twitter Accounts . In: Statistical Language and Speech Processing (SLSP). Lecture Notes in Computer Science . Springer, Cham ( 2020 )

21. Ghanem , B. , Rosso , P. , Rangel , F. : An Emotional Analysis of False Information in Social Media and News Articles . ACM Transactions on Internet Technology (TOIT) 20(2) , 1 { 18 ( 2020 )

22. Giachanou , A. , R ssola, E.A. , Ghanem , B. , Crestani , F. , Rosso , P. : The Role of Personality and Linguistic Patterns in Discriminating Between Fake News Spreaders and Fact Checkers . In: International Conference on Applications of Natural Language to Information Systems . pp. 181 { 192 . Springer ( 2020 )

23. Giachanou , A. , Rosso , P. , Crestani , F. : Leveraging Emotional Signals for Credibility Detection . In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval . pp. 877 { 880 ( 2019 )

24. Giglou , H.B. , Razmara , J. , Rahgouy , M. , Sanaei , M.: LSACoNet: A Combination of Lexical and Conceptual Features for Analysis of Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

25. Gilbert , C. , Hutto , E.: Vader: A parsimonious rule-based model for sentiment analysis of social media text . In: Eighth International Conference on Weblogs and Social Media (ICWSM-14) . Available at ( 20 /04/16) http://comp. social. gatech. edu/papers/icwsm14. vader. hutto. pdf. vol. 81 , p. 82 ( 2014 )

26. Gollub , T. , Potthast , M. , Beyer , A. , Busse , M. , Rangel , F. , Rosso , P. , Stamatatos , E. , Stein , B. : Recent trends in digital text forensics and its evaluation . In: International Conference of the Cross-Language Evaluation Forum for European Languages . pp. 282 { 302 . Springer ( 2013 )

27. Gollub , T. , Stein , B. , Burrows , S. : Ousting ivory tower research: Towards a web framework for providing experiments as a service . In: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval . pp. 1125 { 1126 ( 2012 )

28. Gollub , T. , Stein , B. , Burrows , S. , Hoppe , D. : Tira: Con guring, executing, and disseminating information retrieval experiments . In: 2012 23rd International Workshop on Database and Expert Systems Applications . pp. 151 { 155 . IEEE ( 2012 )

29. Guo , C. , Cao , J. , Zhang , X. , Shu , K. , Liu , H. : Dean: Learning dual emotion for fake news detection on social media . arXiv preprint arXiv: 1903 . 01728 ( 2019 )

30. Hashemi , A. , Zarei , M.R. , Moosavi , M.R. , Taheri , M. : Fake News Spreader Identi cation in Twitter using Ensemble Modeling Notebook for PAN at CLEF 2020 . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

31. Hortennhuemer, C. , Zangerle , E.: A Multi-Aspect Classi cation Ensemble Approach for Pro ling Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

32. Ikade , C. , Savoy , J.: UniNE at PAN-CLEF 2020 : Pro ling Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

33. Johansson , F. : Supervised Classi cation of Twitter Accounts Based on Textual Content of Tweets . In: Cappellato, L. , Ferro , N. , Losada , D. , Muller, H. (eds.) CLEF 2019 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2019 ), http://ceur-ws. org/ Vol- 2380 /

34. Koloski , B. , Pollak , S. , Skrlj , B. : Multilingual Detection of Fake News Spreaders via Sparse Matrix Factorization . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

35. Koppel , M. , Argamon , S. , Shimoni , A.R. : Automatically categorizing written texts by author gender . Literary and linguistic computing 17(4) , 401 { 412 ( 2002 )

36. Labadie , R. , Castro , D.C. , Bueno , R.O. : Fusing Stylistic Features with Deeplearning methods for Pro ling Fake News Spreader . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

37. Lichouri , M. , Abbas , M. , Benaziz , B. : Pro ling Fake News Spreaders on Twitter based on TFIDF Features and Morphological Process Notebook for PAN at CLEF 2020 . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

38. Maharjan , S. , Shrestha , P. , Solorio , T. , Hasan , R.: A straightforward author proling approach in mapreduce . In: Ibero-American Conference on Arti cial Intelligence . pp. 95 { 107 . Springer ( 2014 )

39. Majumder , S.B. , Das , D. : Detecting Fake News Spreaders on Twitter Using Universal Sentence Encoder . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

40. Manna , R. , Pascucci , A. , Monti , J.: Pro ling Fake News Spreaders through Stylometry and Lexical Features. UniOR NLP @PAN2020 . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

41. Mohammad , S.M. , Turney , P.D.: Nrc emotion lexicon . National Research Council, Canada 2 ( 2013 )

42. Niven , T. , Kao , H.Y. , Wang , H.Y.: Pro ling Spreaders of Disinformation on Twitter: IKMLab and Softbank Submission . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEURWS.org (Sep 2020 )

43. Pinnaparaju , N. , Indurthi , V. , Varma , V. : Identifying Fake News Spreaders in Social Media . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

44. Pizarro , J.: Using N-grams to Detect Bots on Twitter . In: Cappellato, L. , Ferro , N. , Losada , D. , Muller, H. (eds.) CLEF 2019 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2019 ), http://ceur-ws. org/ Vol- 2380 /

45. Pizarro , J.: Using N-grams to detect Fake News Spreaders on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

46. Potthast , M. , K opsel, S. , Stein , B. , Hagen , M. : Clickbait detection . In: European Conference on Information Retrieval . pp. 810 { 817 . Springer ( 2016 )

47. Rangel , F. , Rosso , P. : On the implications of the general data protection regulation on the organisation of evaluation tasks . Language and Law= Linguagem e Direito 5 ( 2 ), 95 { 117 ( 2019 )

48. Rangel , F. , Rosso , P. : Overview of the 7th Author Pro ling Task at PAN 2019: Bots and Gender Pro ling in Twitter . In: CLEF 2019 Labs and Workshops , Notebook Papers ( 2019 )

49. Rangel , F. , Rosso , P. , Franco-Salvador , M. : A Low Dimensionality Representation for Language Variety Identi cation . In: In 17th International Conference on Intelligent Text Processing and Computational Linguistics , CICLing'16 . Springer-Verlag, LNCS(9624) . pp. 156 { 169 ( 2018 )

50. Rangel , F. , Rosso , P. , Montes-y Gomez , M. , Potthast , M. , Stein , B. : Overview of the 6th author pro ling task at pan 2018: Multimodal gender identi cation in twitter . In: Working Notes Papers of the CLEF ( 2018 )

51. Rashkin , H. , Choi , E. , Jang , J.Y. , Volkova , S. , Choi , Y. : Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking . In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . pp. 2931 { 2937 . EMNLP ' 17 ( 2017 )

52. Ruchansky , N. , Seo , S. , Liu, Y. : CSI: A Hybrid Deep Model for Fake News Detection . In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management . pp. 797 { 806 . CIKM ' 17 ( 2017 )

53. Russo , I. : Sadness and Fear: Classi cation of Fake News Spreaders Content on Twitter . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

54. Saeed , U. , Fahim , H. , Shirazi , F. : Pro ling Fake News Spreaders on Twitter using Deep Learning LSTM and BI-LSTM Approach . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

55. Sandoval , L.G.M. , Puertas , E. , Quimbaya , A.P. , Valencia , J.A.A. : Assembly of Polarity, Emotion and User Statistics for Detection of Fake Pro les . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

56. Schwartz , H.A. , Eichstaedt , J.C. , Kern , M.L. , Dziurzynski , L. , Ramones , S.M. , Agrawal , M. , Shah , A. , Kosinski , M. , Stillwell , D. , Seligman , M.E. , et al.: Personality, gender, and age in the language of social media: The open-vocabulary approach . PloS one 8 ( 9 ), e73791 ( 2013 )

57. Shashirekha , H.L. , Balouchzahi , F. : ULMFiT for Twitter Fake News Pro ling . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

58. Shrestha , A. , Spezzano , F. , Joy , A. : Detecting Fake News Spreaders in Social Networks via Linguistic and Personality Features . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

59. Shu , K. , Liu , H.: Detecting fake news on social media . Synthesis Lectures on Data Mining and Knowledge Discovery 11 ( 3 ), 1 { 129 ( 2019 )

60. Shu , K. , Wang , S. , Liu, H.: Understanding User Pro les on Social Media for Fake News Detection . In: 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) . pp. 430 { 435 . IEEE ( 2018 )

61. Valencia , A.V. , Gomez , H.A. , Rhodes , C.S. , Fuentes , G.P. : Bots and Gender Identi cation Based on Stylometry of Tweet Minimal Structure and n-Grams Model . In: Cappellato, L. , Ferro , N. , Losada , D. , Muller, H. (eds.) CLEF 2019 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2019 ), http://ceur-ws. org/ Vol- 2380 /

62. Vo , N. , Lee , K. : Learning from Fact-checkers: Analysis and Generation of Factchecking Language . In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval . pp. 335 { 344 ( 2019 )

63. Vogel , I. , Meghana , M. : Fake News Spreader Detection on Twitter using Character N-Grams . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )

64. Vosoughi , S. , Roy , D. , Aral , S.: The Spread of True and False News Online . Science 359 ( 6380 ), 1146 { 1151 ( 2018 )

65. Wang , W.Y.: Liar, Liar Pants on Fire: A New Benchmark Dataset for Fake News Detection . In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . pp. 422 { 426 . ACL ' 17 ( 2017 )

66. Wu , S.H. , Chien , S.L. : A BERT based Two-stage Fake News Spreader Pro ling System . In: Cappellato, L. , Eickho , C. , Ferro , N. , Neveol , A . (eds.) CLEF 2020 Labs and Workshops, Notebook Papers . CEUR-WS.org (Sep 2020 )