This paper synthesizes our participation in the CLEF conference 2020 regarding the Pro ling Fake News Spreaders on Twitter task, organized at the PAN lab on digital text forensics and stylometry. The models that we suggested obtained one of the two best results, based on an average accuracy of 0.7775 {0.7350 for English and 0.8200 for Spanish. In summary, we propose a Support Vector Machine (SVM) classi er with character and word n-gram features to determine whether the author of a Twitter feed is keen to be a spreader of fake news.
In the past few years, social media has been changing how people communicate and interact. Currently, we use these platforms daily for a variety of purposes {searching for information, buying products, reaching out to bank representatives, or as a marketing and commercialization channel. For example, in the rst quarter of 2019, Twitter reported an average of 330 million monthly active users [22].
As well as social media platforms have reached popularity, they have become tools that directly in uence the perception of events, people, or products. Certain people and organizations have been reaching this goal through the spread of fake news, rumors, and misinformation.
This paper presents our participation in the Author Pro ling task at PAN.
That work aims to identify users in two categories: faker user (fake news spreader)
and legitimate user (real news spreader) [
Author Pro ling distinguishes between classes of authors by studying shared
language among people. Therefore, through this perspective, it is possible to
address fake news by studying stylistic deviations of users that tend to spread
them [
As follows, a brief timeline regarding Author Pro ling research achievements.
In 2002, the use of function words and part of speech tagging served to identify
the author's gender on a corpus consisting of 920 labeled documents [
In 2006, an accuracy of 0.80 was obtained in the task of gender identi
cation in a corpus of 85:000 blogs using style and content words [19]. During the
2010 U.S. midterm elections, primitive social bots played the part of
supporters of some candidates and attacked their opponents [
More recently, in 2018, it was reported a case of political manipulation on
social media that used sentiment analysis [20]. Finally, in 2019, the use of a variety
of di erent semantic and stylistic features, and a neural recurrent model helped
to detect fake news on Twitter's accounts [
Related to Author Pro ling task at PAN in CLEF 2017, the gender
identication task obtained an accuracy of 0.8233 for English and 0.8321 in Spanish
[
The following section presents our method. Name Description pres-case whether to maintain letter case or
downcase for everything except for emoticons red-len whether to replace repeated character sequences rpl-dgt whether to replace numbers by xxdgt demojify whether to replace emojis by
word representations rpl-anon whether to replace anonymized tags #URL# by xxurl #USER# by xxusr #HASHTAG# by xxhst The corpus considers the text of tweets whose authors qualify as fake news spreaders or legitimate users. At rst, each author's tweet groups together on a long chain. Subsequently, a preprocessing strategy applies to them.
A preprocessing strategy consists of a stack of text transformations that creates a complete list of preprocessing options. For example, emoji1 replaces emojis, xxdgt instead of numbers, and lower case for speci c cases. Table 1 shows a complete list of the preprocessing options. N-grams of characters and words are generated with di erent n-gram orders. Each document, composed by each set of tweets per author, portraits the use of Term Frequency { Inverse Document Frequency (TF-IDF). Afterward, the n-grams vectors group together to obtain one feature vector per author. Linear Support Vector Machine operates as a classi er due to the positive results obtained in the previous years. 4
This section presents the dataset and the experimental setup. To accomplish our
method goals, we choose to use nltk[
A TIRA [16] account is given in order to evaluate our models and get the results in a hidden test set. Additionally, task organizers permit to evaluate models in an early bird software submission phase that enables the con guration environment subjected to veri ed and provides an early approximation of the model's accuracy.
During the training phase, the 5-fold cross-validation strategy supported the choice of the best model. Nevertheless, as shown in Table 6, the results of our best models in the early bird software submission phase di er from the training. For that reason, the experiments evaluated during the training changed to the 10-fold cross-validation method.
For English, our model uses the v2 1 1 preprocessing strategy {for both char and word n-grams. We apply a variety of replacement strategies, such as exchanging digits for custom tags, emojis for words, anonymous tags for custom tags without the character #, and downcasting letters. As well, we use char ngrams of orders between 1 and 6, and unigrams and bigrams for word n-grams.
For Spanish, our model uses the v3 1 preprocessing strategy {for both char and word n-grams. However, the di erence regarding the English's model is that custom tags do not replace digits in Spanish. Furthermore, we use chars n-grams of orders between 2 and 6, and unigrams, bigrams, and trigrams for words ngrams.
The models that we suggested obtained one of the two best results on the Pro ling Fake News Spreaders on Twitter task at the 8th Author Pro ling Task at PAN 2020 [18], based on an average accuracy of 0.7775 {0.7350 for English and 0.8200 for Spanish. 6
In this paper, we described the submitted models for the Pro ling Fake News Spreaders on Twitter task at PAN 2020. These consist of SVM as a classi er, and TF-IDF of char and word n-grams as features.
Similar to previous editions of the Author Pro ling task in PAN's conferences, 2020 research shows that SVM classi ers with n-grams and TF-IDF features performed positively as our proposed models achieved one of the two best average accuracies.
Therefore, despite the similarity of the methods applied to the Author Proling task in previous years, the best results for 2020 are for Spanish tweets.
The accuracy obtained by our models in the early bird dataset di ers from the accuracy obtained during training. Consequently, to get a more accurate estimation of the model's performance, we shifted from 5-fold cross-validation to 10-fold cross-validation for the training.
Finally, the use of hyperparameter tuning tools was a crucial step to obtain positive results during the model construction process. 14. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825{2830 (2011) 15. Pizarro, J.: Using N-grams to detect Bots on Twitter. In: Cappellato, L., Ferro, N., Losada, D., Muller, H. (eds.) CLEF 2019 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings, CEUR-WS.org (Sep 2019), http://ceur-ws.org/Vol-2380/ 16. Potthast, M., Gollub, T., Wiegmann, M., Stein, B.: TIRA Integrated Research Architecture. In: Ferro, N., Peters, C. (eds.) Information Retrieval Evaluation in a Changing World. Springer (Sep 2019) 17. Rangel, F., Giachanou, A., Ghanem, B., Rosso, P.: Overview of the 8th Author Pro ling Task at PAN 2020: Pro ling Fake News Spreaders on Twitter. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings (Sep 2020), CEUR-WS.org 18. Rangel, F., Giachanou, A., Ghanem, B., Rosso, P.: Overview of the 8th Author Pro ling Task at PAN 2020: Pro ling Fake News Spreaders on Twitter. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Labs and Workshops, Notebook Papers. CEUR-WS.org (Sep 2020) 19. Schler, J., Koppel, M., Argamon, S., Pennebaker, J.W.: E ects of age and gender on blogging. In: AAAI spring symposium: Computational approaches to analyzing weblogs. vol. 6, pp. 199{205 (2006) 20. Stella, M., Ferrara, E., De Domenico, M.: Bots increase exposure to negative and in ammatory content in online social systems. Proceedings of the National Academy of Sciences 115(49), 12435{12440 (2018). https://doi.org/10.1073/pnas.1803470115, https://www.pnas.org/content/115/49/12435 21. Tellez, E.S., Miranda-Jimenez, S., Moctezuma, D., Gra , M., Salgado, V., Ortiz-Bejar, J.: Gender identi cation through multi-modal tweet analysis using microtc and bag of visual words. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018) (2018) 22. Twitter, Inc.: Q1 2019 Selected Company Metrics and Financials (2019), https://s22.q4cdn.com/826641620/ les/doc nancials/2019/q1/
Q1-2019-Selected-Company-Metrics-and-Financials.pdf 23. Yang, K.C., Varol, O., Davis, C.A., Ferrara, E., Flammini, A., Menczer, F.: Arming the public with arti cial intelligence to counter social bots. Human Behavior and Emerging Technologies 1(1), 48{61 (2019). https://doi.org/10.1002/hbe2.115, https://onlinelibrary.wiley.com/doi/abs/10.1002/hbe2.115