This article describes the participation in the Bots and Gender Pro ling shared task in PAN at CLEF 2019. We propose a Support Vector Machine Classi er with character and word n-grams features. Our model achieved the best average performance of 88:05% at the 7th International Competition on Author Pro ling. In the task of determining whether the author of a set of tweets is a bot or a human, our model obtained an accuracy of 93:60% for English and 93:33% for Spanish. In the task of Gender Identi cation, obtained an accuracy of 83:56% for English and 81:72% for Spanish.
Nowadays we communicate and interact through social media platforms on a daily basis, we use it as a source of information, a commercial or marketing channel, to buy products online even to speak with our bank representatives. Thus, social networks have a great in uence on our lifestyle and a ect the way decisions are made. It is known that social media bots (software controlled accounts) pose as humans to intervene in elections and decisions that a ect many people [17]. They are also used to in uence the perception of products through fake reviews.
The objective of this year's Author pro ling tasks [15] in PAN [5] at CLEF 2019 is to determine if the author of a set of tweets in English or Spanish, is a human or a bot, and in case of being human, determining its gender.
The paper is organised as follows. Section 2 presents the related work, Section 3 describes the corpus, environment setup, preprocessing, features and models, and Section 4 shows the trained models with its accuracy on the dev set. Then in Section 5 we discuss about the obtained results in the test set. After that, Section 6 presents an overview of some experiments using deep learning. Finally, Section 7 draws some conclusions.
In 2002, Koppel et al. [8] evaluate the use of function words and part of speech tagging to identify the author gender and document genre of a corpus consisting of 920 labelled documents. Pang et al. [12] explore the use of unigrams and bigrams, with Naive Bayes, maximum entropy classi cation, and support vector machines to classify the sentiment on movie-data. In 2004, Pang et al. [11] employ support vector machine and Naive Bayes to classify movie reviews as either positive or negative. In 2006, Schler et al. [16] were able to obtain an accuracy of 80:0% in gender identi cation in a corpus of 85:000 blogs using style and content words. Metaxas et al. [10] [9] found social bots supporting some candidates and attacking their opponents [19]. In 2014, Dickerson et al. [7] studied the problem of identifying bots on all of Twitter and identi ed 19 of the 25 top features they use are sentiment-related. They use grid search to nd the best hyperparameters for each of the classi ers. In 2016, Bessi et al. [2] found social bots generating a large amount of content, possibly distorting online conversations, they noted that bots tweeting about Donald Trump generated the most positive tweets [19]. In 2018, Stella et al. [17] report a case of political manipulation on social media using sentiment analysis.
In the task of gender identi cation in PAN at CLEF 2017, Basile et al. [1] obtained 82:33% for English and 83:21% in Spanish using an SVM classi er trained with combinations of character and tf-idf n-grams. In the same task of gender identi cation in PAN at CLEF 2018, Daneshvar et al. [6] obtained 82:21% for English and 82:00% for Spanish using char and word n-grams as features, with a SVM classi er. Tellez et al. [18] obtained 81:21% for English and 80:05% for Spanish using a similar strategy. 3
This section presents the methods and materials applied in the experiments. Subsection 3.1 describes the corpus, Subsection 3.2 shows the environment setup, Subsection 3.3 explain the preprocessing, Subsection 3.4 provides a description of the feature representations. Finally, the models are presented in Subsection 3.5 and in Subsection 3.6 all the hyperparameter are shown. 3.1
The corpus consists of a set of les in the XML format, containing of 100 tweets, one le per author. It is balanced and annotated if the author is human or robot, and in case on human its gender, male or female. It is recommended to use the corpus partitions shown in Table 1 to avoid over- tting (more than one le could be written by the same author). Also, it shows that there are more tweets for English than for Spanish. The models were trained on a Jupyter notebook environment known as Colaboratory1. We opted to use mainly the next software tools to build our models: nltk2, sklearn3, hyperopt4. 3.3
The XML les are parsed using the Python 3 library xml.etree.ElementTree5 to be able to work with its content. Then, for each author, their 100 tweets are concatenated forming a long string, and a custom tag is used to separate each of the tweets. After that, we applied a lowercase conversion and the strings are tokenized using nltk TweetTokenizer [3], each URL, user mention and hashtag are replaced by one xed tag respectively, following what was done by Daneshvar et al. [6]6. 3.4
Based on a quick experimentation, we choose to evaluate char and word ngrams with di erent n-gram orders. Also we opted to represent each document using term frequency{inverse document frequency (TF-IDF). Finally, to join both TF-IDF feature representations, the char and the word n-grams, we employ FeatureUnion7, in order to use Pipelines8 obtaining an end to end model.
Taking into account our hardware resource limitation and the reason that we want to try several hyperparameters by each model, we opted to include only a few classical machine learning algorithms. 1 https://colab.research.google.com 2 https://www.nltk.org/ 3 https://scikit-learn.org/ 4 http://hyperopt.github.io/hyperopt/ 5 https://docs.python.org/3/library/xml.etree.elementtree.html 6 https://github.com/pan-webis-de/daneshvar18/blob/5542895062f2404fd5b5a07493 098132308457/ pan18ap/train model.py 7 https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.FeatureUnion.
html 8 https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html A model was trained for each task and language separately, 4 models in total. The best 5 con gurations by language and task are shown in Table 6. The models were trained with the training set and evaluated with the dev set as shown in Table 1.
The best results for the task of determining whether the author is a bot or
a human in English were obtained using SVM with char n-grams with range (
LogisticRegression.html 11 https://scikit-learn.org/stable/modules/generated/sklearn.naive bayes.
MultinomialNB.html 12 http://hyperopt.github.io/hyperopt/
In the Spanish task the same con guration allowed for obtaining the best
result on both tasks, a SVM classi er with char n-grams with range (
In order to evaluate our model on the test set, a TIRA [13] account was given, to install the software dependencies and to deploy our model in testing mode.
The task organisers o er us to evaluate our models in an early birds dataset, allowed us to verify the con guration of the environment and give us an early approximation of our model behaviour. Table 7 shows the results in the early birds dataset (dataset1) and Table 8 shows the results the nal datatest (dataset2).
It is possible to see that in the gender identi cation task the results for the dataset2 are a couple of points better than for the dataset1, but in the task of determining the author, the results are similar for both languages and both datasets.
Finally, the results obtained with our proposed model, are better than the baselines de ned by the task organisers. The baseline models as shown in Table 8 are majority, random and LDSE [14].
Lang Task Classi er Loss Feats word.ngram range char.ngram range
en human or bot LinearSVC -0.945968 word char (
After the submission of our run, we carried out a couple of further experiments that we could not evaluated on the test dataset and that we present in this section.
We conduct several experiments of di erent deep learning architectures with Keras [4] and the data partitions shown in Table 1 (train and dev). The same preprocessing process shown in Subsection 3.3 was done. In addition to that, each emoji was replaced with a word using emoji13.
The model shown in Figure 3 obtained an accuracy of 94:524 0:00167 evaluated in 10 runs. We also opted to evaluate 100 experiments with di erent hyperparameters for the same architecture. Figure 1 shows the accuracy on the training set and Figure 2 shows the accuracy on the dev set. It is possible to see that the results were good, independently of the hyperparameters used. 13 https://github.com/carpedm20/emoji/
Similarly as in previous years of the Author Pro ling shared task in PAN, the
SVM classi er with n-grams and TF-IDF features obtained very good results.
The use of hyperparameter tuning tools showed to be one of the crucial parts of
the model building process to obtain good results. As future work, it could very
useful to explore the use of more features such as the use on lexicons, transform
the emojis to custom tags, and also to try other feature representations such as
word embeddings with neural networks. In Section 6 we showed the promising
results of the preliminary experiments that we carried out.
12. Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up?: Sentiment classi cation using
machine learning techniques. In: Proceedings of the ACL-02 Conference on
Empirical Methods in Natural Language Processing - Volume 10. pp. 79{86.
EMNLP '02, Association for Computational Linguistics, Stroudsburg, PA, USA
(2002). https://doi.org/10.3115/1118693.1118704,
https://doi.org/10.3115/1118693.1118704
13. Potthast, M., Gollub, T., Wiegmann, M., Stein, B.: TIRA Integrated Research
Architecture. In: Ferro, N., Peters, C. (eds.) Information Retrieval Evaluation in
a Changing World - Lessons Learned from 20 Years of CLEF. Springer (2019)
14. Rangel, F., Franco-Salvador, M., Rosso, P.: A low dimensionality representation
for language variety identi cation. In: Gelbukh, A. (ed.) Computational
Linguistics and Intelligent Text Processing. pp. 156{169. Springer International
Publishing, Cham (2018)
15. Rangel, F., Rosso, P.: Overview of the 7th Author Pro ling Task at PAN 2019:
Bots and Gender Pro ling. In: Cappellato, L., Ferro, N., Losada, D., Muller, H.
(eds.) CLEF 2019 Labs and Workshops, Notebook Papers. CEUR-WS.org (Sep
2019)
16. Schler, J., Koppel, M., Argamon, S., Pennebaker, J.W.: E ects of age and gender
on blogging. In: AAAI spring symposium: Computational approaches to
analyzing weblogs. vol. 6, pp. 199{205 (2006)
17. Stella, M., Ferrara, E., De Domenico, M.: Bots increase exposure to negative and
in ammatory content in online social systems. Proceedings of the National
Academy of Sciences 115(49), 12435{12440 (2018).
https://doi.org/10.1073/pnas.1803470115,
https://www.pnas.org/content/115/49/12435
18. Tellez, E.S., Miranda-Jimenez, S., Moctezuma, D., Gra , M., Salgado, V.,
Ortiz-Bejar, J.: Gender identi cation through multi-modal tweet analysis using
microtc and bag of visual words. In: Proceedings of the Ninth International
Conference of the CLEF Association (CLEF 2018) (2018)
19. Yang, K.C., Varol, O., Davis, C.A., Ferrara, E., Flammini, A., Menczer, F.:
Arming the public with arti cial intelligence to counter social bots. Human
Behavior and Emerging Technologies 1(