In this paper, we describe the participation of the Advanced Data Analysis and Management (ADAM) group of the University of Feira de Santana in the Bots and Gender Profiling Task organized by PAN@CLEF 2019. We used Support Vector Machines (SVM) optimized through nested cross-validation. In bot detection, we used features related to behavior of the account, sentiment and variety of posts, in gender detection function words and emoticons. These features were evaluated both individually and in groups. Before starting the training phase, we preprocessed the data to better adjust it. For bot detection, our method reached approximately 0:9057 for English and 0:8767 for Spanish. For gender detection, 0:7696 for English and 0:7150 for Spanish. Although the results for Spanish are poorer than the ones for English, they are above the random baseline (50%).
Social media companies employ mobile and web-based technologies to create highly
interactive platforms through which individuals and communities share, cocreate,
discuss, and modify user-generated content [
Twitter is used in multiple contexts. It is mainly considered as an information
dissemination tool, but also as a source of data that may support studies in different areas
of knowledge. This feature is specially interesting considering it offers an Application
Programming Interface (API)2 that allows crawling and collecting data. In this context,
a subject that has attracted the attention of researchers is the so-called Author profiling,
in which information like age, cultural background, gender, native language, and
personality can be inferred through textual analysis of users’ posts. This type of analysis
enables numerous applications, such as business intelligence, digital forensics,
psychological profiling, brand reputation monitoring, etc. With regard to forensic applications,
bot detection has gained attention, especially due to bots’ self-controlled ability to
disseminate political, extremist or misinformation material that may negatively influence
a massive amount of users [
In this context, this paper describes the participation of the ADAM team in the
Bots and Gender Profiling Task [
We define a set of features with discrimative power for bot and gender detection; We evaluate the power of optimizing a model through cross-validation; We analyze the effectiveness of each group of features in the task.
The remainder of this paper is organized as follows. Section 2 presents the related work. Section 3 presents the experimental validation setup. The results are discussed in Section 4. Finally, Section 5 presents our conclusions and directions for future work. 2
The use of bots has brought problems in many collaborative systems, e.g., Wikipedia
and OpenStreetMaps. This scenario motivates the study of strategies to identify bots in
such collaborative systems, examining the contributions done by users [
For textual classification, information about function words, n-grams (at word and
character level), quantitative features, orthographic features, part of speech (POS) tags,
and vocabulary richness features are usually used [
in sentiment analysis, the weight of the dictionary used may vary according to location of the author.
In the gender identification task, previous works show that women use question tags
more frequently, more emoticons, and less profanities [
The work in [
In [
Similar research is related to gender identification in e-mail messages [
The work in [
In this section, we present the dataset (Section 3.1), the preprossessing phase (Section 3.2), the feature extraction method (Section 3.3) and the classification model (Section 3.4). 3.1
In order to better adjust the posts to the experiments, we executed three independent operations:
3.3
Multiple classes of features were used for bot and gender detection. The following sections describe each one individually, according to the associated task.
Many of the features used in previous work to detect bots in twitter, as usernames, geodata, tweet intervals and following data, is unavailable in this case. Therefore, the features are limited to the twitter text.
Twitter features: these features are related to the twitter profile of the users. Frequency of hashtags (#), frequency of mentions of users (@), frequency of retweets (occurrence of the string “rt”) and frequency of links (occurrence of “http”). The rationale behind the first two is that bots tend to try to increase their reach inserting trending hashtags in their posts or mentioning multiple users to call their attention. Bots use to retweet content as a way to easily build a profile, and constant posting of links is typical behavior of spam bots.
Sentiment features: in the English corpus, the VADER library3 was used to
identify the sentiments of the corpus. The VADER generates the mean of the four
sentiment metrics: positive, negative, neutral and compound. An additional feature was
also computed, the sentiment flip, defined in [
the texts to English automatically and use the library on the resulting text. We ended
using a machine learning based solution4, which generated only one output in the
range [
Variety features: these features measure how varied is the content generated by the user, under the assumption that bots tend to repeat content in their posts. Two features belong to this group: the ratio between number of words used and total number of words in all posts; and the cleanliness: the ratio between the number of characters after and before preprocessing.
Emoticons: frequency of each term in a list of emoticons, meant to describe a range of emotions.
Function words: frequency of function words: pronouns, determiners, modals and conjunctions in English and in Spanish, conjunctions and determiners.
Sentiment features: the same sentiment features used for bot detection. 3.4
For conducting the experiments we used a SVM classifier. In order to optimize the
model learned, a 5-fold nested cross-validation [
This process is done only in the train dataset. After the model is chosen, it is used to classify the dev dataset, giving a notion of how the model will perform in unseen data, i.e., this set validates the model. To make this last step statistically significant, during the learning process nothing of the dev dataset was used.
In the cross-validation, both linear and RBF kernels were used varying their
respecting hyperparameters (C and in RBF, gamma). To do this, we used the scikit-learn
library [
The training and validation steps were done to each feature group individually at first. After that, the same was done using all the features together, so the impact of each feature group on the overall result could be evaluated.
The final evaluation was performed with the official PAN@CLEF 2019 [
In this paper we described the participation of the ADAM team in the Bots and Gender Profiling Task organized by PAN@CLEF 2019. In this task, focused on Twitter posts, should be determined if the author of a Twitter feed was a bot or human. Moreover, in case of a post from a human, the challenge was to identify the gender. We used a set of features in SVM with cross-validation.
The final outcome suggests that our proposal, in general, achieves good results when compared to the random baseline. The best results were achieved for Bot detection in English. Although the accuracy for gender detection is inferior to the accuracy in Bot detection, it also presents promising results.
As future work, we suggest trying to extract metadata from the tweet text, for example, building networks of citations through user mentions. Another approach is to detect how original are the posts of a user within the corpus, since bots are known to replicate human content to fake authenticity. In addition, it is interesting to experiment different machine learning approaches, such as deep learning. Better tools for features in non-English languages are needed, like the Sentiment ones, which are scarce.