We use a set of feature extraction and transformation methods in conjunction with ensemble classifiers for the PAN Author Profiling task. For the bot identification subtask we use user behaviour fingerprint and statistical diversity measures, while for the gender identification subtask we use a set of text statistics, as well as syntactic information and raw words. Automated user (bot) is a program that mimics a real person's behavior on social media. A bot can operate based on a simple set of behavioral instructions, such as tweeting, retweeting, “liking” posts, or following other users. In general, there are two types of bots based on their purpose: non-malicious and malicious. The non-malicious bots are transparent, with no intent of mimicking real Twitter users. Often, they share motivational quotes or images, tweet news headlines and other useful information, or help companies to respond to users. On the other hand, malicious ones may generate spam, try to access private account information, trick users into following them or subscribing to scams, suppress or enhance political opinions, create trending hashtags for financial gain, support political candidates during elections [2], or create offensive material to troll users. Additionally, some influencers may use bots to boost their audience size. We explore bot and gender identification techniques on PAN 2019 [5] Author Profiling task [19]. We apply a set of feature extraction methods to describe how diverse the user behaviour is over extended period of time and if the style of writing is different between two genders. The systems were hosted and evaluated on TIRA [18], a web service that aims to facilitate software submissions and evaluations for shared tasks. The rest of the paper is organized as follows. Related work is discussed in Section 2. Section 3 briefly shows insights into the datasets. Section 4.1 describes the method we used to extract and encode features in the form of digital fingerprint. In Section 4 we describe a set of features used for user profiling, for both gender and bot identification tasks. Section 5 is dedicated to experiments and results. Finally, in Section 6 we give the conclusions and briefly discuss about future work.
One of the most prominent tasks in recent social media analysis is detection of
automated user accounts (bots). Research on this topic is very active [
Bot problem on social media platforms inspired many competitions and evaluation
campaigns such as DARPA [
When it comes to gender and age user profiling, advances in natural language
processing technology have facilitated the prediction in several text genres using
automatic analysis of the variation of linguistic characteristics. However, in social media
texts, there are a couple of limitations. First, small amount of meta information about
the users’ gender, age, social class, race, geographical location, etc., is available to
researchers. Second, communication in online social networks typically occurs in a form
of very short messages, often containing non-standard language usage, which makes
this type of text a challenging text genre for natural language processing. Finally, given
the speed at which chat language has originated globally and continues to develop,
especially among young people, a third challenge in automatically detecting false profiles
on social networks will be the constant retraining of the machine learning algorithms
in order to learn new variations of chat language. Many researchers have tried to solve
some of these challenges [
An = 8>0; plain >>>>8; retweet > > ><16; reply >1; has hastags > >>>>2; has mentions > >:4; has URLs (1)
Vocabulary, given the code set A, consists of 3 23 = 24 unique characters. Each character, which describes a tweet is constructed by adding up codes for tweet features. First three codes describe the type of the tweet (retweet, reply, or plain) and the rest describe the content of the tweet. For example, if a tweet is neither retweet nor reply, it is plain (with the code = 0). If the tweet contains hashtags, then code = code + 1, If the same tweet contains URLs, then code = code + 4. Final tweet code is 5. We transform it to a character label by using ASCII table character indexes: ASCII_tbl[65 + 5] = F . The number of tweets with attributes encoded with characters determines the length of the sequence. The sequence, in our case, is simply the length of a user timeline, that is, actions in chronological order with the appropriate character encoding.
The example of a user fingerprint generated from their timeline looks like: f puser = (ACBCASSCCAF F ADADF AF ASCB:::) Fingerprint segmentation using n-gram technique To calculate data statistics, we extracted n-grams of different length (1-3 length appeared to work best). Fig. 3 shows the example on 3-gram extraction of sample user fingerprint.
N-gram segments are used to calculate richness and diversity measures, which seem to unveil the difference between genuine user and bot online behaviour.
sity have long history and wide area of application [
Yule’s K Index Yule’s original intention for K use is for the author attribution task, assuming that it would differ for texts written by different authors.
K = C
S2
S2 1
S1 = Ch
m=1 1 N + mXmax V (m; N )( m )2i
N To simplify, S1 = N = Pm V (m; N ), and S2 = Pm m2V (m; N ). C is a constant originally determined by Yule, and it is 104.
Shannon’s H Index The Shannon’s diversity index (H) is a measure that is commonly used to characterize species diversity in a community. Shannon‘s index accounts for both abundance and evenness of the species present. The proportion of species i relative to the total number of species (pi) is calculated, and then multiplied by the natural logarithm of this proportion (ln(pi)). The resulting product is summed across species, and multiplied by -1.
H =
V (N) X piln(pi) i=1 V (N ) is the number of distinct species.
Simpson’s D Index Simpson’s diversity index (D) is a mathematical measure that characterizes species diversity in a community. The proportion of species i relative to the total number of species (pi) is calculated and squared. The squared proportions for all the species are summed, and the reciprocal is taken.
D =
1 PV (N) p2
i=1 i
Honoré’s R Statistic Honoré (1979) proposed a measure which assumes that the ratio
of hapax legomena (1; N ) is constant with respect to the logarithm of the text size:
Sichel’s S Statistic Sichel [
S =
V (2; N )
N
We use this measure to express the constancy of n-gram hapax dis legomena (number of n-grams that occur two times) which we show to be distinct for genuine and bot accounts.
On Fig. 4 we show the comparison of density plots of all measures of bot accounts versus genuine users. We can see that the diversity measures are different for bots and genuine users. We exploit this characteristic to build a good classifier with as few features as possible. 4.2
The feature types used for this task can be split into four categories: Character and Word Features We used simple text metrics, such as total number of characters, total number of words, number of characters/words per message, number of special characters, number of digits.
PoS Tags Features Using spacy2 python library we extracted word unigrams and bigrams, as well as PoS tag bigrams.
Emoji Features We counted the number of emojis, as well as fine-grained distinction between different types of emojis. To distinguish categories of emojis we used the latest standard at the time of experiments3.
Text Readability Measures In 1948, Flesch [
For bot identification subtask we conducted four experiments with five different classifiers (Gradient Boosting, Random Forest, SVM, Logistic Regression, K Nearest Neighbours). The differences between the experiments are more focused on testing the improvement with training data increase, as well as feature set generalization using raw fingerprint n-grams versus statistical diversity measures.
Experiment 1 In Experiment 1 we used character n-grams of user fingerprint described in 4.1. The length of n-grams is a combination of 2, 3 and 4. We can see that some classifiers have fairly similar results (Table 1, column E1). The best classifier is Random Forest for both languages. In this experiment we used the training subsets for English and Spanish separately.
Experiment 2 In Experiment 2 we used the diversity measures calculated on character n-grams of user fingerprint described in 4.1. The length of n-grams is a combination of 1, 2 and 3. The best classifier is Random Forest for both languages. In this experiment we used the training subsets for English and Spanish separately.
Experiment 3 In Experiment 3 (Table 2, column E3) we used the same features as in Experiment 1. The best classifier is Gradient Boosting ensemble for both languages. In this experiment we used the training subsets for English and Spanish combined. Because the features are language independent, we combined training dataset into one, and tested it on both languages. The final model is same for both subsets. 2 https://spacy.io/ 3 https://unicode.org/Public/emoji/12.0/emoji-test.txt Dataset Classifier Precision
GBy 0.9252 ilshg SVLMR 00..99102914 nE KNN
RF 0.9189
GBy 0.8896 sh SVM 0.8588 ian LR 0.8478 pS KNN
RF 0.8764
E1 Recall 0.9153 0.9161 0.8750 Experiment 4 In Experiment 4 (Table 2, column E4) we used the same features as in Experiment 2. The best classifier for English is Gradient Boosting ensemble and K Nearest Neighbours for Spanish. As in Experiment 3, we combined training dataset into one, and tested it on both languages.
Although a better performance was obtained on separately trained models for two languages (Random Forest, Table 1) with raw features, we opted for Gradient Boosting ensemble which was trained on combined dataset (Spanish portion slightly dropped in performance). The classifier from Experiment 4 was used for the official ranking.
For the gender identification subtask we used the same set of classifiers as for bot detection. The results in Table 3 show that Gradient Boosting classifier performed the best for both languages. This task was language dependent, so each language had its own model. The official results are shown in Table 4. Bot detection for English performed with similar results as in our experiments with development set, while for Spanish performed better. Similar improvement was obtained with Spanish dataset for gender identification. The models for the final evaluation are trained on both, training and development sets. 6
We show that automated accounts have less diverse behaviour than genuine user accounts and these measures can help in detecting automated behaviour without diving into language-specific analyses. For the gender identification task we used a standard set of features usually used in stylometry analysis, with the addition of emoji features on a more granular level.