When participating in the “bots and gender” subtask (both in English and Spanish), our aim is to automatically detect different text sources (sequence of tweets sent by a bot or a human). When a text is identified as being sent by humans, the system must determine the author's gender (author profiling). To solve these questions, we focus on a simple classifier (k-NN, k = 5) usually able to produce a correct answer but not in an efficient way. Thus, we apply a feature selection procedure to reduce the number of terms (around 200 to 500). We also propose to apply a Zeta model to reduce the number of decisions taken by the kNN classifier. In this case, we focus on terms used in one category and ignored or used rarely by the second. In addition, the Type-Token Ratio of the lexical density (LD) presents some merit to discriminate between tweets sent by a bot (TTR < 0.2, LD ≥ 0.8) or humans (TTR ≥ 0.2, LD < 0.8).
In the last two decades, UniNE has participated in different CLEF evaluation
campaigns with the objective of creating new test collections on the one hand and, on
the other, to promote research in different NLP domains. This year, our team takes part
in the CLEF-PAN in the subtask “bots and gender profiling” using both the English and
Spanish corpus
Within this track, given a set of tweets, the computer must identify whether this
sequence was sent by a bot or a human. In the latter case, the author gender must be
determined. This author profiling question is not new (Schwartz et al., 2016) and has
been the subject of previous evaluation campaigns
The rest of this paper is organized as follows. Section 2 describes the text datasets while Section 3 describes our feature selection procedure. Section 4 exposes our combined Zeta and k-NN classifier and Section 5 shows some of our evaluation results. A conclusion draws the main findings of our experiments.
When faced with a new dataset, a first analysis is to extract an overall picture of the data, their relationships, and to detect and explore some simple patterns related to the different categories. A statistical overview of these PAN datasets is provided in Section 2.1 while Section 2.2 focuses on the emoticon distribution across the different categories. The distribution of the Type-Token ratio values is exposed in Section 2.3. Section 2.4 proposes to use the lexical density to discriminate between bots and humans. Finally, Section 2.5 exposes a brief overview of the distribution of positive and negative emotions.
To design and implement our classification system, a training corpus was available in the English and Spanish languages. As depicted in Table 1, the training data contains the same number of documents (one document = a sequence of 100 tweets) in the bots and human categories. In the latter case, one can find exactly the same number of documents written by men and women (1,030 in English, 750 in Spanish).
These values are obtained by concatenating the two subsets made available by the organizers, namely the train and dev parts. To be precise, the train subset is composed of 2,880 English documents and the dev by 1,240 items (for a grand total of 4,120). For the Spanish corpus, one can count respectively 2,080 and 920 documents (total: 3,000).
As each document is not a single tweet (but usually 100), the mean number of tokens per document is around 2,097 for the English language (median: 1,920; sd: 961.4; min: 100; max: 5,277). For the Spanish language, the mean length is 1,889 (median: 1,925.5; sd: 619.2; min: 100; max: 4,933). In this computation, the punctuation symbols and emoticons (or sequences of them) count as tokens. For example, from the expression “Paul’s books!!!”, our tokenizer returns {paul ’ s book !!!}. As we can see, a light stemmer was applied, removing only the plural form ‘-s’ (Harman, 1981). This choice is justified to keep the word meaning as close as possible to the original one (which is not the case, for example, with Porter’s stemmer reducing “organization” to “organ”).
Nb. doc.
Nb tweets Mean length |Voc|
2060 205,919 2,097
Looking at the mean length for both genders, Table 1 does not corroborate the common assumption that “women are more talkative than men”. For the English language, the mean is slightly higher for women (2,123 vs. 2,014) but not for the Spanish corpus (1,821 vs. 1,964).
As text categorization problems are known for having a large and sparse feature set
Of course, the tweets produced by bots are not really generated by computers but correspond to retweets or tweets showing text excerpts extracted from a larger corpus. To illustrate this, Table 2a exposes six examples of tweets generated by three bots, while Table 2b and 2c present four tweets written by two women and men. 2.2 Emoticons
An interesting aspect of web communication
As bots could be deployed to send a repetitive message (maybe with a slight
modification), one can assume that the TTR value (the number of distinct word-types
divided by the number of word-tokens) should be smaller than for a sequence of tweets
written by a human. Of course, the text genre has a clear impact on this estimation,
with a lower TTR value for an oral production compared to a written message. As a
comparison basis, the TTR achieved by Trump was 0.297 vs. 0.362 for Hillary Clinton
(oral form, primaries debates)
Based on the training set (English language), the TTR values have been computed for documents sent by bots and humans. The two resulting distributions are depicted in Figure 1. In this case, one can see that messages sent by bots tend to contain the same or similar expressions resulting in a lower TTR value, even lower than 0.2 (usually producing a boring message). A similar picture can be obtained with the Spanish language (see the Appendix).
With the English training data, one can count 398 documents generated by a bot having a TTR value smaller than 0.2 (over 2,060 or 19.3%). On the other hand, only 13 documents having a TTR smaller than 0.2 have been written by humans. For the Spanish corpus, one can find 843 documents generated by bots with a TTR values smaller than 0.2 (over 1500, or 56.2%), and none by human beings.
cyne u q e r F 0 0 1 0 5 0 0 0 5 0 0 4 0 0 3 0 0 2 0 0 1 0
Histogram Type-Token Ratio (English corpus)
TTTTRR BHoutman 0.0 0.1 0.2 0.3 0.4 TTR values 0.5 0.6 0.7
The lexical density measures the percentage of content words in a text. This percentage can also be estimated by considering the number of functional words in a text and assuming that a word could be either a content word or a functional one (see Eq. 1). In our implementation, the English language has 571 functional words while for Spanish such a wordlist counts 350 entries.
LD(T) = ()* () = 1 = − ()*() (1) Histogram Lexical Density (Spanish corpus)
LLDD BHoutman 0.2 0.4 0.8
1.0 0.6
LD values As shown in Figure 2, bots tend to present a higher LD value than the set of tweets sent by humans. For example, by assuming that the maximum value for a document written by a human is 0.8, the system can consider documents having a larger value as sent by bots. On the training set, one can count 322 English documents or 216 Spanish ones (sent by bots) for one single English document written by a human (and none in the Spanish corpus).
With the English language, we have a list of words corresponding to positive (159
entries) and negative (151 entries) emotions (extracted from the LIWC (Linguistic
Inquiry and Word Count)
Median Stdev
According to our point of view, the key function of a successful classifier is to be able to generate a good feature set. Moreover, we also want to understand the proposed attribution and be able to explain it in plain English. Therefore, one of our main objectives is to reduce the feature space into one to three orders of magnitude compared to a solution based on all possible isolated words. As shown in Table 5, the vocabulary size (|Voc|) is large for all categories and languages. If text categorization can be characterized by such huge feature spaces, they are also sparse (when considering isolated words, n-grams of words or letters). Many terms occur just once (hapax legomenon) or twice (dis legomena). Ignoring those words reduces the vocabulary size by around 50%.
To reduce this feature set and based on the training data, terms (isolated words or
punctuation symbols in this study) having a tweet frequency (df) smaller than a
predefined threshold (fixed at 9 in our experiments) are ignored
Using the term frequency difference, we can observe the term more employed in each category. For example, the terms “urllink” (replacing the sequence “http://aref”), “job”, “developer”, “and”, “hiring” or “swissmade” appear more frequently in tweets sent by bots than by humans. As other examples, we can mention that men used more frequently: “the”, “that” “it”, “he”, “a”, “is” and the punctuation symbols “.”, “,”. Woman tweets contain more “rt” (retweets), “you”, “to”, “my”, “your”, “thank”, “me”, “love” and the punctuation symbols “:”, “&”. These short examples tend to confirm part of Pennebaker’s (2011) findings, indicating that definite articles are more frequently used by men while personal pronouns and emotions tend to appear more often in female messages. |Voc| with df > 9 with tf
To go further in this space reduction, one can then add a final third step by applying
a feature selection procedure. For example, one can reduce the feature space to a value
between 200 to 500, allowing a manageable space to explain the proposed decision.
For example, previous studies indicate that odds ratio, mutual information or
occurrence frequency tends to produce effective reduced term sets for different text
categorization tasks
In addition, our classifier will also consider terms used infrequently in one category
and ignored or used rarely by the other (Zeta model)
It is also interesting to analyze the distribution of definite articles and some pronouns
The frequencies depicted in Table 6a confirm Pennebaker’s findings. Definite articles (“the”, “a”) are more frequent with male writers, and personal pronouns (“i”, “you”, “me”, etc.) are more often used by women. Exceptions can be found. The English pronoun “he” is clearly employed more often by men. Bots frequently adopt the pronouns “you” or “we” and use more infrequently “she”. Is the bot style more feminine?
The Spanish corpus also confirms Pennebaker’s conclusions. Definite articles (“el”, “un”, “una”, etc.) appear more frequently with men, and personal pronouns (“yo”, “tu”, “ella”, etc.) are more associated with the woman’s style. The Spanish pronoun “nosotros” (we) or “vosotros” (you, plural) are usually not present but this indication is often implicit with the verbal suffixes (e.g., “podemos” we can). (A linguist will also infer that frequencies of such pronouns will be rather small due to their spelling composed of 8 letters, not a length reflecting the less effort principle).
Finally, when analyzing the popularity of some countries (see Table 6a), one can see that “france” is the most popular while “swissmade” appears only with bots. For the other names, “italy” is more associated with women, while all the others are with men (except “swiss” that is associated with bots) (due to soccer, a sport popular in Spain, Italy and Germany?).
Our solution is based on a three-stage function. In the first, the needed variables are initialized (function preProcessing() in Figure 3) and they correspond to the unique vocabulary used in the two categories (VocUnC1, VocUnC2) and to the document representations belonging to the two categories (PtC1, PtC2).
Based on the training data, the system extracts the vocabulary (isolated terms with their frequency) appearing in both categories (function defineVoc()). From them, one can determine the terms appearing frequently in one category but absent (or occurring rarely) in the second (in our implementation, such a term can appear up to three times (min=3) in the second category). To rank them, the term frequency (tf) or the tweet frequency (df) statistics are applied. Instead of returning two wordlists, the system selects the top 200 most frequent ones in the underlying category and merges them (function topVoc()). In Steps #5 and #6, the system represents the documents belonging to Category #1 or #2 as vectors (generating the PtC1 and PtC2 variables).
After this initialization, each document belonging to the test sample can be processed (see function binaryClassifier() in Figure 3). In Step #1, the Zeta model is applied. This function counts the number of distinct terms appearing in VocUnC1 (denoted N1) and in VocUnC2 (or N2). If (N1 > N2+q), the test identifies the given document as belonging to Category #1. On the other hand, if (N2 > N1+q), it is assumed that the document must be labeled with the second category (e.g. Human). If the Zeta reaches a decision (e.g., dec=1 for Bot, dec=2, for Human), this value is returned.
preProcessing(trainDoc) 1 vocC1 = defineVoc(trainDoc) 2 vocC2 = defineVoc(trainDoc) 3 VocUnC1 = topVoc(vocC1, vocC2, top=200, min=3) 4 VocUnC2 = topVoc(vocC2, vocC1, top=200, min=3) 5 PtC1 = definePoints(trainDoc, C1) 6 PtC2 = definePoints(trainDoc, C2)
return(VocUnC1, VocUnC2, PtC1, PtC2) binaryClassifier(newD, VocUnC1,VocUnC2,PtC1,PtC2): decision = 0 1 dec = Zeta(newD, VocUnC1, VocUnC2, q=3) 2 if (dec == 1) or (dec == 2): return(dec) 3 aTTR = TTR(newD) 4 if (aTTR < 0.2): return(dec=1) 5 dec = k-NN(newD, PtC1, PtC2, k=13) return(dec)
Otherwise, Zeta is unable to achieve a clear decision (dec=0). For those cases, the TTR value (Type-Token Ratio) is computed (Step #3 and4). When this value is smaller than 0.2, the decision is taken as “Bot” (dec=1). In addition, we might have computed the lexical density value and returned “Bot” if this value is larger than 0.8. This step was not included in our final submission (due to time constraints).
In general, the Zeta model (together with the TTR value) cannot always propose a
clear answer. In this case, the system calls the k-NN function (with the new document,
and the set of points corresponding to Category #1 (PtC1) or #2 (PtC2)). In our
experiment, the k value was fixed to 13 and the distance between two text surrogates is
computed according to the Manhattan function
When the document type is found to be sent as human, the system re-applies the binaryClassifier() function but with Category #1 corresponding to male and Category #2 to female (but ignoring the TTR computation).
Table 7 depicts the accuracy rate achieved with our model under different conditions
and for both the type (bot vs. human) and the gender (male vs. female). These results
were achieved with the English corpus using the dev test set. In the first row, all words
have been used to build the document surrogates. In the second line, the vocabulary
size was reduced to consider only terms having a df value larger than 9. In the next row
labelled “FS”, our feature selection is applied. Finally, the last five lines correspond to
a feature space reduced to 100, 200, 300, 400, or 500 terms selected by the information
gain function
To compute the accuracy rates, only the train subset is used to define the needed wordlists and document surrogates (in other words, based on 2,880 English documents, and 1,240 Spanish ones). During the evaluation, only the dev subset was needed to derive the performance values (or with 2,080 English documents, and 980 Spanish ones).
All voc with df > 9 with tf 100 IG 200 IG 300 IG 400 IG 500 IG
The important conclusion that can be drawn from Table 7 is that it is possible to reduce the feature set to a few hundred words and to still have a good overall effectiveness. Considering k=13 neighbors tends to produce better results (and this solution is less prone to over-fitting).
Table 8 reports our official results achieved with the TIRA system
TIRA test set 1
k=5 type 0.8939 gender 0.7689 type 0.8939
0.9125 gender 0.7371 Using the CLEF-PAN datasets of the “bots and gender profiling” written in English and Spanish, we were able to achieve the following main findings. First, the text genre associated with bots can be viewed as repetitive, showing a low TTR value (usually lower than 0.25). After fixing a threshold (e.g., 0.2) for this value, one can detect 9.6% to 55% tweet sequences sent by bots (see Figure 1) with a low error rate (around 3%). For a large majority however (90% for the English corpus), documents present a higher TTR value and no decision can be reached with this simple rule. Similarly, one can compute the lexical density value and one can see that values larger than 0.8 correspond very often to bot tweets.
Second, analyzing the emoticon distribution, or the most frequent ones, we can infer that humans tend to employ them more frequently than bots. In tweets sent by machines, the used emoticons indicate directions or appear to draw reader attention (see Table 3). If humans have adopted the emoticons in their web communications, it is not clear whether we can easily distinguish their usage between men and women.
Third, our attribution approach is based on a cascade classifier. In a first step, the Zeta classifier is used to determine the category (bots vs. human, male vs. female) based on terms occurring infrequently in the first class and never (or very rarely) in the second. When the test sample is strongly correlated to the training set, such a strategy works well and can accurately determine close to 85% when a decision can be computed. As the main drawback, this approach fails to propose an answer when the vocabulary appearing in the new document is not associated clearly with one of the predefined wordlists. In such cases, a second classifier must be used (k-NN in our experiments, with k = 5, Manhattan distance).
Fourth, removing terms occurring rarely or in a few documents corresponds to our
first step in the proposed reduction procedure. In addition, we impose that terms
appearing more frequently in a given category must be selected for that class. This
strategy can be further improved by applying a term filter (e.g., mutual information,
odds ratio