We present our system for the author profiling task at PAN 2018 on gender identification on Twitter. The submitted system uses word unigrams, character 1- to 5-grams and emoji unigrams as features to train a logistic regression classifier. We explore the impact of three different word unigram weighing schemes on our system's performance. Our submission achieved accuracies of 77:42% for English, 74:64% for Spanish, and 73:20% for Arabic tweets. It ranked 15th out of 23 competitors.
The rise of the internet and social media has brought a plethora of user generated content. Since an important amount of users post in pseudonymity, author profiling tasks, such as age and gender identification, have become a compelling area of study. For example, in the case of online harassment, one might be interested in identifying the perpetrator. This naturally extends to other applications in fields of forensics and security. Similarly, social sciences might be interested to use this as a jumping-off point to study how different demographics interact with media.
In this work, we describe our submission to the author profiling task at PAN 2018 [
We participated in the gender_txt subtask for all three languages. 1.2
This year’s author profiling task is a continuation of a series of related tasks from
previous years [
Our system uses the same approach for every language. First we preprocess the tweets to handle idiosyncrasies such as hashtags and user handles. Then we extract word unigram features, character n-gram features and emoji unigram features. Finally, we train a logistic regression classifier with those features. 2.1
We use the same basic preprocessing pipeline for all languages.
First we substitute user mentions, email addresses, and URLs with special tokens.
We use the regular expression ‘@\S+‘ to find and replace user mentions and ‘\S+@\S+‘
for email addresses. Inspired by the URLExtract 3 library we identify top-level domain
names in the text and check the boundaries to find URLs. To handle Twitter’s hashtags,
we remove all ‘#’ characters from the text and replace ‘_’ (underscore) by a space
character. Next we tokenize the text using the WordPunctTokenizer provided by the Natural
Language Toolkit (NLTK, version 3:3) [
TF-IDuF: The TF-IDuF score was introduced in [
Word Features: For every tweet of a user, we compute T F IDuF features as described above. We compute the vocabulary of considered terms by retaining all terms that appear in the document collections of at least 2 users. In addition we set all nonzero term frequencies to 1 as we expect this to be less noisy than full term frequencies for short texts such as tweets.
Character Features: We extract character n-gram features for n ranging from 1 to 5.
Every n-gram is considered at most once per tweet, and we use the hashing trick [
We use HashingVectorizer, the implementation provided by the Scikit-learn (sklearn,
version 3:3) framework [
Emoji Features: Using the emoji 4 library, we extract emoji from tweets and weigh them using TF-IDuF with the same settings as for word tokens. 2.3
We train a separate logistic regression classifier for each language, applying the
LogisticRegression implementation provided by sklearn [
We performed ablation experiments to compare different ways of weighing the terms in a document. First we examine the performance of the different unigram weighing approaches on their own: tf is the system that uses raw term frequencies directly, tf-idf is the standard TF-IDF weighing of terms, and tf-iduf is the TF-IDuF as described
en es ar en es ar tf tf-idf tf-iduf tf-iduf & chars full in Section 2.2. tf-iduf & chars uses word unigrams weighed by TF-IDuF and character 1- to 5-grams. Finally, we refer as full to the system incorporating all features as described in Section 2. This is the system that we submitted to the competition.
To run the experiment we split the provided training data randomly into a training set and validation set. The split ratio of training to validation size is 80:20, i.e. 2400 authors for training and 600 for validation in the case of English and Spanish and 1200 authors for training and 300 for validation for Arabic.
Each experiment is run 5 times and we report mean and standard deviation. The numeric results are shown in Table 1 and Figure 1 gives a qualitative overview of the results. The bars show the mean accuracy on the validation split for each system and language. The error bars indicate the standard deviation. The horizontal lines show the results of our submission in the competition for reference.
There seem to be no qualitative differences between the explored feature sets and weighing schemes. The mean accuracies stay mostly within the error bars of each other per language. Furthermore for English and Arabic the validation accuracy is close to the accuracy attained by our submission. For Spanish the validation accuracy is apparently lower than the accuracy of our submission, but this might well be due to random chance. 4
We have given an overview of our classification system for gender identification. Our system attained accuracies of 77:42% for English, 74:64% for Spanish, and 73:20% for Arabic Twitter messages at the author profiling task at PAN 2018. We explored different word unigram weighing schemes and found that they all give similar performance when applied to our system.