Author profiling deals with the study of various profile dimensions of an author such as age and gender. This work describes our methodology proposed for the task of cross-genre author profiling at PAN 2016. We address gender and age prediction as a classification task and approach this problem by extracting stylistic and lexical features for training a logistic regression model. Furthermore, we report the effects of our cross-genre machine learning approach for the author profiling task. With our approach, we achieved the first place for gender detection in English and tied for second place in terms of joint accuracy. For Spanish, we tied for first place.
Author profiling deals with the study of various profile dimensions of an author [
Due to its applications in fields such as security, forensics and marketing, the study
of various profile aspects of an author has gained more importance in recent years. This
in turn has attracted the attention of the scientific community [
Author profiling is useful in a context where missing information about authors is
relevant for an organization. For instance, a company might want to know how old
their target group in social media is in order to customize advertising campaigns. In
other contexts, such as political online participation [
Two profile aspects, namely age and gender, have been the focus of the PAN author
profiling competitions. The focus of the 2016 shared task [
Author profiling has been a recurrent PAN task since 2013. Until today, the age and
gender classification tasks have always been part of the author profiling challenge. The first
challenge in 2013 [
Since there have been 53 participating teams in the last 3 years, various approaches
to author profiling have been tested. The teams have used different preprocessing steps,
features and classifiers. [
In our approach, we need to keep in mind that the evaluation is cross-genre. This means that we cannot use features that are specific for Twitter, such as hashtags. Furthermore, we have to take into account that tweets are limited to 140 characters in length and our evaluation genre may be comprised of longer text. Features based on absolute length, like word counts, were, therefore, not relevant in this year’s challenge. As a result, all of our features are normalized to account for the domain change.
Furthermore, there has also been research into author profiling outside of PAN,
for instance, to predict demographic information [
This section describes our approach to this year’s PAN Author Profiling challenge. First, we outline preprocessing steps that we used to clean the data. Then, we describe the features that we used in our machine learning approach. Afterwards, we briefly explain why we chose logistic regression as our classifier. The dockerized source code of our profiler is available on GitHub1. 3.1
As the genre of the training and test sets are not the same, we processed the documents in the training set in such a way that most of the genre-specific information was eliminated. In this way, the risk of overfitting on genres other than Twitter was reduced. This was accomplished by a composition of multiple preprocessors that each map an input document d (string of characters) to a modified document d0. The individual preprocessors are defined as follows: – p1(d): Returns a string in which all case-based characters have been lowercased. – p2(d): Filters all occurrences of URLs in the string. This is an important step toward creating genre-neutral documents. – p3(d): A mention is a tweet that contains another user’s @username anywhere in the body of the tweet and does not occur in other genres. This function eliminates all mentions in the document. – p4(d): Hashtags are used to categorize tweets. Although hashtags may contain important information about the profile of an author, obtaining a meaningful representation of it is not always trivial (e.g., #timetoact). This function eliminates all hashtags from the document. – p5(d): A retweet is a re-posting of someone else’s tweet. As this feature is also tweet-specific and may not generalize to other domains, we eliminate all retweets. – p6(d): Although the training set is claimed to be only consisting of English, Spanish, and Dutch tweets, it also contains tweets in other languages such as Arabic and Persian. We consider these tweets as noise by eliminating all non-latin characters from the input document. – p7(d): Specific lexical features such as unigrams and bigrams result in better accuracies when accents are removed from the input document. This is accomplished by means of this function. – p8(d): Eliminates all non-alphabetic characters from an input document. This function is applied when dealing with token-based features. – p9(d): Eliminates all stop-words from the document using a language-specific predefined list.
The composition of preprocessors is feature-specific, meaning that for each feature a distinct set of functions is composed in order to preprocess the document (A detailed description of features is provided in Section 3.2). Table 1 lists the preprocessors per feature (category). 3.2
After preprocessing the tweets, we need to extract features for a vector representation. The challenge in our particular task is the need for features that are genre-independent.
Even though we use tweets from Twitter for training our classifier, we cannot use features that are specific for Twitter because the evaluation dataset is from another domain. Features that depend on absolute text length are not good candidates because tweets are limited to 140 characters, whereas the text length in the evaluation domain is probably unrestricted.
Since the tweets are in three different languages, we can either find languagespecific features or language-independent features. Given that we are not familiar with all languages in this task, we decided to find language-independent features.
We experimented with multiple features in the course of our experiments. In the end, we decided to use the combination of features that worked best on the Blog dataset from 2014 as test set (with the dataset from 2016 as training set): – Word unigrams that occur at least two times – Word bigrams – Character 4-grams within word boundaries – We utilize Hunspell2 with LibreOffice dictionaries for all three languages to measure an average spelling error by determining a relative value for correctly spelled words. – In addition, we make use of four token-based punctuation features: average comma count, average dot count, average exclamation count, and average question mark count.
All these features were used for the age subtask. We omitted the punctuation features for the gender subtask.
We also experimented with punctuation n-grams and used polyglot3 to retrieve
L2normalized POS-Tag distributions of UTS tags [
We used logistic regression [
3 http://polyglot-nlp.com/ properties make the logistic regression classifier a suitable choice for our cross-genre classification problem.
As the classification task is a multiclass problem, we use the one-vs-rest scheme for logistic regression. Moreover, we set C = 10 3 as the regularization strength.
Additionally, we also experimented with random forest [
As the focus of this year’s competition is cross-genre author profiling, we only used the tweets dataset provided by the organizers to train our models. For age, the following classes are provided: 18-24, 25-34, 35-49, 50-64 and 65-xx. Moreover, gender consists of the two classes: male and female.
The provided dataset is in the three languages English, Spanish and Dutch. The corpus was annotated with the age and gender information of the authors, except for Dutch, which was only annotated with gender information. For each individual author, there exists an XML document consisting of several tweets. For English there are 436 documents, for Spanish 250 documents, and for Dutch 384 documents. In our approach, we concatenated all tweets of an author into a single document.
In order to evaluate our models, we used stratified k-fold cross validation (k = 10)
on the tweets dataset. For this we used the implementation provided by scikit [
For each language (except Dutch, which contained only the age annotation), two distinct models were trained: one for gender and one for age. For both labels we used the same set of features, except for punctuation features that were only used for age.
For the final evaluation, two test datasets were provided by the task organizers, where the first dataset is a subset of the second one. The official results for the first and second test datasets are reported in Table 2 and Table 3 accordingly.
In general, higher accuracies are achieved on the second dataset. On the second test dataset, for both English and Spanish, the accuracies for age prediction are slightly above 0.5. The highest accuracy (0.7564) is achieved by gender classification for the English language. Outstanding is the joint accuracy of 0.4286 for the Spanish language. Also, the lowest accuracies are reported for Dutch. Unfortunately, at the time of authoring this work, no access to the test datasets was granted to explain this behavior. As all features used for gender classification are token centric, we assume that the outof-vocabulary rate is too high in the prediction phase and this leads to the unsatisfiable results for Dutch. 4.2
One of the main intentions behind using simple features in our approach is to avoid overfitting on genres other than Twitter. We also performed tests on the training datasets from PAN 2014 which are publicly available. The training dataset of PAN 2014 is also annotated with age and gender information and both labels have exactly the same categories as in PAN 2016. In comparison with PAN 2016, the PAN 2014 corpus only contains English and Spanish documents belonging to four different genres, namely blogs, Twitter, social media and hotel reviews. The accuracies of our model on blogs and Twitter are reported in Table 4 and Table 5 respectively.
Among all genres, the highest joint accuracies are achieved for blogs with 0.3878 for English and 0.4091 for Spanish. These values are even higher than the ones obtained during k-fold cross validation on PAN 2016 tweet dataset and signal that no overfitting occurred in case of blogs. It can also be observed that the accuracies for English tweets are extremely high with a score above 0.9. This is most probably due to the high overlap between the datasets from 2014 and 2016.
Unlike in the case of blogs, the accuracies for the genres social media and reviews are not satisfactory (see Table 6 and Table 7). The lengths of the documents in social media and reviews are much greater than the length of the documents in Twitter. This leads to a high out-of-vocabulary rate and consequently to unsatisfactory results.
As the measure of accuracy is not suitable to study the performance of our approach for each individual category, we also exemplarily provide the confusion matrix for a model trained on PAN 2016 tweets and tested on PAN 2014 English blogs on the category age (see Table 8). The confusion matrix shows that no instance from the categories 18-24 and 65-xx is correctly classified. The reason for this is the low support of these categories, which implies that the classifier has not enough data to learn from. Another interesting point is the high similarity between the categories 25-34 and 35-49. From 60 instances in the category 25-34, 32 instances are incorrectly classified to the class 35-49. This indicates that the features defined in our approach are not capable of discriminating between the aforementioned categories. 5
We have presented our approach for the cross-genre PAN 2016 author profiling task. Our best results for the gender and age classification tasks in terms of accuracy are 0.7564 for English and 0.5179 for Spanish, respectively. Furthermore, we evaluated our approach on multiple genres to explore the effects of cross-genre machine learning.
The training set for age was imbalanced (see the confusion matrix in Table 8), which resulted in poor performance. We could use techniques such as sampling or SMOTE to tackle this problem.
In our experiments, we tested different feature combinations. It turned out to be difficult to find good genre- and language-independent features. For instance, the POS distribution turned out not to be a good genre-independent feature. In our future work, we will include more language-dependent features to better capture the characteristics of each language. Additionally, we will include lists of sentiment-bearing words in our features.
This work was partially funded by the PhD program Online Participation, supported by the North Rhine-Westphalian funding scheme Fortschrittskollegs and by the German Federal Ministry of Economics and Technology under the ZIM program (Grant No. KF2846504).