This paper describes the participation of the Cross-genre Author Profiling System (CAPS) in the PAN16 shared task [15]. The classification system considers parts-of-speech, collocations, connective words and various other stylometric features to differentiate between the writing styles of male and female authors as well as between different age groups. The system achieves the second best score - 74.36% accuracy (with the best performing system (BPS) reaching 75.64%) for gender identification on the official test set (test set 2) for English. Further, for age classification, we report accuracy of 44.87% (BPS: 58.97%). For Spanish, CAPS reaches performance of 62.50% (BPS: 73.21%) for gender and 46.43% (BPS: 51.79) for age, while for Dutch, the accuracy for gender (the task did not target age) is lowest - 55.00% (BPS: 61.80%). For comparison, we also tested CAPS on single genre classification of author gender and age on the PAN14 and PAN15 datasets achieving comparable performance.
Author profiling is the process of analyzing the sociolect aspect of the writer of an
anonymous text sample. The classification tasks in this field range from revealing the
author’s age and gender to determining the native language or even personality traits
of the writer [
Profiling social media authors may be utilized for various purposes. It can, for instance, be used for user-based analysis to find out the demographic characteristics of an average consumer or for targeted advertising to place an ad, tailored to the user’s age and gender, next to their written online text. Apart from online texts, profiling the author may also be useful in the field of forensics, for example, to analyze a ransom note.
Previous research into author profiling always concentrated on a single genre. A
new view on this problem was proposed by the “Plagiarism Analysis, Authorship
Identification, and Near-Duplicate Detection” (PAN) 2016 [
In this paper, we present the participation of the Cross-genre Author Profiling System (CAPS) in the PAN16 shared task on cross-genre author profiling. Further, in section 2, we provide an overview of relevant work and in section 3 we describe the experimental setup we used. In section 4, we present the results CAPS achieved at the PAN16 task, as well as its evaluation on the PAN14 and PAN15 datasets. Finally, in section 5, we conclude our findings.
The work by Koppel et al. [
Since 2013, the yearly PAN shared task includes author profiling as one of its
subtasks. PAN13 [
The following year PAN14 [
PAN15 [
The PAN16 author profiling task concentrates on cross-genre gender and age classification, implying the use of one genre for training and an unseen genre to test the classification model. Such modification of the task can be helpful for underrepresented genres for which no or only small amount of training data is available. However, this modification also increases the complexity of the author profiling task immensely. Not only would a system need to be developed in a general enough manner, but also, to achieve good results, the training set needs to be as similar or close to the test dataset genre as possible.
Our work further demonstrates the discrepancies between the datasets as well as the need to use similar text genres for both training/development and the testing phase. In the following section, we describe our experimental setup and the CAPS system with which we participated in the PAN16 shared task.
CAPS includes the following main pipeline processes: preprocessing, TF-IDF representation, topic modeling, chi-square term extraction and custom feature extraction. Figure 1 gives a visual representation of the classification pipeline.
Preprocessing: The HTML content and the Bulletin Board Code present in the data
are removed as a first preprocessing step. Furthermore, all links are normalized to the
special token [URL]. Additionally, all user mentions of the form @username are
translated to [USER]. Since the dataset included a considerable number of duplicate
text samples, probably due to its automatic collection with the help of web scraping,
we excluded all duplicate samples. Table 1 shows the training dataset distribution into
each gender and age class as well as a detailed breakdown of the number of authors
and the underlying text samples after all duplicate text samples have been discarded.
Lemmas and part-of-speech (POS) tags are produced using the TreeTagger [
Term Frequency-Inverse Document Frequency (TF-IDF): TF-IDF is a
procedure widely used in information retrieval and data mining to measure the
importance of each word in a corpus which was first formulated in [
At the early stage of model training, grid search 5-fold cross-validation was performed for TF-IDF optimization. The TF-IDF vector for the lemma representation shows best results when considering unigrams, bigrams, and trigrams. The part-ofspeech TF-IDF vector additionally considers four-gram POS sequences. The categorical character TF-IDF representation considers only trigram characters. Fig. 2 shows the most informative lemma, uni-, bi- and trigrams used by male and female authors. The representation is based on the PAN16 Twitter dataset merged with the PAN14 training datasets. Fig. 2 indicates that mentioning the opposite gender provides reliable information about the gender of the author, suggested by the presence of lemmas/n-grams, such as male, my husband, boyfriend, husband (used by female authors) and women, wife, my wife, girlfriend (used by male authors) amongst the most informative lemmas and bigrams.
Topic Modeling: We apply Latent Dirichlet Allocation (LDA) for topic modeling
of the lemmas. LDA is a generative statistical model that assigns probability weights
to words and according to those probabilities, assigns the word to a certain
automatically generated topic [
Chi-Square Term Selection: Since the use of TF-IDF vector representations for the lemma, part-of-speech and character n-grams produces exceedingly high dimensional vectors, for the final classification model only a smaller percentile of the features is selected. For this purpose, the chi-square test is used which tests the independence of term and class occurrences. Subsequently, the terms that are independent of the class are eliminated. This technique ensures the selection of classdependent features and shrinks the number of the latter, allowing custom features to influence the classification significantly more.
Apart from TF-IDF vector representation and topic modeling about 40 additional custom features have been developed for age and gender cross-genre profiling. Most of these are stylometric features representing the linguistic style of the author. The features are grouped into four clusters: dictionary-based, POS-based, text structure and stylistic.
Dictionary-Based: The purpose of this set of features is to check whether the raw text representation includes words from a predefined list of tokens. The feature cluster Feature Cluster
Feature Name Dictionary-based
Connective Words Emotion Words Contractions Familial Words
Collocations Abbreviations and
Acronyms Stop Words
English furthermore, firstly, moreover, hence … sad, bored, angry, nervous, upset … I’d, let’s, I’ll, he’d, can’t,
he’d … wife, husband, gf, bf, mom
… dodgy, telly, awesome,
freak, troll … a.m., p.m., Mr., Inc.,
NASA, asap …
did, we, ours, you, who,
these, because …
consists of dictionaries of connective words, emotion words, contractions, family
related words (as proposed by Maharjan et al. [
POS-Based: This cluster captures the distribution of the various parts-of-speech.
Additionally, this set includes a more complex F-Measure feature, first introduced by
Heylighen et al. [
Text Structure: This feature cluster attempts to analyze the structure of the text and consists of such features as type/token ratio, average word length, and the amount of punctuation signs used in each text sample.
Stylistic: This set of features counts the frequency of use of different adjectival and
adverbial suffixes in the text samples. First introduced by Corney et al. [
After all custom feature vectors are extracted, they are scaled. There are several techniques used for feature scaling. Some of the most widely used are normalization and standardization. In the case of cross-genre author profiling, feature values of the training set differ greatly from the feature values of the test set. For example, the lengths of text samples throughout various writing styles range from short, one to two sentence long tweets, to very long samples, as for instance, blog posts. Using normalization or similar form of rescaling would be most suitable to cope with these differences, but the information about outlier values, which can be best captured using standardization, would be lost.
To be able to use standardization for this particular task, a form of feature vector pre-scaling is introduced. Pre-scaling rescales the feature values, which count the number of occurrences of a certain token or a stylistic characteristic, relative to the sample length. A simple solution to this problem could be a division of the feature value by the length of the text in tokens. A more comprehensive approach is presented in Equation 1. It rescales the sample length relative to the lowest mean length of a text sample throughout all possible writing styles that could be represented in both training and test sets and divides the feature value by this rescaled sample length. The rescaled sample length represents the amount of possible smallest sample entities that would fit into the text sample under review. Using this technique, the feature value is always scaled relative to the minimum mean length of all text samples of all represented writing styles. The average length is used instead of minimum or maximum length to better represent the sample length distribution of the writing style that has the shortest text samples on average in the dataset.
( ) − = ( ( ) ( ) ( 1… )| ≔ ( 1)… ( )) (1)
Equation 1 gives a mathematical formulation of the feature pre-scaling approach, where ( ) represents the current feature value, is the current text sample, stands for mathematical mean, represents a genre and is the text sample of the genre . len() is a function which, given a text sample, returns its length either in tokens or in characters, which makes this interpretation suitable for both types of features that work on the level of tokens and the ones dealing with character representation.
There are various approaches to the author profiling classification. The task implies
classifying the author of a given text sample, but in many cases, there is a whole set of
documents belonging to one author, which raises the question of how to handle the
big number of samples per author. It is possible to concatenate all text samples of
each author into one uniform sample, as demonstrated by Ṣulea et al. [
In fact, some works, including [
Gender classification is performed using LinearSVC [
4.1 PAN16
All of the test datasets used in the PAN shared tasks since 2014 are not openly
available and can only be used through the TIRA [
It is also important to note that the PAN16 task included two different test sets, although no further information is available on them until the final evaluation results are announced. This section reports system results on both these datasets.
Table 3 gives a detailed overview of CAPS' official participation in the PAN16 shared task. CAPS reaches highly competitive performance to the best participating systems in the task, despite the actual complexity of the problem, achieving 74.36% accuracy (on test set 2, which is the official test set used for system ranking) for gender classification on English, closely following the best performing system in this setting that reached 75.64%. However, there is about 20 percent points discrepancy between the system performance on test set 1 (53.74%) and test set 2 (74.36%). This big performance gap is also observed across the performance of the majority of the other participating systems, which shows that the overall performance on this task is highly biased towards the actual test setup. In fact, test set 1 was assembled in a way, such that none of the systems managed to beat the official baseline for this setting – 56.41%. Our assumption was that test set 1 and test set 2 differ based on their genre and that test set 1 is a genre that is more dissimilar to the training set than test set 2. However, this is not the case, since after the evaluation phase it was revealed that both datasets partially overlap which can only be the case if they are from the same genre. Yet, further details on the actual datasets would need to be provided (presumably contained in the task overview paper presented at the PAN16 workshop during the CLEF Conference in September 2016) in order to be able to understand this difference better.
The results for age classification reached only 44.87% for English which reflects the complexity of author profiling when five age groups are targeted. Additionally, the change of genre also poses a difficulty to this task. While the age indicators for one group should stay consistent within the same genre, a different genre might pose a change in the style of writing within the same age group. Specifically, within twitter data, younger age groups tend to make increased use of acronyms, abbreviations, special symbols, etc. which is learned during training. Different genres (expected during testing in this particular task), such as blogs or reviews directly pose a limitation on the use of such an excessive amount of Twitter-specific style as there is no text length limit and in general, all age groups tend to write closer to the standard language. Such a change would be particularly hard to capture within a classification approach. One remedy for this problem could be not to use a single genre for training, but to look into a range of genres that would represent well the different writing styles one particular age group could have.
Apart from the overall lower results reached on the age classification subtask, once again a significant discrepancy between the performance on the two test sets is observed with accuracy scores of 29.02% and 44.87% for each set respectively (an issue co-occurring across all participating systems).
CAPS' results on Spanish and Dutch are much lower, which is easily explained by the fact that the main focus during system development and training lied on English. For Spanish (gender), CAPS achieves 62.50% on test set 2, while for Spanish (age), similar to English, the result is considerably lower – 46.43%. The latter numbers show that Spanish follows the general tendencies observed for English with large discrepancies in performance between test set 1 and 2 and with test set 1 leading to lower performance. For Dutch (gender (age was not targeted during the task)), however, the situation is slightly different – the gap between the performance on both test sets is only 1 percent point, which is considerably lower than the gap observed for English (20.62%) and Spanish (6.25%) for the same setting. The latter is also observed among all other systems. These changes seem to correlate with the size of the datasets available which presumably is also an indicator that once gender and age indicators are well learned for one genre, the change in genre only leads to higher error rates based on the discrepancies of the writing styles between the genders and age groups across the different domains. 4.2 PAN14 and PAN15
For comparison purposes and in addition to the participation in the PAN16 shared task, CAPS was also trained and tested on the datasets of the PAN14 and PAN15 shared tasks through the TIRA evaluation system.
Within the PAN15 setup, CAPS successfully performed on the single genre datasets and produced results also highly competitive to the state-of-the-art systems presented in the task. For instance, CAPS achieved 81.69% for gender profiling on the English dataset, which is only 4.23 percent points lower than the best system presented in that shared task. Table 4 compares the results of CAPS with the best results achieved for gender and age classification in the PAN15 shared task across all three languages (English, Spanish, and Dutch). For CAPS, as for the PAN16 system participation, we provide detailed numbers for both test sets separately, as well as an averaged accuracy score.
A further comparison between CAPS' performance and the best system results in the PAN14 is given in Table 5. Due to the limited amount of Random Access Memory on the TIRA Virtual System, not all PAN14 datasets could be evaluated in time. Subsequently, due to time constraints no additional resources have been requested from the organizers.
Altogether, Table 5 demonstrates the system’s effectiveness on various single genre classifications. The best result of 71.32% in terms of accuracy is achieved on the English hotel reviews dataset for the gender classification, falling about 1 percent point short of the best system’s performance (72.59%).
Overall, the results indicate that the current system may also be used in a single genre setting, especially evident by the results on the PAN15 datasets where the model reaches around 70% of average accuracy on both gender and age classification throughout all three languages.
Language (Setting) English (Gender) English (Age) English (Gender) English (Age) English (Gender) English (Age) Spanish (Gender) Spanish (Age) Spanish (Gender)
Spanish (Age) 5
The task of cross-genre author profiling is highly complex in comparison to single genre profiling. For the task presented at PAN16, the classification model needs to be very robust and able to adapt the observations learned from short and usually stylistically and grammatically malformed tweet samples to work on any possible form of text as well as samples of any length, for example, hotel reviews, blogs or any other writing style.
CAPS showed promising results in a cross-genre aspect of author profiling regardless of the complexity of the task. The performance of the system on gender identification reached 74.36% for English, 62.50% for Spanish and 55.00% for Dutch displaying great variation across both datasets and across all three languages. With respect to age, CAPS reached 44.87% for English and 46.43% for Spanish (Dutch was not included in the age setting). These low results do outperform the corresponding baseline, but also show that five age groups are probably a too finegrained distinction to automatically deal with in addition to the change of genre which also increases the complexity of the task. For comparison, we also evaluated CAPS on single genre classification of author gender and age on the PAN14 and PAN15 datasets demonstrating that our system is highly competitive to the state-of-the-art systems applied across all genres. On the PAN15’s English dataset CAPS achieved 81.69% for gender classification with the best PAN15 participating system reaching a performance of 85.92%.
CAPS can be further improved in various ways. Firstly, more attention needs to be paid to Spanish and Dutch since the included custom features are only tailored for English and simply adjusted to function on other languages represented in the shared task. Second, the current model considers each text sample as a separate entity that does not correlate with the other text samples belonging to the author. Some form of text sample-author profile interrelation could improve the model performance.