This technical notebook describes the methodology used - and results achieved - for the PAN 2015 Author Profiling Challenge by the team from Xerox Research Centre Europe (XRCE). This year, personality traits are introduced alongside age and gender in a corpus of tweets in four languages - English, Spanish, Italian and Dutch. We describe a largely language agnostic methodology for classification which uses language specific linguistic processing to generate features. We also report on experiments in which we use machine translation to accommodate for languages in which there is less training data. Native language results are successful, but socio-demographic signals in language seem to be lost under MT conditions.
Personal Language Analytics is a branch of text mining in which the object of analysis
is the author of a document rather than the document itself. Language use in text (or
indeed, speech) can reveal a great deal about a person: it can reveal one’s gender, age
or nationality, among other demographic traits. It can also provide clues as to one of
the most important individual differences: personality. For example, when writing
personal emails, out-going, social Extraverts are more likely to start by saying ‘hi’ while
Introverts opt for ‘hello’ [
Personality traits (and indeed the other human attributes mentioned) are a valuable
source of information for applications such as user modeling or social media
engagement. Work in this area, particularly in the computational recognition of personality,
is garnering increasing interest with a number of workshops being organized in recent
years (e.g. [
This paper presents the contribution of Xerox Research Centre Europe to the Author Profiling challenge 2015. We leverage our experience in multi-lingual processing by using language specific tools for each the four languages of the data set (see section 2 for more details). However, our methodology beyond this processing is broadly language agnostic: as much as possible we use a comparable feature set across each language; we also use the same parameters in our experiments.
One notable aspect of the dataset is the varying size of the corpora for the different
languages. Therefore, in addition to exploring classification within each language in
isolation, we have also used statistical machine translation in order to generate larger
datasets. MT has shown to be of use with NLP tasks such as sentiment analysis [
The data for the Author Profiling task is drawn from Twitter. For each user, the data
consists of a number of tweets (the average is approximately 100 per subject) and a
series of gold standard labels: gender (Male or Female), age-class (one of 18-24, 25-34,
35-49, 50-xx) and personality. The labels are provided by the author, with scores on five
personality traits being calculated via self-assessment responses to the short big5 test
(BFI-10, [
Language English (EN) Spanish (ES) Italian (IT) Dutch (NL) There are a number of differences between the data provided for the challenge and data typically collected directly from twitter.
– The data has been anonymised to the extent that all user mentions have been replaced with ‘@username’ – Unicode characters typically representing ‘emojis’ – a commonly occurring phenomena in Tweets – have not been encoded in the data. Thus, their use have been replaced by unknown character markers, e.g. ‘?????’
Other features of tweets, such as URLs and hastags, remain as per the original data. 2.2
The task of the Author Profiling Challenge is to predict an author’s demographics from their tweets. Performance will be evaluated on the prediction of gender and personality traits in the four languages, along with age for the larger corpora, English and Spanish. For the official challenge, age and gender will be ranked by accuracy, the personality traits by Root Mean Squared Error (RMSE).
In this section we describe the methods we have combined to form the core pipeline of our Personal Language Analytics engine: firstly we report on the linguistic analysis which forms our pre-processing and feature extraction steps; secondly, the techniques of the learning framework are outlined. We also introduce the machine translation (MT) process we employed (as introduced earlier) to explore the effect on classification of using translated data to boost smaller corpora. 3.1
In order to feed the prediction models, we use a robust dependency syntactic parser [
Processing Steps Processing here includes tokenization, morpho-syntactic analysis, POS tagging – which is performed via a combination of hand-written rules and HMM – Named Entity Detection, chunking and finally, extraction of dependency relations such as subject, object and modifiers between lexical nodes.
This is the stage of processing in which, as mentioned previously, we use-language specific tools. Several grammars have been developed for this parser, among which are the grammars for the PAN languages, i.e. English, Spanish, Italian and Dutch. These grammars are in different stages of development, the English one being more advanced than the others. Consequently, the set of features extracted is different from one language to another (see Table 2).
This parser has also been customized to parse social media data, and detects
hashtags, mentions, and (ASCII) emoticons, along with labelling the latter with their
polarity. For English, we have integrated a normalization dictionary (from [
Feature extraction We apply the parser on the different sets of PAN input data, and select a broad set of linguistically interesting features. In order to be closer to our aim of language independence, we do not draw on the deepest level of morpho-syntactic analysis which our toolset provides. For example, Spanish adjectives can have gender inflections, while English adjectives typically do not at the same level. We recognise that in doing this we may not be using the features of a given language as much as possible. However – tool performance aside – this is in-line with our broader aims.
The features extracted are of two types: word-level or class-based features. Wordlevel features are associating information to the surface and lemma forms of the words directly, while class-based features are more abstract and more generalised: they encode the presence of a given POS, semantic type, hashtags, etc, without tying the feature to the surface form.
– word-level features: unigram, bigram and trigram of surface and lemmatized form of the words; part-of-speech of surface and normalized word; words with negation, words with at least three repeated letters; bigram of repeated character (cc), trigram of repeated character (ccc), quadrigram of repeated characters (cccc); – class-based features: named entities (places, persons, organisation, dates and time expressions); unigram, bigram and trigram of POS tags, positive emoticons, negative emoticons, other emoticons; hashtags, mentions and http links; use of feminine or masculine firstnames and pronouns; capitalized words. Our learning framework is composed of 3 elements. First, the exhaustive tag-set produced by the linguistic preprocessing step is pruned using frequency thresholding determined by cross-validation. This reduces the occurrences of heavily under-used features. In the second step, the resulting index of features is compressed using truncated singular value decomposition. Finally, ensemble models are produced for each personality and demographic trait.
Truncated Singular Value Decomposition Singular Value Decomposition (SVD) [
In the truncated version, the purpose of the method is to compute an approximation
of X instead of the exact decomposition such as, for instance, in Principal Components
Analysis (PCA) [
The resulting reconstruction U SkkV T has a rank k. Neglecting all but the first k
components is justified since the data noise perturbs the small eigenvalues, whereas the
first k components supposedly capture the underlying structure of the data. Selecting the
cutoff value k defines the so-called model-order selection problem of truncated SVD.
In our framework, the selection has been determined through cross-validation.
Ensemble decision models Ensemble methods [
Sub-data classification Our framework enables these ensemble classifiers to operate at different levels for any given data point – in the case of this challenge a data point is considered to be a single author. In the first instance, a ‘user-level’ decision consists in inferring a given trait from the compressed representation of an aggregatied view of the features of the entire dataset, i.e. the full set of tweets.
A second level – in this setting ‘tweet-level’ – is to submit each sub-data point (i.e. each tweet) for a decision from the inference model. These sub-decisions are then combined to produce an expected decision at the higher level. 3.3
We created machine translation models from English to each one of Spanish, Italian and Dutch, in order to increase the size of the training data of these languages. The details of these models are described below.
Parallel corpora We wished to use the same setting for all language-pairs. To that
end, we chose parallel corpora that are available for all language pairs, namely: the
European Parliament proceedings [
Translation System Moses [
Preprocessing We used the standard Moses tools to preprocess the data, including tokenization, lowercasing and removal of bi-sentences where at least one of the sentences is empty or longer than 80 tokens.
Recasing and Language models We used SRILM [20] version 1.7.1 to train 5-gram
language models on the target side of the parallel corpus, with modified Kneser-Ney
discounting [
Tuning We tuned the translation models using MERT [
Translation and post-processing Each of the tweets of the PAN training set was preprocessed in the same fashion as was the training data. It was then translated with the trained model of the corresponding language-pair, and finally underwent quick postprocessing, namely recasing and detokenization. 4
In this section we outline our own internal evaluations of our system. First we report experiments into the parameters of our core pipeline. Following this, our experiments in using machine translation to improve performance of the smaller language subsets. 1 Version 7, from: http://www.statmt.org/europarl/ 2 http://www.ted.com 3 Data from IWSLT 2014 evaluation campaign: https://wit3.fbk.eu/mt.php? release=2014-01. 4 Version 3.0, downloaded 16 Feb 2015 from http://www.statmt.org/moses/. 4.1
Training data is first passed through the linguistic processing pipeline as described in section 3.1. Subsequently, the data encoded as features, along with the labels are passed to the learning framework. We experimented with a number of parameters which included: – numeric representation of the features: binary, normalised, or absolute frequency – feature thresholding (only including features with a frequency greater than a set value). – dimensionality of the compressed feature space (see section 3.2 for more details) – the level of classification decisions: per user, or per tweet.
For each combination of settings, we employed the following conditions: – We use leave-one-out cross-validation on the training data – Due to random seeding in the bagging used in the cross-validation of the SVD calculations (see section 3.2) we run each setting five times, and average the result. – Since age is a scale, we use regression as our classification model. To do this, we convert the classes into an ordered scale: 0, 1, 2, 3. Performance on age is reported as mean-squared error, similar to the personality traits.
Results In the interests of space, we do not report all runs here. Generally, we found that thresholding the feature space at n >= 5 provided the best results, balancing model performance and computational runtime. Similarly, while increasing the dimensionality generally improved performance, too great an increase significantly impacts runtime. We report only those experiments on the optimum value across all settings of 500. Results are reported in table 3.
The most distinct result is gender: across all languages it is the model trained at a per-tweet level using binary representation of feature frequency that performs the best. Conversely, age – though limited to two languages – shows best performance with normalised frequency at a per-user level.
Results for personality traits are less clear. Overall, the same conditions as for age – per-user, normalised frequency – perform best. In many of the cases in which they do not, the difference in performance is insignificant. 4.2
Personality labelled data sets can often be smaller than the ones typically used for text classification tasks. This is largely due to the personal nature of the information and the complexity of collecting such labels. This issue of size is particularly clear in the Dutch and Italian datasets (see table 2).
One alternative approach to collecting personality labels is the use of perception ratings – wherein the personality labels are judgements made by third parties. In this work, we explored another approach to answering the data sparsity question, namely machine translation.
Our main approach was to use the largest corpus – the English – to supplement the remaining smaller datasets. The intention was to see if increasing the size of the dataset, by leveraging non-native labelled data, would improve results. The experiments where conducted thus: – The English dataset was translated (using the models described in section 3.3) into each of Spanish, Italian and Dutch. – Each enlarged data set was processed using the linguistic pipeline configured for that language. – Using the same settings as described for the native language experiments, similar trait classification models were trained using the combined dataset. The results reported here are on the same "leave-one-out" approach, though only the original native non-English dat was used to compute the reported results.
Results The results of these tests can be seen in table 4 along with the results from the previous native experiments for comparison. In the interests of space, we have selected the best result for each language-trait pair. Although Spanish is closest in size to the English corpus, it is also included for completeness.
Overall, the results suggest that translation does not help in the classification of socio-demographic traits. In fact, in many cases – particularly gender – it is considerably detrimental to performance. Despite previous finding that SMT to assist NLP tasks provides at least ‘comparable’ results, the effect here is worse than expected.
One issue with working in automatic personality classification is understanding how
the manifestation of traits varies between data sources [
In delivering models for the official PAN review, we chose models based on our desire to be as language agnostic as we could be. With this in mind, we chose a single, optimised combination of parameters across all traits and languages. The only variation on this is with gender, for which the settings had a significant – and consistent – impact. As per the settings discussed in section 3, the final parameters for the models uploaded to the evaluation platform are listed in table 5. Additionally, we retain the model dimensionality value of 500.
Trait Encoding n Decision level Decision model Gender binary 5 tweet-level SVM Age normalised 5 user-level linear regression Personality Trait normalised 5 user-level linear regression
Note that though age is a regression in our setting, for the challenge it is converted to a class, rounding the value. The global results can be found in the overview paper for the PAN 2015 Author Profiling challenge [18]. Here we report the results of our system on the evaluation data in table 1. As discussed previously age and gender are measured by accuracy, the personality traits by Root Mean Squared Error (RMSE).
The difference between these results and those of our own tests naturally vary. Gender performance on evaluation is overall lower, but personality traits sees both improvements and worsening of performance. There is no clear pattern in this across language or trait, so there are no general conclusions which can be drawn. Large decreases in expected performance of any trait-language pair (for example NL Extraversion, testing: 0.088, evaluation: 0.135) suggests an overfitting of features under training. Despite attempting to minimise this outcome, with corpora of the sizes of Italian and Dutch, this is to be expected.
We cannot directly compare age, because we used a different metric (MSE compared with accuracy). However, when we compare our performance to others, we see that for English age prediction, we ranked among the lowest in the challenge. It is expected that this is largely due to our choice of regression modelling. As an ordered trait, even performing class-based learning as a regression makes sense. It is clear, however, that our naive approach of rounding our predicted value to a class label does not perform well. 6
In this paper we have presented details of XRCE’s Personal Language Analytics engine for multi-lingual author profiling. The system we have described leverages our capabilities in natural language processing and machine learning. We have chosen a largely language agnostic approach to this task, which has shown good performance on the four datasets.
We expect to continue this work, further refining our models. In particular we intend to explore the contribution of the individual categories of linguistic features to classification across languages and traits. This, we expect, will also lead to a further understanding of the nature of the relationship between language and personality traits in Twitter.
Related to this, we have also discussed the use of machine translation as a potential
means to accommodate for the difficulty of acquiring labelled data of this nature. In
the limited context explored here, this has not shown to be helpful. This suggests that
though sentiment signals can often be maintained under translation (c.f [
on Spoken Language Processing (INTERSPEECH 2002). pp. 257–286 (2002) 21. Tkalcic, M., Carolis, B.D., de Gemmis, M., Odic´, A., Kosir, A.: Preface: Empire 2014. In: Proceedingsof the 2nd Workshop Emotions and Personality in Personalized Services (EMPIRE 2014). CEUR-WS.org (July)