Author profiling is the task of determining the attributes for a set of authors. This paper presents the design, approach, and results of our submission to the PAN 2015 Author Profiling Shared Task. Four corpora, each in a different language, were provided. Each corpus consisted of collections of tweets for a number of Twitter users whose gender, age and personality scores are know. The task was to construct some system capable of inferring the same attributes on as yet unseen authors. Our system utilizes two sets of text based features, n-grams and topic models, in conjunction with Support Vector Machines to predict gender, age and personality scores. We ran our system on each dataset and received results indicating that n-grams and topic models are effective features across a number of languages.
Author profiling is the problem of determining the characteristics of a set of authors
based on the text they produce, how they behave and with whom they interact. An
author profiling task will typically centre on predicting one or more attributes of one
or more authors. An attribute can represent any element of a persons self, ranging from
obvious outward characteristics such as gender and age, to more personal qualities such
as personality, political leaning or sexual orientation [
A range of potential applications exist for author profiling techniques, many of
which give rise to deep ethical considerations. A company or organisation could use
an author profiling tool to identify their core user-base. Marketers could further target
advertisement to social media users who are determined to hold particular
characteristics. Law enforcement could potentially use such a system to link on-line criminal
behaviour with individuals. Studies have already investigated the use of author
profiling techniques in identifying on-line grooming [
A machine learning approach was employed for this task in order to predict
gender, age and personality. Topic models, implemented using Latent Dirichlet Allocation
(LDA) [
For the Author Profiling task at PAN 2015, a set of Twitter users whose gender, age and
personality is known is provided. These users are further divided into four languages:
Italian, English, Dutch and Spanish. The task is, given a single set of these users, some
judgement of age, gender and personality must be made on as yet unseen users [
Four corpora of tweets of different languages are provided. The corpora are balanced by author gender, such that there is an equal number of male and female authors present in each corpus. There is no guarantee that each author has the same number of tweets, and as such over-fitting to particular authors is a risk. For age there is definite imbalance, with particular age groups containing many more authors.
The task of determining age in this case has been converted to a classification problem, where a range of ages is to be predicted rather than a continuous value. Gender is also a classification problem; binary selection of male or female.
Personality prediction in this task was to estimate each user’s “Big 5” personality scores, in the range of 0:5 to 0:5, and is treated as a regression problem. The personality dimensions considered are all of the Big 5: openness, conscientiousness, extraversion, agreeableness, and neuroticism. 3
The main pre-processing step undertaken was tokenisation using a Twitter specific
tokeniser [
In early experiments on the data, all short-links present in the text were followed
and converted to the domain name of the website found, as previous author profiling
studies have identified website use as a potential analogue for some attributes [
The Twitter specific step of eliminating “retweets” was also considered, although the provided data contains so few retweets this step was deemed unnecessary. In most other Twitter profiling tasks this would be included. Another consideration is that some Tweets are in the form “shared via some app”, and do not register as retweets. These are not considered in the scope of this shared task, but may be a useful addition in future experiments. 4
In the final approach word n–grams and topics from topic models were used as features. Other features were experimented with in early development, but discarded due to poor performance. In this section the features experimented with are presented and discussed. In order to assess the affect of various features a 10-fold cross validation was performed on the training data. n–gram language model Throughout early experiments it became apparent that unigrams and bigrams together produced the most reliable results and as such would form the basis of any system developed. n–grams were weighted using the tf-idf term weighting scheme, where a term’s rating is based not only on its frequency in a document, but also against how common the term is in the whole set of documents, rating very common terms lowly and uncommon terms highly.
A stop-list was not used in building the n–gram feature vectors due to the
multilingual nature of the problem, instead all tokens that appeared in more than 70% of the
documents, as this is a roughly analogous, language independent technique.
Topic model Topic models are a group of algorithms that identify hidden themes
(topics) in collections of documents. The topic model used in this approach is Latent
Dirichlet Allocation [
As part of the training process an LDA topic model is trained on the input data, with a target of 10 topics. Ideally the model would be trained on a large additional corpus to produce more robust topics, sadly due to time and computational constraints this was not possible in the scope of this shared task.
The trained model is then used to infer topics, labelled as present or not, on unseen
documents. There is also the option to weight a topic feature by the likelihood that it
belongs to the input text, although early experiments showed that this added no benefit.
Parts–of–speech In early experiments all tweets were POS tagged as part of the pre–
processing step using a Twitter specific part–of–speech tagger [
As official results were not consistently available throughout development, 10-fold cross validations was used throughout development, to assess the affect of different features on classifier accuracy. Results from this cross-validation, which motivated feature choice in the final submission, are presented in Table 2. The feature(s) with the best score for each attribute for each language is highlighted in bold.
Results are presented in each language for n–gram features, LDA features, and the two in conjunction. In the English case, results for POS tagged n–grams are also included. These results show POS tagged n–grams as being the best feature for English gender and age prediction; despite this they were not used in the final submission, as a comparable POS tagger could not be found for Spanish, Dutch and Italian tweets.
In most cases n–gram features provided the best results, but not by a significant margin, with n–grams in conjunction with LDA topics performing similarly. LDA topics on their own proved to be a very poor quality for the English and Spanish datasets, and gave the worst results in all cases.
The final submission included n–grams in conjunction with LDA topics, as these judgements proved to be more stable across folds than n–grams on their own. The architecture of the submitted system is presented in Figure 1. The system comprises two main components: a model generation module, and one which uses a pre-trained model to infer the attributes it contains on unseen documents.
For model generation the training data is fed through several feature extraction modules. Firstly, an LDA model is trained which is then used in the “Topic Extraction” module. The same data is also passed through an “n–gram Extraction” module. The resulting feature vectors are then used to train a machine learning model.
The machine learning algorithm used in the final submission is Support Vector Machines (SVM) as they have been repeatedly shown to produce better results than other algorithms. Experiments were performed with ensemble methods and other algorithms, but none beat the results achieved by the SVM implementation.
For age and gender a Support Vector Classifier with a linear kernel was used. For
the personality recognition element Support Vector Regressors were used, again with a
linear kernel. All implementations were provided in Scikit-learn [
The resulting model can then be presented with previous unseen documents, and perform judgements on the author attributes it was trained with. The results of the final run are presented in Table 2. The system performed best on the Italian dataset, achieving a global score above 0.8, where scores for submitted systems ranged from 0:8658 to 0:6024. For the English and Spanish corpora scores were in the ranges 0:7906 to 0:5217 and 0:8215 to 0:5049 respectively, with the results obtained by our system falling roughly in the middle of these ranges. The worst performance was obtained for the Dutch dataset, scoring on the bottom end of the range 0:9406 to 0:6703.
In most cases the final results are worse than those observed by applying crossvalidation to the training data. However similar or better results were observed for some personality elements across languages. English age prediction and Spanish gender prediction also achieved reasonable scores compared to the cross-validation.
The results show that n–grams and topic models are a useful element in developing author profiling systems across a number of languages and provide reasonable results without any additional features. In order to improve the system without adding any other features the LDA topic model could be trained on a large external corpus of text, in theory leading to a more robust model. Additional stylometric features such as readability and text structure could also be applied to assess their affect on performance. It would also be interesting to asses the effect of network and behavioural features on performance should an additional dataset containing appropriate information become available. In this document we have presented our approach to the PAN 2015 Author Profiling shared task. We used Support Vector Machine classifiers and regressors in conjunction with n–gram and topic features, in order to provide judgements on age, gender and personality.
In future work we would like to investigate the effect of additional text and non-text features on classifier performance, as well as an investigation into system performance on larger datasets.
This material is based upon work supported by the Air Force Office of Scientific Research, Air Force Material Command, USAF under Award No. FA9550-14-1-0333.