This paper describes our participation at PAN 2014 author profiling task. Our idea was to define, develop and evaluate a simple machine learning classifier able to guess the gender and the age of a given user based on his/her texts, which could become part of the solution portfolio of the company. We were interested in finding not the best possible classifier that achieves the highest accuracy, but to find the optimum balance between performance and throughput using the most simple strategy and less dependent of external systems. Results show that our software using Naive Bayes Multinomial with a term vector model representation of the text is ranked quite well among the rest of participants in terms of accuracy.
PAN1 is a competitive evaluation lab on uncovering plagiarism, authorship and social
software misuse, held as part of CLEF2 conference. PAN 2014 offers three different
main tasks: 1) plagiarism detection, 2) author identification and 3) author profiling.
describes our participation at the PAN 2014 author profiling scenario [
The task is focused on author profiling, i.e., the problem to distinguish between classes of authors studying how language is shared by people, allowing to identify aspects such as gender, age, native language, or personality type. Specifically, the focus is on author profiling in social media messages. Author profiling is a problem of growing importance in different applications such as forensics, security, and marketing, for instance, to know the demographics of people that like or dislike their products, based on the analysis of blogs and online product reviews.
Given a document, the task is to determine its author's age and gender. Participants are provided with a training data set that consists of blog posts, Twitter tweets and social media texts written in both English and Spanish as well as hotel reviews written in English. Gender is a binary classification (male or female) and with regard to age, the following 5 classes are considered: 18-24, 25-34, 35-49, 50-64, >65. Differently to other CLEF labs, participants must not submit the results of their experiments using a provided test corpus, but else upload a software that runs within TIRA evaluation platform4.
The idea behind our participation was to define, develop and evaluate a simple machine learning classifier able to guess the gender and the age of a given user based on his/her texts, which could become part of the solution portfolio of the company. We were interested to find not the best possible classifier that achieves the best accuracy, but to find the best balance between performance and throughput using the most simple strategy and less dependent of external systems. Our system and results achieved are presented and discussed in the following sections. 2
The provided training data covers 1) four different types of corpus with presumably different language usage, 2) two different languages (English and French), and 3) two attributes to guess (gender and age). After several preliminary analysis using cross validation on the training corpora, we decided to build a machine learning classifier specifically trained for each combination of corpus-language-attribute, so 14 classifiers in all. instance Twitter corpus), we decided to design a two-level classifier: first, a document-oriented classifier, which guesses the gender and age of a given text, and then, an author-oriented classifier, which predicts the gender and age of a given user by aggregating the output of the first classifier for each text written by a given user. All corpora are equally balanced for gender and age, so the training is not affected by any class unbalance problem.
All 14 classifiers are trained with all texts for each combination of corpus, language and attribute. We used Weka 3.7 for performing our experiments and for developing our software to run in TIRA. Texts were tokenized using WordTokenizer to obtain a simple bag of words representation. The tokenizer allows to define split characters that are removed from the term vector space representation of the text. Besides the usual split symbols, spaces and some punctuation marks, we use some specific delimiters such as hashtags (#), usernames (@), emoticons, slashes, ampersands, question marks and hyphens that are used to separate words in SEO optimized URLs. Finally, as a high number of terms were low frequency numerals we decided to add numbers as well to help in normalization.
Regarding the document-oriented classifiers, a number of supervised algorithms
were evaluated using cross validation, and finally, for its performance, we selected
Multinomial Naive Bayes (NBM) classifier [
Results of this document-oriented classifier on training data using cross validation are shown in Table 2.
The author-oriented classifier reads the output of the document-oriented classifier for each text written by a given author and predicts the gender and age using a simple voting strategy, i.e., returns the most frequent value among all texts, selected after some preliminary tests. Some other strategies were tested, such as a voting approach using a confusion matrix with different cost for each decision values, depending on the estimated accuracy for each class, but unfortunately we did not find any definite conclusion or improvement due to lack of time.
The final submission consists in a script written in PHP that reads the input test corpus and the output directory, and, using a loop, processes every file in the test corpus, reading all documents and creating two files in the arff format suitable for Weka, one for gender and another one for age. Then Weka is called to obtain the predictions and then the output is aggregated to select the most frequent value that is chosen as the final output prediction for the author. 3
The gender and age predictions have been evaluated as a classification problem, so accuracy measure over each class are reported. Results achieved by our software are shown in Table 3.
In general, classifiers for Spanish achieve better results than classifiers for English, except for the case of blogs where English works better.
Although apparently gender attribute achieves a higher precision than age attribute, the classifier for gender is quite useless, as, taking into account that the range of values for the attribute is just two (male vs female), a random choice would achieve a 0.50 accuracy (assuming an equally balanced test corpus, the same as the training corpus). Thus classifiers for age outperform classifiers for gender in terms of lift (increment with regards to the random choice): for instance, 59% vs 50% for gender in English, 37% vs 20% for age in English (5 possible classes), etc.
Table 4 shows the comparison with other participants. This table shows, for each corpus, language and attribute, the maximum, minimum and average values, and the position of our software in the ranking of participants.
In general, we achieve average results just above the middle of the table, except for same cases were our software outperforms other participants, such as social media or reviews in English.
As it can be also noticed in the table, our results for Spanish are worse than the average for all participants in Spanish, though the approach is the same as for English. We do not have any explanation for this issue yet. However, we have a feeling that a stemming or lemmatization step should have been considered for Spanish, as inflection processes are important in this language and affect other tasks such as information retrieval or named entity recognition.
Corpus Blog Review Socialmedia English Twitter Blog Twitter Socialmedia Spanish Results show that our quite simple approach using a two-level classifier composed of a document-oriented Naive Bayes Multinomial classifier with a term vector model representation of the text and then a voting strategy for predicting the author age achieves acceptable results in terms of accuracy. Despite of the difficulty of the task, results somewhat validate the fact that this technology may be already included into an automated workflow process for the first step towards social media mining and author profiling for supporting marketing activities.
However, in general, classifiers for gender (for all participants) are quite useless as they achieve a very low improvement over the random choice. Classifiers for age are worse in absolute accuracy but better in terms of lift with respect to the random choice. Obviously a different approach must be investigated to predict gender more robustly.
We already include a module for extraction user demographics in our portfolio of
solutions [
Based on the results achieved in PAN, our initial idea to find a strategy that offers
a good balance between performance and throughput using the most simple approach
and less dependent of external systems gets validated and developing such classifier is
within our immediate plans. In the short term, we plan to carry out some tests using
our software for text classification [
Regretfully, due to lack of time and resources, we have not been able yet to carry out an individual analysis by language, by corpus and a detailed analysis per class (confusion matrix) so we do not understand yet the effect of each component in the final result.
Specifically for the age attribute, we think that in a real business scenario, accuracy as defined in the task, i.e., a binary decision between right or not, could be somewhat relaxed using a cost matrix, considering that a miss classification between adjacent age ranges is less serious than between more distant ranges, specially for users who are near the end of the interval. So, we suggest to consider a modified evaluation metric that considers this cost matrix for future editions of PAN.
This work has been supported by several Spanish R&D projects: Ciudad2020: Towards a New Model of a Sustainable Smart City (INNPRONTA IPT-20111006), MA2VICMR: Improving the Access, Analysis and Visibility of Multilingual and Multimedia Information in Web (S2009/TIC-1542) and MULTIMEDICA: Multilingual Information Extraction in Health Domain and Application to Scientific and Informative Documents (TIN2010-20644-C03-01).