This overview presents the framework and results for the Author Profiling task at PAN 2013. We describe in detail the corpus and its characteristics, and the evaluation framework we used to measure the participants performance to solve the problem of identifying age and gender from anonymous texts. Finally, the approaches of the 21 participants and their results are described.
In classical authorship attribution, we are given a closed set of candidate authors
and are asked to identify which one of them is the author of an anonymous text.
Author profiling, on the other hand, distinguishes between classes of authors, rather
than individual authors. Thus, for example, profiling is used to determine an author’s
gender, age, native language, personality type, etc. Author profiling is a problem of
growing importance in a variety of areas, including forensics, security and marketing.
For instance, from a forensic linguistics perspective, being able to determine the
linguistic profile of the author of a suspicious text solely by analyzing the text could
be extremely valuable for evaluating suspects. Similarly, from a marketing viewpoint,
companies may be interested in knowing, on the basis of the analysis of blogs and
online product reviews, what types of people like or dislike their products. Here we
consider the problem of author profiling in social media, with particular focus on the
use of everyday language and how this reflects basic social and personality processes.
Our starting point is the seminal work of Argamon et al. [
In PAN 20131 we consider the gender and age aspects of the author profiling
problem, both in English and Spanish. So far research work in computational linguistics
[
In Section 2 we present the state of the art, describing related work and how the task has been approached. In Section 3 we describe the details of the collection used and the evaluation measures. In Section 4 we present the authors’ approaches and we discuss the results in Section 5, concluding the overview in Section 6. 2
The study of how certain linguistic features vary according to the profile of their authors
is a subject of interest for several different areas such as psychology, linguistics and,
more recently, natural language processing. Pennebaker et al. [27] connected language
use with personality traits, studying how the variation of linguistic characteristics
in a text can provide information regarding the gender and age of its author.
Argamon et al. [
With the rise of the social media, the focus is on other kind of writings, more colloquial, less structured and formal, like blogs or fora. Koppel et al. [16] studied the problem of automatically determining an author’s gender by proposing combinations of simple lexical and syntactic features, and achieving approximately 80% accuracy. Schler et al. [30] studied the effect of age and gender in the style of writing in blogs; they gathered over 71,000 blogs and obtained a set of stylistic features like non-dictionary words, parts-of-speech, function words and hyperlinks, combined with content features, such as word unigrams with the highest information gain. They obtained an accuracy of about 80% for gender identification and about 75% for age identification. They demonstrated that language features in blogs correlates with age, as reflected in, for example, the use of prepositions and determiners. Goswami et al. [12] added some new features as slang words and the average length of sentences, improving accuracy to 80.3% in age group detection and to 89.2% in gender detection.
It is to be noted that the previously described studies were conducted with texts of at least of 250 words. The effect of data size is known, however, to be an important factor in machine learning algorithms of this type. In fact, Zhang and Zhang [33] experimented with short segments of blog post, specifically 10,000 segments with 15 tokens per segment, and obtained 72.1% accuracy for gender prediction, as opposed to more than 80% in the previous studies. Similarly, Nguyen et al. [22] studied the use of language and age among Dutch Twitter users, where the documents are really short, with an average length of less than 10 terms. They modelled age as a continuous variable (as they had previously done in [21]), and used an approach based on logistic regression. They also measured the effect of the gender in the performance of age detection, considering both variables as inter-dependent, and achieved correlations up to 0.74 and mean absolute errors between 4.1 and 6.8 years.
One common problem when investigating the author profiling problem is the need to obtain labelled data for the authors, for example, to obtain their age and gender. Studies in classical literature deals with a small number of well-known authors, where manual labelling can easily be applied, however for the dimensions of the actual social media data this is a more difficult task, which should be automated. In some cases, researchers manually label the collection [22] with some risk of bias. In other cases, as in the vast majority of the aforementioned studies, researchers took into account information provided by the authors themselves. For example, in blog platforms, the contributors self-specify their profiles. This is the case for Peersman et al. [25] who retrieved a dataset from Netlog2,where authors report their gender and exact age, and Koppel et al. [16], who retrieved the dataset from Blogspot3. In these cases we have to be aware of a common issue, the use of these media (mainly blogs) to promote web positions in search engines through the use of false profiles. This is likely to introduce noise to the evaluation corpus, but it also reflects the realistic state of the available data. 3
In this section we describe the data collection obtained for the task, its properties, challenges and novelties as well the evaluation measures. 3.1
We built the corpus with thousands of blog posts taking into account that: – The variety of themes provides a wide spectrum of topics, making the task of determining age and gender more realistic. The ample diversity of topics allows to investigate standard cliches, for example, men speaking a lot about beer or football and women about nails or shopping, for breaking or reinforcing them. – Blog posts are used daily for search engine optimization and can be automatically generated by robots or be advertisements (chatbots). – People may use social media to talk also about sex and few can also break the line and use these systems to misbehave and engage in conversations that may result into sexual harassment. For this reason and due to the importance of unveiling fake profiles, we decided to test the robustness of the author profiling approaches including in our collection some texts from last year PAN task on sexual predator identification. – We wanted to carry out the task in a multilingual setting, therefore, in addition to English we included a Spanish part in our collection. Spanish and English are two of the most used languages in the world4.
3 http://blogspot.com 4 http://www.internetworldstats.com/stats7.htm en
es We looked online for open and public repositories such as Netlog with posts labelled with author demographics such as gender and age. Once found, we decided to group posts by author, selecting those authors with at least one post, and chunking in different files those authors with more than 1,000 words in their posts. We also included authors with very few and possibly short posts in order to maintain a realistic evaluation framework. We divided the collection into the following parts: training, early bird evaluation and final testing. Authors were randomly split into these parts, making sure that each author is included in exactly one part. For age detection, we followed what was previously done in [30] and considered three classes: 10s (13-17), 20s (23-27) and 30s (33-47). The collection is balanced by gender and imbalanced by age group. Additionally, trying to preserve a real-world scenario5 , we incorporated a small number of samples from conversations of sexual predators [14] together with samples from adult-adult conversations about sex. In Table 1 we illustrate the statistics of English and Spanish collections.
In the training part of the English collection, numbers inside parentheses for male 20s and 30s correspond to the number of samples of sexual predator conversations 5 E.g. There are statistics of about 200 tweets per hour in English from sexual predators (http://www.mirror.co.uk/news/uk-news/paedophiles-using-twitter-to-find-victims-1253833). Twitter issued about 200 million tweets per day (https://blog.twitter.com/2011/200million-tweets-day) in 2011, achieving 400 million tweets per day in 2013 (http://www.webpronews.com/twitter-turns-7-boasts-400m-tweets-per-day-2013-03). This is about 0.0012% while numbers inside parenthesis for female 20s correspond to the adult-adult sexual conversation samples. We provided these samples for training purposes. In the collection for early bird evaluation, we did not include any sample of this kind. The final collection was built adding a 20% of samples over the early bird dataset this time including samples from sexual predator conversations for male 20s and 30s, and samples from adult-adult conversations for female 20s.
As can be seen, there are significant differences between the two languages. More than 80% of Spanish posts are about 15-word long (e.g. greetings, especially for teenagers). On the other hand, English speakers seem to describe situations, experiences or thoughts, but in a more elaborated way. 3.2
For evaluating participants’ approaches we have used accuracy. Concretely, we calculated the ratio between the number of authors correctly predicted by the total number of authors. We calculated separately the accuracy for each language, gender, and age group. Moreover, we combined accuracy for the joint identification of age and gender. The final score used to rank the participants is the average for the combined accuracies for each language.
We also calculated the total number of correctly identified gender and age for predator samples, in order to determine what approaches are more robust to this kind of outliers. Finally, we calculated the total time needed to process the test data, in order to investigate the difficulties of processing big volumes of data in the framework of a real-world application. 4
We received 21 submissions for the task of Author Profiling and a total of 18 notebook papers: 8 long papers and 10 short papers. We present the analysis of the 18 approaches we received a description of.
Pre-processing . Only few participants preprocessed the data. Various participants
[23][20][19][32][24] cleaned HTML to obtain plain text, one participant [
Features . Many participants [17] [
Different content features (e.g. Latent Semantic Analysis, bag of words, TF-IDF,
dictionary-based words, topic-based words, entropy-based words, and so on) were also
used by many participants [29] [23] [17] [31] [
A different approach based on information retrieval was presented by one
participant [32]. In such approach, the text to be identified was used as a query for a
search engine. One participant [
Classification approaches . All the approaches used supervised machine learning
methods. The vast majority of them [28] [23] [31] [11] [32] used decision trees. Three
approaches [17] [
We divided the evaluation in two steps, an early bird option for those who wanted to test their approaches before the final submission in order to have some feedback, and the final evaluation. There were 5 early bird submissions and 21 for final evaluation. We could not evaluate one early bird submission due to runtime errors on the TIRA6 platform. A baseline was provided in order to compare the different approaches with. This baseline was programmed as two random classifiers for each variable (gender and age group), obtaining 50% of accuracy for gender identification and 33% for age identification, and 16.5% for joint identification.
In Table 2 the performance of early bird submissions is shown. In Table 3 the final ranking for each language is presented. We show the accuracy for gender and age group and the accuracy for the joint identification. The difficulty of the task is reflected in the low values of such measure, especially for gender identification with close to the baseline. In addition, the joint identification shows a dramatic decrease in the result, highlighting the even greater difficulty of the joint identification.
It is difficult to establish a correlation between the used features in the different
approaches and the obtained results, due mainly to the amount of shared features
in all of them. It is to be noted the usage of second order representations based on
relationships between documents and profiles by the winner of the task [18] and the
use of collocations for the winner of the English task [19], features that do not seem
to be as good for Spanish (or maybe more difficult to tune). Stylistic and content
features were used for the vast majority of approaches and the obtained values for
accuracy show results in different positions of the ranking. POS features were used in
five different approaches, e.g. by systems in the first position for English [19] and in
the first position for the Spanish [28], with values under the median of the ranking for
the rest of the approaches. Such features seem to improve the performance on the task.
Readability is another feature widely used for the vast majority of the approaches. We
can compare the performance of this feature with the rest because there is an approach
[11] based only on such feature, achieving the 8th position in English and the 13th in
Spanish. Except one approach [19], those which used n-gram features did not achieve
very good results, all of them over the median of the ranking. The use of sentiment
words [23] and emotion words [
Regarding employing some kind of preprocessing, it is interesting that except two cases [19] [17] the rest get worse performance, although it may be probably due to the features used not to the preprocessing itself.
In Table 4 the identification of fake profiles for sexual predators is shown. The first group of columns shows the number of correctly identified profiles for adult-adult sexual conversations and the second group shows the number of correctly identified fake profiles for sexual predators. In brackets the ratio is shown.
The vast majority of participants identified correctly cases of adult-adult sexual
conversations but what is more surprising is that all the participants identified the
right age and gender of many predator samples. At least 7 participants identified more
than 50% of such cases, 10 participants identified gender for more than 95% of the
cases and 7 participants identified age for more than 50% of them. Best results were
obtained by 3 participants who combined content and stylistic features [
Finally, in Table 5 we show the time each participant needed to finish the task, reversely ordered by runtime. Runtime is shown in milliseconds. The differences between the fastest (10.26 minutes) [11] and the slowest (11.78 days) [31] is enormous. The fastest [11] approached the task only with the readability features, obtaining the 8th position in English and 13th in Spanish. The slowest [31] approached the task with content features, obtaining the 3rd. position in English and 21th in Spanish. The vast majority of approaches took a few hours. The slowest participants used collocations [19], POS [17], n-grams [20] and performed preprocessing such as html removal [20] [19], detection of chatbots [19] and Principal Component Analysis [17]. 6
In this paper we present the results of the 1st International Author Profiling Task at PAN-2013 within CLEF-2013. Given a large and realistic collection of blog posts and chat logs, the 21 participants of the task had to identify gender and age of anonymous authors.
Participants used several different features to approach the problem, being able to be grouped into content-based (bag of words, named entities, dictionary words, slang words, contractions, sentiment words, emotion words, and so on), stylistic-based (frequencies, punctuations, POS, HTML use, readability measures and many different statistics), n-grams based, IR-based and collocations-based. Results show the difficulty of the task, mainly for the gender identification and for the joint identification of gender and age.
We introduced some conversations from sexual predators in order to check the robustness of the approaches, and we were pleasantly surprised by the high amount of such cases correctly identified by all the participants.
The author profiling task @PAN-2013 was an activity of the WIQ-EI IRSES project (Grant No. 269180) within the FP 7 Marie Curie People Framework of the European Commission. We want to thank the Forensic Lab of the Universitat Pompeu Fabra Barcelona for sponsoring the award for the winner team. The work of the first author was partially funded by Autoritas Consulting SA and by Ministerio de Economía y Competitividad de España under grant ECOPORTUNITY IPT-2012-1220-430000. The work of the second author was in the framework the DIANA-APPLICATIONS-Finding Hidden Knowledge in Texts: Applications (TIN2012-38603-C02-01) project, and the VLC/CAMPUS Microcluster on Multimodal Interaction in Intelligent Systems. The work of fifth author was funded in part by the Swiss National Science Foundation (SNF) project "Mining Conversational Content for Topic Modelling and Author Identification (ChatMiner)" under grant number 200021_130208. [10] Pamela Forner, Roberto Navigli, and Dan Tufis, editors. CLEF 2013 Evaluation Labs and Workshop – Working Notes Papers, 23-26 September, Valencia, Spain, 2013. [11] Lee Gillam. Readability for author profiling?—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [12] Sumit Goswami, Sudeshna Sarkar, and Mayur Rustagi. Stylometric analysis of bloggers’ age and gender. In Eytan Adar, Matthew Hurst, Tim Finin, Natalie S. Glance, Nicolas Nicolov, and Belle L. Tseng, editors, ICWSM. The AAAI Press, 2009. [13] Janet Holmes and Miriam Meyerhoff. The Handbook of Language and Gender.
Blackwell Handbooks in Linguistics. Wiley, 2003. ISBN 9780631225027. [14] Giacomo Inches and Fabio Crestani. Overview of the International Sexual Predator Identification Competition at PAN-2012. In Pamela Forner, Jussi Karlgren, and Christa Womser-Hacker, editors, CLEF 2012 Evaluation Labs and Workshop – Working Notes Papers, 17-20 September, Rome, Italy, September 2012. [15] Magdalena Jankowska, Vlado Keselj, and Evangelos Milios. CNG Text Classification for Authorship Profiling Task—Notebook for PAN at CLEF 2013.
In Forner et al. [10]. [16] Moshe Koppel, Shlomo Argamon, and Anat Rachel Shimoni. Automatically categorizing written texts by author gender, 2003. [17] Wee Yong Lim, Jonathan Goh, and Vrizlynn L. L. Thing. Content-Centric Age and Gender Profiling—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [18] A. Pastor Lopez-Monroy, Manuel Montes-Y-Gomez, Hugo Jair Escalante, Luis Villasenor-Pineda, and Esau Villatoro-Tello. INAOE’s Participation at PAN’13: Author Profiling task—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [19] Michal Meina, Karolina Brodzinska, Bartosz Celmer, Maja Czokow, Martyna Patera, Jakub Pezacki, and Mateusz Wilk. Ensemble-based Classification for Author Profiling Using Various Features—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [20] Erwan Moreau and Carl Vogel. Style-based Distance Features for Author
Profiling—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [21] Dong Nguyen, Noah A. Smith, and Carolyn P. Rosé. Author age prediction from text using linear regression. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, LaTeCH ’11, pages 115–123, Stroudsburg, PA, USA, 2011. Association for Computational Linguistics. [22] Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and Theo Meder. "how old do you think i am?"; a study of language and age in twitter. Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media, 2013. [23] Braja Gopal Patra, Somnath Banerjee, Dipankar Das, Tanik Saikh, and Sivaji Bandyopadhyay. Automatic Author Profiling Based on Linguistic and Stylistic Features—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [24] Aditya Pavan, Aditya Mogadala, and Vasudeva Varma. Author Profiling Using LDA and Maximum Entropy—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [25] Claudia Peersman, Walter Daelemans, and Leona Van Vaerenbergh. Predicting age and gender in online social networks. In Proceedings of the 3rd international workshop on Search and mining user-generated contents, SMUC ’11, pages 37–44, New York, NY, USA, 2011. ACM. [26] James W. Pennebaker. The Secret Life of Pronouns: What Our Words Say About
Us. Bloomsbury USA, 2013. ISBN 9781608194964. [27] James W. Pennebaker, Mathias R. Mehl, and Kate G. Niederhoffer.
Psychological aspects of natural language use: Our words, our selves. Annual review of psychology, 54(1):547–577, 2003. [28] K Santosh, Romil Bansal, Mihir Shekhar, and Vasudeva Varma. Author Profiling: Predicting Age and Gender from Blogs—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [29] Upendra Sapkota, Thamar Solorio, Manuel Montes-Y-Gomez, and Gabriela Ramirez-De-La-Rosa. Author Profiling for English and Spanish Text—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [30] Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W. Pennebaker.
Effects of age and gender on blogging. In AAAI Spring Symposium:
Computational Approaches to Analyzing Weblogs, pages 199–205. AAAI, 2006. [31] Mechti Seifeddine, Jaoua Maher, and Hadrich Belghith Lamia. Author Profiling Using Style-based Features—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [32] Edson Weren, Viviane P. Moreira, and Jose Oliveira. Using Simple Content Features for the Author Profiling Task—Notebook for PAN at CLEF 2013. In Forner et al. [10]. [33] Cathy Zhang and Pengyu Zhang. Predicting gender from blog posts. Technical report, Technical Report. University of Massachusetts Amherst, USA, 2010.