We describe the submission of the So a University team for the Author Pro ling Task, part of the PAN 2015 Challenge. Given a set of writing samples by the same person, the task asks to predict some demographical information such as age and gender, as well as the personality type of that person. We experimented with SVM classi ers using variety of features extracted from publicly available resources, achieving the second-best score for Spanish out of 21 submissions, and the sixthbest for English out of 22 submissions.
Social media applications such as Facebook and Twitter have hundreds of millions of users who share information at an unprecedented scale. Naturally, this has attracted business and research interest from various elds including marketing, political science, and social studies, among others. Initially, the primary research and practical interest was in the opinion and the sentiment users express, e.g., towards products such as iPhone6, policies such as ObamaCare, and events such as Pope's visit to Palestine.
Soon, companies realized that knowing the overall public opinion towards their products was not enough; for marketing purposes, it was also important to be able to break this opinion by demographic factors such as gender and age. Similarly, public policy and security experts wanted to further know the native language and some personality traits of the users expressing particular opinions. The personality traits were also important for human resources experts when considering to hire somebody in a team: they wanted to make sure the person's personality would t well in the target team.
While some demographic information was sometimes directly extractable from the public user pro les, there was no guarantee it was correct and upto-date, which motivated research in trying to predict it automatically from the text of the messages posted by a given user.
Early work on detecting sentiment focused on newswire text [
A similar research trend followed with respect to author pro ling. While
there were several publications that were trying to predict some
demographical information such as gender, age, and native language [
Below we discuss the participation of the So a University team, registered
as kiprov15, in the 2015 edition of the task, which ran as part of the PAN
2015 [
The task was o ered in English, Spanish, Dutch and Italian, but we participated with a system for the rst two languages only. We experimented with SVM classi ers using variety of features extracted from publicly available resources, achieving the second-best score for Spanish out of 21 submissions, and the sixth-best for English out of 22 submissions.
We built our system from scratch, as a M.Sc. class project; our research, the lessons we learned and some observations about the topic and potentially relevant references, datasets, and resources are presented in the next sections. We should note that we have saved a lot of time by reusing the GATE infrastructure and by performing feature extraction on top of it. We focused most of our e orts on feature engineering: we implemented some previously-proposed features, and we further analyzed the training data in an attempt to design some new ones.
The remainder of this paper is organized as follows: Section 2 gives an overview of our approach, including a description of the preprocessing, the features, and the learning algorithm we used. Section 3 presents our experimental setup and the o cial results our system achieved. Section 4 discusses our results and provides some deeper analysis. Finally, Section 5 concludes and points to possible directions for future work. 2
We use the GATE framework [
We integrated a pipeline of various resources for text analysis that are already
available in GATE such as a Twitter-speci c tokenizer [
Lexicon Features. For English, we used several lexicons, both
manuallycrafted and automatically generated:
{ NRC Hashtag Emotion Lexicon [
For each lexicon, we found in the tweets the terms that were listed in it, and then we added as features the total terms count, normalized by the total number of tweets. We instantiated separate features for the di erent lexicons.
For the Hashtag Emotion Lexicon, we also calculated the total score of all matching terms for each emotion type: anticipation, fear, anger, trust, surprise, sadness, joy, and disgust.
For the World Well-Being Project Personality Lexicon, we calculated a total of 30 features, i.e., the following six features for each of the big ve types (extroverted, stable, agreeable, conscientious, and open): 5 https://opennlp.apache.org/ 6 http://www.wdyl.com/ { Total positive terms score and count; { Total negative terms score and count; { Total terms score and count.
Twitter-speci c Features. We used the following Twitter-speci c features: { Letter case: the number of lower-case, all-caps, and mix-case words; { Hashtags: the number of hashtags; { URLs: the number of URLs posted; { Retweets: the number of retweets; { User mentions: the number of mentions of users using the pattern @username; { User mentions start: the number of tweets starting with a user mention; { Picture share: the number of shared pictures using the pattern [pic]. This feature was eventually dropped because there were too few users sharing this type of content, and thus using it did not yield improvements.
All of the above counts are normalized by the total number of available tweets
for the target user; so they could be viewed as \average number per tweet".
Orthographic Features. We used the following orthographic features:
{ Elonged words: the number of words with a sequence of more than two
identical characters;
{ Average sentence length: the average length of a sentence.
Term-level Features. We used the following term-level features:
{ n-grams: presence and count of unigrams and bigrams. This feature helps
to nd similar users based on their vocabulary overlap. For tokenizing the
text, we used the GATE Twitter-speci c tokenizer, which is aware of URLs,
emoticons, Twitter tags, etc.
{ Vocabulary size: the number of di erent words used by a user in all his
tweets.
{ POS tagging: We used a specialized POS tagger for tweets in English,
TwitIE, which is available in GATE [
We used the above features and support vector machines (SVM) as implemented in LibSVM. We trained a separate classi er for each language and for each target category: one for age, one for gender. As the ve personality traits (extroverted, stable, agreeable, conscientious, and open) were real values, we predicted them using support vector regression. For both classi cation and regression, we used the features as they were generated in the feature-extraction phase without any further scaling or normalization, as most of them were in reasonable ranges, e.g., because they were already normalized by the number of tweets.
Since the challenge contained documents from di erent languages and topics, we aimed to avoid as much as possible the use of language-speci c markers. Note that most of our features are token-based.
For training, we used linear kernels, which are known to be su cient for text classi cation: as we had a very large number of unigrams and bigrams, there was no need to use a kernel in order to make the two classes linearly separable. 3
3.1
During development, we trained on training datasets provided by the organizers: { pan15-author-pro ling-training-dataset-english-2015-03-02 { pan15-author-pro ling-training-dataset-spanish-2015-03-02
{ pan15-author-pro ling-test-dataset2-english-2015-04-23 { pan15-author-pro ling-test-dataset2-spanish-2015-04-23
While developing the system, we randomly selected 15 percent of our training data to try out new features and to measure progress. For the o cial submission, we trained our model on all provided training data, and we tested on the test datasets. 3.2
O
cial Results
We were ranked 6th out of 22 submissions for English, and 2nd out of 21
submissions for Spanish. A summary of our o cial results is shown in Table 1, where
we have included our GLOBAL and RMSE scores, as well as our ranking among
the other participants in PAN-2015 Author pro ling task[
Although we also experimented with Dutch and Italian, we did not have enough time to ne-tune our systems for them, and thus we eventually decided not to submit results for them. In our experiments above, we tried to solve the author pro ling task as a standard n-gram based classi cation/regression problem, treating the target classes as nominal (age, gender) or as real-valued (the big ve traits).
We started with the same types of features for both language, only adding dictionaries for English. Thus, our general method is in principle languageindependent and applicable to any language for which data is available.
We further did a lot of extra feature engineering and selection for English, but we did not have time to do anything special in that respect for Spanish. Yet, we performed better for Spanish, which was a surprise for us.
Over the development of our nal model, we tested various combinations of
features, and we eventually excluded groups of features that did not actually
improve the performance. One such feature group were n-grams with length
more than 2; our observation is that 3-grams and 4-grams, although helpful
for sentiment analysis [
Most of the orthographic features and dictionaries did not improve signi cantly the big ve score, although improving the age and gender accuracy. Removing all dictionaries drops the average age and gender F1 score by 2.5 percent while not changing any of the big ve RMSEs by more than 0.005.
We further tested the assumption that advertising and posting titles and news articles introduces noise regarding personality detection by removing all upper-case words from the n-grams. However, this worsened the RMSE by 0.02 on average.
We believe the LIWC [
We have described the system built by So a University's kiprov15 team for the PAN-2015 Author Pro ling task, which was ranked 2nd in Spanish and 6th in English, according to the o cial GLOBAL score.
We have made some interesting observations about the impact of the di erent features. Among the best feature groups were POS-tag counts and word unigrams and bigrams. These had the most sustainable performance over the provided test datasets.
Even though we managed to achieve good ranking for the two languages we made submissions for, we feel that there is a lot more to gain. For example, we would like to try using di erent word clusters, thesaurus, the Linguistic Inquiry and Word Count (LIWC) lexicons for di erent languages, named entity recognition and normalization, e.g., locations, dates, numbers, money, person names, etc.
In addition to adding extra features, we are interested in using the social media to generate more training examples. In particular, we would like to explore the way personality is expressed in Twitter and whether it is dependent upon language usage in general. For instance, would the model improve if we have di erent training sets for English-speaking users in USA vs. UK vs. Canada vs. Australia vs. India, etc. 6