In this paper we go through our approach at solving the PAN Author Profiling task. We introduce a novel way of computing the type/token ratio of an author and show that, although strong correlations have been observed between high extroversion and low type/token ratios in the past, this ratio is not necessarily a strong indicator of extroversion. Since the text of a person is influenced by all 7 features (gender, age, and big five personality traits) that are required to be automatically identified in this task, we used this ratio, along with Term frequency-Inverse document frequency (tf-idf ) matrices, in all 7 subtasks and all 4 corpora and obtained good results.
While the importance of age and gender is a more familiar notion in user or author
profiling, automatic personality detection is a relatively new task [10]. Since many
correlations between personality traits and consumer preferences have been reported ([
For features, we tried several approaches, but eventually settled on using two: first, the tf-idf matrix at character level, with various n-gram ranges and parameter tuning, depending on the language and subtask, and second, the type/token ratio of a user or verbosity rate. These two features were combined using scikit-learn’s FeatureUnion().
The tf-idf scores were extracted using scikit-learn’s TfidfVectorizer(). This vectorizer was applied either on all tweets of one user put together, or on each tweet pertaning to one user. More precisely, in the sparse matrix created by the TfidfVectorizer(), the columns represented, in both cases, all the character n-grams extracted from all the tweets in one of the four datasets, while each line represented either all tweets of one user concatenated, or one tweet of a user. Our cross-validation results, which will be presented further, showed that the former method was consistently more appropriate for the classfication tasks and the latter, for regression. An intuitive answer would be that gender and age specific features change less often, while personality traits may influence each tweet.
The verbosity ratio of a user was only inspired by the type/token ratio and is not one per se, since distinguishing between a linguistic manifestation of a conceptual type (bicycle in sentence 1.a), and its token (bicycle in sentence 1.b), implies deep semantic analysis which is far from trivial with today’s tools in Natural Language Processing. (1) Type/token distinction a. The bicycle is more popular now. b. The bicycle is in the garage.
What we did to echo the idea of a type/token ratio was to compute, for each user, the
ratio between the total number of unique stems and the total number of words used
after applying stemming. From this ratio we excluded stop words. Stemming was done
using the nltk implementation of the Snowball algorithm since it offered a version for
each of the four languages present in this year’s task. For stopwords lists, we used
nltk.corpus.stopwords. The motivation for using this feature was the often observed
correlation between extroversion and type/token ratio [
However, our preliminary analysis, by computing both Pearson and Spearman correlation coefficients on verbosity ratios versus personality scores, showed no clear-cut linear relationships. The fact that Spearman correlation coefficient was better than the Pearson correlation coefficient only goes to show that the relationship is rather a monotonic one than linear. Below are the top three statistically significant correlation scores on all corpora and all personality scores, computed with python’s scipy.stats package. The plots were drawn using seaborn python module.
For the Dutch corpus (figure 1 on page 3), we found that there was a -0.46 Pearson correlation with a p-value < 0.001 and -0.49 Spearman rank correlation with a p-value < 0.001 between verbosity ratios and openness scores. Also, given a verbosity ratio, males tended to have higher openness scores than females.
Also for the Dutch corpus (figure 2 on page 4), we found that there was a -0.34 Pearson correlation with a p-value < 0.001 and -0.45 Spearman rank correlation with a p-value < 0.001 between verbosity ratios and neuroticism (stable) scores. Regarding gender separation, males tended to be more stable at a given verbosity ratio.
For the Italian corpus (figure 3 on page 4), we found that there was a -0.33 Pearson correlation with a p-value < 0.001 and -0.40 Spearman rank correlation with a p-value < 0.001 between verbosity ratios and agreeableness scores. Apparently, on average, Italian females were more agreeable than Italian males at a given verbosity ratio.
We also present the results of verbosity ratios for the classification tasks.
In the English training corpus (table 1 on page 3), across all age groups, males had a slightly higher verbosity ratio than females. We also observed that verbosity ratios increased slightly with age, across both genders.
The difference betweem male and female verbosity ratios was minimal on the Spanish training corpus (table 2 on page 5). However, we observed a larger difference when it came to age groups, with the highest verbosity ratio being for age group 25-34 (with a median of 0.70) and the lowest for age group 35-49 (with a median of 0.67).
A similar difference we also observed on the Italian corpus (table 3 on page 5). Female users tended to have a lower verbosity ratio (with a median of 0.69), while males had a median verbosity ratio of 0.72.
As for the Dutch training corpus (table 4 on page 5), the difference between male and female verbosity ratios was again minimal, with a difference between medians of under 2 percentage points.
Given these inconclusive findings, we decided to use a combination of tf-idf on character n-grams with verbosity scores, which improved cross-validation results over models based on the same features taken separately.
Comparing the cross-validation results to the final test results, we can see signs of overfitting only in some of the cases in which we used relatively more features. Overall, our models generalized well when the number of features was smaller. LinearSVC() and Ridge() allowed us to use sparse matrices, which meant we did not have to transform to dense matrices (which would have occupied too much memory) or reduce dimensions (which is a computationally expensive operation).
As we stated before, we concatenated each user’s tweets for the classification tasks, while for the regression tasks we used each individual tweet. This led, on average, to a smaller vocabulary for the classification tasks.
On the English corpus (table 5 on page 6), our system over-fitted slightly on the gender, stable and open tasks. On the Spanish corpus (table 6 on page 6), our system over-fitted slightly on extroverted and conscientious tasks. On the Dutch corpus (table 7 on page 7), our system over-fitted slightly on extroverted and conscientious tasks.
The biggest difference between cross-validation and test results was on the Italian corpus (table 8 on page 7), where our system over-fitted on all tasks, but extroverted. The biggest overfit was for the gender task, with a difference of 15 percentage points between cross-validation results and test corpus results. Based on our results, we conclude that a combination of simple features like tf-idf and verbosity ratios obtain reasonable results that generalize well. Comparing our approach across all corpora, we found that this solution worked best as a regressor for the Dutch corpus and as a classifier for the Spanish corpus. We found that the best tf-idf features are those at character-level ngrams, with ngram ranges of up to 2; 6. Above this threshold, the system seemed to overfit. We also found that there is at best a monotone relationship between verbosity ratios and personality scores. Nevertheless, combining them with other, many-dimensional features, like tf-idf matrices, improves results and generalizes well. 6. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (Oct 2011) 7. Quercia, D., Kosinski, M., Stillwell, D., Crowcroft, J.: Our twitter profiles, our selves: Predicting personality with twitter. In: Proceedings of the Third International Conference on Social Computing (SocialCom) and the Third International Conference on Privacy, Security, Risk and Trust (PASSAT). pp. 180–185. IEEE (Oct 2011), http: //ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6113111&tag=1 8. Rangel, F., Rosso, P., Potthast, M., Stein, B., Daelemans, W.: Overview of the 3rd author profiling task at pan 2015. In: Cappellato, L., Ferro, N., Gareth, J., San Juan, E. (eds.) CLEF 2015 Labs and Workshops, Notebook Papers. CEUR-WS.org (2015) 9. Roozmand, O., Ghasem-Aghaee, N., Nematbakhsh, M., Baraani, A., Hofstede, G.: Computational modeling of uncertainty avoidance in consumer behavior. International Journal of Research and Reviews in Computer Science pp. 18–26 (April 2011) 10. Vinciarelli, A., Mohammadi, G.: A survey of personality computing. T. Affective Computing 5(3), 273–291 (2014), http://dx.doi.org/10.1109/TAFFC.2014.2330816