This paper describes the methods employed to solve the Author Pro ling task at PAN-2015. The main goal was to test the use of features derived from Information Retrieval to identify the personality traits of the author of a given text. This paper describes the features, the classi cation algorithms employed, and how the experiments were run. Also, I provide a comparative analysis of my results compared to those of other groups.
Author Pro ling, which has a growing importance in applications in forensics,
marketing, and security[
This paper reports on the my participation at the third edition of the
Author Pro ling task, organized in the scope of the PAN Workshop series, which
is collocated with CLEF2015. More details about the task and the workshop
can be found in the overview paper [
The underlying assumption was that authors from the same gender, age group or personality traits tend to use similar terms and that the distribution of these terms would be di erent across genders, age groups and personality traits. To implement this notion, all conversations were indexed using an Information Retrieval engine and then, the conversation to be classi ed was treated as a query. The idea is that the conversations retrieved (i.e., the most similar to the query) are the ones from the same gender, age group, and personality traits.
The training dataset was composed of conversations (XML les) about various topics grouped by author. Conversations were in English, Spanish, Italian, and Dutch and were annotated with the gender, age group, and the personality traits of the author. A complete description of the dataset may be found in http://pan.webis.de/. 2.1
Features The texts from each author, or the documents, were represented by a set of 288 features (or attributes).
The complete set of texts was indexed by an Information Retrieval (IR) System in a manner similar to that used in [4{6]. Then, the text that to be classi ed was used as a query and the k most similar texts were retrieved. The ranking is given by the Cosine or Okapi metrics as explained below. Cosine These features are computed as an aggregation function over the topk results for each age, gender, and personality trait obtained in response to a query composed by the keywords in the text to be classi ed. Three types of aggregation functions were tested, namely: count, sum, and average. For this featureset, queries and documents were compared using the cosine similarity (Eq. 1). For example, if we retrieve 10 documents in response to a query composed by the keywords in q, and 5 of the retrieved documents were in the 18-24 age group, then the value for 18-24 cosine avg is the the average of the 5 cosine scores for this class. Similarly, 18-24 cosine sum is the summation of such scores, and 18-24 cosine count simply counts how many retrieved documents fall into the 18-24 cosine count category.
COSIN E = (c; s) c:q jcjjqj where !c and !q are the vectors for the document and the query, respectively. The vectors are composed of tfi;c idfi weights where tfi;c is the frequency of term i in document c, and IDFi = log nN(i) where N is the total number of documents in the collection, and n(i) is the number of documents containing i. Okapi Similar to the previous, these features compute an aggregation function (average, sum, and count) over the retrieved results from each gender, age, and personality traits group that appeared in the top-k ranks for the query composed by the keywords in the document. For this featureset, queries and documents were compared using the Okapi BM25 score (Eq. 2).
n BM 25(c; q) = X IDFi i=1
tfi;c (k1 + 1) tfi;c + k1(1 b + b ajvDgdjl ) where tfi;c and IDFi are as in Eq. 1 jdj is the length (in words) of document c, avgdl is the average document length in the collection, k1 and b are parameters (1) (2) that tune the importance of the presence of each term in the query and the length of the text. In my experiments, i used k1 = 1:2 and b = 0:75. 2.2
Experiments The steps taken to process the datasets and run the experiments were the following: 1. Pre-process the conversation in the training data to tokenize (only during testing, stemming and stopword removal was performed but without significant gains). 2. Use each conversation as queries. 3. Index 100% of the pre-processed conversations with a retrieval engine. Zettair1, which is a compact and fast search engine developed by RMIT University (Australia), was used for indexing and querying. Zettair implements several methods for ranking documents in response to queries and calculates cosine and Okapi BM25. 4. Compute the features using the results from the queries submitted to Zettair.
The top-10 scoring conversations were retrieved.
5. Train the classi ers and generate the models. Weka [
Training the Classi ers Twenty-six classi ers are necessary, since there are four languages and seven dimensions in each (age [only English and Spanish], gender, and personality traits [extroverted, stable, agreeable, conscientious, and open]). All results in this section refer to experiments run on the training data only. The predictions of the classi ers were compared two settings (i) using all 288 features an (ii) using just a subset of 6 to 22 features (produced by BestFirst subset evaluator).
Figure 1 shows the results comparing the accuracy of the runs that use all features and the runs that use just the subset. Using the subsets had advantages in nearly all cases. The only exception was for age. This can be con rmed by Table 1, which shows results of paired t-tests that assess the signi cance of the di erence between the runs that use all features and the runs that use a subset only. For the majority of the learning algorithms, the results of all runs are very close, with a slight advantage in favor of the runs with the selected subset of attributes.
Figure 2 shows the most accurate classi ers on the training data grouped by language. We can see that some languages had better performance than 1 http://www.seg.rmit.edu.au/zettair/ others. While Italian had the best scores, English had the lowest. This di erence could be explained by the fact that Italian has a more diverse morphology and a greater vocabulary compared to English and this may provide the classi er with more distinctive features. Regarding the choice of classi er, we can see that di erent languages had di erent classi ers as best performers. Classi cation via Regression, Random Committee, and Rotation Forest were among the top 5 in two cases each.
Figure 3 shows the most accurate classi ers for Age and Gender prediction. For age, we notice that a number of algorithms achieved similar results (around 0.8). For gender, RBFNetwork was the best.
Figure 4 shows the best classi ers for modelling personality traits. The Multilayer Perceptron is among the top performers in three out of ve traits. The 1.00 0.95 0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50 1.00 0.95 0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50 1.00 0.95 0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.921 0.895 0.895 0.684 0.658 0.853 0.824 0.824 0.824 0.824 (c) Italian (d)
most notable cases in which there were large classi ers were for Conscientious and Open. di erences in the accuracies of the
A pairwise comparison of the accuracies of the twenty-two teams which participated on the Author Pro ling task was held, considering age, gender, personality traits and language. In this work, for the systems to be signi cantly di erent from each other, they had to have p < 0.05.
As a result, system proposed in study is not signi cantly di erent from systems that scored best, considering as an example the small set of training data used: English, Spanish, Italian, and Dutch - 152, 100, 38 and 34 les, respectively.
Comparing the results on the training and test datasets, a drop of about ten percentage points was observed. Overall results on the training data were 0.8171 and on the test data the nal score was 0.7223. 4
In this paper, i presented an empirical evaluation of a number of features and learning algorithms for the task of identifying author pro les. More speci cally, the task here is, for a given text, to identify gender, age group and personality traits of its author.
The goal was to validate the use of Information Retrieval-based features to identify personality traits. The results show that they are suitable to the task. Acknowledgments. I thank to Viviane Pereira Moreira for their help in the nal revision of this paper.