This paper describes the methods we have employed to solve the author profiling task at PAN-2014. Our goal was to rely mainly on features from Information Retrieval to identify the age group and the gender of the author of a given text. We describe the features, the classification algorithms employed, and how the experiments were run. Also, we provide an analysis of our results compared to other groups.
The texts from each author, or documents, were represented by a set of 64 features (or attributes), which were divided into five groups. Next, we explain each of these groups. Length These are simple features that calculate the absolute length of the text. – Number of Characters; – Number of Words; and – Number of Sentences.
Information Retrieval This is the group of features that encode our assumption that
authors from the same gender or age group tend to use similar terms and that the
distribution of these terms would be different across genders and age groups. The process
here was the same as in [
– Cosine female_cosine_sum, male_cosine_sum, female_cosine_count, male_cosine_count, female_cosine_avg, male_cosine_avg, 18-24_cosine_sum, 25-34_cosine_sum, 35-49_cosine_sum, 50-64_cosine_sum, 65-xx_cosine_sum, 18-24_cosine_count, 25-34_cosine_count, 35-49_cosine_count, 50-64_cosine_count, 65-xx_cosine_count, 18-24_cosine_avg, 25-34_cosine_avg, 35-49_cosine_avg, 50-64_cosine_avg, 65-xx_cosine_avg. These features are computed as an aggregation function over the top-k results for each age/gender group obtained in response to a query composed by the keywords in the text that we wish to classify. We tested three types of aggregation functions, namely: count, sum, and average. For this featureset, queries and documents were compared using the cosine similarity (Eq. 1). For example, if we retrieve 100 documents in response to a query composed by the keywords in q, and 50 of the retrieved documents were in the 18-24’s age group, then the value for 18-24_cosine_avg is the the average of the 50 cosine scores for this class. Similarly, 18-24_cosine_sum is the summation of such scores, and 18-24_cosine_count simply counts how many retrieved documents fall into the 18-24_cosine_count category.
cosine(c; q) = !c !q j!c jj !qj (1) where !c and !q are the vectors for the document and the query, respectively. The vectors are composed of tfi;c idfi weights where tfi;c is the frequency of term i in document c, and IDFi = log nN(i) where N is the total number of documents in the collection, and n(i) is the number of documents containing i. – Okapi BM25 female_okapi_sum, male_okapi_sum, female_okapi_count, male_okapi_count, female_okapi_avg, male_okapi_avg, 18-24_okapi_sum, 25-34_okapi_sum, 35-49_okapi_sum, 50-64_okapi_sum, 65-xx_okapi_sum, 18-24_okapi_count, 25-34_okapi_count, 35-49_okapi_count, 50-64_okapi_count, 65-xx_okapi_count, 18-24_okapi_avg, 25-34_okapi_avg, 35-49_okapi_avg, 50-64_okapi_avg, 65-xx_okapi_avg .
Similar to the previous featureset, these features compute an aggregation function (average, sum, and count) over the the retrieved results from each gender/age group that appeared in the top-k ranks for the query composed by the keywords in the document. For this featureset, queries and documents were compared using the Okapi BM25 score (Eq. 2).
BM25(c; q) = Xn IDFi tfi;c (k1 + 1) (2)
i=1 tfi;c + k1(1 b + b ajvDgdjl ) where tfi;c and IDFi are as in Eq. 1 jdj is the length (in words) of document c, avgdl is the average document length in the collection, k1 and b are parameters that tune the importance of the presence of each term in the query and the length of the text. In our experiments, we used k1 = 1:2 and b = 0:75.
Readability Readability tests indicate the comprehension difficulty of a text. – Flesch-Kincaid readability tests
We employ two tests that indicate the comprehension difficulty of a text: Flesch
Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) [
F RE = 206:835 1:015
#words #sentences 84:6 #syllables #words F KGL = 0:39
#words #sentences + 11:8 #syllables #words 15:59 (3) (4) Correctness This group of features aims at capturing the correctness of the text. – Words in the dictionary: ratio between the words from the text found in the OpenOffice US dictionary1 and the total number of words in the text. – Cleanliness: ratio between the number of characters in the preprocessed text and the number of characters in the raw text. The idea is to assess how "clean" the original text is. – Repeated Vowels: in some cases, authors use words with repeated vowels for emphasis. e.g. "I am soo tired". This group of features counts the numbers of repeated vowels (a, e, i, o, and u) in sequence within a word. – Repeated Punctuation: this features compute the number of repeated punctuation marks (i.e., commas, semi-colons, full stops, question marks, and exclamation marks) in sequence in the text.
Style – HTML tags: this feature consists in counting the number of HTML tags that indicate line breaks <br>, images <img>, and links <href>. – Diversity: this feature is calculated as the ratio between the distinct words in the text and the total number of words in the text. 1 http://extensions.openoffice.org/en/project/ english-dictionaries-apache-openoffice Corpus
Lang Twitter Twitter Blog Blog
EN ES EN
ES SocialMedia
EN SocialMedia
ES Reviews
EN In order to evaluate how discriminant each of the 64 features described in Section 2 is, we calculated their information gain with respect to the class. The five highest ranking features for each corpus and each class are shown in Table 1. The vast majority of the most discriminative features is from the IR group. Style, length, and correctness also appear, but at a much lower frequency. For Age-Blogs-EN, none of our features had a good score for information gain. Interestingly, we got the best scores for this corpus on the test data, compared to other groups.
Information gain evaluates each feature independently from each other. However,
when selecting the best group of features, we wish to avoid redundant features by
keeping features that have at the same time a high correlation with the class and a low
intercorrelation. With this aim, we used Weka’s [
Gender male_okapi_sum male_cosine_sum male_cosine_count words_in_dictionary repeated_exclamation female_cosine_sum male_cosine_count female_okapi_count repeated_e repeated_exclamation male_cosine_count 18-24_cosine_sum 35-49_cosine_count female_okapi_count FRE <img> repeated_exclamation repeated_interrogation female_cosine_sum male_cosine_avg male_okapi_count 18-24_okapi_count FKGL repeated_a repeated_i repeated_u repeated_exclamation female_cosine_sum 50-64_okapi_count 65-XX_okapi_count diversity repeated_semicolon subsets varied a lot, from one (Gender-Twitter-EN) to 16 (Age-Reviews-EN). Again, we observed that most features in the subsets are IR-based. Surprisingly, readability features (namely FKGL) appear in only two subsets for Age. Style and correctness attributes also appear in the chosen subsets. Also, we noticed that some features that were intended for age, have been selected as useful for gender and vice-versa.
We treated gender and age classification separately. Thus, the features described in the
previous section were used to train one classifier for each corpus for gender and age
resulting in 14 classifiers. We used Weka [
The preprocessing consisted basically in tokenisation, removal of tags, and escape characters. No stemming or stopword removal was performed. All training instances were used to generate the model. No attempt to remove noise was taken.
Table 3 shows our official results for both training and test corpora in terms of accuracy. It also shows which classification algorithm was used and whether all attributes or just a subset were used. Most classifiers (11 out of 14) used just the subset of attributes, as their results on the training data outperformed (or got very close to) the results using all attributes.
As expected, results on the training corpora were superior to the results on the test corpora. The biggest drop was for Age-Blog-ES as in this corpus, in which accuracy dropped by half. Interestingly, the results for three corpora were better on the test data (Age-Twitter-ES, Age-Blogs-EN, and Gender-Twitter-ES). We still need to investigate these differences further.
Corpus Twitter Twitter Blog Blog SocialMedia SocialMedia Reviews 0.15 0.1 0.05
0 -0.05 -0.1 -0.15 -0.2 age gender age gender age gender age gender age gender age gender age gender
We also analysed our results compared against the mean of all participants. These are shown in Figure 1. For 9 out of 14 cases, our results were above the mean. The case with the biggest gain was Age-Blogs-EN, in which the advantage was of 31%. In 5 runs, our results were at or below the mean. Our worst scores were for Age-Blogs-ES, in which our loss was of nearly 66%. Adding up all gains and losses, we get a positive result of 10% in relation to the average. 5
This paper describes our participation in the Author Profiling task run in PAN 2014. We used the training data to build classifiers using several machine learning algorithms. Our focus was on exploring Information Retrieval-based features. The official results show that our scores were above the mean for all participants in most cases (9 times out of 14).
Author profiling is a challenging task. Consequently, there are many possibilities for future work. As a first step, once the test data is released, we will further investigate the cases in which our system fails or succeeds in the classification. The goal is to try and establish patterns. We are also interested in testing methods for instance selection to improve our classification models. In addition, we have treated gender and age classification separately as independent problems. However, since some attributes meant to discriminate gender were found useful for age (and vice-versa), we wish to explore the influence of both types of classification into each other.
Acknowledgements: This work has been partially supported by CNPq-Brazil (478979/2012-6). We thank Anderson Kauer for his help in revising this paper. We thank Martin Potthast, Francisco Rangel, and other members of the PAN organising team for their help in getting our software to run.