This paper describes our participation in the 9th PAN evaluation lab in the author pro ling task. The proposed approach relies on the extraction of stylistic, lexicon and corpus-based features, which were combined with a logistic classi er. These three sets of features contain pairwise intersections and even some features that belong to all categories. A comprehensive comparison of the contribution of several feature subsets is presented. In particular, a set of features based on Bayesian inference provided the most important contribution. We developed our system in the Spanish training corpus, once developed it was used, with minor changes, for the English documents, too. The proposed system was ranked 6th in the o cial ranking for Spanish documents among 17 submitted systems. This result shows that our approach is meaningful and competitive for predicting demographics from text.
Due to the large amount of textual information on the internet, it is now possible
to carry out di erent research problems about the texts, either in connection with
their authors, the registers involved, and the varieties of texts, among others. In
the framework of the international conference, CLEF 2013 [
In order to identify the author pro le from written texts, the use of stylistic
and content features is a common practice [
The content-based features and the mechanism used for its selection also vary
from one author to another. The extraction of corpus words for its comparison
between the classes of interest [
In our study, each document is represented in a vector space, where each
feature adds one unit to the dimension, including stylistic and lexicon-based
attributes, relevant to distinguish the gender and age-range of the authors.
Furthermore, we explore a new subset of features that involve the use of some
statistics measures (corpus statistics features). These three subsets of features,
as shown in Fig. 1, are intersected, and therefore some of them are located in
more than one class. We used a machine-learning approach to build classi
cation models to produce the predictions. The details of the task, documents and
evaluation are presented in [
In the remainder of the paper, we begin with a description of the features (Section 2) and of the system used in this campaign (Section 3). Section 4 focuses on the main results of our work, while the nal sections present the discussion and the conclusion that can be drawn from this study. 2
The set of features extracted from each text contains components of one or more of the following categories: `S' (style), `C' (corpus statistics) and `L' (lexicon). Fig. 1 shows a Venn diagram depicting the number of features extracted for each category combination. In the following subsections these features are described and the labels in Fig. 1 are used to clarify their categories, i.e. `SL' for Style and Lexicon categories. Besides, the features in the `C' category are presented separately by their supervised or unsupervised nature. This set of 6 features is built from statistics gathered from the training corpus, ignoring the demographic categories age and gender associated to each document. These corpus-based statistics use collection and document frequencies of the words in the entire training English and Spanish collections. The motivation for the use of document frequencies is to prevent very long documents from generating biased results.
IR features (2 `C' features). Using the tf.idf term weighting approach used P idf(w) in the information retrieval eld we obtained two features: IDF (d) = w2d len(d)
P tf(w;d) idf(w) and T F:IDF (d) = w2d len(d) , where len(d) is the number of words in the document d, idf (w) = log dfD(w) , df (w) is the number of documents where the word w occurs, D is the number of documents in the corpus and tf (w; d) is the number of times that w occurs in the document d. Tf.idf weight measures the informative character (for retrieval purpose) of the words given a particular document and the whole corpus. Thus, these features measure the density of that notion for each document.
Pd(w)
v2d Qd(d) = P Pd ln w2d
QPdd((ww)) .
Entropy (2 `C' features) measures the amount of information in a set of random variables, i.e. occurrences of words in a document. The probability of occurrence of a word is given by Pf (w) fM(w) where f (w) is the number of occurrences of w in the corpus, and M is the total number of words in the corpus. Alternatively, these probabilities can be obtained from document frequencies by Pdf (w) dfD(w) . Thus, the entropy of a document is given by Hf (d) = P Pf (w) log2 (Pf (w)). Hdf (d) is obtained with the same formula w2d but using Pdf (w).
Kullback-Leibler (KL)-divergence (1 `C' feature) measures the information loss when a document probability distribution Q is used to approximate the \true" corpus distribution P . The probability Q for a word in a document is given by Qd(w) dlfe(nw(;dd)) . The corpus probability distribution P is given by
Pf(fw()v) . Thus, the KL-divergence of a document is given by Pd k log2 (Qd(w)).
Cross entropy (1 `C' feature), similarly to the KL-divergence, compares P and Q measuring the ability of the former for predicting the latter. The cross entropy of a document is given by the following expression: H(Pd; Qd) = P Pd(w) w2d 2.2
Unlike the previous set of features, this collection was built taking into account the age and gender of the authors of the training documents.
Gender score (2 `C' features). We developed the gender score (GS), a measure that aggregates the di erences between the probabilities of a word w estimated in the corpus of documents written by males and females. Let Pf (wjmale) fmale(w) be the probability of w estimated only in the corpus written by males, Mmale where fmale(w) is the number of occurrences of w in the \male" subset of the corpus and Mmale is the total number of words in that same subset. Pf (wjf emale) is calculated analogously. Thus, GS is given by: GSf (d) = P (Pf (wjmale) Pf (wjf emale)). GSdf is obtained using Pdf (wjmale) w2d dfmale(w) where dfmale(w) is the number of documents written by males where w
Dmale occurred and Dmale is the total number of documents written by males. Again, Pdf (wjf emale) is calculated analogously.
Bayes score (10 `C' features). We proposed a score for each one of the ve demographic categories male, female, 10's, 20's and 30's using the Bayes theorem. These scores are given by the expression BSf;cat:(d) = P Pf (cat:jw) w2d having cat: 2 fmale, female, 10s, 20s, 30sg, Pf (cat:jw) = Pf (wjcat:) P (cat:)) and Pf (w) P (cat:) DDcat: . Similarly, BSdf;cat: is obtained analogously but using probabilities subscripted by df . This way, we obtained 10 features from the 5 categories (cat:) and the 2 types of probabilities Pf and Pdf .
Supervised KL-divergence (5 `C' features) can also be used to build supervised attributes. In this case, it measures the information loss when Qd is used to predict the probability distribution of the subset of documents written by authors of the demographic category cat.This probability distribution is given by Pd:cat(w)
Pf(fw(jvcjacta:)t:) , and the KL divergences are given by P jjQcat:(d) = v2d P Pd:cat ln PdQ:cda(tw(w)) : w2d Supervised cross entropy (5 `C' features). As it can be expected, crossentropy can also be calculated based on probability distributions of each individual demographic category. In this case, it measures how predictable Q is when using Pd:cat:. The equation to do so is H(P; Q)cat = P Pd:cat(w) log2 (Qd(w)). w2d Supervised lexicon extraction using T-test (20 `CL' features). The Students t-test, frequently used in text mining, allows us to determine the most characteristic words of each demographic category by measuring the signi cance of the di erences in the occurrences of the words on each category (gender) or between the category and the whole corpus (age). We used critical values in the T-table to build ve lexicons, one for each gender and age range. This word lists contain the words that have an absolute T-value greater than 2 for the given category, which are equivalent to around the three percent most relevant words of each demographic group. The construction method is di erent for gender and age categories. However, the following de nitions are used in both cases: S = pPf (w) Pf (w)2 and Scat: = pPf (wjcat) Pf (wjcat)2. In the gender T-function, as in the gender score, values greater than zero are characteristic of males and those less than zero are more often used by female. This value is given by Tg = Pf (wjmale) Pf (wjfemale) .
q Smale2 + Dfemale
Sfemale2
Dmale
Since the comparison cannot be made the same way when having three categories, a T-function was used for each age range, comparing the category with the general corpus. This function is given by the following equation, where cat: can only be an age range category: Tcat: = Pf (wjmale) Pf (w) . This procedure q Scat:2 + SD2
Dcat: provides 5 lexicons of words characterizing each demographic category. 2.3
The 5 lexicons built using T-test, as well as other pre-fabricated lexicons are
used to generate 4 features for each one:
Lexical density (1 feature) is the ratio of content words to the total number
of words. Ure, according to Johansson, introduced it in order to distinguish
between words with lexical properties, and those without [
Weighted density (1 feature). The Spanish Emotion Lexicon [
The used lexicons and their sources are listed in Table 1. Manual preprocessing was applied to some lexicons by deduplicating and adding the gender variation for some Spanish words. Twenty `CSL' features result from the 5 Ttest lexicons, the two entropy-related attributes of bad words, Internet and stopwords add 6 `CSL' features. Similarly, their densities add six `SL' features. For the remaining lexicons, their entropies generate `CL' features, and their densities `L' features. This generates on 22 `L' and 22 `CL' features for Spanish, and 8 `L' and 8 `CL' features for English. The stylistic features are classi ed in three subsets: character-based, wordbased, and syntactic features. The character-based features contain 50 features, such as character density, uppercase or lowercase characters, letters, and special characters like the use of asterisk. All of them, except the letter count, have been used by other researchers for identifying the pro le of the author. The wordbased features include 11 measures for vocabulary richness, the length of words and density of hapax legomenas, dislegomenas, 3-legomenas until 5-legomenas. Syntactic features involve 9 attributes related to the regular punctuation such as colon, semicolon and question marks, among others. We also considered as stylistic features those obtained from lexicons such as stopwords, Internet and bad words. 3
The submitted system was built by extracting the features described in the previous section for each one of the rst 20,000 documents in the English and Spanish training sets. That is, 166 features for English and 198 in Spanish; the di erence is due to the di erent number of lexicons used on each language. For obtaining words from the character sequences in the documents xml tags were removed. Then each consecutive sequence of characters in the English or Spanish alphabet that was delimited by space, tab, enter or any punctuation mark, produced a word.
The statistics used in the calculation of the features that contain the label C were gathered using all the documents in the training set, i.e. 236,600 documents in English and 75,900 in Spanish. For each word w in the vocabulary we obtained: f (w), fmale(w), ffemale(w), f10s(w), f20s(w), f30s(w), df (w), dfmale(w), dffemale(w), df10s(w), df20s(w) and df30s(w).
These datasets were used to train 4 logistic classi ers [
In this section the o cial results obtained by the proposed system for predicting authors age and gender in unseen documents are presented in Table 2. To assess the contribution of the di erent feature sets, additional experiments were carried out using a subset comprised of the rst 20,000 documents from the training set. Each feature subset was evaluated using 10-fold cross validation and the average of ten di erent random folds is reported. Tables 3 through 6 show the results of these experiments.
As shown in Tables 3-6, the best results for distinguishing gender were obtained in English and Spanish using all features, while the supervised attributes were better predictors for age-range. The age and gender were more appropriately identi ed using statistical features, although they were more suitable for typifying gender. The best statistic predictor in all cases was the features based on the Bayes theorem. The lexical and stylistic features were more useful to distinguish age than gender. Finally, the pre-established lists of words do not distinguish gender although they are useful for discriminating age. 6
We participated in the 9th PAN evaluation campaign with an author pro ling system based on a set of features extracted from documents that were combined with machine learning. The features were designed in such a way that each one could contain at least one of the following components: stylometry, usage of pre-fabricated lexicon and corpus statistics. We developed this system for Spanish obtaining the 6th place in the o cial results among 17 participant systems. However, the same system adapted for English (replacing Spanish lexicons) performed poorly in unseen documents.
In a comprehensive comparison of di erent features we concluded that the features that provided the larger contribution were the ones obtained from corpus statistics. Particularly, the proposed score obtained using the Bayes theorem. To the extent of our readings such (or similar) features have not been used in the past.