We describe the approach that we submitted to the 2015 PAN competition [5] for the author profiling task1. The task consists in predicting some attributes of an author analyzing a set of his/her Twitter tweets. We consider several sets of stylometric and content features, and different decision algorithms: we use a different combination of features and decision algorithm for each language-attribute pair, hence treating it as an individual problem.
instances in the testing set are compared against the actual values and the comparison outcome is expressed in terms of accuracy for gender and age, and in terms of Rootmean-square error (RMSE) for the personality traits. 2
We chose to handle the prediction of each attribute for each language as an individual problem: in particular, we consider gender and age group prediction as 2 classification tasks and personal traits prediction as 5 regression tasks. Since we had tweets written in four languages and we had to predict age groups for those written in English and Spanish only, we hence considered 26 different problems.
We propose a machine learning approach based on a number of different stylometric and content features which are processed by one among three different decision algorithms—we used SVM and random forests as classifiers and regressors. We carried out an extensive experimental campaign for systematically assessing a large number of the possible combinations, through leave-one-out cross validation on the available training data. 2.1
Training set analysis and repetitions
During preliminary analysis, we noticed that the training set included some subsets of
problem instances for which L and the solution were the same, i.e., the attributes values
for all the problem instances in a subset were the very same, despite being D different.
We call repetitions those problem instances. We argued that the tweets of the problem
instances in each of those subsets were authored by the same person. For this reason, we
decided to build a new training set by replacing each of those subsets with a single
problem instance in which D is the union of all the tweet sets of the subset—i.e., we merged
the repetitions. Table 1 shows the sizes of the training set portions corresponding to
each language before and after merging repetitions. We later experimentally verified
that this transformation did affect the learned classifiers and regressors.
The feature extraction procedure requires a language-dependent dictionary in which
words are grouped according to their prevalent topic (e.g., “money”, “sports”, or
“religion”) or their function (e.g., “prepositions”, “articles”, or “negations”). To this end,
we used an English dictionary similar to the one used by LIWC [
The feature extraction procedure is also based on the notion of automatic tweet, that we define as follows. We determined a set of ordered sequences of n 1; : : : ; 4 words, that we call templates, based on an analysis of the full training set: 1. we automatically extracted from the full training set all tweets starting with the same ordered sequence of n words; 2. we automatically constructed a set including all word sequences that were the starting sequence of at least 3 different tweets; 3. we manually analyzed each sequence and retained only those which appeared to be the beginning of an automatically-generated tweet.
We say that a tweet is an automatic tweet if its first words correspond to a template. Table 2 provides some examples of templates, along with the presence or absence of corresponding automatic tweets of different languages in the training sets.
The feature extraction procedure is as follows. Given a problem instance xD; Ly, we denote by DM the set of tweets obtained by D by removing all the automatic tweets. We extract several numerical features from each problem instance: the value of all (except of 3) features is obtained by averaging the corresponding computation outcomes on the tweets in D or DM —the remaining three feature values are computed on the whole D and/or DM . For ease of presentation, we group conceptually similar features together; the full list is given in Table 3. 2 http://www.netlingo.com/top50/popular-text-terms.php Stylometric These features tend to capture the structural properties of a tweet in a way largely independent of both the language and the specific semantic content; therefore, they are not based on the dictionaries. Stylometric features are computed on tweets in DM : the reason is because we assume that automatic tweets are not really representative of the tweet writing style of the author.
Content These features are based on the dictionaries categories related to word topic and are computed on tweets in D: the reason is because we assume that the content of automatic tweets is indeed informative of the author profile.
Hybrid These features are based on the dictionaries categories related to word function and are computed on tweets in DM . 2.3
Feature selection
Past studies on author profiling report several correlations between gender, age,
personality traits and writing style. In particular, [
Classifier and regressor
We decided to build a different model for each language-problem pair, for a total of 26,
as described in Section 1. We explored the usage of SVM [
As described in the previous sections, we considered 40 sets of features and 3 classifiers/regressors. We systematically assessed the effectiveness of all the 120 resulting combinations by means of a leave-one-out procedure applied on the training set, separately for each language-attribute pair. That is, for each language-attribute pair, set of features, and classifier/regressor, (i) we built the subset T of the problem instances of the training set with that language, (ii) we removed one element t0 from T , (iii) we computed the values for the features set on the problem instances in T and trained the classifier/regressor, (iv) we applied the trained classifier/regressor to the problem instance t0 and compared the generated answer against the known one. We repeated all but first steps |T | times, i.e., by removing each time a different element, and computed Feature name Description allpunc Number of .,:; commas Number of , exclmar Number of ! questma Number of ? parenth Number of parenthesis numbers Number of numbers cwocount Number of words itrelongwor Number of words longer than 6 letters mupcawor Number of uppercase words ltyocarrret Number of carriage returns (\n, \r, \r\n) satmenti Number of @ mentions extlink Number of links hashtag Number of # posemot Number of positive emoticons negemot Number of negative emoticons emotico Number of emoticons emotiyn Presence of emoticons in D (binary feature) moneywo Number of words in the “money” category jobword Number of words in the “job or work” category sportwo Number of words in the “sports” category televwo Number of words in the “tv or movie” category sleepwo Number of words in the “sleeping” category eatinwo Number of words in the “eating” category sexuawo Number of words in the “sexuality” category familwo Number of words in the “family” category tfrienwo Number of words in the “friends” category tenposemwo Number of words in the “positive emotion” category onnegemwo Number of words in the “negative emotion” category cemotiwo Number of words in the “positive emotion” or “negative emotion” category swearwo Number of words in the “swear words” category affecwo Number of words in the “affective process” category feeliwo Number of words in the “feeling” category religwo Number of words in the “religion” category schoowo Number of words in the “school” category occupwo Number of words in the “occupation” category autotwe Automatic tweets ratio, i.e., |D|zDD|M | autweyn Presence of automatic tweets in D (binary feature) fsipron Number of words in the “I” category fplpron Number of words in the “we” category ssipron Number of words in the “you” category selfref Number of words in the “self” category irdnegpart Number of words in the “negations” category hybasspart Number of words in the “assents” category article Number of words in the “articles” category preposi Number of words in the “prepositions” category pronoun Number of words in the “pronoun” category slangwo Number of words in the “chat acronyms” category
Table 3. Features list. the performance of the method in terms of the indexes defined in Section 1. Finally, we chose, for each language-attribute pair, the best performing combination, in terms of accuracy or RMSE, as appropriate for that attribute. The resulting configurations are summarized in Table 4.
In order to provide a synthetic baseline, we built 3 baseline methods using each of the 3 classifiers/regressors with all the features. The results, obtained by means of the same leave-one-out procedure, are shown in Table 5.
It can be seen from Table 4 that our procedure lead us to chose a different configuration of classifier/regressor and features set for each language-attribute pair. There could be several reason to explain that. First, every language has its own writing rules and culture, so it is possible that a middle aged English man could not have the same interests and the same writing style of a middle aged Italian man. Second, the Spanish, Dutch, and Italian dictionaries we used were not as good as the LIWC English one. Finally, the number of problem instances in the training set was not the same for every language, and so was the number of tweets in the instances within each language subset.
Gen rf2000 commas negemot exclmar Age rf2000 allpunc commas exclmar questma parenth numbers wocount longwor upcawor carrret atmenti extlink hashtag posemot negemot emotico autotwe Ext svm wocount questma parenth familwo Neu svm selfref fsipron chatacr affecwo emotiwo hashtag posemot pronoun wocount Con rf500 extlink longwor numbers hashtag fsipron selfref Agr svm questma atmenti allpunc ssipron article longwor jobword chatacr Ope rf2000 commas extlink hashtag exclmar questmar parenth wocount ssipron negpart article feeliwo moneywo jobword eatinwo familwo negemwo religwo Gen svm allpunc commas exclmar questma parenth numbers wocount longwor upcawor carrret atmenti extlink hashtag posemot negemot fsipron fplpron ssipron selfref negpart asspart article preposi pronoun slangwo moneywo jobword sportwo televwo sleepwo eatinwo sexuawo familwo frienwo posemwo negemwo affecwo feeliwo Age svm extlink hashtag numbers sleepwo sexuawo Ext rf2000 longwor carrret questma preposi autweyn emotico Neu rf2000 posemot ssipron exclmar selfref extlink Con rf500 extlink longwor numbers hashtag fsipron selfref affecwo emotiwo Agr svm allpunc commas exclmar questma parenth numbers wocount longwor upcawor carrret atmenti extlink hashtag posemot negemot + fsipron fplpron ssipron selfref negpart asspart article preposi pronoun slangwo moneywo jobword sportwo televwo sleepwo eatinwo sexuawo familwo frienwo posemwo negemwo swearwo religwo Ope rf2000 autotwe hashtag preposi wocount religwo
Gen rf500 asspart fsipron selfref exclmar extlink hashtag emotiyn IT NEexut rfs2v0m00 calolmpumnacswlooncoguwnotrhfapslhptraognqcuheasttamcraautweyn
Con svm commas extlink hashtag exclmar questmar parenth wocount ssipron negpart article feeliwo moneywo jobword eatinwo familwo negemwo religwo Agr svm posemot exclmar moneywo hashtag pronoun autweyn Ope svm negpart hashtag atmenti exclmar longwor Gen rf2000 negemot upcawor preposi Ext svm questma atmenti allpunc ssipron article longwor jobword chatacr extlink autweyn Neu rf2000 atmenti preposi longwor emotiyn Con svm hashtag questma exclmar atmenti posemot wocount extlink longwor Agr svm atmenti commas exclmar hashtag autweyn emotiyn Ope svm negpart hashtag atmenti exclmar longwor Table 4. Chosen classifier/regressor and features set for each language-attribute pair.
L Attribute svm rf500 rf2000 Our conf.
Gen 0:566 0:619 0:619 0:735 EN AEgxet 00::611845 00::611872 00::610851 00::619625
Neu 0:243 0:226 0:226 0:208 Con 0:167 0:158 0:158 0:146 Agr 0:173 0:183 0:183 0:162 Ope 0:157 0:149 0:149 0:143
Gen 0:760 0:760 0:760 0:820 ES AEgxet 00::148050 00::147074 00::147166 00::155860
Neu 0:243 0:220 0:220 0:202 Con 0:161 0:163 0:162 0:154 Agr 0:162 0:169 0:169 0:157 Ope 0:183 0:183 0:183 0:168
Gen 0:632 0:705 0:737 0:853 IT NEexut 00::125092 00::126125 00::126125 00::112710
Con 0:126 0:135 0:136 0:113 Agr 0:159 0:165 0:165 0:150 Ope 0:186 0:178 0:177 0:102
Gen 0:611 0:344 0:333 0:633 NL NEexut 00::123016 00::124005 00::123094 00::110556
Con 0:122 0:125 0:125 0:101 Agr 0:163 0:161 0:162 0:130
Ope 0:121 0:122 0:122 0:104 Table 5. Results of our configuration and the synthetic baselines. Accuracy is reported for Gen and Age, RMSE is reported for Ext, Neu, Con, Agr, and Ope.