=Paper=
{{Paper
|id=Vol-1391/104-CR
|storemode=property
|title=Tweets Classification using Corpus Dependent Tags, Character and POS N-grams
|pdfUrl=https://ceur-ws.org/Vol-1391/104-CR.pdf
|volume=Vol-1391
|dblpUrl=https://dblp.org/rec/conf/clef/Gonzalez-Gallardo15
}}
==Tweets Classification using Corpus Dependent Tags, Character and POS N-grams==
https://ceur-ws.org/Vol-1391/104-CR.pdf
Tweets Classification Using Corpus Dependent Tags,
Character and POS N-grams
Notebook for PAN at CLEF 2015
Carlos E. González-Gallardo, Azucena Montes, Gerardo Sierra,
J. Antonio Núñez-Juárez, Adolfo Jonathan Salinas-López, Juan Ek
Grupo de Ingeniería Lingüística, Instituto de Ingeniería, UNAM, México
{cgonzalezg,amontesr,gsierram,anunezj,asalinasl,jekc}
@iingen.unam.mx
Abstract. This paper is part of the Author Profiling task at PAN 2015 contest;
in witch participants had to predict the gender, age and personality traits of
Twitter users in four different languages (Spanish, English, Italian and Dutch).
Our approach takes into account stylistic features represented by character N-
grams and POS N-grams to classify tweets. The main idea of using character N-
grams is to extract as much information as possible that is encoded inside the
tweet (emoticons, character flooding, use of capital letters, etc.). POS N-grams
were obtained using Freeling and certain token were relabeled with Twitter de-
pendent tags. Obtained results were very satisfactory; our global ranking score
was of 83.46%.
1 Introduction
Author Profiling focused on Internet texts has been growing for the last years, one of
the reasons is the big amount of information produced every minute in social net-
works or blogs. These Internet texts have their own characteristics that are hardly
comparable with literary texts, documentaries or essays; this is because of the necessi-
ty of having a quick communication and the liberty of publishing unrevised content.
For March 2015, Facebook reported having about 936 million daily active users on
average [1].
As part of the PAN 2015 contest, this year, the Author Profiling (AP) task was about
tweets in Spanish, English, Italian and Dutch [2]; for Spanish and English, the objec-
tive was to predict gender, age and personality traits of a Twitter [3] user. Moreover,
for Italian and Dutch was only needed to predict gender and personality traits.
Twitter has its own rules and characteristics that users explode to express themselves
and communicate to each other. These rules can be extracted to create dependent tags
(corpus dependent tags) that will help the classifier to improve its performance.
2 Dataset
The dataset provided this year consisted of tweets in Spanish, English Italian and
Dutch. Regard to gender, the corpus was balanced in all four languages (50% of
tweets were label as “female” and the other half as “men”).
Table 1. Female and male distribution of the corpus
Female Male
Language Total samples
Samples Percentage Samples Percentage
Spanish 50 50% 50 50% 100
English 76 50% 76 50% 152
Italian 19 50% 19 50% 38
Dutch 17 50% 17 50% 34
For the case of Spanish and English, age classes were defined in four groups (18-24,
25-34, 35-49 and 50-xx). In this case the corpus was not balanced, having a lot of
“25-34” samples (around 40%) and just a few samples for “50-xx” (around 10%).
Table 2. Proportion of age-group samples for Spanish and English
Spanish English
Samples 22 58
18-24
Percentage 22% 38%
Samples 46 60
25-34
Percentage 46% 40%
Samples 22 22
35-49
Percentage 22% 14%
Samples 10 12
50-xx
Percentage 10% 8%
Total samples 100 152
There were five personality traits to predict: extroverted, stable, open, conscientious
and agreeable; each one of them with a possible value between -0.5 and +0.5. It is
important to mention that the samples for personality traits were totally imbalanced.
For example: in Italian, for the conscientious personality trait there were just 5 labels
of the 11 possible ones (-0.5, -0.4, … , +0.4, +0.5), and the number of samples of
these existing labels varied a lot.
Table 3. Number of samples per label in each personality trait
-0.5 -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5
Extroverted 3 5 5 28 32 9 9 9
Stable 2 10 26 9 12 19 10 10 2
Spanish Agreeable 3 16 6 16 40 12 2 5
Conscientious 2 21 7 20 12 21 17
Open 7 10 37 15 9 14 8
Extroverted 1 4 10 17 41 37 20 13 9
Stable 11 5 22 9 19 37 19 18 12
English Agreeable 5 2 12 19 44 46 13 7 4
Conscientious 1 4 30 38 27 33 12 7
Open 2 1 47 39 23 19 21
Extroverted 8 13 9 3 5
Stable 1 3 3 8 4 12 5 2
Italian Agreeable 1 3 11 9 7 7
Conscientious 3 18 6 5 6
Open 1 14 9 2 7 5
Extroverted 3 5 11 7 6 2
Stable 1 5 3 3 4 6 8 4
Dutch Agreeable 2 1 5 10 10 2 4
Conscientious 2 4 15 6 5 2
Open 4 11 4 12 3
3 Features
One of the main characteristics of the AP task for PAN 2015 was that it was not about
classifying tweets but classifying Twitter users based on a group of their tweets;
let us call each one of these groups a document. So based on this fact, the vectors for
the training algorithm and the vector for each one of the tests were a document-vector
formed by a group of individual tweets; considering this, if a tweet can have a maxi-
mum length of 140 characters and in average each document was about 100 tweets,
the length of each document could be about 14000 characters; a length quite accepta-
ble for extracting a good number of features [4].
Because of the nature of the task that involved four different languages and our idea
of making the algorithm the most language independent as possible, it was going to be
difficult and unpractical to use content-based features; so we opted for the stylistic
features and just keep it simple. It is possible to divide the used features in two
groups: character N-grams and POS N-grams.
Using character N-grams could be seen very basic and naive, but it has shown to be
very useful and practical in previous experiments [5]. Besides the usefulness shown in
the past by the character N-grams, implicitly a great amount of stylistic features are
extracted; for example: if 3-grams are used with in a document the frequency of all
punctuation marks, characters flooding (!!!, ???, …, etc.), word flexion and deriva-
tion, diminutives, superlatives and prefixes are being extracted [6,7].
Regard to the POS N-grams, it is possible to obtain the grammatical sequence of the
writer. For the POS tagging we used Freeling [8] with its corresponding configuration
for Spanish, English and Italian; for Dutch, Freeling doesn't have a module so in this
case we set up a basic assumption: if we use the English Freeling module, errors
are going to be performed; but if these errors follow a certain stable pattern, it is
possible that some grammatical information is being extracted from the tweets.
Depending of the input language to analyze, our software selects the best configura-
tion of extraction parameters (based on a series of tests) that maximize its own per-
formance. These configurations take into account the number of N in the character N-
grams (num_gramas) and POS N-grams (num_POS), if they are retroactive or not
(retro_gramas and retro_POS), the N-gram representation (modo) and if the N-grams
should be represented in a logarithmic scale (frec_log).
Table 4. Extraction parameters and its possible values
PARAMETER VALUES
num_gramas 0≤𝑛
num_POS 0≤𝑛
0 NO
retro_gramas
1 YES
0 NO
retro_POS
1 YES
frec frequency mode
modo
bin binary mode
0 NO
frec_log
1 YES
The best configuration established is shown in the next table.
Table 5. Best configuration of the extraction parameters (GA: gender and age, P: personality
traits)
num_gramas num_POS retro_gramas retro_pos modo frec_log
GA 3 3 1 1 frec 1
Spanish
P 3 3 1 1 frec 0
GA 2 3 1 1 frec 1
English
P 3 3 1 1 frec 1
GA 3 1 1 1 frec 1
Italian
P 3 1 1 1 frec 1
GA 3 1 1 1 frec 1
Dutch
P 3 1 1 1 frec 1
4 Algorithm
As mentioned before, the objective of the task was to predict gender, age and person-
ality traits of a Twitter user in four different languages: Spanish, English, Italian and
Dutch (for these last two languages was only needed to predict gender and personality
traits). For the case of gender and age it is obvious that the type of learning algorithm
to implement is a classifier. By the other hand, given the nature of the personality
traits values that can be considered continue real values it is possible to think in a
regression problem. This is a good approach, but considering the few samples of cer-
tain points (values) for the regression in some of the personality traits and the bad
performance of some regression algorithms we used to check the behavior of the cor-
pus, we decided to make the prediction of personality traits problem a classification
task.
For Spanish and English the goal was to predict age and gender of a Twitter user, so
we decided to classify both characteristics at the same time getting the next 8 possible
classes: _F_20s, _F_30s, _F_40s, _F_50s, _M_20s, _M_30s, _M_40s and _M_50s.
Age groups are explained in the next table. For Italian and Dutch two classes where
just created: _F and _M.
Table 6. Age groups for Spanish and English
Age group Age range
20s [18,25)
30s [25,35)
40s [35,50)
50s [50, +oo)
The training phase is divided in 5 entities: extraction, labeled, POS generation, vec-
tors creation and training.
Extraction: The truth file (truth.txt) is read and analyzed. One file is created for
each one of the possible classes: class file. The tweets are preprocessed based on
the substitution rules showed in Table 7; hashtags are not preprocessed because
we consider they can provide stylistic information.
Table 7. Substitution rules
Token Token explanation Substitution
@username Reference to another Twitter user @us
http[s]://… Link to an external site htt
\n New line character Space character
Note: Each one of these class files is separated by documents (group of tweets of
an author).
Labeled: For each class file created by the extraction phase, a local instance of
Freeling is called to obtain a JSON [9] list1 that contains each one of the tweets of
each one of the documents.
POS generation: Once the JSON list is obtained, the POS generation phase cre-
ates a POS file with the same structure of the class file and relabels certain tokens
(adding our own tags) that are needed to extract extra grammatical information.
Table 8. Corpus dependent tags
Token Label
@us REF#USERNAME
htt REF#LINK
#{something} REF#HASHTAG
Vectors creation: Character N-grams and POS N-grams are extracted from each
document from each class file and POS file respectively based on the extraction
parameters (Table 5.) to produce the document-vectors. Once all the document-
vectors are created, a general features-vector is generated and each document-
vector is expanded to the features-vector length obtaining the features-matrix that
will be used to train the system.
Training: The features-matrix is now passed to the learning algorithm to train
the system. The algorithm we used to classify age, gender and personality traits is
an implementation of a Support Vector Machine (SVM) with a linear kernel
called LinearSVC [10]. Once the system has been trained, the learning model and
the features-vector is serialized and saved into disk for their later use.
The test phase is also divided in 5 entities: extraction, labeled, POS generation, vector
creation and testing.
1 The Freeling instance is called using an interface that converts the output of Freeling into a
JSON string. This interface was developed by Grupo de Ingeniería Lingüística, Instituto de
Ingeniería, UNAM
Extraction: The xml file of the Twitter user is read and processed based on the
substitution rules mentioned in (Table 7.) obtaining a preprocessed text file;
hashtags are not preprocessed because we consider they can provide stylistic in-
formation.
Labeled: The preprocessed file done by the extraction phase is passed to a local
instance of Freeling to obtain a JSON string.
POS generation: Once the JSON string is obtained, the POS generation phase
creates a POS file with the same structure of the preprocessed file and relabels
certain tokens (adding our own tags) that are needed to extract extra grammatical
information (Table 8.).
Vector creation: Character N-grams and POS N-grams are extracted from the
preprocessed file and the POS file respectively based on the extraction parame-
ters (Table 4.) to produce a document-vector. Then the features-vector is loaded
to expand the document-vector.
Testing: The learning model created by the training phase is loaded so that the
document-vector can be tested. Once all 6 classifications (gender/gender_age, ex-
troverted, stable, agreeable, conscientious and open) are done the output xml file
is created and written to disk.
5 Results
Two measures were used to evaluate the submissions: accuracy was used to measure
age and gender, leaving Root Mean Squared Error (RMSE) to measure personality
traits. For the case of Spanish and English, an average between age and gender was
performed (both).
Table 9. Global result of our approach
gonzalesgallardo15
Global Results
Personality
(
(1−𝑅𝑀𝑆𝐸)+
language
GLOBAL
conscientious
runtime
extroverted
gender
agreeable
place
both
age
RMSE
stable
open
2
𝑔𝑒𝑛𝑑𝑒𝑟+𝑎𝑔𝑒
2
)
Spanish 0.1168 0.1709 0.1406 0.1398 0.2094 0.1555 0.7273 0.8977 0.7045 0.7745 00:04:25 3
English 0.1480 0.1101 0.1303 0.1422 0.2151 0.1491 0.7817 0.8521 0.6972 0.7740 00:06:29 2
Italian 0.0745 0.1269 0.0764 0.1572 0.2121 0.1294 - 0.8611 - 0.8658 00:01:31 1
Dutch 0.0952 0.1299 0.0901 0.0637 0.0661 0.0890 - 0.9375 - 0.9242 00:01:20 2
0.8346
If we compare our approach against the other two best participants (tables 7.1-7.4),
ours is slightly slower because of the grammatical analysis made by Freeling.
Each one of the corpus languages had different number of samples and different dis-
tribution of data, so maybe it is not objective to compute a global average but we
think is important to do this because the task involved analyzing all four languages; so
we took the freedom of computing it obtaining a global score of 83.46%.
Some extra information related to our results is presented in Section 8. Extended Re-
sults.
6 Conclusions
Internet has made of communication something very quickly and fluid; example of
these is the social network Twitter, in which users have to transmit a complete mes-
sage in just 140 characters. To accomplish this, it is necessary to compact as much
information as possible, making each one of the tweets a dense text (short text with a
lot of information).
The use of N-grams of characters and POS N-grams, as shown in the results, is a good
option with dense texts because of theirs extraction capacity. In the case of N-grams it
was possible to extract emoticons, exaggeration of punctuation marks (character
flooding), use of capital letters and all kind of emotional information encoded in the
tweet. With POS N-grams, in Spanish and English we were able to capture the most
representative series of two and three grammatical elements; in Italian and Dutch we
were able to capture the most frequent grammatical elements.
7 Future Work
Our approach showed to be good for the gender classification task but not too good
for age classification. We will focus in finding some characteristics that we are prob-
ably missing and try to give them more emphasis to make them more representative.
Probably is a good idea to separate the classification and just classify age and gender
separately.
8 Extended Results
Table 10. PAN 2015 Author Profiling results for Spanish
Personality
GLOBAL
conscientious
runtime
gender
extroverted
place
agreeable
both
user
age
RMSE
stable
open
1 alvarezcarmona15 0.1113 0.1168 0.1319 0.1257 0.1631 0.1297 0.7955 0.9659 0.7727 0.8215 00:00:44
2 kiprov15 0.1249 0.1386 0.1625 0.1334 0.1884 0.1495 0.7841 0.9091 0.7273 0.7889 00:02:46
3 gonzalesgallardo15 0.1168 0.1709 0.1406 0.1398 0.2094 0.1555 0.7273 0.8977 0.7045 0.7745 00:04:25
Table 11. PAN 2015 Author Profiling results for English
Personality
GLOBAL
conscientious
runtime
gender
extroverted
place
agreeable
both
user
age
RMSE
stable
open
1 alvarezcarmona15 0.1305 0.1172 0.1278 0.1202 0.2253 0.1442 0.8380 0.8592 0.7254 0.7906 00:00:59
2 gonzalesgallardo15 0.1480 0.1101 0.1303 0.1422 0.2151 0.1491 0.7817 0.8521 0.6972 0.7740 00:06:29
3 teisseyre15 0.1480 0.1309 0.1371 0.1351 0.1990 0.1500 0.7535 0.8310 0.6479 0.7489 00:03:15
Table 12. PAN 2015 Author Profiling results for Italian
Personality
GLOBAL
conscientious
runtime
gender
extroverted
place
agreeable
both
user
age
RMSE
stable
open
1 gonzalesgallardo15 0.0745 0.1269 0.0764 0.1572 0.2121 0.1294 - 0.8611 - 0.8658 00:01:31
2 grivas15 0.1389 0.2461 0.1350 0.1586 0.1930 0.1743 - 0.8333 - 0.8295 00:00:29
3 kocher15 0.1302 0.1093 0.1000 0.1344 0.1555 0.1259 - 0.7778 - 0.8260 00:00:01
Table 13. PAN 2015 Author Profiling results for Dutch
Personality
GLOBAL
conscientious
runtime
gender
extroverted
place
agreeable
both
user
age
RMSE
stable
open
1 alvarezcarmona15 0.0000 0.1075 0.0750 0.0354 0.0637 0.0563 - 0.9375 - 0.9406 00:00:24
2 gonzalesgallardo15 0.0952 0.1299 0.0901 0.0637 0.0661 0.0890 - 0.9375 - 0.9242 00:01:20
3 grivas15 0.1427 0.2278 0.1467 0.0973 0.1711 0.1571 - 0.9688 - 0.9058 00:00:29
Acknowledgements: This work was funded by project CONACyT-México No.
215179 “Caracterización de huellas textuales para el análisis forense”.
References
1. Facebook. «Facebook newsroom», http://newsroom.fb.com/company-info
2. Rangel, F., Rosso, P., Potthast, M., Stein, B., Daelemans, W.: Overview of the 3rd
author profiling task at pan 2015. In: Cappellato L., Ferro N., Gareth J. and San
Juan E. (Eds). (Eds.) CLEF 2015 Labs and Workshops, Notebook Papers. CEUR-
WS.org, (2015)
3. Twitter, Inc., https://twitter.com
4. Schwartz, R., Tsur, O., Rappoport, A., & Koppel, M. (2013). Authorship Attribu-
tion of Micro-Messages. In EMNLP (pp. 1880-1891).
5. Doyle, J., & Kešelj, V. (2005). Automatic Categorization of Author Gender via N-
Gram Analysis. In The 6th Symposium on Natural Language Processing, SNLP.
6. Stamatatos, E. (2006, August). Ensemble-based author identification using charac-
ter n-grams. In Proceedings of the 3rd International Workshop on Text-based In-
formation Retrieval (pp. 41-46).
7. Stamatatos, E. (2009). A survey of modern authorship attribution methods. Journal
of the American Society for information Science and Technology, 60(3), 538-556.
8. Freeling, http://nlp.lsi.upc.edu/freeling/
9. JSON (JavaScript Object Notation), http://JSON.org/
10. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ...
& Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. The Journal of
Machine Learning Research, 12, 2825-2830.