=Paper=
{{Paper
|id=Vol-2150/MEX-A3T_paper_6
|storemode=property
|title=INGEOTEC at MEX-A3T: Author Profiling and Aggressiveness Analysis in Twitter Using μTC and EvoMSA
|pdfUrl=https://ceur-ws.org/Vol-2150/MEX-A3T_paper6.pdf
|volume=Vol-2150
|authors=Mario Graff,Sabino Miranda-Jiménez,Eric S. Tellez,Daniela Moctezuma,Vladimir Salgado,José Ortiz-Bejar,Claudia N. Sánchez
|dblpUrl=https://dblp.org/rec/conf/sepln/GraffMTMSOS18
}}
==INGEOTEC at MEX-A3T: Author Profiling and Aggressiveness Analysis in Twitter Using μTC and EvoMSA==
https://ceur-ws.org/Vol-2150/MEX-A3T_paper6.pdf
INGEOTEC at MEX-A3T:
Author profiling and aggressiveness
analysis in Twitter using µTC and EvoMSA
Mario Graff1, Sabino Miranda-Jiménez1, Eric S. Tellez1, Daniela
Moctezuma2, Vladimir Salgado1, José Ortiz-Bejar1,3, and Claudia N. Sánchez1,4
1
CONACyT - INFOTEC, Aguascalientes, Mexico
{mario.graff,sabino.miranda,eric.tellez,vladimir.salgado}@infotec.mx
2
CONACyT - CentroGEO, Aguascalientes, Mexico
dmoctezuma@centrogeo.edu.mx
3
Universidad Michoacana de San Nicolás de Hidalgo, México
jortiz@umich.mx
4
Universidad Panamericana. Facultad de Ingenierı́a. Aguascalientes, México
cnsanchez@up.edu.mx
Abstract. This paper describes our participation in the MEX-A3T challenge
for Aggressiveness Detection and Author Profiling tasks for Mexican Spanish
language. We used two approaches, µTC and EvoMSA systems. The first one
is a minimalistic text categorization system, and the second one is a two-level
architecture for Sentiment Analysis using information from different models
on the current text analyzed to get a final prediction by a consensus view.
Keywords: emotion classification, text categorization, author profiling.
1 Introduction
Author profiling and aggressiveness detection are essential tasks for marketing, digital
text forensics, cyber-bullying, security, among others. Aggressiveness detection allows
us to identify offenses and misbehavior expressed in text and commonly shared in social
networks. Author profiling is related to extract information from author’s texts such
as gender, age, and other kinds of personality traits. To increase the research in those
areas, several international competitions have been organized to deal with them, such
as PAN [14], SemEval [12], and TASS [11]. As part of this, recently, the MEX-A3T5 [3]
contest which is part of IBEREVAL’186 workshop has been launched on the research
community. The purpose of MEX-A3T is deal with author profiling and aggressiveness
detection in Spanish language focusing on Mexican Twitter users. MEX-A3T contest
presents two tasks to classify Twitter text. The first one is aggressiveness detection task
where systems have to determine whether a tweet is aggressive or not automatically.
The second task is author profiling, where systems have to automatically determine
the occupation and location (place of residence) of users from their tweets [3].
In the literature, several approaches have been proposed to tackle both author
profiling and aggressiveness detection. Such is the case of [7] where a system based
5
https://mexa3t.wixsite.com/home
6
https://sites.google.com/view/ibereval-2018
Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)
2 M. Graff. et al.
on lexicon, fuzzy logic, and statistical approaches is proposed to detect aggressiveness
in a text, or the proposed in [6] where a Lexical Syntactic Feature is used to detect
offensive content and then be able to identify a potential offensive user in social
media. Agrawal & Gonçalves [1] propose a combination of classifiers to identify gender
associated with a set of texts. This propose includes TFIDF representation, and a
dimension reduction of it, to finally employs Naive Bayes and Linear SVM as classifiers.
In [4] several stylometric features are considered for identifying males from females
in several age groups. Stylometric features are also used in [5] where tri-grams and
complementary-weighted Second Order Attributes are employed.
In this work, we present the methodology proposed to deal with profiling and aggres-
siveness detection, which includes two approaches, µTC and EvoMSA systems. µTC
is a minimalistic text categorization system, and EvoMSA is a two-level architecture for
Sentiment Analysis using information from different models getting the final prediction
by consensus. Both systems will be more detailed in following sections. The rest of the
paper is organized as follows. Section 2 describes our system and the general approach
to model the problem. Section 3 detail the experimental methodology and the achieved
results. Finally, conclusions and future work are given in Section 4.
2 System Description
As commented, we use two systems to tackled the author profiling and the aggressiveness
text detection tasks: µTC and EvoMSA, respectively. On the one hand, µTC is used
mainly to evaluate author profiling task because in our experiments it obtained the best
performance in this tasks. On the other hand, EvoMSA is used to evaluate aggressiveness
task. In the following paragraphs, we describe these approaches.
2.1 EvoMSA
EvoMSA7 has two stages. The first one, namely B4MSA [16], uses SVMs to predict
their decision function values of a given text. On the second hand, EvoDAG [9, 10]
is a classifier based on Genetic Programming with semantic operators which makes the
final prediction through a combination of all the decision function values. Furthermore,
EvoMSA is open to being fed with different models such as µTC, and lexicon-based
models, and EvoDAG. It is an architecture of two phases to solve classification tasks, see
Figure 1. In the first part, a set of different classifiers are trained with datasets provided
by the contests and others as additional knowledge, i.e., whatever knowledge could be
integrated into EvoMSA. In this case, we used tailor-made lexicons for the aggressiveness
task: aggressiveness words and affective words (positive and negative), see Section 2.3.
The precise configuration of our benchmarked system is described in Section 3.
2.2 µTC
µTC8 (a.k.a. B4MSA) is a minimalistic system able to tackle general text classification
tasks independently of domain and language. For complete details of the model see [17].
Roughly speaking, µTC creates text classifiers searching for the best models in given
7
https://github.com/INGEOTEC/EvoMSA
8
https://github.com/INGEOTEC/microTC
129
Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)
INGEOTEC at MEX-A3T 3
Fig. 1. Architecture of our EvoMSA framework
configuration space. A configuration consists of instructions to enable several preprocess-
ing functions, a combination of tokenizers among the power set of several possible ones
(character q-grams, n-word grams, and skip-grams), and a weighting scheme such as TF,
TFIDF, or several distributional schemes. µTC uses an SVM classifier with a linear kernel.
A text transformation feature could be binary (yes/no) or ternary (group/delete/none)
option. Tokenizers denote how texts must be split after applying the process of each text
transformation to texts. Tokenizers generate text chunks in a range of lengths, all tokens
generated are part of the text representation. In Table 1, we can see details of text trans-
formations used in our solution for detecting aggressiveness and profiling. For example,
Tokenizers used for Profiling are unigrams, bigrams, trigrams of words, and q-grams
of 1 and five characters length, and skip-grams of two words with a gap between them.
Table 1. Example of set of configurations for text modeling
Text transformation Aggressiveness Profiling
Text transformation Aggressiveness Profiling
remove diacritics yes yes
remove duplicates yes yes Term weighting
remove punctuation yes false TF-IDF yes no
emoticons group none Entropy no yes
lowercase yes true
numbers group group Tokenizers
urls group group n-words {1,2} {1,2,3}
users group delete q-grams {2,3,4} {1,5}
hashtags none none skip-grams — (2,1)
entities none none
2.3 Lexicon-based models
To introduce extra knowledge into our approach for aggressiveness task, we used two
lexicon-based models. The first, Up-Down model produces a counting of affective words,
i.e., for a given text, it is produced in two indexes one for positive words, and another
for negative words. We created a positive-negative lexicon based on the several Spanish
affective lexicons [2, 15, 13] and enriched with Spanish WordNet [8]. The other Bernoulli
130
Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)
4 M. Graff. et al.
Table 2. Results for Aggressiveness Detection
System macro-F1 macro-Recall Accuracy F1-aggressiveness
EvoMSA+LexB+UpDown 0.7941 0.8044 0.8061 0.7446
EvoMSA+UpDown 0.7926 0.8023 0.8048 0.7421
EvoMSA+LexB 0.7888 0.7982 0.8014 0.7373
EvoMSA 0.7830 0.7866 0.7992 0.7238
µTC 0.7900 0.7915 0.8070 0.7304
model was created to predict aggressiveness using a lexicon with aggressive words. We
created this lexicon gathering common aggressive words for Mexicans. These indexes
and prediction along with B4MSA’s (µTC) outputs are the input for EvoDAG system.
2.4 EvoDAG
EvoDAG9 [9, 10] is a Genetic Programming system specifically tailored to tackle
classification problems on very high dimensional vector spaces and large datasets.
EvoDAG uses the principles of Darwinian evolution to create models represented as
a directed acyclic graph (DAG). Due to lack of space, we refer the reader to [9] where
EvoDAG is broadly described. It is important to mention that EvoDAG does not have
information regarding whether input Xi comes from a particular class decision function,
consequently from EvoDAG point of view all inputs are equivalent.
3 Results
As mentioned, we split the dataset provided by organizers into 70-30 partition for training
and test. We run several configurations of our systems. In Table 2 and Table 3 results are
shown. In the case of the aggressiveness task, Table 2, we use the F1-aggressiveness score
to measure the performance. The basic configuration of EvoMSA is one model based
on B4MSA’s predictions using the training set provided by the competition. In case of
EvoMSA, plus symbol indicates the model added to the EvoMSA basic configuration.
In our experiments, the best performance we obtained is the combination of basic
EvoMSA along with a Lexicon-based Bernoulli model (LexB), and a counting model
of affective words (UpDown). This configuration was used to evaluate on the gold
standard that our approach obtained 0.4883 in F1-aggressiveness class, see Table 4,
INGEOTEC team.
In the case of author profiling task, the best performance was µTC system for
Occupation classes. Thus, we decided to apply the same approach to Location classes.
Table 3 shows the results of author profiling in our experiments. Our best system was
used to evaluate on the gold standard that our approach obtained 0.4470 of F1-score
for Occupation, 0.8155 of F1-score for Location and 0.6312 of F1-score on average of
both, see Table 5, INGEOTEC team .
Tables 4 and 5 list the top-final rankings for aggressiveness detection task and
user profiling task, respectively, more details of all results of the contest see [3]. Our
INGEOTEC team reached the first place in aggressiveness detection and the third
place in the author profiling task.
9
https://github.com/mgraffg/EvoDAG
131
Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)
INGEOTEC at MEX-A3T 5
Table 3. Results for Author Profiling
System macro-F1 macro-Recall Accuracy
Occupation
µTC 0.4369 0.4018 0.7006
EvoMSA 0.2946 0.3013 0.5758
Location
µTC 0.8086 0.7841 0.8522
Table 4. Final scores of the aggressiveness detection task
F1-score F1-score (non-
Rank Team Accuracy
(aggressive class) aggressive class)
1 INGEOTEC 0.4883 0.7535 0.6673
2 CGP 0.4500 0.7612 0.667
3 GeoInt-b4msa 0.434 0.7842 0.6876
4 aragon-lopez 0.4312 0.8069 0.7117
5 Trigrams (baseline) 0.4304 0.786 0.6888
Table 5. Final scores of the Author profiling task
Rank Team F1-score (Occupation) F1-score (Location) Average
1 MXAA 0.5122 0.8301 0.6711
2 aragon-lopez 0.4910 0.8388 0.6649
3 INGEOTEC 0.4470 0.8155 0.6312
4 CIC-GIL-run2 0.4894 0.7363 0.6128
5 CIC-GIL-run1 0.4727 0.7310 0.6018
4 Conclusions
In this paper was presented our solution for the MEX-A3T challenge. For Aggressiveness
Detection task, we applied EvoMSA system which can integrate different models as
additional knowledge as we have shown. Also, we applied our generic text classifier, µTC,
for author profiling task. Both systems are designed to be multilingual, language and
domain independent as much as possible. For the training step, we use extra knowledge
coded into affective and aggressiveness lexicons our robust solution (EvoMSA) performs
well for the aggressiveness task; however, there is room for further improvements in
performance for author profiling task using another sort of knowledge such as semantic
information into our architecture.
References
1. Agrawal, M., Gonçalves, T.: Age and gender identification using stacking for classification.
notebook for pan at clef 2016 (2016)
2. de Albornoz, J.C., Plaza, L., Gervás, P.: Sentisense: An easily scalable concept-based
affective lexicon for sentiment analysis. In: Proceedings of LREC 2012. pp. 3562–3567 (2012)
3. Álvarez-Carmona, M.Á., Guzmán-Falcón, E., Montes-y Gómez, M., Escalante, H.J.,
Villaseñor-Pineda, L., Reyes-Meza, V., Rico-Sulayes, A.: Overview of mex-a3t at ibereval
2018: Authorship and aggressiveness analysis in mexican spanish tweets. In: Notebook
Papers of 3rd SEPLN Workshop on Evaluation of Human Language Technologies for
Iberian Languages (IBEREVAL), Seville, Spain, September (2018)
132
Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)
6 M. Graff. et al.
4. Bilan, I., Zhekova, D.: Caps: A cross-genre author profiling system (2016)
5. Bougiatiotis, K., Krithara, A.: Author profiling using complementary second order
attributes and stylometric features (2016)
6. Chen, Y., Zhou, Y., Zhu, S., Xu, H.: Detecting offensive language in social media to
protect adolescent online safety. In: 2012 International Conference on Privacy, Security,
Risk and Trust and 2012 International Confernece on Social Computing. pp. 71–80 (Sept
2012). https://doi.org/10.1109/SocialCom-PASSAT.2012.55
7. Del Bosque, L.P., Garza, S.E.: Aggressive text detection for cyberbullying. In: Gelbukh, A.,
Espinoza, F.C., Galicia-Haro, S.N. (eds.) Human-Inspired Computing and Its Applications.
pp. 221–232. Springer International Publishing, Cham (2014)
8. Fernández-Montraveta, A., Vázquez, G., Fellbaum, C.: The spanish version of wordnet
3.0. Text Resources and Lexical Knowledge. Mouton de Gruyter pp. 175–182 (2008)
9. Graff, M., Tellez, E.S., Miranda-Jiménez, S., Escalante, H.J.: Evodag: A seman-
tic genetic programming python library. In: 2016 IEEE International Autumn
Meeting on Power, Electronics and Computing (ROPEC). pp. 1–6 (Nov 2016).
https://doi.org/10.1109/ROPEC.2016.7830633
10. Graff, M., Tellez, E.S., Escalante, H.J., Miranda-Jiménez, S.: Semantic Genetic
Programming for Sentiment Analysis. In: Schà 41 tze, O., Trujillo, L., Legrand, P.,
Maldonado, Y. (eds.) NEO 2015, pp. 43–65. No. 663 in Studies in Computational
Intelligence, Springer International Publishing (2017), dOI: 10.1007/978-3-319-44003-3 2
11. Martınez-Cámara, E., Dıaz-Galiano, M., Garcıa-Cumbreras, M., Garcıa-Vega, M.,
Villena-Román, J.: Overview of tass 2017. In: Proceedings of TASS 2017: Workshop on
Semantic Analysis at SEPLN (TASS 2017). vol. 1896 (2017)
12. Mohammad, S.M., Bravo-Marquez, F., Salameh, M., Kiritchenko, S.: Semeval-2018 Task
1: Affect in tweets. In: Proceedings of International Workshop on Semantic Evaluation
(SemEval-2018). New Orleans, LA, USA (2018)
13. Perez-Rosas, V., Banea, C., Mihalcea, R.: Learning sentiment lexicons in spanish. In:
LREC. vol. 12, p. 73 (2012)
14. Rangel, F., Rosso, P., Potthast, M., Stein, B.: Overview of the 5th author profiling task
at pan 2017: Gender and language variety identification in twitter. Working Notes Papers
of the CLEF (2017)
15. Sidorov, G., Miranda-Jiménez, S., Viveros-Jiménez, F., Gelbukh, A., Castro-Sánchez,
N., Velásquez, F., Dı́az-Rangel, I., Suárez-Guerra, S., Treviño, A., Gordon, J.: Empirical
study of machine learning based approach for opinion mining in tweets. In: Proceedings
of the 11th Mexican International Conference on Advances in Artificial Intelligence -
Volume Part I. pp. 1–14. MICAI’12, Springer-Verlag, Berlin, Heidelberg (2013)
16. Tellez, E.S., Miranda-Jiménez, S., Graff, M., Moctezuma, D., Suárez, R.R., Siordia, O.S.:
A simple approach to multilingual polarity classification in Twitter. Pattern Recognition
Letters 94, 68–74 (2017)
17. Tellez, E.S., Moctezuma, D., Miranda-Jiménez, S., Graff, M.: An automated text
categorization framework based on hyperparameter optimization. Knowledge-Based
Systems 149, 110–123 (2018)
133