INGEOTEC solution for Task 4 in TASS'18 competition

-

News classi cation problem is closely related to traditional text classi cation applications such as topic classi cation (e.g., classifying a news-like text into sports, politics, or economy). Knowing any kind of categorization of news can re ect the problems of society in several domains. For instance, the discovery of negative news over time can be helpful to have a reference for leaders or decisionmakers to do something about the current situation (Wang et al., 2018) .

In this year, in TASS competition (Mart nez-Camara et al., 2018) , a new task was proposed (Task 4), this task is related to an emotional categorization of news articles.

With the purpose for knowing if a new article is SAFE or UNSAFE, a corpus was built from RSS feeds of a number of online newspapers in di erent varieties of Spanish (Argentina, Chile, Colombia, Cuba, Spain, USA, Mexico, Peru, and Venezuela). For the purpose of classifying these news, as SAFE or UNSAFE, the headlines were provided.

From Task 4, two sub-tasks were speci ed: Subtask-1 Monolingual classi cation and Subtask-2 Multilingual classi cation.

The main di erence between these two tasks is that, in the rst case, the algorithm must be trained and tested with the same Spanish variety. In contrary, in the second case, the algorithm can be trained with a Spanish variety and tested with a di erent one. More information about details from Task 4 please see (Mart nez-Camara et al., 2018) .

Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes. In this paper, the solution proposed for the INGEOTEC team is presented. This solution is based on our B4MSA classi er and a number of specialized resources related to aggressiveness and a ectivity detection. Finally, our EvoMSA classi er based on Genetic Programming is used to combine all the resources and the available training data. It is worth to mention that our scheme to create our systems for Task 1 (monolingual and cross-lingual polarity classi cation) and Task 4 (this one) is pretty similar; of course, we use the given task's training set to learn and optimize for each task.

The manuscript is organized as follows. In Section 2 the description of our solution is detailed. In Section 3 our results achieved in task 4 is presented. Finally, the conclusions are commented in Section 4. 2

System Description

As commented before, we use a combination of several sub-systems to tackle the (un)safeness categorization of the given news. Firstly, we use our generic text classi er B4MSA (Tellez et al., 2017) and a vocabulary of pre-trained vectors of FastText (Mikolov et al., 2013) . Also, we use two di erent domain-speci c lexicon resources, one of them designed to detect aggressiveness and the other one designed to detect emotions in text. All these sub-systems and resources are combined using our genetic programming scheme (EvoMSA) over the decision functions of several classi ers built on top of these resources. The rest of this section details the use of these sub-systems and resources. 2.1

EvoMSA

EvoMSA1 has two stages. The rst one, namely B4MSA (Tellez et al., 2017) , uses SVMs to predict their decision function values of a given text. On the second hand, EvoDAG (Gra et al., 2016; Gra et al., 2017) is a classi er based on Genetic Programming with semantic operators which makes the nal prediction through a combination of all the decision function values. Furthermore, EvoMSA is open to being fed with di erent models such as B4MSA (Tellez et al., 2018) , and lexicon-based models, and EvoDAG. It is an architecture of two phases to solve classi cation tasks, see Figure 1. In the rst part, a set of di erent classi ers are trained with datasets provided by the contests and others as additional knowledge, i.e., whatever knowledge could be integrated into EvoMSA. In this case, we used tailor-made lexicons for identifying aggressiveness, positiveness, and negativeness in texts, see Section 2.2 for more details. The precise conguration of our benchmarked system is described in Section 3. 2.1.1 B4MSA B4MSA2 (a.k.a. TC) is a minimalistic system able to tackle general text classi cation tasks independently of domain and language. For complete details of the model see (Tellez et al., 2018) . Roughly speaking, TC creates text classi ers searching for the best models in given con guration space. A conguration consists of instructions to enable several preprocessing functions, a combination of tokenizers among the power set of several possible ones (character q-grams, nword grams, and skip-grams), and a weighting scheme such as TF, TFIDF, or several distributional schemes. TC uses an SVM (Support Vector Machine) classi er with a linear kernel. A text transformation feature could be binary options (yes/no) or ternary options (group/delete/none). Tokenizers denote how texts must be split after applying the process of each text transformation to texts, all tokens generated are part of the text representation. In Table 1, we can see details of the preprocessing, tokenizers, and term weighting scheme. 2.2

Lexicon-based models

To introduce extra knowledge into our approach, we used two lexicon-based models. The rst, Up-Down model produces a counting of a ective words, that is, it produces two indexes for a given text: one for positive words, and another for negative words. We created the positive-negative lexicon based on the several Spanish a ective lexicons (de Albornoz, Plaza, y Gervas, 2012; Sidorov et al., 2013; Perez-Rosas, Banea, y Mihalcea, 2012) ; we also enriched this lexicon with Spanish WordNet (FernandezMontraveta, Vazquez, y Fellbaum, 2008) . The other Bernoulli model was created to predict aggressiveness using a lexicon with aggressive words. We created this lexicon 1https://github.com/INGEOTEC/EvoMSA 2https://github.com/INGEOTEC/microTC

EvoDAG

EvoDAG3 (Gra et al., 2016; Gra et al., 2017) is a Genetic Programming system speci cally tailored to tackle classi cation problems on very large and high dimensional vector spaces. EvoDAG uses the principles of Darwinian evolution to create models represented as a directed acyclic graph (DAG). Due to lack of space, we refer the reader to (Gra et al., 2016) where EvoDAG is broadly described. It is important to mention that EvoDAG does not have information regarding whether input Xi comes from a particular class decision function, consequently from EvoDAG point of view all inputs are equivalent.

2.4 FastText

FastText (Joulin et al., 2017) is a tool to create text classi ers and learn a semantic vocabulary, learned from a given collection of documents; this vocabulary is represented with a collection of high dimensional vectors, one per word. It is worth to mention that FastText is robust to lexical errors since outvocabulary words are represented as the combination of vectors of sub-words, that is, a kind of character q-grams limited in context to words. Nonetheless, the main reason of in3https://github.com/mgra g/EvoDAG

Team's name

Macro-F1

Accuracy

ELiRF rbnUGR

INGEOTEC MeaningCloud

SINAI TNT-UA-WFU

lone wolf INGEOTEC ELiRF-UPV

rbnUGR MeaningCloud ITAINNOVA cluding FastText as part of our system is to overcome the small train set that comes with Task 4, which is ful lled using the pre-trained vectors computed in the Spanish content of Wikipedia (Bojanowski et al., 2016) . We use these vectors to create document vectors, one vector per document. A document vector is, roughly speaking, a linear combination of the word vectors that compose the document into a single vector of the same dimension. These document vectors were used as input to an SVM with a linear kernel, and we use the decision function as input to EvoMSA. 3

Experiments and results

In order to test all the approaches in the Task-4, the SANSE (Spanish brANd Safe Emotion) corpus was established. The SANSE corpus is composed of 2,000 headlines of news written in the Spanish language along several Spanish speaker countries Spain, Mexico, Cuba, Chile, Colombia, Argentina, Venezuela, Peru, and U.S.A.

In the case of Subtask-1, Monolingual Classi cation, the goal was training with a Spanish variety, e.g., Mexico, and then testing with the same Spanish variety. In this case, our results and the results of the best ve teams ranked by Macro-F1 metric, are presented in Table 2.

Team's name Macro-F1 Accuracy

INGEOTEC ELiRF-UPV

rbnUGR MeaningCloud

SINAI lone wolf TNT-UA-WFU 0.795 0.79 0.774 0.767 0.728 0.700 0.492 Our solution based on Genetic Programming reached the best result in SubTask1 Monolingual Classi cation SANSE-TEST500 and SubTask-2 Multilingual Classi cation SANSE 408 corpus. In the largest corpus in SubTask-2 (SANSE-TEST-13152) our system reached the third best team solution.

Our approach, EvoMSA, is able to deal with several data sources through an ensemble of decision functions from each di erent bunch of data such as extra knowledge coded into lexicons for sentiment analysis and aggressiveness identi cation, and semantic information from word vectors.

Acknowledgements

The authors would like to thank Laboratorio Nacional de GeoInteligencia for partially funding this work.

References

Bojanowski , P. , E. Grave, A . Joulin, y T. Mikolov . 2016 . Enriching word vectors with subword information . arXiv preprint arXiv:1607 . 04606 .

de Albornoz , J. C. , L. Plaza, y P. Gervas . 2012 . Sentisense: An easily scalable concept-based a ective lexicon for sentiment analysis . En Proceedings of LREC 2012 , paginas 3562 { 3567 .

Fernandez-Montraveta , A. , G. Vazquez, y C. Fellbaum . 2008 . The spanish version of wordnet 3.0 . Text Resources and Lexical Knowledge . Mouton de Gruyter, paginas 175 { 182 .

Gra , M. , E. S. Tellez , S. Miranda-Jimenez, y H. J. Escalante . 2016 . Evodag: A semantic genetic programming python library . En 2016 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC), paginas 1{6 , Nov .

Gra , M. , E. S. Tellez , H. J. Escalante , y S. Miranda-Jimenez . 2017 . Semantic Genetic Programming for Sentiment Analysis . En O. Schutze L. Trujillo P. Legrand, y Y. Maldonado, editores, NEO 2015 , numero 663 en Studies in Computational Intelligence . Springer International Publishing, paginas 43 { 65 . DOI: 10 .1007/978-3- 319 -44003-3 2 .

Joulin , A. , E. Grave , P. Bojanowski, y T. Mikolov . 2017 . Bag of tricks for e cient text classi cation . En Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2 , Short Papers , paginas 427 { 431 . Association for Computational Linguistics, April.

Mart nez-Camara, E., Y. Almeida-Cruz , M. C. D az- Galiano , S. Estevez-Velarde , M. A. Garc a-Cumbreras, M. Garc aVega, Y. Gutierrez , A. Montejo Raez , A. Montoyo , R. Mun~oz, A. PiadMor s, y J. Villena-Roman . 2018 . Overview of TASS 2018: Opinions, health and emotions . En E. Mart nezCamara Y. Almeida-Cruz M. C. D azGaliano S. Estevez-Velarde M. A. Garc aCumbreras M. Garc a-Vega Y. Gutierrez A. Montejo Raez A. Montoyo R. Mun ~oz A. Piad-Mor s , y J. Villena-Roman, editores , Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN (TASS 2018 ), volumen 2172 de CEUR Workshop Proceedings, Sevilla, Spain, September. CEUR-WS.

Mikolov , T. , I. Sutskever, K. Chen , G. S. Corrado, y J. Dean . 2013 . Distributed representations of words and phrases and their compositionality . En Advances in neural Perez-Rosas , V., C. Banea, y R. Mihalcea . 2012 . Learning sentiment lexicons in spanish . En LREC , volumen 12 , pagina 73.

Sidorov , G., S. Miranda-Jimenez , F. ViverosJimenez , A. Gelbukh , N. Castro-Sanchez , F. Velasquez , I. D az- Rangel , S. SuarezGuerra , A. Trevin~o, y J. Gordon . 2013 . Empirical study of machine learning based approach for opinion mining in tweets . En Proceedings of the 11th Mexican International Conference on Advances in Articial Intelligence - Volume Part I, MICAI'12, paginas 1 { 14 , Berlin, Heidelberg. Springer-Verlag.

Tellez , E. S. , S. Miranda-Jimenez , M. Gra , D. Moctezuma , R. R. Suarez , y O. S. Siordia . 2017 . A simple approach to multilingual polarity classi cation in Twitter . Pattern Recognition Letters , 94 : 68 { 74 .

Tellez , E. S. , D. Moctezuma , S. MirandaJimenez, y M. Gra . 2018 . An automated text categorization framework based on hyperparameter optimization . Knowledge-Based Systems , 149 : 110 { 123 .

Wang , B. , L. Gao , T. An , M. Meng, y T. Zhang . 2018 . A method of educational news classi cation based on emotional dictionary . En 2018 Chinese Control And Decision Conference (CCDC), paginas 3547 { 3551 , June.