TASS 2018: Workshop on Semantic Analysis at SEPLN, septiembre 2018, págs. 29-35 Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation Atalaya en TASS 2018: Análisis de Sentimiento con Embeddings de Tweets y Aumentación de Datos Franco M. Luque1 , Juan Manuel Pérez2 1 Universidad Nacional de Córdoba & CONICET 2 Universidad de Buenos Aires & CONICET francolq@famaf.unc.edu.ar, jmperez@dc.uba.ar Resumen: El workshop TASS 2018 propone diferentes desafı́os de análisis semántico del Español. Este trabajo presenta nuestra participación con el equipo Atalaya en la tarea de clasificación de polaridad de tweets. Seguimos técnicas estándar de preprocesamiento, representación y clasificación, y también exploramos algunas ideas novedosas. En particular, para obtener embeddings de tweets entre- namos word embeddings con información de subpalabras, y usamos un esquema de pesaje para promediarlos. Para lidiar con problemas de sobreajuste causados por la escasez de datos de entrenamiento, probamos una estrategia de aumentación de datos basada en traducción automática bidireccional. Experimentos con clasi- ficadores lineales y modelos neuronales muestran resultados competitivos para las diferentes subtareas propuestas en el desafı́o. Palabras clave: Análisis de Sentimiento, Clasificación de Polaridad, Embeddings, Aumentación de Datos, Modelos Lineales, Redes Neuronales Abstract: TASS 2018 workshop proposes different challenges on semantic analy- sis in Spanish. This work presents our participation as team Atalaya in the task of polarity classification of tweets. We followed standard techniques in preprocessing, representation and classification, and also explored some novel ideas. In particu- lar, to obtain tweet embeddings we trained subword-aware word embeddings and use a weighted scheme to average them. To deal with overfitting problems caused by training data scarcity, we tried a data augmentation strategy based on two-way machine translation. Experiments with linear classifiers and neural models show competitive results for the different subtasks proposed in the challenge. Keywords: Sentiment Analysis, Polarity Classification, Embeddings, Data Aug- mentation, Linear Models, Neural Networks 1 Introduction an iterative experimental development pro- cess. We tried different approaches for tweet The TASS workshop presents every year dif- preprocessing, vector representation and po- ferent challenges related to sentiment analy- larity classification models. Standard pre- sis in Spanish. One of the main tasks is polar- processing techniques, including text sim- ity classification of tweets and tweet aspects. plification, stopword filtering, lemmatization In particular, task 1 of TASS 2018 (Martı́nez- and negation handling were used. Tweets Cámara et al., 2018) proposes polarity clas- were represented with bag-of-words, bag-of- sification on tweet datasets from three differ- characters, tweet embeddings and combina- ent Spanish speaking countries: Spain (ES), tions of these. As classification models, we Costa Rica (CR) and Perú (PE). This arti- considered linear classifiers and neural net- cle describes our participation in TASS 2018 works. task 1 with team Atalaya. We present polar- ity classification systems using standard tech- We used fastText subword-aware word niques and propose improvements based on vectors using tweet datasets specifically pre- ISSN 1613-0073 Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes. Franco M. Luque y Juan Manuel Pérez pared for the task. Tweet vectors were com- • Lemmatization using TreeTagger puted from word vectors using a weighted av- (Schmid, 1995). eraging scheme, with weights inversely pro- • Simple negation handling: We find nega- portional to word frequency. tion words and add the prefix ’NOT ’ to To cope with scarcity of training data, we the following tokens. Up to three tokens experimented with a data augmentation trick are negated, or less if a non-word token based on translation of training data to other is found. (Das et al., 2001; Pang, Lee, languages and back to Spanish. and Vaithyanathan, 2002) Embedding weighting and data augmen- tation represent novel approaches in the con- • Removal of punctuation. text of TASS. In experiments, both ideas showed improvements in prediction quality • Removal of consecutive repetitions of for some configurations. handles and URLs. The rest of the paper is as follows: Next No treatment was performed to hash- section describes the main techniques and re- tags, emojis, interjections and onomatopeias. sources we tried; section 3 presents the exper- Moreover, no spelling correction nor any imental development of the systems, describ- other additional normalization was applied. ing explored configurations and final models selection; and section 4 summarizes our fi- 2.2 Bags of Words and Characters nal results for the competition, and addresses The simplest approach we considered to build conclusions and future work. tweet representations was the bag-of-words 2 Techniques and Resources encoding. A bag-of-words (BOW) builds fea- ture vectors for each token seen in training This section describes the main techniques data. For a particular tweet, its BOW vec- and resources we used to define the basic tor contains the number of ocurrences of each components to build our systems. token in the tweet. Resulting vectors are 2.1 Preprocessing high-dimensional and sparse. Variations of BOWs include counting not only single to- Preprocessing is crucial in NLP applica- kens but also n-grams of tokens, binarizing tions, specially when working with noisy counts, and limiting the number of features. user-generated data. Character usage in tweets may also hold We divided preprocessing into a two-stage useful information for sentiment analysis. process: First, we defined basic tweet pre- Character n-grams —such as presence and processing, using well-known standard and repetition of uppercase letters, emoticons and general purpose techniques; then, we defined exclamation marks— may indicate strong sentiment-oriented preprocessing, using tech- presence of sentiment of some kind, where niques that try to emphasize semantic infor- others may indicate a more formal writing mation. style, and therefore an absence of sentiment. Basic tweet preprocessing includes: To capture this information, we consid- • Tokenization using NLTK tweet tok- ered a bag-of-characters (BOC) representa- enizer (Bird and Loper, 2004). tion that encodes counts of character n-grams for some values of n. These vectors are com- • Replacement of handles with token puted from original texts of tweets, with no ’@USER’, URLs with ’URL’, and e-mails preprocessing at all. BOCs have the same with ’user@mail.com’. variants and parameters as BOWs. • Replacement of four or more repeated letters with three letters. 2.3 Word Embeddings Word embeddings are low-dimensional dense Sentiment-oriented preprocessing includes vector representations of words (Mikolov et the following additional steps: al., 2013). These representations encode syn- • Lowercasing. tactical and semantical relations of words, useful for NLP tasks, and they can be learned • Removal of stopwords, using NLTK in an unsupervised fashion using large quan- Spanish stopword list. tities of plain text, providing high vocabulary • Removal of numbers. coverage. When precomputed embeddings 30 Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation are used as features in supervised tasks, they ignores token repetitions in tweets; and nor- provide robust information for words that are malization, which scales resulting tweet vec- rare or unseen in training data. This is par- tors to have unit norm. ticularly useful when training data is scarce, as in this competition. 2.5 Data Augmentation Recent work on embeddings introduced As the amount of training instances was the usage of subword information to com- small, we paid special attention to model pute word vectors. Informative representa- regularization. A technique used to ad- tions for out-of-vocabulary (OOV) words can dress this is data augmentation, which con- be obtained from subword embeddings. OOV sists of creating new synthetic instances out words are an important issue when working of real ones by applying label-preserving with highly noisy data such as user generated transformations. This overfitting-reduction data in social networks. Here, the need for strategy is widely used in Computer Vision text normalization in preprocessing can be (Krizhevsky, Sutskever, and Hinton, 2012; alleviated with subword-based embeddings. Simard, Steinkraus, and Platt, 2003) and In our work, we used fastText subword- Speech Recognition (Jaitly and Hinton, 2013; based embeddings library (Bojanowski et al., Ko et al., 2015). For instance, images can be 2016). Instead of using pretrained vectors, zoomed, cropped, rotated, etc., while keeping we decided to train our own embeddings on the objects in it still recognizable. Twitter data. Data augmentation in NLP is a more sub- To address the multilingual character of tle problem: there are no straightforward the challenge, we first collected a database invariant-transformations such as in Com- of ∼90 million tweets from various Spanish- puter Vision. A common technique (Zhang, speaking countries, including the ones con- Zhao, and LeCun, 2015) is to replace words cerning the challenge. Then, we prepared with synonyms using a thesaurus. two versions of the data, one using only basic In this work we adopted a novel tech- preprocessing, and the other one using senti- nique successfully used in a recent Kaggle ment oriented preprocessing (only excepting NLP competition.1 The technique consists of lemmatization). For these two datasets, we translating the texts to a different language, trained skipgram embeddings using different and then translating them back to the orig- parameter configurations, including the num- inal one. This process results in tweets that ber of dimensions, size of word and subword vary lexically and syntactically, while mostly n-grams and size of context window. keeping its meaning. The tool selected to do this work was 2.4 Tweet Embeddings Google Translate, and the languages used There are a number of ways of using word as intermediates were English, French, Por- embeddings for sentiment analysis on tweets: tuguese and Arabic. We discarded other op- approaches go from simple averaging of vec- tions (e.g. Mandarin Chinese) as they greatly tors for each word in the tweet, to the use altered the meaning of tweets. Table 1 dis- of more complex architectures such as CNNs plays examples of tweets and the resulting or RNNs. In this work, we used averag- artificial instances. ing to compute a single tweet embedding 3 Systems Development of same dimensionality as the original word embeddings. We followed two simple ap- This section describes the polarity classifica- proaches: plain averaging and weighted av- tion systems we developed using the tools in- eraging. For weighted averaging, we used a troduced in the previous section. scheme that resembles Smooth Inverse Fre- We worked on two type of classifiers: lin- quency (SIF) Arora, Liang, and Ma (2017), ear classifiers and neural networks. In both inspired by TF-IDF reweighting. Each word cases, we tried to do some kind of model se- a w is weighted with a+p(w) , where p(w) is the lection, at times using development as the op- word unigram probability, and a is a smooth- timization target, and at other times using ing hyper-parameter. Big values of a means cross-validation on the combination of train more smoothing towards plain averaging. and development. We also considered two options that af- 1 https://www.kaggle.com/c/jigsaw-toxic- fect tweet embeddings: binarization, which comment-classification-challenge/discussion/52557 31 Franco M. Luque y Juan Manuel Pérez Original Tweet Data-augmented tweets Gracias por la información. Parece que el olor ha - Gracias por la información. Parece que el olor se ha detenido. Ayer cesado. Ayer pasó lo mismo sobre la misma hora sucedió lo mismo al mismo tiempo - Gracias por la información. Parece que el olor se ha detenido. Ayer, lo mismo ocurrió al mismo tiempo Muy buenas amigos! Como podemos contactar - ¡Muy buenos amigos! ¿Cómo podemos ponernos en contacto con usted? con ustedes - Muy buenos amigos! ¿Cómo podemos contactarlo? La verdad es que tiene buena pinta. Investigaré, - La verdad es que parece bueno. Voy a investigar, gracias gracias - La verdad es que se ve bien. Voy a investigar, gracias - El hecho es que se ven bien. Lo comprobaré, gracias Table 1: Data augmentation examples. Left column shows original tweets, right column shows results of two-way translations for several intermediate languages. Next subsections describe the experimen- Model BOW BOC M-F1 Acc. tal development and the best configurations LR (1, 2) (1, 3) 0.496 0.634 we found for both types of system LR+DA (1, 2) (1, 3) 0.490 0.615 LR (1, 5) (1, 6) 0.493 0.634 3.1 Linear Classifiers LR+DA (1, 5) (1, 6) 0.529 0.648 We first built a classifying pipeline using sim- ple linear classifying models —such as logis- Table 2: Experiments with logistic regres- tic regressions and SVMs— that were imple- sions (LR), showing the interaction of train- mented with scikit-learn (Pedregosa et al., ing data augmentation (DA) with n-gram 2011). Next, we describe the model selec- size ranges for bags of words and characters tion process, done almost entirely using the (BOW and BOC, resp.). Results are on In- InterTASS ES corpus. terTASS ES development set. As input features, we combined the three representations described in the previous sec- Macro-F1 over InterTASS ES development tion: bag-of-words, bag-of-characters and set. In particular, the best regularization pa- tweet embeddings. rameters found were C = 1.0 for LRs, and For the bag of words and characters, early C = 0.05 for SVMs. Logistic regressions were experiments showed a clear advantage of bi- selected over SVMs as they performed consis- nary values over counts, together with TF- tently better in all experiments. IDF re-weighting. First choices for n-gram When adding augmented data, first re- ranges were (1, 2) for words and (1, 3) for sults showed a significant degradation in ac- characters. curacy. However, an exploration of parame- For the embeddings, sentiment-oriented ter values showed that it allowed an improve- word vectors showed an advantage over ba- ment in performance when increasing the sic vectors. We tried embeddings of dimen- range of n-gram sizes considered for BOWs sions 50, 100, 200 and 300. Best results were and BOCs. Best results were found with up found with 50 dimensions, and there were no to 5-grams for words, and up to 6-grams for statistically significant differences. characters. Tab. 2 shows how data augmen- To compute tweet embeddings, we tried tation combined with bigger n-gram ranges basic averaging (as provided by fastText) and improved results. the weighted averaging scheme described in Most previous parameter selection was re- section 2.4. We experimented with smooth- viewed after data augmentation, confirming ing values a = 10n for n ∈ {−3, . . . , 3} re- selected values. We also tried adding train- sulting in a significant advantage of using ing data from General TASS corpus, to find a = 0.1. Here, binarization and normaliza- that this was harmful for our models. With tion as described in section 2.4 showed better the optimal models found in this process we results. submitted final results for the Spanish (ES) For the classifier, we tried logistic regres- monolingual task. sions (LRs) and linear-kernel SVMs. To al- For Costa Rica (CR) and Perú (PE) leviate the class imbalance problem, train- monolingual tasks, same values than for ing items were weighted according to the in- ES were used for most parameters. Only verse of the class frequency. Both LR and weighted averaging, data augmentation and linear SVM hyper-parameters were selected n-gram ranges were explored. In CR data, targeting the optimization of accuracy and weighting improved results, with the peak at 32 Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation InputLayer Task Model M-F1 Acc. MLP 0.476 0.544 Mono ES LR 0.468 0.599 MLP 0.451 0.562 dense_1: Dense Mono CR LR 0.475 0.582 MLP 0.437 0.520 Mono PE LR 0.462 0.451 dropout_1: Dropout Cross Lingual ES 0.441 0.485 Cross Lingual PE MLP 0.438 0.523 Cross Lingual CR 0.453 0.565 dense_2: Dense Table 3: Submitted results for each subtask. rons and keep-probabilities2 . dropout_2: Dropout Results of this search showed that bag-of- words features and embedding weighting did not improve performance. Regarding the Output: Softmax MLP architecture, we selected 256 as the size of the first layer and 128 for the sec- ond, and keep-probabilities of 0.25 and 0.55 Figure 1: Architecture of the MLP. respectively. This configuration was used in all subtasks. a = 0.5. Data augmentation also was good, Data augmentation in combination with with the best results using up to 4-grams for MLPs showed mixed results. For the mono- words and 6-grams for characters. In PE lingual ES subtask, using synthetic data re- data, neither weighting nor data augmenta- sulted in a Macro-F1 gain while for monolin- tion were helpful. Best results were found gual PE it degraded the results. using up to 2-gram for words and 5-grams All monolingual models were trained us- for characters. ing the respective train sections of InterTASS datasets. General TASS was not used as it 3.2 Multilayer Perceptron not showed improvements. For cross-lingual In the second set of experiments we used mul- tasks, models for each language were trained tilayer perceptrons (MLP) neural networks. using the datasets for the two other lan- MLPs performed well in previous editions of guages. the challenge (Dı́az-Galiano et al., 2018). We used Keras (Chollet and others, 2015) to implement the model and scikit-learn (Pe- Fig. 1 displays the chosen architecture, dregosa et al., 2011) to perform the cross- consisting of two hidden layers and a softmax validation. output. ReLU units were used as activation functions in the hidden layers. To avoid over- 4 Conclusions and Future Work fitting, we tried dropout (Srivastava et al., 2014) and early stopping. We presented our participation on TASS 2018 To find the best configurations, we per- task 1 as team Atalaya. We explored stan- formed random search (Bergstra and Ben- dard approaches as well as some simple but gio, 2012) using 5-fold cross-validation over original recent ideas such as data augmenta- the InterTASS ES training and development tion and word embedding weighting. Table datasets. The explored configurations and 3 displays results for each subtask. Our sys- hyperparameters were: tems ranked among the first three in all the subtasks. • BOW features: No BOW features at all, Experiments show that competitive re- top-50 or top-150. sults can be achieved without having to re- sort to complex neural architectures such as • Tweet embeddings: Basic or weighted CNNs, RNNs, LSTMs, etc. Even simple lo- averaging. 2 Probability of keeping the value of a neuron when • Hidden layers: Different number of neu- training with dropout. 33 Franco M. Luque y Juan Manuel Pérez gistic regressions were able to rank among the Krizhevsky, A., I. Sutskever, and G. E. Hin- top performing systems. ton. 2012. Imagenet classification with Future work includes further exploration deep convolutional neural networks. In on data augmentation, tweet embedding Advances in neural information processing techniques, and sentiment-oriented word em- systems, pages 1097–1105. beddings. We also aim at improving prepro- cessing and adopting modern neural classify- Martı́nez-Cámara, E., Y. Almeida Cruz, ing models. M. C. Dı́az-Galiano, S. Estévez Velarde, M. A. Garcı́a-Cumbreras, M. Garcı́a- References Vega, Y. Gutiérrez Vázquez, A. Mon- tejo Ráez, A. Montoyo Guijarro, Arora, S., Y. Liang, and T. Ma. 2017. A R. Muñoz Guillena, A. Piad Morffis, simple but tough-to-beat baseline for sen- and J. Villena-Román. 2018. Overview tence embeddings. of TASS 2018: Opinions, health and Bergstra, J. and Y. Bengio. 2012. Ran- emotions. In E. Martı́nez-Cámara, dom search for hyper-parameter optimiza- Y. Almeida Cruz, M. C. Dı́az- tion. Journal of Machine Learning Re- Galiano, S. Estévez Velarde, M. A. search, 13(Feb):281–305. Garcı́a-Cumbreras, M. Garcı́a-Vega, Bird, S. and E. Loper. 2004. Nltk: the nat- Y. Gutiérrez Vázquez, A. Montejo Ráez, ural language toolkit. In Proceedings of A. Montoyo Guijarro, R. Muñoz Guillena, the ACL 2004 on Interactive poster and A. Piad Morffis, and J. Villena-Román, demonstration sessions, page 31. Associa- editors, Proceedings of TASS 2018: Work- tion for Computational Linguistics. shop on Semantic Analysis at SEPLN (TASS 2018), volume 2172 of CEUR Bojanowski, P., E. Grave, A. Joulin, and Workshop Proceedings, Sevilla, Spain, T. Mikolov. 2016. Enriching word vectors September. CEUR-WS. with subword information. arXiv preprint arXiv:1607.04606. Mikolov, T., I. Sutskever, K. Chen, G. Cor- rado, and J. Dean. 2013. Distributed Chollet, F. et al. 2015. Keras. representations of words and phrases Das, S. R., M. Y. Chen, T. V. Agar- and their compositionality. CoRR, wal, C. Brooks, Y. shee Chan, D. Gib- abs/1310.4546. son, D. Leinweber, A. Martinez-jerez, Pang, B., L. Lee, and S. Vaithyanathan. P. Raghubir, S. Rajagopalan, A. Ranade, 2002. Thumbs up? sentiment classifi- M. Rubinstein, and P. Tufano. 2001. Ya- cation using machine learning techniques. hoo! for amazon: Sentiment extraction In Proceedings of the 2002 Conference on from small talk on the web. In 8th Asia Empirical Methods in Natural Language Pacific Finance Association Annual Con- Processing, pages 79–86. Association for ference. Computational Linguistics, July. Dı́az-Galiano, M. C., E. Martı́nez-Cámara, M. Ángel Garcı́a Cumbreras, M. G. Vega, Pedregosa, F., G. Varoquaux, A. Gram- and J. V. Román. 2018. The democratiza- fort, V. Michel, B. Thirion, O. Grisel, tion of deep learning in tass 2017. Proce- M. Blondel, P. Prettenhofer, R. Weiss, samiento del Lenguaje Natural, 60(0):37– V. Dubourg, J. Vanderplas, A. Passos, 44. D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Jaitly, N. and G. E. Hinton. 2013. Vocal Machine learning in Python. Journal tract length perturbation (vtlp) improves of Machine Learning Research, 12:2825– speech recognition. In Proc. ICML Work- 2830. shop on Deep Learning for Audio, Speech and Language, volume 117. Schmid, H. 1995. Improvements in part- of-speech tagging with an application to Ko, T., V. Peddinti, D. Povey, and S. Khu- german. In In Proceedings of the ACL danpur. 2015. Audio augmentation for SIGDAT-Workshop, pages 47–50. speech recognition. In Sixteenth Annual Conference of the International Speech Simard, P. Y., D. Steinkraus, and J. C. Platt. Communication Association. 2003. Best practices for convolutional 34 Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation neural networks applied to visual docu- ment analysis. In null, page 958. IEEE. Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929– 1958. Zhang, X., J. Zhao, and Y. LeCun. 2015. Character-level convolutional networks for text classification. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28. Cur- ran Associates, Inc., pages 649–657. 35