=Paper=
{{Paper
|id=Vol-2172/p1_atalaya_tass2018
|storemode=property
|title=Atalaya at TASS 2018: Sentiment Analysis with
Tweet Embeddings and Data Augmentation
|pdfUrl=https://ceur-ws.org/Vol-2172/p1_atalaya_tass2018.pdf
|volume=Vol-2172
|authors=Franco M. Luque,Juan Manuel Pérez
|dblpUrl=https://dblp.org/rec/conf/sepln/LuqueP18
}}
==Atalaya at TASS 2018: Sentiment Analysis with
Tweet Embeddings and Data Augmentation==
https://ceur-ws.org/Vol-2172/p1_atalaya_tass2018.pdf
TASS 2018: Workshop on Semantic Analysis at SEPLN, septiembre 2018, págs. 29-35
Atalaya at TASS 2018: Sentiment Analysis with
Tweet Embeddings and Data Augmentation
Atalaya en TASS 2018: Análisis de Sentimiento con
Embeddings de Tweets y Aumentación de Datos
Franco M. Luque1 , Juan Manuel Pérez2
1
Universidad Nacional de Córdoba & CONICET
2
Universidad de Buenos Aires & CONICET
francolq@famaf.unc.edu.ar, jmperez@dc.uba.ar
Resumen: El workshop TASS 2018 propone diferentes desafı́os de análisis
semántico del Español. Este trabajo presenta nuestra participación con el equipo
Atalaya en la tarea de clasificación de polaridad de tweets. Seguimos técnicas
estándar de preprocesamiento, representación y clasificación, y también exploramos
algunas ideas novedosas. En particular, para obtener embeddings de tweets entre-
namos word embeddings con información de subpalabras, y usamos un esquema
de pesaje para promediarlos. Para lidiar con problemas de sobreajuste causados
por la escasez de datos de entrenamiento, probamos una estrategia de aumentación
de datos basada en traducción automática bidireccional. Experimentos con clasi-
ficadores lineales y modelos neuronales muestran resultados competitivos para las
diferentes subtareas propuestas en el desafı́o.
Palabras clave: Análisis de Sentimiento, Clasificación de Polaridad, Embeddings,
Aumentación de Datos, Modelos Lineales, Redes Neuronales
Abstract: TASS 2018 workshop proposes different challenges on semantic analy-
sis in Spanish. This work presents our participation as team Atalaya in the task of
polarity classification of tweets. We followed standard techniques in preprocessing,
representation and classification, and also explored some novel ideas. In particu-
lar, to obtain tweet embeddings we trained subword-aware word embeddings and
use a weighted scheme to average them. To deal with overfitting problems caused
by training data scarcity, we tried a data augmentation strategy based on two-way
machine translation. Experiments with linear classifiers and neural models show
competitive results for the different subtasks proposed in the challenge.
Keywords: Sentiment Analysis, Polarity Classification, Embeddings, Data Aug-
mentation, Linear Models, Neural Networks
1 Introduction an iterative experimental development pro-
cess. We tried different approaches for tweet
The TASS workshop presents every year dif- preprocessing, vector representation and po-
ferent challenges related to sentiment analy- larity classification models. Standard pre-
sis in Spanish. One of the main tasks is polar- processing techniques, including text sim-
ity classification of tweets and tweet aspects. plification, stopword filtering, lemmatization
In particular, task 1 of TASS 2018 (Martı́nez- and negation handling were used. Tweets
Cámara et al., 2018) proposes polarity clas- were represented with bag-of-words, bag-of-
sification on tweet datasets from three differ- characters, tweet embeddings and combina-
ent Spanish speaking countries: Spain (ES), tions of these. As classification models, we
Costa Rica (CR) and Perú (PE). This arti- considered linear classifiers and neural net-
cle describes our participation in TASS 2018 works.
task 1 with team Atalaya. We present polar-
ity classification systems using standard tech- We used fastText subword-aware word
niques and propose improvements based on vectors using tweet datasets specifically pre-
ISSN 1613-0073 Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes.
Franco M. Luque y Juan Manuel Pérez
pared for the task. Tweet vectors were com- • Lemmatization using TreeTagger
puted from word vectors using a weighted av- (Schmid, 1995).
eraging scheme, with weights inversely pro-
• Simple negation handling: We find nega-
portional to word frequency.
tion words and add the prefix ’NOT ’ to
To cope with scarcity of training data, we
the following tokens. Up to three tokens
experimented with a data augmentation trick
are negated, or less if a non-word token
based on translation of training data to other
is found. (Das et al., 2001; Pang, Lee,
languages and back to Spanish.
and Vaithyanathan, 2002)
Embedding weighting and data augmen-
tation represent novel approaches in the con- • Removal of punctuation.
text of TASS. In experiments, both ideas
showed improvements in prediction quality • Removal of consecutive repetitions of
for some configurations. handles and URLs.
The rest of the paper is as follows: Next No treatment was performed to hash-
section describes the main techniques and re- tags, emojis, interjections and onomatopeias.
sources we tried; section 3 presents the exper- Moreover, no spelling correction nor any
imental development of the systems, describ- other additional normalization was applied.
ing explored configurations and final models
selection; and section 4 summarizes our fi- 2.2 Bags of Words and Characters
nal results for the competition, and addresses The simplest approach we considered to build
conclusions and future work. tweet representations was the bag-of-words
2 Techniques and Resources encoding. A bag-of-words (BOW) builds fea-
ture vectors for each token seen in training
This section describes the main techniques data. For a particular tweet, its BOW vec-
and resources we used to define the basic tor contains the number of ocurrences of each
components to build our systems. token in the tweet. Resulting vectors are
2.1 Preprocessing high-dimensional and sparse. Variations of
BOWs include counting not only single to-
Preprocessing is crucial in NLP applica-
kens but also n-grams of tokens, binarizing
tions, specially when working with noisy
counts, and limiting the number of features.
user-generated data.
Character usage in tweets may also hold
We divided preprocessing into a two-stage
useful information for sentiment analysis.
process: First, we defined basic tweet pre-
Character n-grams —such as presence and
processing, using well-known standard and
repetition of uppercase letters, emoticons and
general purpose techniques; then, we defined
exclamation marks— may indicate strong
sentiment-oriented preprocessing, using tech-
presence of sentiment of some kind, where
niques that try to emphasize semantic infor-
others may indicate a more formal writing
mation.
style, and therefore an absence of sentiment.
Basic tweet preprocessing includes:
To capture this information, we consid-
• Tokenization using NLTK tweet tok- ered a bag-of-characters (BOC) representa-
enizer (Bird and Loper, 2004). tion that encodes counts of character n-grams
for some values of n. These vectors are com-
• Replacement of handles with token
puted from original texts of tweets, with no
’@USER’, URLs with ’URL’, and e-mails
preprocessing at all. BOCs have the same
with ’user@mail.com’.
variants and parameters as BOWs.
• Replacement of four or more repeated
letters with three letters. 2.3 Word Embeddings
Word embeddings are low-dimensional dense
Sentiment-oriented preprocessing includes
vector representations of words (Mikolov et
the following additional steps:
al., 2013). These representations encode syn-
• Lowercasing. tactical and semantical relations of words,
useful for NLP tasks, and they can be learned
• Removal of stopwords, using NLTK in an unsupervised fashion using large quan-
Spanish stopword list. tities of plain text, providing high vocabulary
• Removal of numbers. coverage. When precomputed embeddings
30
Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation
are used as features in supervised tasks, they ignores token repetitions in tweets; and nor-
provide robust information for words that are malization, which scales resulting tweet vec-
rare or unseen in training data. This is par- tors to have unit norm.
ticularly useful when training data is scarce,
as in this competition. 2.5 Data Augmentation
Recent work on embeddings introduced As the amount of training instances was
the usage of subword information to com- small, we paid special attention to model
pute word vectors. Informative representa- regularization. A technique used to ad-
tions for out-of-vocabulary (OOV) words can dress this is data augmentation, which con-
be obtained from subword embeddings. OOV sists of creating new synthetic instances out
words are an important issue when working of real ones by applying label-preserving
with highly noisy data such as user generated transformations. This overfitting-reduction
data in social networks. Here, the need for strategy is widely used in Computer Vision
text normalization in preprocessing can be (Krizhevsky, Sutskever, and Hinton, 2012;
alleviated with subword-based embeddings. Simard, Steinkraus, and Platt, 2003) and
In our work, we used fastText subword- Speech Recognition (Jaitly and Hinton, 2013;
based embeddings library (Bojanowski et al., Ko et al., 2015). For instance, images can be
2016). Instead of using pretrained vectors, zoomed, cropped, rotated, etc., while keeping
we decided to train our own embeddings on the objects in it still recognizable.
Twitter data. Data augmentation in NLP is a more sub-
To address the multilingual character of tle problem: there are no straightforward
the challenge, we first collected a database invariant-transformations such as in Com-
of ∼90 million tweets from various Spanish- puter Vision. A common technique (Zhang,
speaking countries, including the ones con- Zhao, and LeCun, 2015) is to replace words
cerning the challenge. Then, we prepared with synonyms using a thesaurus.
two versions of the data, one using only basic In this work we adopted a novel tech-
preprocessing, and the other one using senti- nique successfully used in a recent Kaggle
ment oriented preprocessing (only excepting NLP competition.1 The technique consists of
lemmatization). For these two datasets, we translating the texts to a different language,
trained skipgram embeddings using different and then translating them back to the orig-
parameter configurations, including the num- inal one. This process results in tweets that
ber of dimensions, size of word and subword vary lexically and syntactically, while mostly
n-grams and size of context window. keeping its meaning.
The tool selected to do this work was
2.4 Tweet Embeddings Google Translate, and the languages used
There are a number of ways of using word as intermediates were English, French, Por-
embeddings for sentiment analysis on tweets: tuguese and Arabic. We discarded other op-
approaches go from simple averaging of vec- tions (e.g. Mandarin Chinese) as they greatly
tors for each word in the tweet, to the use altered the meaning of tweets. Table 1 dis-
of more complex architectures such as CNNs plays examples of tweets and the resulting
or RNNs. In this work, we used averag- artificial instances.
ing to compute a single tweet embedding
3 Systems Development
of same dimensionality as the original word
embeddings. We followed two simple ap- This section describes the polarity classifica-
proaches: plain averaging and weighted av- tion systems we developed using the tools in-
eraging. For weighted averaging, we used a troduced in the previous section.
scheme that resembles Smooth Inverse Fre- We worked on two type of classifiers: lin-
quency (SIF) Arora, Liang, and Ma (2017), ear classifiers and neural networks. In both
inspired by TF-IDF reweighting. Each word cases, we tried to do some kind of model se-
a
w is weighted with a+p(w) , where p(w) is the lection, at times using development as the op-
word unigram probability, and a is a smooth- timization target, and at other times using
ing hyper-parameter. Big values of a means cross-validation on the combination of train
more smoothing towards plain averaging. and development.
We also considered two options that af- 1
https://www.kaggle.com/c/jigsaw-toxic-
fect tweet embeddings: binarization, which comment-classification-challenge/discussion/52557
31
Franco M. Luque y Juan Manuel Pérez
Original Tweet Data-augmented tweets
Gracias por la información. Parece que el olor ha - Gracias por la información. Parece que el olor se ha detenido. Ayer
cesado. Ayer pasó lo mismo sobre la misma hora sucedió lo mismo al mismo tiempo
- Gracias por la información. Parece que el olor se ha detenido. Ayer, lo
mismo ocurrió al mismo tiempo
Muy buenas amigos! Como podemos contactar - ¡Muy buenos amigos! ¿Cómo podemos ponernos en contacto con usted?
con ustedes - Muy buenos amigos! ¿Cómo podemos contactarlo?
La verdad es que tiene buena pinta. Investigaré, - La verdad es que parece bueno. Voy a investigar, gracias
gracias - La verdad es que se ve bien. Voy a investigar, gracias
- El hecho es que se ven bien. Lo comprobaré, gracias
Table 1: Data augmentation examples. Left column shows original tweets, right column shows
results of two-way translations for several intermediate languages.
Next subsections describe the experimen- Model BOW BOC M-F1 Acc.
tal development and the best configurations LR (1, 2) (1, 3) 0.496 0.634
we found for both types of system LR+DA (1, 2) (1, 3) 0.490 0.615
LR (1, 5) (1, 6) 0.493 0.634
3.1 Linear Classifiers LR+DA (1, 5) (1, 6) 0.529 0.648
We first built a classifying pipeline using sim-
ple linear classifying models —such as logis- Table 2: Experiments with logistic regres-
tic regressions and SVMs— that were imple- sions (LR), showing the interaction of train-
mented with scikit-learn (Pedregosa et al., ing data augmentation (DA) with n-gram
2011). Next, we describe the model selec- size ranges for bags of words and characters
tion process, done almost entirely using the (BOW and BOC, resp.). Results are on In-
InterTASS ES corpus. terTASS ES development set.
As input features, we combined the three
representations described in the previous sec- Macro-F1 over InterTASS ES development
tion: bag-of-words, bag-of-characters and set. In particular, the best regularization pa-
tweet embeddings. rameters found were C = 1.0 for LRs, and
For the bag of words and characters, early C = 0.05 for SVMs. Logistic regressions were
experiments showed a clear advantage of bi- selected over SVMs as they performed consis-
nary values over counts, together with TF- tently better in all experiments.
IDF re-weighting. First choices for n-gram When adding augmented data, first re-
ranges were (1, 2) for words and (1, 3) for sults showed a significant degradation in ac-
characters. curacy. However, an exploration of parame-
For the embeddings, sentiment-oriented ter values showed that it allowed an improve-
word vectors showed an advantage over ba- ment in performance when increasing the
sic vectors. We tried embeddings of dimen- range of n-gram sizes considered for BOWs
sions 50, 100, 200 and 300. Best results were and BOCs. Best results were found with up
found with 50 dimensions, and there were no to 5-grams for words, and up to 6-grams for
statistically significant differences. characters. Tab. 2 shows how data augmen-
To compute tweet embeddings, we tried tation combined with bigger n-gram ranges
basic averaging (as provided by fastText) and improved results.
the weighted averaging scheme described in Most previous parameter selection was re-
section 2.4. We experimented with smooth- viewed after data augmentation, confirming
ing values a = 10n for n ∈ {−3, . . . , 3} re- selected values. We also tried adding train-
sulting in a significant advantage of using ing data from General TASS corpus, to find
a = 0.1. Here, binarization and normaliza- that this was harmful for our models. With
tion as described in section 2.4 showed better the optimal models found in this process we
results. submitted final results for the Spanish (ES)
For the classifier, we tried logistic regres- monolingual task.
sions (LRs) and linear-kernel SVMs. To al- For Costa Rica (CR) and Perú (PE)
leviate the class imbalance problem, train- monolingual tasks, same values than for
ing items were weighted according to the in- ES were used for most parameters. Only
verse of the class frequency. Both LR and weighted averaging, data augmentation and
linear SVM hyper-parameters were selected n-gram ranges were explored. In CR data,
targeting the optimization of accuracy and weighting improved results, with the peak at
32
Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation
InputLayer
Task Model M-F1 Acc.
MLP 0.476 0.544
Mono ES
LR 0.468 0.599
MLP 0.451 0.562
dense_1: Dense
Mono CR
LR 0.475 0.582
MLP 0.437 0.520
Mono PE
LR 0.462 0.451
dropout_1: Dropout
Cross Lingual ES 0.441 0.485
Cross Lingual PE MLP 0.438 0.523
Cross Lingual CR 0.453 0.565
dense_2: Dense Table 3: Submitted results for each subtask.
rons and keep-probabilities2 .
dropout_2: Dropout Results of this search showed that bag-of-
words features and embedding weighting did
not improve performance. Regarding the
Output: Softmax MLP architecture, we selected 256 as the
size of the first layer and 128 for the sec-
ond, and keep-probabilities of 0.25 and 0.55
Figure 1: Architecture of the MLP. respectively. This configuration was used in
all subtasks.
a = 0.5. Data augmentation also was good, Data augmentation in combination with
with the best results using up to 4-grams for MLPs showed mixed results. For the mono-
words and 6-grams for characters. In PE lingual ES subtask, using synthetic data re-
data, neither weighting nor data augmenta- sulted in a Macro-F1 gain while for monolin-
tion were helpful. Best results were found gual PE it degraded the results.
using up to 2-gram for words and 5-grams All monolingual models were trained us-
for characters. ing the respective train sections of InterTASS
datasets. General TASS was not used as it
3.2 Multilayer Perceptron not showed improvements. For cross-lingual
In the second set of experiments we used mul- tasks, models for each language were trained
tilayer perceptrons (MLP) neural networks. using the datasets for the two other lan-
MLPs performed well in previous editions of guages.
the challenge (Dı́az-Galiano et al., 2018). We used Keras (Chollet and others, 2015)
to implement the model and scikit-learn (Pe-
Fig. 1 displays the chosen architecture,
dregosa et al., 2011) to perform the cross-
consisting of two hidden layers and a softmax
validation.
output. ReLU units were used as activation
functions in the hidden layers. To avoid over-
4 Conclusions and Future Work
fitting, we tried dropout (Srivastava et al.,
2014) and early stopping. We presented our participation on TASS 2018
To find the best configurations, we per- task 1 as team Atalaya. We explored stan-
formed random search (Bergstra and Ben- dard approaches as well as some simple but
gio, 2012) using 5-fold cross-validation over original recent ideas such as data augmenta-
the InterTASS ES training and development tion and word embedding weighting. Table
datasets. The explored configurations and 3 displays results for each subtask. Our sys-
hyperparameters were: tems ranked among the first three in all the
subtasks.
• BOW features: No BOW features at all, Experiments show that competitive re-
top-50 or top-150. sults can be achieved without having to re-
sort to complex neural architectures such as
• Tweet embeddings: Basic or weighted CNNs, RNNs, LSTMs, etc. Even simple lo-
averaging.
2
Probability of keeping the value of a neuron when
• Hidden layers: Different number of neu- training with dropout.
33
Franco M. Luque y Juan Manuel Pérez
gistic regressions were able to rank among the Krizhevsky, A., I. Sutskever, and G. E. Hin-
top performing systems. ton. 2012. Imagenet classification with
Future work includes further exploration deep convolutional neural networks. In
on data augmentation, tweet embedding Advances in neural information processing
techniques, and sentiment-oriented word em- systems, pages 1097–1105.
beddings. We also aim at improving prepro-
cessing and adopting modern neural classify- Martı́nez-Cámara, E., Y. Almeida Cruz,
ing models. M. C. Dı́az-Galiano, S. Estévez Velarde,
M. A. Garcı́a-Cumbreras, M. Garcı́a-
References Vega, Y. Gutiérrez Vázquez, A. Mon-
tejo Ráez, A. Montoyo Guijarro,
Arora, S., Y. Liang, and T. Ma. 2017. A
R. Muñoz Guillena, A. Piad Morffis,
simple but tough-to-beat baseline for sen-
and J. Villena-Román. 2018. Overview
tence embeddings.
of TASS 2018: Opinions, health and
Bergstra, J. and Y. Bengio. 2012. Ran- emotions. In E. Martı́nez-Cámara,
dom search for hyper-parameter optimiza- Y. Almeida Cruz, M. C. Dı́az-
tion. Journal of Machine Learning Re- Galiano, S. Estévez Velarde, M. A.
search, 13(Feb):281–305. Garcı́a-Cumbreras, M. Garcı́a-Vega,
Bird, S. and E. Loper. 2004. Nltk: the nat- Y. Gutiérrez Vázquez, A. Montejo Ráez,
ural language toolkit. In Proceedings of A. Montoyo Guijarro, R. Muñoz Guillena,
the ACL 2004 on Interactive poster and A. Piad Morffis, and J. Villena-Román,
demonstration sessions, page 31. Associa- editors, Proceedings of TASS 2018: Work-
tion for Computational Linguistics. shop on Semantic Analysis at SEPLN
(TASS 2018), volume 2172 of CEUR
Bojanowski, P., E. Grave, A. Joulin, and Workshop Proceedings, Sevilla, Spain,
T. Mikolov. 2016. Enriching word vectors September. CEUR-WS.
with subword information. arXiv preprint
arXiv:1607.04606. Mikolov, T., I. Sutskever, K. Chen, G. Cor-
rado, and J. Dean. 2013. Distributed
Chollet, F. et al. 2015. Keras. representations of words and phrases
Das, S. R., M. Y. Chen, T. V. Agar- and their compositionality. CoRR,
wal, C. Brooks, Y. shee Chan, D. Gib- abs/1310.4546.
son, D. Leinweber, A. Martinez-jerez,
Pang, B., L. Lee, and S. Vaithyanathan.
P. Raghubir, S. Rajagopalan, A. Ranade,
2002. Thumbs up? sentiment classifi-
M. Rubinstein, and P. Tufano. 2001. Ya-
cation using machine learning techniques.
hoo! for amazon: Sentiment extraction
In Proceedings of the 2002 Conference on
from small talk on the web. In 8th Asia
Empirical Methods in Natural Language
Pacific Finance Association Annual Con-
Processing, pages 79–86. Association for
ference.
Computational Linguistics, July.
Dı́az-Galiano, M. C., E. Martı́nez-Cámara,
M. Ángel Garcı́a Cumbreras, M. G. Vega, Pedregosa, F., G. Varoquaux, A. Gram-
and J. V. Román. 2018. The democratiza- fort, V. Michel, B. Thirion, O. Grisel,
tion of deep learning in tass 2017. Proce- M. Blondel, P. Prettenhofer, R. Weiss,
samiento del Lenguaje Natural, 60(0):37– V. Dubourg, J. Vanderplas, A. Passos,
44. D. Cournapeau, M. Brucher, M. Perrot,
and E. Duchesnay. 2011. Scikit-learn:
Jaitly, N. and G. E. Hinton. 2013. Vocal Machine learning in Python. Journal
tract length perturbation (vtlp) improves of Machine Learning Research, 12:2825–
speech recognition. In Proc. ICML Work- 2830.
shop on Deep Learning for Audio, Speech
and Language, volume 117. Schmid, H. 1995. Improvements in part-
of-speech tagging with an application to
Ko, T., V. Peddinti, D. Povey, and S. Khu- german. In In Proceedings of the ACL
danpur. 2015. Audio augmentation for SIGDAT-Workshop, pages 47–50.
speech recognition. In Sixteenth Annual
Conference of the International Speech Simard, P. Y., D. Steinkraus, and J. C. Platt.
Communication Association. 2003. Best practices for convolutional
34
Atalaya at TASS 2018: Sentiment Analysis with Tweet Embeddings and Data Augmentation
neural networks applied to visual docu-
ment analysis. In null, page 958. IEEE.
Srivastava, N., G. Hinton, A. Krizhevsky,
I. Sutskever, and R. Salakhutdinov. 2014.
Dropout: a simple way to prevent neural
networks from overfitting. The Journal of
Machine Learning Research, 15(1):1929–
1958.
Zhang, X., J. Zhao, and Y. LeCun. 2015.
Character-level convolutional networks for
text classification. In C. Cortes, N. D.
Lawrence, D. D. Lee, M. Sugiyama, and
R. Garnett, editors, Advances in Neural
Information Processing Systems 28. Cur-
ran Associates, Inc., pages 649–657.
35