TASS 2018 workshop proposes di erent challenges on semantic analysis in Spanish. This work presents our participation as team Atalaya in the task of polarity classi cation of tweets. We followed standard techniques in preprocessing, representation and classi cation, and also explored some novel ideas. In particular, to obtain tweet embeddings we trained subword-aware word embeddings and use a weighted scheme to average them. To deal with over tting problems caused by training data scarcity, we tried a data augmentation strategy based on two-way machine translation. Experiments with linear classi ers and neural models show competitive results for the di erent subtasks proposed in the challenge.
The TASS workshop presents every year different challenges related to sentiment analysis in Spanish. One of the main tasks is polarity classi cation of tweets and tweet aspects.
In particular, task 1 of TASS 2018
We used fastText subword-aware word vectors using tweet datasets speci cally pre
Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes. pared for the task. Tweet vectors were computed from word vectors using a weighted averaging scheme, with weights inversely proportional to word frequency.
To cope with scarcity of training data, we experimented with a data augmentation trick based on translation of training data to other languages and back to Spanish.
Embedding weighting and data augmentation represent novel approaches in the context of TASS. In experiments, both ideas showed improvements in prediction quality for some con gurations.
The rest of the paper is as follows: Next section describes the main techniques and resources we tried; section 3 presents the experimental development of the systems, describing explored con gurations and nal models selection; and section 4 summarizes our nal results for the competition, and addresses conclusions and future work. 2
This section describes the main techniques and resources we used to de ne the basic components to build our systems. 2.1
Preprocessing is crucial in NLP applications, specially when working with noisy user-generated data.
We divided preprocessing into a two-stage process: First, we de ned basic tweet preprocessing, using well-known standard and general purpose techniques; then, we de ned sentiment-oriented preprocessing, using techniques that try to emphasize semantic information.
Basic tweet preprocessing includes:
Tokenization using NLTK tweet
tokenizer
Replacement of handles with token '@USER', URLs with 'URL', and e-mails with 'user@mail.com'.
Replacement of four or more repeated letters with three letters.
Sentiment-oriented preprocessing includes the following additional steps:
Removal of stopwords, using NLTK Spanish stopword list.
using
Simple negation handling: We nd negation words and add the pre x 'NOT ' to the following tokens. Up to three tokens are negated, or less if a non-word token is found. (Das et al., 2001; Pang, Lee, and Vaithyanathan, 2002)
Removal of consecutive repetitions of handles and URLs.
No treatment was performed to hashtags, emojis, interjections and onomatopeias. Moreover, no spelling correction nor any other additional normalization was applied. 2.2
The simplest approach we considered to build tweet representations was the bag-of-words encoding. A bag-of-words (BOW) builds feature vectors for each token seen in training data. For a particular tweet, its BOW vector contains the number of ocurrences of each token in the tweet. Resulting vectors are high-dimensional and sparse. Variations of BOWs include counting not only single tokens but also n-grams of tokens, binarizing counts, and limiting the number of features.
Character usage in tweets may also hold useful information for sentiment analysis. Character n-grams |such as presence and repetition of uppercase letters, emoticons and exclamation marks| may indicate strong presence of sentiment of some kind, where others may indicate a more formal writing style, and therefore an absence of sentiment.
To capture this information, we considered a bag-of-characters (BOC) representation that encodes counts of character n-grams for some values of n. These vectors are computed from original texts of tweets, with no preprocessing at all. BOCs have the same variants and parameters as BOWs. 2.3
Word embeddings are low-dimensional dense
vector representations of words
Recent work on embeddings introduced the usage of subword information to compute word vectors. Informative representations for out-of-vocabulary (OOV) words can be obtained from subword embeddings. OOV words are an important issue when working with highly noisy data such as user generated data in social networks. Here, the need for text normalization in preprocessing can be alleviated with subword-based embeddings.
In our work, we used fastText
subwordbased embeddings library
To address the multilingual character of the challenge, we rst collected a database of 90 million tweets from various Spanishspeaking countries, including the ones concerning the challenge. Then, we prepared two versions of the data, one using only basic preprocessing, and the other one using sentiment oriented preprocessing (only excepting lemmatization). For these two datasets, we trained skipgram embeddings using di erent parameter con gurations, including the number of dimensions, size of word and subword n-grams and size of context window. 2.4
There are a number of ways of using word
embeddings for sentiment analysis on tweets:
approaches go from simple averaging of
vectors for each word in the tweet, to the use
of more complex architectures such as CNNs
or RNNs. In this work, we used
averaging to compute a single tweet embedding
of same dimensionality as the original word
embeddings. We followed two simple
approaches: plain averaging and weighted
averaging. For weighted averaging, we used a
scheme that resembles Smooth Inverse
Frequency (SIF)
We also considered two options that affect tweet embeddings: binarization, which ignores token repetitions in tweets; and normalization, which scales resulting tweet vectors to have unit norm. 2.5
As the amount of training instances was
small, we paid special attention to model
regularization. A technique used to
address this is data augmentation, which
consists of creating new synthetic instances out
of real ones by applying label-preserving
transformations. This over tting-reduction
strategy is widely used in Computer Vision
Data augmentation in NLP is a more
subtle problem: there are no straightforward
invariant-transformations such as in
Computer Vision. A common technique
In this work we adopted a novel technique successfully used in a recent Kaggle NLP competition.1 The technique consists of translating the texts to a di erent language, and then translating them back to the original one. This process results in tweets that vary lexically and syntactically, while mostly keeping its meaning.
The tool selected to do this work was Google Translate, and the languages used as intermediates were English, French, Portuguese and Arabic. We discarded other options (e.g. Mandarin Chinese) as they greatly altered the meaning of tweets. Table 1 displays examples of tweets and the resulting arti cial instances. 3
This section describes the polarity classi cation systems we developed using the tools introduced in the previous section.
We worked on two type of classi ers: linear classi ers and neural networks. In both cases, we tried to do some kind of model selection, at times using development as the optimization target, and at other times using cross-validation on the combination of train and development.
1https://www.kaggle.com/c/jigsaw-toxiccomment-classi cation-challenge/discussion/52557 Original Tweet Data-augmented tweets Gracias por la informacion. Parece que el olor ha - Gracias por la informacion. Parece que el olor se ha detenido. Ayer cesado. Ayer paso lo mismo sobre la misma hora sucedio lo mismo al mismo tiempo - Gracias por la informacion. Parece que el olor se ha detenido. Ayer, lo mismo ocurrio al mismo tiempo Muy buenas amigos! Como podemos contactar - <Muy buenos amigos! >Como podemos ponernos en contacto con usted? con ustedes - Muy buenos amigos! >Como podemos contactarlo? La verdad es que tiene buena pinta. Investigare, - La verdad es que parece bueno. Voy a investigar, gracias gracias - La verdad es que se ve bien. Voy a investigar, gracias - El hecho es que se ven bien. Lo comprobare, gracias
Next subsections describe the experimental development and the best con gurations we found for both types of system 3.1
We rst built a classifying pipeline using
simple linear classifying models |such as
logistic regressions and SVMs| that were
implemented with scikit-learn
As input features, we combined the three representations described in the previous section: bag-of-words, bag-of-characters and tweet embeddings.
For the bag of words and characters, early experiments showed a clear advantage of binary values over counts, together with TFIDF re-weighting. First choices for n-gram ranges were (1, 2) for words and (1, 3) for characters.
For the embeddings, sentiment-oriented word vectors showed an advantage over basic vectors. We tried embeddings of dimensions 50, 100, 200 and 300. Best results were found with 50 dimensions, and there were no statistically signi cant di erences.
To compute tweet embeddings, we tried basic averaging (as provided by fastText ) and the weighted averaging scheme described in section 2.4. We experimented with smoothing values a = 10n for n 2 f 3; : : : ; 3g resulting in a signi cant advantage of using a = 0:1. Here, binarization and normalization as described in section 2.4 showed better results.
For the classi er, we tried logistic regressions (LRs) and linear-kernel SVMs. To alleviate the class imbalance problem, training items were weighted according to the inverse of the class frequency. Both LR and linear SVM hyper-parameters were selected targeting the optimization of accuracy and
LR+DA
BOW BOC (1, 2) (1, 3) (1, 2) (1, 3) (1, 5) (1, 6) (1, 5) (1, 6)
Macro-F1 over InterTASS ES development set. In particular, the best regularization parameters found were C = 1:0 for LRs, and C = 0:05 for SVMs. Logistic regressions were selected over SVMs as they performed consistently better in all experiments.
When adding augmented data, rst results showed a signi cant degradation in accuracy. However, an exploration of parameter values showed that it allowed an improvement in performance when increasing the range of n-gram sizes considered for BOWs and BOCs. Best results were found with up to 5-grams for words, and up to 6-grams for characters. Tab. 2 shows how data augmentation combined with bigger n-gram ranges improved results.
Most previous parameter selection was reviewed after data augmentation, con rming selected values. We also tried adding training data from General TASS corpus, to nd that this was harmful for our models. With the optimal models found in this process we submitted nal results for the Spanish (ES) monolingual task.
For Costa Rica (CR) and Peru (PE) monolingual tasks, same values than for ES were used for most parameters. Only weighted averaging, data augmentation and n-gram ranges were explored. In CR data, weighting improved results, with the peak at
InputLayer dense_1: Dense dropout_1: Dropout
dense_2: Dense dropout_2: Dropout
Output: Softmax a = 0:5. Data augmentation also was good, with the best results using up to 4-grams for words and 6-grams for characters. In PE data, neither weighting nor data augmentation were helpful. Best results were found using up to 2-gram for words and 5-grams for characters. 3.2
In the second set of experiments we used
multilayer perceptrons (MLP) neural networks.
MLPs performed well in previous editions of
the challenge
Fig. 1 displays the chosen architecture, consisting of two hidden layers and a softmax output. ReLU units were used as activation functions in the hidden layers. To avoid overtting, we tried dropout (Srivastava et al., 2014) and early stopping.
To nd the best con gurations, we
performed random search
BOW features: No BOW features at all, top-50 or top-150.
Tweet embeddings: Basic or weighted averaging.
Hidden layers: Di erent number of neu
MLP LR MLP LR MLP LR MLP
rons and keep-probabilities2.
Results of this search showed that bag-ofwords features and embedding weighting did not improve performance. Regarding the MLP architecture, we selected 256 as the size of the rst layer and 128 for the second, and keep-probabilities of 0:25 and 0:55 respectively. This con guration was used in all subtasks.
Data augmentation in combination with MLPs showed mixed results. For the monolingual ES subtask, using synthetic data resulted in a Macro-F1 gain while for monolingual PE it degraded the results.
All monolingual models were trained using the respective train sections of InterTASS datasets. General TASS was not used as it not showed improvements. For cross-lingual tasks, models for each language were trained using the datasets for the two other languages.
We used Keras
Conclusions and Future Work We presented our participation on TASS 2018 task 1 as team Atalaya. We explored standard approaches as well as some simple but original recent ideas such as data augmentation and word embedding weighting. Table 3 displays results for each subtask. Our systems ranked among the rst three in all the subtasks.
Experiments show that competitive results can be achieved without having to resort to complex neural architectures such as CNNs, RNNs, LSTMs, etc. Even simple lo2Probability of keeping the value of a neuron when training with dropout. gistic regressions were able to rank among the top performing systems.
Future work includes further exploration on data augmentation, tweet embedding techniques, and sentiment-oriented word embeddings. We also aim at improving preprocessing and adopting modern neural classifying models. neural networks applied to visual document analysis. In null, page 958. IEEE.