In this article we describe our participation in TASS 2019, a shared task aimed at the detection of sentiment polarity of Spanish tweets. We combined di erent representations such as bag-of-words, bagof-characters, and tweet embeddings. In particular, we trained robust subword-aware word embeddings and computed tweet representations using a weighted-averaging strategy. We also used two data augmentation techniques to deal with data scarcity: two-way translation augmentation, and instance crossover augmentation, a novel technique that generates new instances by combining halves of tweets. In experiments, we trained linear classi ers and ensemble models, obtaining highly competitive results despite the simplicity of our approaches.
TASS is a shared task organized every year, since 2012, with challenges related
to Sentiment Analysis in Spanish. In TASS 2019 [
Five datasets are o ered for the task, each one from a di erent Spanish speaking country: CR (Costa Rica), ES (Spain), MX (Mexico), PE (Peru) and UY (Uruguay). Each corpus is divided into train, development and test sections. No other supervised datasets can be used, but external linguistic resources such as embeddings and lexicons are allowed.
The challenge is divided into two subtasks. In monolingual subtask 1, systems must be trained and tested on the same dataset. In cross-lingual subtask 2,
Training
Data Translation Augmented
Data Crossover Augmented
Data
BoC
BoW
Embedding Preprocess
Classifier systems must be trained using datasets from countries others than the one used for testing.
In this article, we describe our participation in TASS 2019 as team Atalaya.
We based our systems on our previous work [
To represent tweets, we used a combined approach of bag-of-words,
bag-ofcharacters and tweet embeddings. Tweet embeddings were computed from word
embeddings using a weighted averaging scheme. For word embeddings, we used
fastText subword-aware vectors [
Our fastText embeddings are robust to noise since they can compute embeddings for unseen words by using subword embeddings. Moreover, we trained them using a database of 90M tweets from various Spanish-speaking countries, giving wide domain-speci c vocabulary coverage. We achieved additional robustness by doing preprocessing using several text normalization and noise reduction techniques.
To cope with training data scarcity, we experimented with data augmentation techniques. As in our previous work, we did augmentation using machine translation to and from several other languages.
We also tried a novel augmentation technique we called instance crossover, loosely inspired by the crossover operation from genetic algorithms. This technique combines halves of tweets to generate new instances. Despite its simplicity, this idea showed to be useful in our experiments.
For the classifying models, we used logistic regressions and also bagging ensembles of logistic regressions.
The rest of the paper is as follows. The next section presents the components of our systems and the ideas we used to build them. Section 3 presents the experiments and results for both subtasks. Section 4 concludes the work with some observations about our experience. 2
The components and general architecture of our systems is shown in Fig. 1. In this section, we describe the techniques and resources we used to build them. 2.1
Preprocessing is important to reduce noise from tweets. We follow our previous work, applying two levels of preprocessing. Basic tweet preprocessing includes tokenization, replacement of handles, URLs, and e-mails, and shortening of repeated letters. Further preprocessing is done, aimed at semantic tasks. It includes removal of punctuation, stopword and numbers, lowercasing, lemmatization, and negation handling.
For negation handling, we followed a simple approach [
No treatment was performed to hashtags, emojis, interjections and onomatopoeias. Moreover, no spelling correction nor any other additional normalization was applied. 2.2
A simple way to represent textual data as feature vectors is to use bag-of-words (BoWs). A bag-of-words represents a tweet as a vector with the counts of words occurring in it. Resulting vectors are high-dimensional and sparse. The BoW representation can be extended to count also word n-grams. In this work, we used BoWs, together with count binarization and TF-IDF re-weighting, both useful for semantic tasks such as sentiment analysis.
For more robustness, we also used a bag-of-characters (BoC) representation. BoCs have exactly the same properties and variants than BoWs but are applied to characters instead of word tokens. Character usage in tweets holds useful information for sentiment analysis. In our work, the BoC representation is computed over the original raw text of tweets, with no preprocessing at all. 2.3
A more interesting way to represent text is using embeddings. Word embeddings are low-dimensional dense vector representations of words. These representations are learned in an unsupervised fashion using large quantities of plain text, providing high vocabulary coverage.
For our systems, we used fastText embeddings [
We did not use a pretrained fastText model but trained our own using a big preprocessed dataset of 90 million tweets from various Spanish-speaking countries. This dataset is mostly composed of tweets we collected for previous work, and also includes the tweets from all sections of all TASS 2019 datasets. 2.4
To use word embeddings in sentiment analysis, the embeddings of the individual tokens must be aggregated in some way to obtain a complete tweet representation.
A simple approach is to do averaging to obtain a single vector. A bit more interesting is to add weights to the averaging scheme. This way, some words may be considered more relevant than others for the classi cation task.
In this work, we used Smooth Inverse Frequency (SIF), a simple weighted
averaging scheme from [
In [
An important limitation of this tweet embedding scheme is that word order is completely ignored. Only preprocessing may allow the in uence of ordering in the result. In particular, the negation handling trick from section 2.1 is a useful, although naive, way to let words be a ected by previous negations. 2.5
One of the main successful approaches from our previous work on TASS was the use of data augmentation techniques. Data augmentation helps to cope with training data scarcity. Augmentation aims at the introduction of data variability using label-preserving transformations on real data. When correctly used, it contributes to data robustness and acts as a regularizer for the models.
Our approach for TASS 2018 (also as team Atalaya) was to use two-way translation augmentation. In two-way translation, an external machine translation service is used to translate tweets to other \pivot" languages and then back to Spanish. This augmentation technique helps to introduce lexical and syntactical variations to tweets, most times preserving their meaning.
In [
In this work, we explored two-way translation further, applying it to all the datasets using 20 di erent pivot languages. To get translations, we used Google's Cloud Translation API service.
Pivot languages were selected by hand from the list of available languages that the API can translate from/to Spanish. The selection was done trying to pick representative languages from di erent language families. 2.6
We also tried a new augmentation idea that aims at the generation of new data by combining pairs of instances with the same label. We call it the instance crossover augmentation technique, inspired by the chromosome crossover operation from genetic algorithms.
Our approach is simply to split tokenized tweets into two halves, and then randomly sample and combine rst halves with second halves. Resulting instances will probably be ungrammatical and semantically unsound, but our hypothesis is that what is left of semantics, for instance at the lexical level, will preserve sentiment polarity most of the times.1
Fig. 2 shows an example of instance crossover using two tweets with positive sentiment. In this example, crossover is successful in the sense that the resulting instances can be clearly judged as having a positive sentiment. In other cases, crossover may fail to preserve polarity, for instance, because of an unfortunate combination involving a negation. Resulting instances may even be completely nonsensical, introducing noise to the data.
For this work, we chose to directly validate in experiments this augmentation idea. In our experiments, we applied augmentation over the training tweets after basic preprocessing and before semantic preprocessing (as de ned in section 2.1). We tried di erent levels of augmentation, multiplying the size of original training datasets by factors of 4, 8, 12, 16 and 20. We preserved the original distribution over labels and therefore the class imbalance.
Instance crossover is a very rough and naive augmentation technique. However, it may be useful to introduce more data variability than two-way translation. With translation, new data points may fall very close to the original ones, while crossover introduces new points in the \spaces" between the original ones. Moreover, this is done in a representation agnostic fashion. It can be used with bag-of-words, embeddings, or even neural based representations.
Another clear advantage of instance crossover is that it does not rely on any external resource or system. Unlike this, translation requires an external service, at a cost, and other techniques such as synonym replacement require thesauruses or word similarity databases. 3
In this section, we describe our experiments. We implemented all our systems
using scikit-learn [
For simplicity, most of our work was centered on subtask 1 and on the ES dataset, looking for model con gurations and hyperparameter values that gave the best 1 Grammaticality and semantic soundness are already rare in the original tweets, so it is not something we should worry about very much.
ORIGINAL @USER fue genial debemos organizar mas cosas as sin necesidad de que nadie abandone el pa s @USER me alegro mucho ! ! es importante darnos cuenta del gran valor que podemos aportar y encontrar nuestra mision
AUGMENTED @USER fue genial debemos
organizar mas cosas as del gran valor que podemos aportar y encontrar nuestra mision @USER me alegro mucho ! ! es importante darnos cuenta sin necesidad de que nadie abandone el pa s results over the development section of the ES dataset. The optimization process was done using a mixed approach of grid search and by-hand tuning.
We targeted the maximization of both macro-F1 and accuracy scores. The macro-F1 score is the main metric of TASS 2019, but it is very unstable and sensible to small changes in predictions for minority classes. On the other hand, accuracy is more stable and reliable for the development process.
As a starting point, we used our optimal model and con guration from TASS
2018 [
Arabic (EN+FR+PT+AR) as pivot languages. { BoW: All word n-grams for n 5. { BoC: All character n-grams for n 6. { Tweet embeddings: 50 dimension fastText vectors. Weighted averaging with a = 0:1. { Logistic regression: liblinear solver with primal formulation, L2 regularization with inverse strength C = 1:0, and class-balanced reweighting. Table 1 shows a detailed evaluation of this baseline model.
The rst idea we explored was augmentation using two-way translation with the new 20 pivot languages. We tried adding all new data, but also adding some subsets of it, by grouping pivot languages in packs of four datasets. However, we could not nd any improvement from the original EN+FR+PT+AR augmentation, sometimes nding important drops in model quality.
Next, we explored augmentation using instance crossover. We tried 4x, 8x, 12x, 16x and 20x factor augmentations from the original ES training corpus, with and without additional translation-based augmentation. In every case results were improved w.r.t. not using crossover augmentation. The best result was found for 8x augmentation.
Last, we tuned the hyperparameters of the logistic regression. The best conguration found was a liblinear solver with primal formulation, L2 regularization with inverse strength C = 0:2, with no class-balanced reweighting.
We also tried an ensemble of logistic regressions by using bagging. Bagging was found to be useful for the ES dataset. The best con guration found was using a bag of 40 logistic regressions.
Table 2 shows a detailed evaluation for our best model for ES found following the development process. 3.2
To build a submission for subtask 1, we rst ran the nal test on the ES dataset using the best model described in the previous section.
To build submissions for the other datasets, we followed a similar development approach, but with most hyperparameters xed with the optimal values for ES. We focused the optimization process in the usage of translation and crossover augmentations, in the logistic regression hyperparameters and in the usage of bagging. Tuning was done mostly by hand and sometimes using gridsearch. Table 3 shows the optimal con gurations found for each dataset.
The nal results for the complete submission for subtask 1 are shown in Table 4. 3.3
To build submissions for subtask 2 we did a minimal set of experiments. For each language, we started from the optimal model con guration found for subtask 1 and then trained it using the union of the training datasets of every other language. We then proceeded to optimize the main hyperparameters of the logistic regression, mostly doing by-hand tuning.
We did some preliminary experiments with data augmentation and bagging for the ES dataset. However, results were not improved, so we didn't do further experimentation with the other datasets.
augmentation translation crossover logistic regression
C class-weight bagging ES EN+FR+PT+AR PE EN+FR+PT+AR CR no UY no MX EN+FR+PT+AR As a complementary post-competition experiment, we performed ablation tests for each of the components of our systems, to assess the relevance of each of the techniques used in this work.
The ablation tests were done using the best system for subtask 1 on the ES dataset. The results are displayed in Table 6.
It can be seen that all the techniques have a positive impact. Among representations, tweet embedding is the most important representation, way above BoW and BoC representations. Also, it is interesting to observe that crossover augmentation has an impact on the F1 but not on the accuracy, indicating that it is helping mostly on the minority classes NEU and NONE. 4
Robust representations and data augmentation play a strong role in sentiment analysis with small-sized training datasets. As in our previous experience with TASS 2018, we are still able to obtain top ranking results without having to resort to complex models such as deep neural networks.
We observe that, for this edition of TASS, most of our work was on the application of general ML techniques, and not on particular task/domain speci c engineering. In particular, we successfully tried instance crossover augmentation, a novel technique that, despite its simplicity, showed a positive impact on results. This idea can be useful to augment small datasets for other short text classi cation tasks without the need for external resources.