This paper describes the JACERONG system proposed to participate in TASS-2017 Task 1. For such a benchmark evaluation, two ensemble methods widely utilized because of their proved ability to increase prediction accuracy were implemented, namely: averaging and stacking. First of all, (relatively) highly correct classi ers utilize supervised learning algorithms to predict a class label or probability estimates. Then, predictions from these classi ers are optimally combined in order to obtain a better nal prediction. Finally, how to choose which classi ers constitute an ensemble was also an important issue addressed in this work. Experimental results show that the proposed system is top-ranked on the test set of the InterTASS corpus, according to the accuracy metric. Together with this, results indicate that the predictive performance on the whole test set of the General Corpus of TASS outperforms the best result achieved in the four-label evaluation of the previous edition of TASS, in terms of the Macro-F1 metric.
Nowadays, `tweeting' has become an activity par excellence to say what one thinks or feels.
Thus, the large amount of user-generated content on Twitter, in the form of short texts limited to 140 characters that are known as tweets, has led to develop new methods to explore the human subjectivity at large scale.
Sentiment analysis, as one of these
methods is known, has been widely utilized to
gauge public opinion regarding important
issues of people's everyday life, the society, and
the word in general, e.g. a political election
TASS is a workshop aimed at fostering research on sentiment analysis of Spanish tweets, which provides a benchmark evalu
Copyright © 2017 by the paper's authors. Copying permitted for private and academic purposes. ation to compare the latest advances in the
eld
This paper describes the JACERONG
system proposed to participate in TASS-2017
Task 1. For this sixth edition, classi er
ensembles based on stacking were developed, in
addition to the ones based on averaging, with
several improvements, that were presented in
the previous edition
Regarding the ensembles, they are constituted by (relatively) highly correct classi ers that utilize Logistic Regression and Support Vector Machines as the supervised learning algorithms to predict a class label or probability estimates. Then, predictions from these classi ers are optimally combined in order to obtain a better nal prediction. Finally, how to choose which classi ers constitute an ensemble was also an important issue addressed in this work.
The remainder of this paper is organized as follows. Section 2 explains the system architecture. Next, the submitted runs and the obtained results are discussed in Section 3.
Lastly, Section 4 concludes the paper. 2
The system architecture can be viewed as a pipeline consisting of several pre-processing modules, a vectorizer that transforms a text into a feature vector, machine learning classi ers, and an ensemble combiner that takes level-one predictions and then optimally combines them to obtain a better nal prediction. Figure 1 illustrates the system architecture. In addition to this, code of the system is publicly available to enable reproducibility.1 2.1
This is a rule-based normalizer as listed below:
Removing URLs and emails.
HTML entities are mapped to their textual representation (e.g., \<" ! \<"). Speci c Twitter terms such as mentions (@user) and hashtags (#topic) are replaced by placeholders.
Unknown characters are mapped to their closest ASCII variant, using the Python Unidecode module for the mapping. Consecutive repetitions of a same character are reduced to one occurrence. Emoticons are recognized and then classi ed into positive and negative, according to the sentiment they convey (e.g., \:)" ! \EMO POS", \:(" ! \EMO NEG").
Uni cation of punctuation marks
An open-source spell checker for Spanish
texts is used to normalize non-standard
word forms, i.e. out-of-vocabulary (OOV)
words, to their standard lexical form
Inspired by the approach proposed by Pang
et al.
Once the text has been normalized as described above, it is transformed into a feature vector that feeds a rst-level classi er. The feature vector is formed by concatenating basic and n-gram features.
Some of the following features are computed before the text normalization is performed.
The number of words completely in uppercase.
The number of words with more than two consecutive repetitions of a same character.
The number of consecutive repetitions of exclamation marks, question marks, and both punctuation marks (e.g., \!!", \??", \?!"), and whether the text ends with an exclamation or question mark.
The number of occurrences of each class of emoticons, i.e. positive and negative, and whether the last token of the text is an emoticon.
The number of positive and negative
words, relative to the ElhPolar lexicon
The number of negated contexts.
The number of occurrences of each Partof-Speech tag.
The xed-length set of basic features is
always extracted from a text. However, a text
varies from another in terms of length,
number of tokens, and vocabulary. For that
reason, a process that transforms textual data
into numerical feature vectors of xed length
is required. This process, known as
vectorization, is performed by applying the
TfIdf weighting scheme
At this stage, a machine learning classi er, or
rst-level classi er, receives the feature
vector and predicts a class label or probability
estimates, i.e. the probability of the tweet
to be of a certain class. Whichever the
prediction be, it is denominated level-one
prediction. Logistic Regression and Support Vector
Machines (SVM) with `linear' kernel are the
algorithms utilized to develop a supervised
learning classi cation approach; Scikit-learn
Two ensemble methods were implemented to
take level-one predictions and then optimally
combine them in order to obtain a better nal
prediction, namely: averaging and stacking
(Li et al., 2014). The former chooses the class
with the highest unweighted average
probability from probability estimates predicted by
rst-level classi ers. In spite of its simplicity,
it has proved to be a competitive method that
allows to achieve top results
Firstly, the training data were used to t
8,774 rst-level classi ers (4,387 of which
were learned from the training set of the
InterTASS corpus, while to learn the remaining
ones the training set of the General Corpus
of TASS was used) via 5-fold cross validation
in order to nd the best parameter settings,
namely: scope of the negated context,
polarity lexicon, order of word and character
n-grams, and other parameters related to the
vectorizer (e.g., frequency thresholds).
Secondly, these classi ers were ranked
according to their predictive performance on cross
validation, i.e. the (out-of-fold) prediction
accuracy obtained by averaging among the
k iterations; out-of-fold predictions in the
k-th iteration are the predictions obtained
by applying a rst-level classi er, which was
trained on k 1 folds, to the remaining
one subset. Thus, the best 100 rst-level
classi ers for each training set were ltered.
Thirdly, how to choose which rst-level
classi ers constitute an ensemble was an
important issue tackled in this work.
Empirical ndings indicate that the less-correlated
combination of classi ers achieves top results
In order to evaluate the predictive
performance of the system, the test set of the
InterTASS corpus and the two test sets of the
General Corpus of TASS (the whole set and the
strati ed sample of 1,000 tweets) were used.
Speci cally, given a tweet from any of the test
sets, its polarity should be predicted; the
polarity, or class label, can be P, N, NEU, or
NONE. Macro-F1 and Accuracy are the
ofcial metrics used to rank the participating
systems. Regarding the provided corpora,
and the way these are split into training and
test sets, the reader is referred to
In the same way, the runs submitted to evaluate the system on the test set of the InterTASS corpus are described below: InterTASS-run-1: the less-correlated combination of 3 rst-level classi ers learned from the training set of the InterTASS corpus, which constitute an ensemble based on averaging.
InterTASS-run-2: the less-correlated combination of 19 rst-level classi ers learned from the training set of the InterTASS corpus, which constitute an ensemble based on stacking.
InterTASS-run-3: the less-correlated combination of 14 rst-level classi ers learned from the training set of the General Corpus of TASS, which constitute an ensemble based on averaging.
In summary, it is worth to state that
ensembles based on averaging are signi cantly
better than the ones based on stacking. And
this signi cance does not only correspond to
the slightly better results achieved by the
former, but also to their ability to increase
prediction accuracy given their simplicity and
their computational e ciency. Thus, the
proposed system outperforms all the
participating systems in predictive performance on the
test set of the InterTASS corpus, in terms
of the accuracy metric; likewise, the
predictive performance on the whole test set of
the General Corpus of TASS turns out to be
slightly better than the best result achieved
in the four-label evaluation of the previous
edition
As a nal point, class imbalance is a major problem that has not been properly tackled yet. Speci cally, the overall performance of the system was signi cantly a ected by the low discriminative power for the NEU class, on both the test set of the InterTASS Corpus and the two test sets of the General Corpus of TASS. With this in mind, future research e orts should be focused on dealing with the low representativeness of the NEU class.
This paper has described the JACERONG system proposed to participate in TASS-2017 Task 1. For such a benchmark evaluation, two ensemble methods were implemented, namely: averaging and stacking. Findings indicate that ensembles based on averaging are signi cantly better than the ones based on stacking. This signi cance corresponds to the former's ability to increase prediction accuracy given their simplicity and their computational e ciency, in addition to the slightly better results achieved by them. Moreover, ndings show that the less-correlated combination of classi ers achieves top results. The experimental evaluation on the test set of the InterTASS corpus showed that the proposed system is top-ranked. Together with this, results indicated that the predictive performance on the whole test set of the General Corpus of TASS outperforms the best result achieved in the four-label evaluation of the previous edition of TASS. Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN co-located with the 32nd SEPLN Conference (SEPLN 2016), pages 13{21.