This paper describes an ensemble-based approach developed to participate in TASS-2016 Task 1 on sentiment analysis of Spanish tweets at global level. Ensembles are built on the combination of systems with the lowest absolute correlation with each other. The systems are able to deal with non-standard lexical forms in tweets, in order to improve the quality of natural language analysis. To support the polarity classi cation, the approach uses basic features that have proved their discriminative power, as well as word and character n-gram features. Then, outputs from Logistic Regression classi ers, which may be either class labels or probabilities for each class, are used to build ensembles. Experimental results show that the less-correlated combination of 25 systems, which chooses the class with the highest unweighted average probability, is the setting that best suits to the task, achieving an overall accuracy of 62.0% in the six-labels evaluation, and of 70.5% in the fourlabels evaluation.
What people say on social media about
issues of their everyday life, the society, and
the world in general, has turned into a rich
source of information to understand social
behavior. Twitter content, in particular,
has caught the attention of researchers who
have investigated its potential for conducting
studies on the human subjectivity at large
scale, which was not feasible using
traditional methods. Around election time,
sentiment analysis of political tweets has been
widely used to capture trends in public
opinion regarding important issues such as
voting intention
TASS is a workshop aimed at fostering
research on sentiment analysis of Spanish
Twitter data, which provides a benchmark
evaluation to compare the latest advances in the
eld
This paper presents an ensemble-based approach to polarity classi cation of Spanish tweets, developed to participate in Task 1 proposed by the organizing committee of the TASS workshop. The ensemble members are (relatively) highly correct classi ers with the lowest absolute correlation with each other. The output from each classi er, which may be either a class label or probabilities for each class, is used to assign the polarity to a tweet based on a majority rule or on the highest unweighted average probability. Moreover, classi ers are adapted to deal with non-standard lexical forms in tweets, in order to improve the quality of natural language analysis.
The remainder of this paper is organized as follows. Section 2 describes the common architecture of the ensemble members (i.e., classi ers). Next, the submitted experiments, as well as the obtained results, are discussed in Section 3. Finally, Section 4 concludes the paper. 2
The tweet text is passed through the pipeline of each system in order to assign it a class label or a probability to be of a certain class. The pipeline, which goes from text preprocessing to machine learning classi cation, is described below. Note that the system term is preferred over the classi er term, because a machine learning classi er receives a feature vector and produces a class label or probabilities for each class; instead, the system term enables to conceive the whole process, from preprocessing to machine learning classi cation. 2.1
The process of text cleaning and normalization is performed in two phases: basic preprocessing and advanced preprocessing.
The following simple rules are implemented as regular expressions:
Removing URLs and emails.
HTML entities are mapped to textual representations (e.g., \<" ! \<"). Speci c Twitter terms such as mentions (@user) and hashtags (#topic) are replaced by placeholders.
Unknown characters are mapped to their closest ASCII variant, using the Python Unidecode module for the mapping. Consecutive repetitions of a same character are reduced to one occurrence. Emoticons are recognized and then classi ed into positive and negative, according to the sentiment they convey (e.g., \:)" ! \EMO POS", \:(" ! \EMO NEG").
Uni cation of punctuation marks
Once the set of simple rules has been applied,
the tweet text is tokenized and
morphologically analyzed by FreeLing
Lexical normalization. Each token is
passed through a set of basic modules
of FreeLing (e.g., dictionary lookup,
sufxes check, detection of numbers and
dates, and named entity recognition)
for identifying standard word forms and
other valid constructions. If a token
is not recognized by any of the
modules, it is marked as out-of-vocabulary
(OOV) word. Then, a confusion set
is formed by normalization candidates
which are identical or similar to the
graphemes or phonemes that make the
OOV word. These candidates are
elements of the union of a dictionary
of Spanish standard word forms and a
gazetteer of proper nouns. The best
normalization candidate for the OOV word
is which best ts a statistical language
model. The language model was
estimated from the Spanish Wikipedia
corpus. Lastly, the selected candidate is
capitalized according to the
capitalization rules of the Spanish language.
Extensive research on lexical normalization
of Spanish tweets can be read in
approach proposed by Pang et al.
In this stage, the normalized tweet text is transformed into a feature vector that feeds the machine learning classi er. The features are grouped into basic features and n-gram features.
Some of these features are computed before the process of text cleaning and normalization is performed.
The number of words completely in uppercase.
The number of words with more than two consecutive repetitions of a same character.
The number of consecutive repetitions of exclamation marks, question marks, and both punctuation marks (e.g., \!!", \??", \?!") and whether the text ends with an exclamation or question mark.
The number of occurrences of each class of emoticons (i.e., positive and negative) and whether the last token of the tweet is an emoticon.
The number of positive and negative
words, relative to the ElhPolar lexicon
The number of negated contexts.
The number of occurrences of each Partof-Speech tag.
The xed-length set of basic features is
always extracted from tweets. However, the
tweet text varies from another in terms of
length, number of tokens, and vocabulary
used. For that reason, a process that
transforms textual data into numerical feature
vectors of xed length is required. This process,
known as vectorization, is performed by
applying the tf-idf weighting scheme
At the last stage, the sentiment analysis
system classi es a given tweet as either P+, P,
NEU, N, N+, or NONE, or assigns
probabilities for each class. After receiving as input
the feature vector, a L2-regularized Logistic
Regression classi er assigns a class label to
the tweet or a probability to be of a certain
class. The classi er was trained on the
training set, using the Scikit-learn
1,720 di erent sentiment analysis systems were trained on the training set via 5-fold cross validation, in order to nd the best parameter settings, namely: negation handling, Experiment Accuracy run-1 run-2 run-3 run-1 run-2 run-3
P
NEU
N
NONE
polarity lexicon, order of word and
character n-grams, and others parameters related
to the vectorization process (e.g.,
lowercasing, frequency thresholds, etc.). The systems
were sorted by their mean cross-validation
score, and thus the top 50 ranked were
ltered to build the ensemble. The training
set is a collection of 7,219 tweets, each of
which is tagged with one of six labels (i.e.,
P+, P, NEU, N, N+, and NONE). Note that
the systems were trained for the six-labels
evaluation, and therefore the P+ and P
labels were merged into P, as well as the N+
and N labels were merged into N, to produce
an output in accordance with the four-labels
evaluation. Further description of the
provided corpus, as well as of the training and
test sets, can be read in
Next, the top 50 systems assigned a class label to each tweet in a collection of 1,000, which was drawn from the untagged test set with a similar class distribution to the training set. In this stage, the objective was to nd the systems with the lowest absolute correlation with each other; therefore, the performance was not evaluated. Then, the less-correlated combinations of 5, 10, and 25 systems, were used to build the ensembles, whose outputs correspond to the submitted experiments. These experiments are described below: run-1: the less-correlated combination of 5 systems, which chooses the class label that represents the majority in the predictions made by the ensemble members. run-2: the less-correlated combination of 10 systems, which chooses the class with the highest unweighted average probability. run-3: the less-correlated combination of 25 systems, which chooses the class with the highest unweighted average probability.
Tables 1 and 2 show the performance evaluation on the test set (i.e., a collection of 60,798 tweets) for six and four labels, respectively. Accuracy has been de ned as the o cial metric for ranking the systems. In summary, the main gain occurs among the \run1" and \run-2" experiments, with an increment of 0.5% in accuracy in the six-labels evaluation, and of 0.2% in the four-labels evaluation; instead, a negligible gain occurs among the \run-2" and\ run-3" experiments, taking additionally into account the computational cost of running the latter.
As a nal point, Table 3 shows how the overall performance is a ected by the low discriminative power of the ensembles (in this case, the one that correspond to \run-3") for the NEU class. With this in mind, it is proposed as future work to deal with the low representativeness of the NEU class in the training data (i.e., 9.28% of tweets), in order to properly characterize this kind of tweets. 4
This paper has described an ensemble-based approach for sentiment analysis of Spanish Twitter data at global level, developed in order to participate in Task 1 proposed by the organization of TASS workshop. Three ensembles were built on the combination of sentiment analysis systems with the lowest absolute correlation with each other. The systems were adapted to the informal genre and the free writing style that characterize Twitter, in order to improve the quality of natural language analysis. In this way, the predicted class label for a particular tweet was based on a majority rule or on the highest average probability. Experimental results showed that the less-correlated combination of 25 systems, which chose the class with the highest unweighted average probability, was the setting that best suited to the task. However, there is a great room for improvement in the learning of a proper characterization of neutral tweets.