This document describes the participation of the INGEOTEC team in SENTIPOLC 2016 contest. In this participation two approaches are presented, B4MSA and B4MSA + EvoDAG, tested in Task 1: Subjectivity classification and Task 2: Polarity classification. In case of polarity classification, one constrained and unconstrained runs were conducted. In subjectivity classification only a constrained run was done. In our methodology we explored a set of techniques as lemmatization, stemming, entity removal, character-based q-grams, word-based n-grams, among others, to prepare different text representations, in this case, applied to the Italian language. The results show the official competition measures and other well-known performance measures such as macro and micro F1 scores.
Italiano. Questo documento descrive la partecipazione del team INGEOTEC alla competizione SENTIPOLC 2016. In questo contributo sono presentati due approcci, B4MSA e B4MSA + EvoDAG, applicati al Task 1: Subjectivity classification e Task 2: Polarity classification. Nel caso della classificazione della polarit, sono stati sottomessi un run constrained ed un run unconstrained. Per la classificazione della soggettivita, stato sottomesso solo un run constrained. La nostra metodologia esplora un insieme di tecniche come lemmatizzazione, stemming, rimozione di entit, q-grammi di caratteri, n-grammi di parole, ed altri, al fine di ottenere diverse rappresentazioni del testo. In questo caso essa applicata alla lingua italiana. I risultati qui presentati sono due: le metriche della competizione ufficiale ed altre misure note della performance, come macro F1 e micro F1.
Nowadays, the sentiment analysis task has become a problem of interest for governments, companies, and institutions due to the possibility of sensing massively the mood of the people using social networks in order to take advantage in decisionmaking process. This new way to know what are people thinking about something imposes challenges to the natural language processing and machine learning areas, the first of all, is that people using social networks are kindly ignoring formal writing. For example, a typical Twitter user do not follow formal writing rules and introduces new lexical variations indiscriminately, the use of emoticons and the mix of languages is also the common lingo. These characteristics produce high dimensional representations, where the curse of dimension makes hard to learn from examples.
There exists a number of strategies to cope with the sentiment analysis on Twitter messages, some of them are based on the fact that the core problem is fixed: we are looking for evidence of some sentiment in the text. Under this scheme a number of dictionaries have been described by psychologists, other resources like SentiWordNet have been created adapting well known linguistic resources and machine learning. There is a lot of work around this approach; however, all these knowledge is language dependent and must exists a deep understanding of the language being analyzed. Our approach is mostly independent of this kind of external resources while focus on tackling the misspellings and other common errors in the text.
In this manuscript we detail our approach to sentiment analysis from a language agnostic perspective, e.g., no one in our team knows Italian language. We neither use external knowledge nor specialized parsers. Our aim is to create a solid baseline from a multilingual perspective, that can be used as a real baseline for challenges like SENTIPOLC’16 and as a basic initial approximation for sentiment analysis systems.
The rest of the paper is organized in the following sections. Section 2 describes our approach. Section 3 describes our experimental results, and finally Section 4 concludes. 2
This participation is based on two approaches. First, B4MSA method, a simple approach which starts by applying text-transformations to the tweets, then transformed tweets are represented in a vector space model, and finally, a Support Vector Machine (with linear kernel) is used as the classifier. Second, B4MSA + EvoDAG, a combination of this simple approach with a Genetic programming scheme. 2.1
B4MSA is a system for multilingual polarity classification that can serve as a baseline as well as a framework to build sophisticated sentiment analysis systems due to its simplicity. The source code of B4MSA can be downloaded freely1.
We used our previous work, B4MSA, to tackle the SENTIPOLC challenge. Our approach learns based on training examples, avoiding any digested knowledge as dictionaries or ontologies. This scheme allows us to address the problem without caring about the particular language being tackled.
The dataset is converted to a vector space using
a standard procedure: the text is normalized,
tokenized and weighted. The weighting process is
fixed to be performed by TFIDF
1https://github.com/INGEOTEC/b4msa
At a glance, our goal is to find the best performing normalization and tokenization pipelines. We state the modeling as a combinatorial optimization problem; then, given a performance measure, we try to find the best performing configuration among a large parameter space.
The list of transformations and tokenizers are
listed below. All the text transformations
considered are either simple to implement, or there is
an open-source library (e.g.
In order to find the best performing configuration, we used two sort of features that we consider them as parameters: cross-language and languagedependent features.
Cross-language Features could be applied in
most similar languages and similar surface
features. Removing or keeping punctuation
(question marks, periods, etc.) and diacritics from the
original source; applying or not applying the
processes of case sensitivity (text into lowercase) and
symbol reduction (repeated symbols into one
occurrence of the symbol). Word-based n-grams
(nwords) Feature are word sequences of words
according to the window size defined. To compute
the N-words, the text is tokenized and combined
the tokens. For example, 1-words (unigrams) are
each word alone, and its 2-words (bigrams) set are
the sequences of two words, and so on
three language dependent features: stopwords,
stemming, and negation. These processes are
applied or not applied to the text. Stopwords
and stemming processes use data and the
Snowball Stemmer for Italian, respectively, from NLTK
Python package (Bird et al., 2009). Negation
feature markers could change the polarity of the
message. We used a set of language dependent rules
for common negation structures to attached the
negation clue to the nearest word, similar to the
approach used in
The model selection, sometimes called hyperparameter optimization, is the key of our approach. The default search space of B4MSA contains more than 331 thousand configurations when limited to multilingual and language independent parameters; while the search space reaches close to 4 million configurations when we add our three language-dependent parameters. Depending on the size of the training set, each configuration needs several minutes on a commodity server to be evaluated; thus, an exhaustive exploration of the parameter space can be quite expensive that makes the approach useless.
To reduce the selection time, we perform a
stochastic search with two algorithms, random
search and hill climbing. Firstly, we apply
random search
Finally, the performance of the final configuration is obtained applying the above procedure and cross-validation over the training data. 2.4
In the polarity task besides submitting B4MSA
which is a constrained approach, we decided to
generate an unconstrained submission by
performing the following approach. The idea is to
provide an additional dataset that it is automatically
label with positive and negative polarity using the
Distant Supervision approach
We start collecting tweets (using Twitter stream) written in Italian. In total, we collect more than 10; 000; 000 tweets. From these tweets, we kept only those that were consistent with the emoticon’s polarity used, e.g., the tweet only contains consistently emoticons with positive polarity. Then, the polarity of the whole tweet was set to the polarity of the emoticons, and we only used positive and negative polarities. Furthermore, we decided to balance the set, and then we remove a lot of positive tweets. At the end, this external dataset contains 4; 550; 000 tweets, half of them are positive and the another half are negative.
Once this external dataset was created, we decided to split it in batches of 50; 000 tweets half of them positive and the other half negative. This decision was taken in order to optimize the time needed to train a SVM and also around this number the Macro F1 metric is closed to its maximum value. That is, this number of tweets gives a good trade-off between time needed and classifier performance. In total there are 91 batches.
For each batch, we train a SVM at the end of this process we have 91 predictions (it is use the decision function). Besides these 91 predictions, it is also predicted (using as well the decision function) each tweet with B4MSA. That is, at the end of this process we have 94 values for each tweet. That is, we have a matrix with 7; 410 rows and 94 columns for the training set and of 3; 000 rows and 94 columns for the test set. Moreover, for matrix of the training set, we also know the class for each row. It is important to note that all the values of these matrix are predicted, for example, in B4MSA case, we used a 10-fold cross-validation in the training set in order to have predicted values.
Clearly, at this point, the problem is how to
make a final prediction; however, we had built
a classification problem using the decision
functions and the classes provided by the competition.
Thus, it is straight forward to tackle this
classification problem using EvoDAG (Evolving Directed
Acyclic Graph)2
2https://github.com/mgraffg/EvoDAG
This Section presents the results of the
INGEOTEC team. In this participation we did two
runs, a constrained and an unconstrained run with
B4MSA system, and only a constrained run with
B4MSA + EvoDAG. The constrained run was
conducted only with the dataset provided by
SENTIPOLC’16 competition. For more technical
details from the database and the competition in
general see
The unconstrained run was developed with an
additional dataset of 4; 550; 000 of tweets labeled
with Distant Supervision approach. The Distant
Supervision is an extension of the paradigm used
in
For the constrained run we participate in two
task: subjectivity and polarity classification. In
the unconstrained run we only participate in
polarity classification task. Table 1 shows the results of
subjectivity classification Task (B4MSA method),
here, Prec0 is the P recision0 value, Rec0 is the
Recall0 value, FSc0 is F Score0 value and
Prec1, Rec1 and FSc1 the same for F Score1
values and FScavg is the average value from all
F-Scores. The explanation of evaluation measures
can be seen in
Table 2, shows the results on the polarity classification task. In this task our B4MSA method achieves an average F-Score of 0:6054 and our combination of B4MSA + EvoDAG reaches an 0:6075 of average F-Score. These results place us on position 18 (unconstrained run) and 19 (constrained run) of a total of 26 entries.
It is important to mention that the difference between our two approaches is very small; however, B4MSA + EvoDAG is computationally more expensive, so we expected to have a considerable improvement in performance. It is evident that these results should be investigated further, and, our first impression are that our Distant supervision approach should be finely tune, that is, it is needed to verify the polarity of the emoticons and the complexity of the tweets.
Finally, Table 3 presents the measures
employed by our internal measurement, that is Macro
F1 and Micro F1 (for more details see
In this work we describe the INGEOTEC team participation in SENTIPOLC’16 contest. Two approaches were used, first, B4MSA method which combine several text transformations to the tweets. Secondly, B4MSA + EvoDAG, which combine the B4MSA method with a genetic programming approach. In subjectivity classification task, the obtained results place us in seventh of a total of 21 places. In polarity classification task, our results place us 18 and 19 places of a total of 26. Since our approach is simple and easy to implement, we take these results important considering that we do not use affective lexicons or another complex linguistic resource. Moreover, our B4MSA approach was tested internally in irony classification task with a result of 0:4687 of macro f1, and 0:8825 of micro f1. 0.56 FScorepos 0.6414
0.5694 0.6054
0.5944 0.6205 0.6075 Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python.
O’Reilly Media.
Christopher J.C. Burges. 1998. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):121–167. 2005.