English. This paper presents the participation of the IRADABE team to the SENTIPOLC 2016 task. This year we investigated the use of positional features together with the fusion of sentiment analysis resources with the aim to classify Italian tweets according to subjectivity, polarity and irony. Our approach uses as starting point our participation in the SENTIPOLC 2014 edition. For classification we adopted a supervised approach that takes advantage of support vector machines and neural networks.
Italiano. Quest’articolo presenta il lavoro svolto dal team IRADABE per la partecipazione al task SENTIPOLC 2016. Il lavoro svolto include l’utilizzo di caratteristiche posizionali e la fusione di lessici specialistici, finalizzato alla classificazione di tweet in italiano, secondo la loro soggetivita`, polarita` ed ironia. Il nostro approccio si basa sull’esperienza acquisita nel corso della partecipazione all’edizione 2014 di SENTIPOLC. Per la classificazione sono stati adottati dei metodi supervisionati come le macchine a supporto vettoriale e le reti neurali.
In sentiment analysis related tasks, there are
several factors that can be considered in order to
determine the polarity of a given piece of text.
Overall, the presence of positive or negative words is
used as a strong indicator of sentiment.
Nowadays there are many sentiment analysis related
resources that can be exploited to infer polarity from
texts. Recently, this kind of lexicons has been
proven to be effective for detecting irony in Twitter
However, the language barrier is not the only
drawback for these resources. Another issue is
the limited coverage of certain resources. For
instance, AFINN
We observed also that in some cases these
lexicons provide a score for a word but not for one
of their synonyms: in the Hu-Liu lexicon, for
instance, the word ‘repel’ is listed as a negative one,
but ‘resist’, which is listed as one of its synonym
in the Roget’s thesaurus1, is not. SentiWordNet
Therefore, we built a sentiment lexicon which
was aimed to provide the highest possible
coverage by merging existing resources and extending
the scores to synonyms or quasi-synonyms. The
sentiment lexicon was built following a three-step
process:
1. Create a unique set of opinion words from
the AFINN, Hu-Liu and SentiWordNet
lexicons, and merge the scores if multiple scores
are available for the same word; the original
English resources were previously translated
into the Italian language for our participation
1http://www.thesaurus.com/
Roget-Alpha-Index.html
in SENTIPOLC 2014;
2. Extend the lexicon with the WordNet
synonyms of words obtained in step 1;
3. Extend the lexicon with pseudo-synonyms
of words obtained in step 1 and 2, using
word2vec for similarity. We denote them
as “pseudo-synonyms” because the
similarity according to word2vec doesn’t
necessarily means that the words are synonyms, only
that they usually share the same contexts.
The scores at each step were calculated as follows:
in step 1, the weight of a word is the average of
the non-zero scores from the three lexicons. In
step 2, the weight for a synonym is the same of
the originating word. If the synonym is already
in the lexicon, then we keep the most polarizing
weight (if the scores have the same sign), or the
sum of the weights (if the scores have opposed
signs). For step 3 we previously built semantic
vectors using word2vec
In addition to this new resource, we exploited
labMT-English words. It is a list
It is well known that in the context of opinion
mining and summarization the position of
opinion words is an important feature
In fact, we analyzed the training set and noticed that some words tend to appear in certain positions when the sentence is labelled with a class rather than the other one. For example, in the subjective sub-task, ‘non’ (not), ‘io’ (I), auxiliary verbs like ‘potere’ (can), ‘dovere’ (must) tend to occur mostly at the beginning of the sentence if the sentence is subjective. In the positive polarity subtask, words like ‘bello’ (beautiful), ‘piacere’ (like) and ‘amare’ (love) are more often observed at the beginning of the sentence if the tweet is positive.
We therefore introduced a positional Bag-ofWords (BOW) weighting, where the weight of a word t is calculated as:
w(t) = 1 + pos(t)=len(s) where pos(t) is the last observed position of the word in the sentence, and len(s) is the length of the sentence. For instance, in the sentence “I love apples in fall.”, w(love) = 1 + 1=5 = 1:2, since the word love is at position 1 in a sentence of 5 words.
The Bag of Words was obtained by taking all the lemmatized forms w that appeared in the training corpus with a frequency greater than 5 and I(w) > 0:001, where I(w) is the informativeness of word w calculated as: I(w) = p(wjc+) log(p(wjc+)) log(p(wjc )) where p(wjc+) and p(wjc ) are the probabilities of a word appearing in the tweets tagged with the positive or negative class, respectively. The result of this selection consisted in 943 words for the subj subtask, 831 for pos, 991 for neg and 1197 for iro.
The results in Table 3 show a marginal improvement for the polarity and irony classes, while in subjectivity the system lost 2% in F-measure. This is probably due to the fact that the important words that tend to appear in the first part of the sentence pos. BOW std. BOW may repeat later, providing a wrong score for the feature.
With respect to the 2014 version of
IRADABE, we introduced 3 more position-dependent
features. Each tweet was divided into 3 sections,
head, centre and tail. For each section, we
consider the sum of the sentiment scores of the
included words as a separate feature. Therefore, we
have three features, named in Table 3.1 as headS,
centreS and tailS.
We renewed most of the features used for
SENTIPOLC 2014, with the main difference that we
are now using a single sentiment lexicon
instead than 3. In IRADABE 2014 we grouped
the features into two categories: Surface
Features and Lexicon-based Features. We recall the
ones appearing in Table 2, directing the reader
to
Subj http shout sum( ) count( ) headS pers
‘!’ avglabMT ‘mi’ taboo
Pol(+) subj avglabMT ‘grazie’ smileys polSum http ‘?’ sum(+) ‘bello’ ‘amare’
Pol(-) Iro subj subj sum( ) http count( ) ‘governo’ avglabMT mixed length shout polSum ‘Mario’
http ‘che’ centreS ‘#Grillo’
taboo length stdlabMT sum( ) We evaluated our approach on the dataset provided by the organizers of SENTIPOLC 2016. This dataset is composed by up to 10,000 tweets distributed in training set and test set. Both datasets contain tweets related to political and socio-political domains, as well as some generic tweets3.
We experimented with different configurations
for assessing subjectivity, polarity and irony.
We sent two runs for evaluation purposes in
SENTIPOLC-2016:
run 1. For assessign the subjectivity label a
Tensorflow4 implementation of Deep Neural
Ngetwork (DNN) was applied, with 2 hidden
layers with 1024 and 512 states, respectively.
Then, the polarity and irony labels were
determined by exploiting a SVM5.
run 2. In this run, the bag-of-words were
revised to remove words that may have a
differ3Further details on the datasets can be found in the task
overview
From the results, we can observe that the DNN obtained an excellent precision (more than 93%) in subj, but the recall was very low. This may indicate a problem due to the class not being balanced, or an overfitting problem with the DNN, which is plausible given the number of features. This may also be the reason for which the SVM performs better, because SVMs are less afflicted by the “curse of dimensionality”.
Subj 0.9328 0.4575 0.6139 As future work, it could be interesting to exploit the labels for exact polarity as provided by the organizers. This kind of information could help in some way to identify the use of figurative language. Furthermore, we are planning to enrich IRADABE with other kinds of features that allow us to cover more subtle aspects of sentiment, such as emotions. The introduction of the “happiness score” provided by labMT was particularly useful, with the related features being critical in the subjectivity and polarity subtasks. This motivates us to look for dictionaries that may express different feelings than just the overall polarity of a word. We will also need to verify the effectiveness of the resource we produced automatically with respect to other hand-crafted dictionaries for the Italian language, such as Sentix (Basile and Nissim, 2013)
We plan to use a more refined weighting scheme
for the positional features, such as the
locallyweighted bag-of-words or LOWBOW
About the utility of positional features, the current results are inconclusive, so we need to investigate further about how the positional scoring affects the results. On the other hand, the results show that the merged dictionary was a useful resource, with dictionary-based features representing 25% of the most discriminating features.
This research work has been supported by the “Investissements d’Avenir” program ANR-10LABX-0083 (Labex EFL). The National Council for Science and Technology (CONACyT Mexico) has funded the research work of Delia Irazu´ Herna´ndez Far´ıias (Grant No. 218109/313683 CVU-369616).