English. The paper describes our submission to the task 2 of SENTIment POLarity Classification in Italian Tweets at Evalita 2016. Our approach is based on a convolutional neural network that exploits both word embeddings and Sentiment Specific word embeddings. We also experimented a model trained with a distant supervised corpus. Our submission with Sentiment Specific word embeddings achieved the first official score.
Italiano. L’articolo descrive la nostra partecipazione al task 2 di SENTIment POLarity Classification in Italian Tweets a Evalita 2016. Il nostro approccio si basa su una rete neurale convoluzionale che sfrutta sia word embeddings tradizionali che sentiment specific word embeddings. Abbiamo inoltre sperimentato un modello allenato su un corpus costruito mediante tecnica distant supervised. Il nostro sistema, che utilizza Specific Sentiment word embeddings, ha ottenuto il primo punteggio officiale. Task 3: Irony Detection: identify if is present the irony in a tweet.
The state of the art on the polarity
classification of tweets is the application of Deep Learning
methods
We explored Deep Learning techniques for the sentiment analysis of English tweets at Semeval 2016 with good results, where we noticed that use of convolutional neural network and Sentiment Specific word embeddings was promising.
We applied a similar approach for the Italian language, building word embeddings from a big corpus of Italian tweets, sentiment specific word embeddings from positive and negative tweets, using a convolutional neural network as classifier. We also introduced a distant supervised corpus as silver training set.
We report the results of our experiments with this approach on the task Evalita 2016 Sentipolc Task 2 Polarity classification. 2
The architecture of the system consists of the following steps: Oggi non mi sento molto bene EMO_SAD embeddings for each word convolutional layer
with multiple filters max over time pooling
Multilayer percep
tron
with dropout
build word embeddings from a collection
of 167 million tweets collected with the
Twitter API over a period of May to
September 2016, preprocessed as described
later.
build Sentiment Specific word
embeddings using a portion of these tweets split
into positive/negative by distant
supervision.
train a convolutional neural network
classifier using one of the above word
embeddings
The convolutional neural network classifier
exploits pre-trained word embeddings as only
features in various configurations as described
below. The architecture of the classifier consists of
the following layers described in Figure 1: a
lookup layer for word embeddings, a
convolutional layer with a ReLU activation function, a
maxpooling layer, a dropout layer, a linear layer
with tanh activation and a softmax layer. This is
the same classifier described in
In order to build the word embeddings we
preprocessed the tweets using tools from the Tanl
pipeline
Word Embeddings and Sentiment
Specific Word Embeddings
We experimented with standard word
embeddings, in particular building them with the tool
word2vec1
The frequency distribution of classes in the
dataset, as shown in Table 1, seems skewed and not
fully representative of the distribution in a
statistical sample of tweets: negative tweets are
normally much less frequent than positive or neutral
ones
The plain word embeddings were built applying vord2vec to a collection of 167 million Italian unlabeled tweets, using the the skip gram model, and the following parameters: embeddings size 300, window dimension 5, discarding word that appear less than 5 times. We obtained about 450k word embeddings.
The sentiment specific word embeddings (SWE) were built with DeepNL, starting from the word embeddings built at the previous step and tuning them with a supervised set of positive or negative tweets, obtained as follows from 2.3 million tweets selected randomly from our corpus of collected tweets: Positive tweet: one that contains only emoticons from a set of positive emoticons (e.g. smiles, hearts, laughs, expressions of surprise, angels and high fives).
Negative tweet: one that contains only emoticons from a set of negative emotions (e.g. tears, angry and sad).
Integris srl cooperated to the task providing a set of 1.3 million tweets, selected by relying on a lexicon of handcrafted polarized words. This resource is also added to the corpus.
We split the training set provided for the Evalita 2016 SentiPolc Task into a train set (5335 tweets), a validation set (592 tweets) and a test set (1482 tweets). This dataset was tokenized and normalized as described in Section 2.1.
For the take of participating to subtask 2, polarity classification, the 13-value annotations present in the datasets were converted into four values: “neutral”, “positive”, “negative” and “mixed” depending on the values of the fields “opos” and “oneg”, which express the tweet polarity, according to the task guidelines3. We did not take into account the values for “lpos” and “lneg”.
The frequency distribution of these classes turns out to be quite unbalanced, as shown in Table 1. 3 http://www.di.unito.it/~tutreeb/sentipolcevalita16/sentipolcguidelines2016UPDATED130916.pdf The training set is still fairly small, compared for example to the size of the corpus used in SemEval 2106. The “mixed” class in particular is small in absolute numbers, even though not in percentage value, which makes hard to properly train a ML classifier.
Therefore we tried to increase the training set by means of the distant supervision as described above: we selected a maximum of 10,000 tweets for class via regular expressions, then we classified them with the classifier trained with the gold training set. We chose for addition into a silver training set, the tweets which were assigned by the classifier the same class of the regular expression. As reported in Table 2, the silver dataset remains unbalanced; in particular, no “mixed” example was added to the original train set.
Class Train set Dev set Neutral 8505 554 Negative 5987 513 Positive 6813 312 Mixed 337 103
Table 2. Distant supervised dataset distribution.
Table 3 shows the common settings used for training the classifier. We used the same parameters as SemEval-2016.
Word Embeddings Size Hidden Units Dropout Rate Batch size Adadelta Decay Epochs We performed extensive experiments with the classifier in various configurations, varying the number of filters; the use of skip-gram word embeddings or sentiment specific word embeddings; different training sets, either the gold one or the silver one. Results of the evaluation on the validation set allowed us to choose the best settings, as listed in the Table 4. Best Settings.
Embeddings Training set
Filters 4
Run1 WE skipgram Gold Silver Gold 2,3,5 4,5,6,7 7,7,7,7,8,8,8,8
Table 4. Best Settings
Run2 SWE
Silver 7,8,9,10 We submitted four runs for the subtask 2 “polarity classification”:
UniPI_1.c: gold training set, word embeddings with skip-gram model, filters: “2,3,5”.
UniPI_1.u: silver corpus as training set, word embeddings with skip-gram model, filters: “4,5,6,7”.
UniPI_2.c: gold training set, sentiment specific word embeddings, filters: “7,7,7,7,8,8,8,8”.
UniPI_2.u: silver corpus as training set, sentiment specific word embeddings, filters: “7,8,9,10”.
The following table reports the top official results for the subtask 2:
System UniPI_2.c team1_1.u team1_2.u team4_.c team3_.1.c team5_2.c team3_.2.c UniPI_1.u UniPI_1.c
Positive F-score 0.685 0.6354 0.6312 0.644 0.6265 0.6426 0.6395 0.6699 0.6766
Negative F-score 0.6426 0.6885 0.6838 0.6605 0.6743 0.648 0.6469 0.6146 0.6002
Combined
F-score 0.6638 0.662 0.6575 0.6522 0.6504 0.6453 0.6432 0.6422 0.6384 UniPI_2.u 0.6586 0.5654 0.612
Table 5. Top official results for SentiPolc subtask 2. The run UniPI_2.c achieved the top overall score among a total of 26 submissions to task 2. This confirms the effectiveness of sentiment specific word embeddings in sentiment polarity classification also for Italian tweets.
The use of an extended silver corpus did not provide significant benefits, possibly because the resulting corpus was still unbalanced.
In addition to the subtask 2, we submitted one run for the Task 1 “Subjectivity Classification”: given a message, decide whether the message is subjective or objective. We used the same classifier for the subtask 2, using only two classes (subjective, objective), with the same skip-gram word embeddings used for the other task and the configuration listed in Table 3, using the following filters: “7,8,9,10”, without performing extensive experiments. The following table reports the top official results for the subtask 1:
system team1_1.u
Objective F-score 0.6784
Subjective
F-score 0.8105
Combined
F-score 0.7444 team1_2.u team2_.1.c team3_.2.c team4_.c team5_2.c team6_.c team1_.c UniPI_1 team3_.1.c team8_.c We confirmed the validity of the convolutional neural networks in the twitter sentiment classification, also for the Italian language.
The system achieved top score in the task 2 of SENTiment POLarity Classification Task of Evalita 2016.
We gratefully acknowledge the support of the University of Pisa through project PRA and NVIDIA Corporation for a donation of the Tesla K40 GPU used in the experiments.
Integris srl cooperated by providing a corpus of sentiment annotated tweets.