English. In the late years sentiment analysis and its applications have reached growing popularity. Concerning this field of research, in the very late years machine learning and word representation learning derived from distributional semantics field (i.e. word embeddings) have proven to be very successful in performing sentiment analysis tasks. In this paper we describe a set of experiments, with the aim of evaluating the impact of word embedding-based features in sentiment analysis tasks.
In the late years sentiment analysis has reached great popularity among NLP tasks. As reported by Mäntylä et al. (2016) the number of papers on this subject has increased significantly in the first two decades of 21st century, as well as the extent of its applications. A wide variety of technologies has been used to assess sentiment analysis tasks during this period. In the latter years, machine learning techniques proved to be very effective; in particular, in recent years systems based on deep learning techniques represent the state of the art. In this field, word embeddings have been widely used as a way of representing words in sentiment analysis tasks, and proved to be very effective.
A relevant mirror of the state of the art in
sentiment analysis field can be found in the SemEval
workshops. In the 2015 edition
Concerning Italian language, EVALITA
conference well represents the state of the art in the
natural language processing field. In 2016
edition
The purpose of this study is to explore ways of using word embeddings to build meaningful representations of documents in sentiment analysis tasks performed on Italian tweets. 2
In this paper we aimed to evaluate the effect of exploiting word embeddings in sentiment analysis tasks. In particular, we explore the effect of five factors on the performance of a sentiment analysis classification system, to answer five research questions: 1. What is the effect of the size of the corpus used to train the embeddings? 2. Which text domain allows us to train better embeddings (in-domain vs out-of-domain data)? 3. Which type of learning method produces better embeddings (word vs character-based word embeddings)? 4. Which method to combine the word vectors produces a better document vector representation? 5. What are the most important words (in terms of part-of-speech) to produce a better document vector representation?
To answer such questions, we performed
several classification experiments testing our system
on the three sentiment analysis tasks proposed in
the 2016 EVALITA SENTIPOLC campaign
the first of these tasks, the highest accuracy was achieved by the system of Castellucci et al. (2016). Concerning the 2nd task, the most accurate system was the one submitted by Attardi et al. (2016). Regarding the 3rd task, the highest accuracy value was reached by the system of Di Rosa and Durante (2016). Among these systems, Castellucci et al. (2016) and Attardi et al. (2016) use deep learning techniques (convolutional neural networks), while Di Rosa and Durante (2016) use an ensemble of many supervised learning classifiers. 3
We tested our system on the three sentiment analysis tasks proposed in 2016 EVALITA SENTIPOLC campaign. These tasks and the related datasets have been described by Barbieri et al. (2016). We conducted our experiments on the training set provided by the organizers of the evaluation campaign, which is composed of 7921 tweets.
We train our word embeddings on two corpora:
in-domain and out-domain. The in-domain dataset
is a collection of tweets that we collected for this
work, named Tweets. It is composed by almost 80
millions of tweets, resulting in around 1.2 billions
of tokens. The out-of-domain dataset is the Paisà
corpus, a collection of Italian web texts described
by Lyding et al.
For our experiments, we used a classifier based on SVM using LIBLINEAR (Rong-En et al., 2013) as machine learning library. As features, the classifier uses only information extracted combining the word-embeddings of the words of the analyzed tweet.
In all the experiments described in this paper, our system addresses the classification tasks by performing 5-fold cross-validation on the training set provided for the SENTIPOLC 2016 evaluation campaign. The final score is the average score. We evaluate each fold using the Average F-score described by Barbieri et al. (2016).
For what concerns the word embeddings, we
trained two types of word embedding
representations: i) the first one using the word2vec1 toolkit
In both cases, each word is represented by a 100 dimensions vector, computed using the CBOW algorithm – that learns to predict the word in the middle of a symmetric window based on the sum of the vector representations of the words in the window – and considering a context window of 5 words. 5
To answer the questions listed in Section 2, we conducted a great amount of experiments, testing many ways of representing the tweets by exploiting in different manners the word embeddings of 1http://code.google.com/p/word2vec/ 0:59 0:58 0:57 s 0:56 re 0:55 o sc 0:54 -F 0:53 e ag 0:52 ev 0:51 r A 0:5 0:49 0:48 0:47 0 200 400
600 800 1;000 1;200
Millions of tokens the words extracted from the tweets.
To evaluate the impact (in terms of classification accuracy) of the variations of each studied parameter, we report the accuracy for each variation of the parameter calculated as the average accuracy across all the classification experiments that we conducted by varying all the other parameters (in a 5-fold cross-validation scenario).
In all the experiments, we used only features based on word embeddings. 5.1
To answer the question n. 1, we trained several word embedding models on different partitions of Tweets corpus of increasing sizes, using both word2vec and fastText. Ten smaller partitions were obtained starting with just ten millions of tokens (for the smaller one) and adding other ten millions for each new partition, reaching the amount of 100 millions. We created other four bigger partitions, which contain respectively 240, 480, 720 and 960 millions of tokens; the size of the smaller of this four partitions is comparable to the size of Paisà.
Figure 1 reports the results. When we use embeddings trained with word2vec on increasing amounts of data, the average value of F-score grows for all the three subtasks. The amount of this growth is similar for the subtasks Subjectivity Classification (0.016) and Polarity Classification (0.019), while it’s smaller for the subtask Irony Detection, which is the most challenging among the three. In all cases the increase is significantly faster in the first 80 to 100 millions of tokens, particularly as regards the Irony Detection task: in this case, the average F-score basically stops growing after around 80 millions of tokens.
When we use embeddings trained with fastText, the outcome is the opposite: the average F-score values decrease as bigger amounts of data are used to train the embeddings. The decrease of the values is faster when using the first hundreds of millions of tokens.
Lesson learned: these results suggest that, regarding word-based word embeddings, as the training corpus grows the accuracy rises, but it becomes stable quickly. On the other hand, the increase of the size of the training corpus apparently doesn’t influence the accuracy values when the embedding have been produced using fastText (or it even causes a lowering of the accuracy values). 5.2
To answer the question n. 2, we ran a set of experiments using the four models obtained using word2vec and fastText on Paisà and Tweet corpora. Table 1 reports the results of the experiments. As we can see, the embeddings trained with word2vec on the in-domain dataset (Tweets) provide features that allow to achieve a higher average accuracy compared to the features extracted from the out-domain corpus. Differently, there isn’t any variation in terms of accuracy when the embeddings are trained with fastText.
Lesson learned: the in-domain word embeddings are very important in a semantic classification scenario. Apparently, this is not true when character-based word embedding are used.
Subj.
Pol.
Iro.
w2v tw 0.5901 pa 0.572 ft 0.5198 0.5206 For what regards the question n. 3, the type of embeddings learning model (words vs character n-grams) influences considerably the performance of the classifier. Using embeddings trained with word2vec leads to F-score values that are significantly higher in comparison to the accuracy obtained using embeddings trained with fastText (see Table 1).
Lesson learned: this outcome suggests that embeddings learned by methods that treat words as atomic entities provide features that are more useful in a semantic task such as sentiment classification, in comparison with character-based embeddings.
To answer the question n. 4, we tested many methods to combine the embeddings of the words of each document into a document-level vector representation.
We experimented five combining methods: Sum, Mean, Maximum-pooling, Minimumpooling, Product. Each of this methods returns a single vector ~t , such that each tn is obtained by combining the nth components w1n; w2n : : : wmn of the embedding of each tweet word. Figure 2 shows a graphical representation of this process. tweet w~1 w~2 w~3 ... w~n ~ t w11 w21 w31 ... wn1 t1 w12 w22 w32 ... wn2 t2
We tested these methods separately, and all of them jointly as well. When using all methods, the document representation is obtained concatenating the vectors returned by each method.
As we can see in Table 2, the Sum method proved to be the best method for all the tasks, when using embeddings obtained by word2vec. The best results overall are obtained using the concatenation of each of the vectors returned by the used methods (row All in the Table). When using embeddings trained with fastText, the best results are obtained with mean for Subjectivity and Polarity Classification, and with sum for Irony Detection. In this case, the combination of all vector leads to poor results.
Lesson learned: these outcomes suggest that the best combination methods are sum for word vectors obtained by using word-based word embeddings and mean for character-based ones.
Meanwhile, the worst approach is the Product combination. Interestingly, while the concatenation of all the combined word-based word embeddings is surely the best approach to produce the document-level vector representation, this is not true for the character-based ones. To answer the question n. 5, we ran a set of experiments using only a subset of the word embeddings of each document to produce the document vector representation. The word selection is guided by the morpho-syntactic categories of the words. We tested four categories: noun, verb, adjective, adverb. The embeddings of the words belonging to each of these categories were combined in a posbased vector representation document. In addition, we tested the document representation vector obtained through the concatenation of the different pos-based vectors (N, V, Adj, Adv) with and without the all-word document vector All words, which is the only one taking into account emoticons and hash tags.
Table 3 reports the results of the experiments. In the word-based word embedding scenario, regarding the contribution of single morpho-syntactic categories, noun shows the highest performance. Overall, the highest score is yielded by the combination of all the selected categories concatenated with the combined vector of all the word embeddings (All words rows in the table). For what regards the character-based word embeddings, we can see that the noun is the individually best performing category only for the Subjectivity Classification task, while the adjective and the verb are the best performing category for the other two tasks.
Lesson learned: these results show that noun class is the most important grammatical category only in the word-based word embedding scenario; meanwhile the concatenation of all the pos-based vectors and the All words vector yields the best accuracy in both scenarios. 6
In this work we study the impact of word embedding-based features in the sentiment analysis tasks. We performed several classification experiments to investigate the effects on classification performances of five dimensions related to the word embeddings. We tested several different ways of selecting and combining the embeddings and we studied how the performance of a sentiment classifier changes.
Despite the lessons learned from this work, several aspects remain to investigate, such as, for example, the tuning of the parameters used to train the embeddings, and new vector combining strategies.