Aspect-Based Sentiment Analysis is in charge of study the opinion of people about di erent aspects from a certain entity. This task is challenging and highly relevant for the Natural Language Processing community. In this paper, we report the participation of C100T-PUCP team in the TASS 2017 for the second task about sentiment analysis. In this edition, we used word embeddings to get the similarity between words selected from a training set that had tweets about political parties from Spain and made a model to classify each polarity of each aspect for each tweet. The results showed that using more examples to training the model with this approach is more convenient. Moreover, the proposed approach avoids the problem of the classical methods that are oriented to a speci c training data set.
The 6th edition of the TASS workshop
consists of two task in sentiment analysis
focusing of Spanish tweets: (1) polarity classi
cation at global level and (2) aspect-based
sentiment analysis in which the goal is to
predict the polarity of tweets in relation to a set
of identi ed aspects
The task of polarity classi cation has been taken with many di erent approaches. One of them consists in representing a word as a vector and using it to get a similarity with other words. This method is called word embeddings. Word Embeddings is a well-know technique to get a vector of a word in natural language processing. Although this method is widely used in English, there are few implementations of this approach for Spanish.
In this sense, many studies use deep learn
Copyright © 2017 by the paper's authors. Copying permitted for private and academic purposes.
ing as a main approach to tackle sentiment
analysis so they can get a similarity between
a bag of words that represent each sentiment
Our system uses this kind of approach to classify the sentiment of each aspect of each tweet presented in the task.
This paper summarizes the participation of the C100T-PUCP team from Ponti cia Universidad Catolica del Peru in the second task of the workshop. In this edition, we propose a word embedding-based approach to tackle the problem of aspect-level polarity classi cation. Firstly, we obtain a word embeddings set from politics corpus to get similarity between tweets. Then, we explored a feature selection method for unbalanced data. Finally, we built experiments with some classi ers using word embeddings and the obtained features.
This paper is organized as follows: an overview of related works is shown in Section 2. Section 3 presents the analyzed corpus and its class distribution. The system description is described in Section 4. Section 5 shows the experimentation and results, and nally, Section 6 presents some conclusions and future works. 2
In TASS 2014, an Aspect Detection and
Aspect-based Sentiment Analysis Task were
proposed
In TASS 2015, only the Aspect-based
Sentiment Analysis Task was proposed, but a
new corpus was added in the evaluation
In TASS 2016, two proposals were
submitted to the Aspect-based Sentiment
Analysis task
The STOMPOL corpus is composed of tweets
in Spanish about Spanish elections of 2015.
This corpus was presented in TASS 2014
The system presented in this edition of the TASS uses a preprocessing removing stopwords using NLTK tool. Also, words like URL's and special characters are removed from the tweets. After this, all the words are passed through Freeling lemmatizer, version 4.0. Furthermore, hashtags and labels to users are kept in this processing. As using this preprocessing the tokenization for all the tweets is completed.
The Aspect-based Sentiment Analysis was tackled as a classi cation problem. For this, support vector machines (SVM) and adaptive boosting (AdaBoost) classi ers were used because of the precedence in previous works that showed that they behave well in classifying long vector features. For these models, scikit-learn implementations are used from the toolkit. These, are sklearn.ensemble.AdaBoostClassi er for the adaptive boosting and sklearn.svm.SVC for the SVM implementation with a polynomial kernel.
Also each vector was lled with the cosine
similarity between each feature and the top
50 most important words for each sentiment
using a probabilistic appearance metric
A word embeddings model was created
using Mikolov Word2Vec model
Tweets were selected using search queries related to political parties from South America and Spain in a range from 2012 to 2015. Even though many tweets were selected for this approach much more data was needed, In that sense, online news websites were also scrapped. Speci cally, we obtained texts from "El pa s"1, "ABC" 2 and "20 Minutos"3 Spanish newspapers. From these sites only political news were selected and no range in time was used.
After this, we got 400MB of data which included about 5 million sentences, 30 millions words and 1.5 million unique words. The corpus was preprocessed following the next steps: removing stopwords and special characters that are common in tweets as "..." and URLs removing words that have only one character removing numbers lemmatizing words using Freeling4, keeping all mentions to users and hashtags
Additionally, mentions to particular entities in news data were replaced with their user ids from Twitter. This was to make the news data the most similar to the twitter data so the embeddings get the same relations between words. For example, if we have a new that mentions Pablo Iglesias we replace it with his Twitters id Pablo Iglesias and we do the same with other common political persons.
To create the word embeddings from
the corpus, we used the Word2vec
model
Economics Health Education Political Party Others Total
For this task, a window of three words were selected from the previously identi ed aspect in the training corpus to extract the features. Each tweet in the training corpus was also preprocessed in the same way as the word embeddings model, removing stopwords, special characters and URLs. After this, the corpus was lemmatized keeping user tags and hashtags. The analysis was based on the cosine similarity of these words with the words selected as a feature for detecting the sentiment.
In this step, rstly, we needed to create a
dictionary with words that represent in a
better way each sentiment that was classi ed
using the labels for the sentiments in the
training data. Thus, we used two metrics,
TFIDF and probabilistic occurrence
After selecting the features, two vectors were created: (1) based on probabilistic occurrence and (2) based on probabilistic occurrence per aspect. Each vector was lled in two ways, one was only using the traditional approach, i. e., the vector a bag-of-words and lling the vector with 1 if the word feature occurs in the window and the other way is to ll the feature with the most similarity value to the feature. Then each vector was used to train a SVM and AdaBoost models. In general, we sent three runs which are shown as below:
Run 1: The rst run consisted of using vector lled with the cosine similarity between each word and the words in the dictionary of polarities. This vector was used to train a SVM model with gamma value of 0.31622776601683794, C 1 and degree 2.
Run 2: The second run was using the same vector were the hyper parameters were: gamma was 0.0316, C was 1122.018 and degree was 2.
Run 3: The last run was using AdaBoost Classi er with a modi ed vector that have the most representative word of each sentiment but for each aspect. Also this vector has one hot encoding based on each aspect. This model was trained with a Naive Bayes classier as the weak classi er. The AdaBoost model was created with the following hyperparemeters selected by experimentation: learning rate was 0.000001, number of estimators was 100.
For the experiments a SVM classi er and an AdaBoost classi er were tested using each set of features prepared using the dictionaries previously created. In order to test these approaches F1-score (F1), Precision (P) and (R) recall were used. These metrics evaluate how well the model predicts based on how many true-positive it predicts and the inverse how many false-positive it predicts. In the other hand, accuracy just measured how many hits it does in the prediction. Furthermore, to test the model to detect the sentiment polarity the TASS experiment page was used. This page measure a macro F1score, recall, precision and accuracy based on the combination of how well it predicts each aspect in an speci c entity.
The results for the polarity classi cation are presented in Table 3. In this case a SVM classi er and an AdaBoost classi er using a Naive Bayes model were tested. Also, as discussed earlier, the data is not well balanced, so, for these methods a SMOTE oversampling was applied. For these results, the SVM-run2 model was created by using the features using probabilistic weights of the words. In this sense, this model predicts the sentiment based on the similarity of the word selected as the sentiment representatives with each word that represent a feature in the vector, keeping the best score of these words. This vector was also used for the other models except for the ADA-run3 model.
The SVM-run1 considers that every word may have di erent meanings for each context so it uses a vector that has words that represents all the sentiments for each aspect and a feature that represents from which element the tweet came. This model has better results in comparison to its similar SVM that only use top words in all the training set and have a better accuracy compared with the ADA-run3 but not a better F1-Score.
As seen, the model was not trained with a big amount of data but it still had almost all the words inside it's vocabulary. This pointed us that the errors for the sentiment polarity were caused probably by the unbalance data for each sentiment per aspect. The sentiments were in it's majority negative and few were neutral or positive. Although, using this approach gave us results that were close to other participants that uses domain speci c systems.
In the 6th edition of the TASS 2017, we have tried a word embedding approach to classify sentiment of an aspect. With this approach we generate a vector feature of how similar each selected word is to the features in the vector. The performance of the models has been compared with the models of the other participants of the TASS.
Although the model was not trained with a big amount of data that it was required, this got the similarity for most of the words in the training set. Also, the results were close to the other participants. The obtained results may give us an idea about how well word embeddings could perform in sentiment analysis due to word embeddings are not adjusted to a speci c set of words like traditional methods (using Bag-Of-Words).
With these results, we propose to test this approach using a more balanced data for aspects and sentiments for training. Also, getting more sentences to train the word2vec model could improve the di erentiation of the words that are used in the features. Using this improvement could get better results because as we saw the models can learn to predict a class very well if they have a lot of information about it. Workshop on Semantic Analysis at SEPLN (TASS 2016), pages 47{51. volume 1702 of CEUR Workshop Proceedings. CEUR-WS.org.