English. This paper describes our submission to the tasks on Sentiment Analysis of ATE ABSITA (Aspect Term Extraction and Aspect-Based Sentiment Analysis). In particular, we focused on Task 3 using an approach based on combining frequency of words with lexicon-based polarities and uses Boosted Trees to predict the sentiment score. This approach achieved a competitive error and, thanks to the interpretability of the building blocks, allows us to show the what elements are considered when making the prediction. We also joined Task 1 proposing a hybrid model that joins rule-based and machine learning methodologies in order to combine the advantages of both. The model proposed for Task 1 is only preliminary.
Italiano. Questo articolo descrive la nostra sottomissione ai tasks sulla Sentiment Analysis ATE ABSITA (Aspect Term Extraction and Aspect-Based Sentiment Analysis). I nostri sforzi si sono concentrati sul Task 3 per il quale abbiamo adottato gli alberi di predizione (Boosted Trees) utilizzando come features di ingresso una combinazione basata sulla frequenza delle parole con la polarita` derivate da un lessico. L’approccio raggiunge un errore competitivo e, grazie all’interpretabilita` dei moduli intermedi, ci consente di analizzare in dettaglio gli elementi che caratterizzano maggiormente la fase di predizione. Una proposta e` stata realizzata anche per il Task 1, dove abbiamo sviluppato un modello ibrido che Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). combina un approcio basato su regole con tecniche Machine Learning. Il modello sviluppato per il Task 1 e` solo in fase preliminare.
User feedback has become essential for
companies to improve their services and products.
Nowadays, we can find user feedback in textual form as
online reviews, posts on social media and so on.
These resources can express overall opinions but
also opinions about some specific details (aspects)
of the subject. In this scenario, the tools provided
by Sentiment Analysis are crucial to process user
feedbacks, the ongoing research in this field is
focused on creating models that are more and more
accurate and that can also extract fine grained
information for the data. As part of this research, the
ATE ABSITA tasks
It’s important to notice that the dataset released
for the task is one of the few resources for the
Italian language that has annotated aspects and
sentiment at the same time. Others Italian resources
that take into account sentiment with respect to
aspects are
This paper describes our approaches in solving task 1 and task 3. The approach for task 1 is still preliminary.
1http://www.di.uniba.it/ swap/ate absita/index.html
In the last decade top performing approaches to
Sentiment Analysis have shifted from using
classifiers on hand-crafted features, often based on
lexicons
Regarding Task 1 we follow the latest trend of
exploiting linguistic patterns
In Section 2 we will describe our approach for Task 3 and in Section 2.4 we will discuss the results. In Section 3 we will briefly discuss the preliminary model we build for Task 1 and its results. 2
The idea behind our approach is to achieve
competitive results using well known tools that can
be used on commodity hardware. We build the
features representing the text using n-grams and
adding a set of characteristic annotated in
SenticNet
Before extracting the lexical features we remove stop words (apart from words that can be used as negative adverbs) and lemmatized each word. Finally, we extract a set of n-grams from each review. We consider uni-grams, bi-grams and trigrams at the same time. 2.2
To build the polarity features of our model, we have adopted SenticNet, a resource used for concept-level sentiment analysis. It contains a collection of concepts, including common-sense concepts, provided with values for polarity, attention, pleasantness and sensitivity. These are numerical features that are available for a subset of the words in each review. We take in account the average, the minimum and the maximum of all the values available in each review. We also consider the mood tags provided by SenticNet. They are sets of tags as #tristezza, #rabbia, #felicita`2 attached to each word, we consider them as binary features. 2.3
Our final regressor is composed of 800 Decision Trees with a maximum depth of 4 layers. The model was trained using Gradient Boosting with a learning rate of 0.3. The final prediction is computed averaging the output of each tree. The rationale behind our choice is that we have a high number of features that are easy to use with tree based methods for specific cases, hence ensembling allows us to learn a set of shallow trees and each of them can work well for specific cases. 2.4
To build our model we initially focused on the training set using cross-validation to optimize the parameters achieving a root mean square error of 0.852 (the prediction target is on a scale from 1 to 5), we then tested the optimized model on the development set reaching an error of 0.805. We finally achieved an error of 0.795 on the final test set. The difference in the error across the different stages of validation suggests that the model is well trained as the error doesn’t increase when new data is presented. However, it also suggest that the estimation of the error has a wide confidence interval, the standard deviation estimated during cross validation is 0.049.
In Figure 1 we compare the scores predicted and the annotated score on the development set. The chart shows that the model has a tendency to over estimate the error, especially in cases annotated with a low score.
2In English: #sadness, #anger, #happiness perfection line score 1 2
3 annotated score 4 5
We will now examine two reviews for which our regressor has the highest error. This is the text of the first review: “si autospenge proprio quando si necessita di usarla contelecomando”3.
This review was annotated with a score of 2, but the score assigned by our system is 4.75. This highlights a tendency of the system to give higher scores in uncertain cases. In this specific case we have no adjectives and two typing mistakes that result in no information from the lexicon and most of the words being disregarded as rare by our preprocessing pipeline. This suggests that a special treatment is needed for these specific cases where the classifier has fewer elements to take a decision.
The text of the second review is: “Per questo prezzo c’e` di meglio.. restituita.Gli accessori sono ottimi.”4.
This sentence was annotated with a score of 2, but the score assigned by our system is 3.36. We have again a case of over estimation of the score. This time the review has two contrasting sentences. A very negative one where the user states of having returned the item and a very positive one regarding the accessories. This ambiva3In English: It turns off on its own when you need to use it with the remote control. (The original sentence contains a two typos.)
4In English: There’s a better choice for the same price.. I returned it.The accessories are great. term pessimo purtroppo rimborsare non consigliare purtroppo essere cattivo dispiacere pensare sconsigliare dopo non funzionare delusione non riconoscere restituire bruciare lence makes the review a borderline case for our model.
We attribute this tendency to overestimate the target to the fact that the model is optimized to minimize the root-mean-square error, this makes the model predict values closer to the average annotated score. While this is acceptable in an academic competition, it’s less than ideal in an industrial setting. One way to solve the overestimation problem, without changing the formulation of the error to minimize, would be to balance the data so to have a similar number of occurrences for each score. Sub-sampling the data is unpractical as it would reduce the sample size too drastically. This leaves open only the option to add more samples.
In Table 1 we see the 15 terms most influential on the model. Here we note that most of the terms have a negative connotation. Interestingly, all the bi-grams in the list contain the word non (not). Taking in account that the terms reported in the table add up to 37% of the importance of all the features, this highlights the fact that the regressor puts particular attention in the prediction of reviews with a low score even if they are a minority. 3
Task 1 asks to identify terms and phrases that contain an aspect of the customer review when it cooccurs with opinion words that bring information about the sentiment polarity. 5
For this task we have designed a hybrid model that joins a rule-based approach with machine learning. The main idea is to identify a set of plausible aspects via some pre-defined rules, then use a classifier to filter out the wrong candidates. The rules are defined on POS-Tagging patterns. For example the review
“Ottimo rasoio dal semplice utilizzo.” with annotated as aspect “semplice” matches the rule defined by the following pattern
The bold tag indicates the position of the plausible aspect. We have defined a set of about 3000 rules. The rules have been discovered picking the most common POS-Tagging patterns that match the annotated aspects. In particular we have found the position of the aspects in the sentence and selected the POS of close words (three on each side) taking in account the punctuation.
Each aspect found can match one or more rules.
The activation of each rule is used as binary
feature for the final classifier. The final classifier is
implemented using Logistic Regression
This preliminary effort achieves a F1-score of 0.340, which is above the baseline (0.255) but below the average score of the submissions (0.504). 4
The submission confirmed the effectiveness of using a simple approach to predict the sentiment score of customer reviews in Italian (Task 3). The approach consists in combining simple word embedding, specifically tri-grams, and a lexicon as SenticNet to build features for Boosted Trees. Our system achieved a competitive error which is lower than the baseline by 0.209 points and higher than the best model by 0.131 points. The error achieved above the average official score by 0.067 points (the estimates includes baseline models).
The submission also highlights that we were able to beat the baseline for Task 1 with a rudimentary approach. We will build upon this approach in our future work.
http://www.di.uniba.it/ swap/ate absita/task.html task at