English. In this paper we describe and present the results of the two systems, called here X2C-A and X2C-B, that we specifically developed and submitted for our participation to ABSITA 2018, for the Aspect Category Detection (ACD) and Aspect Category Polarity (ACP) tasks. The results show that X2C-A is top ranker in the official results of the ACD task, at a distance of just 0.0073 from the best system; moreover, its post deadline improved version, called X2C-A-s, scores first in the official ACD results. About the ACP results, our X2C-A-s system, which takes advantage of our ready-to-use industrial Sentiment API, scores at a distance of just 0.0577 from the best system, even though it has not been specifically trained on the training set of the evaluation.
Italiano. In questo articolo descriviamo e presentiamo i risultati dei due sistemi, chiamati qui X2C-A e X2C-B, che abbiamo specificatamente sviluppato per partecipare ad ABSITA 2018, per i task Aspect Category Detection (ACD) e Aspect Category Polarity (ACP). I risultati mostrano che X2C-A si posiziona ad una distanza di soli 0.0073 dal miglior sistema del task ACD; inoltre, la sua versione migliorata, chiamata X2C-A-s, realizzata successivamente alla scadenza, mostra un punteggio che lo posiziona al primo posto nella classifica ufficiale del task ACD. Riguardo al task ACP, il sistema X2CA-s che utilizza il nostro standard Sentiment API, consente di ottenere un punteggio che dista solo 0.0577 dal miglior sistema, nonostante il classificatore di sentiment non sia stato specificamente addestrato sul training set della evaluation.
The traditional task of sentiment analysis is the
classification of a sentence according to the
positive, negative, or neutral classes. However, such
task in this simple version is not enough to detect
when a sentence contains a mixed sentiment, in
which a positive sentiment is referred to one
aspect and a negative sentiment to another aspect.
Aspect-based sentiment analysis is focused on the
sentiment classification (negative, neutral,
positive) for a given aspect/category in a sentence.
In nowadays world, reviews became an important
tool widely used by consumers to evaluate
services and products. Given the large amount of
reviews available online, systems allowing to
automatically classify reviews according to
different categories, and assign a sentiment to each of
those categories, are gaining more and more
interest in the market. The former task is called Aspect
Category Detection (ACD) since detects whether
a review speaks about one of the categories
under evaluation; the latter task, called Aspect
Category Polarity (ACP) tries to assign a sentiment
independently for each aspect. In this paper, we
present X2C-A and X2C-B, two different
implementations for dealing with the ACD and ACP
tasks, specifically developed for the ABSITA
evaluation
The official training dataset has been split into our
internal training set (80% of the documents) and
development set (the remaining 20%). We
randomly sampled the examples for each category,
thus obtaining different sets for training/test set,
by keeping the per category distribution of the
samples through the three sets. We submitted
two runs, as the results of the two different
systems we developed for each category, called
X2CA and X2C-B. The former has been developed
on top of the Scikit-learn library in Python
language
X2C-A The X2C-A system has been created by applying an NLP pipeline including a vectorization of the collection of reviews to a matrix of token counts of the bi-grams; then, the count matrix has been transformed to a normalized tf-idf representation (term-frequency times inverse documentfrequency). As machine learning algorithm, an implementation of the Support Vector Machine has been used, specifically the LinearSVC. Such algorithm has been selected as the best performer on such dataset compared to other common implementations available in the sklearn library.
Table 1 shows the F1 score on the positive label in the development set for each category, where the average value on all of the categories is 84.92%. X2C-A shows the lowest performance on the Value category, while shows the best performance on Location, and high score on Wifi and Staff. 2.2
X2C-B In the model selection process, the two best algorithms have been Naive Bayes and SMO. We built a model with both algorithms for each category. We took into account the F1 score on the positive labels in order to select the best algorithm. In this implementation, SMO (Sequential Minimal Optimization) (Platt, 1998) (Keerthi et al., 2001) (Hastie et al., 1998) has been the best performing algorithm on all of the categories, and showed an average F1 score across all categories of 85.08%. Its scores are reported in Table 1, where we also compare its performance with the X2C-A one on the development set.
The two systems are built on different implementation of support vector machines, as previously pointed out, and differ on the features extraction process. In fact, X2C-B takes into account a vocabulary of the 1000 most mentioned words in the training set, according to the size limit parameter available in the StringToWordVector Weka function. Moreover, it uses unigrams instead of the bi-grams extraction performed in X2C-A. The two systems reach similar results, i.e. high scores on Location, Wifi and Staff, and low scores on the Value category. However, the overall weighted performance is very close, around 85% of F1 on the positive labels, and since for some categories is better X2C-A and for others X2C-B, we decided to submit both implementations, in order to understand which is the best one on the test set of the ABSITA evaluation.
Cleanliness Comfort Amenities Staff Value Wifi Location 3.1
is about Location and Staff, but since only a part of it is about Location, the location model of this category would receive a document containing ”noise” from its point of view. In the post deadline runs, we reduce the ”noise” by splitting this example review in The sight is beautiful which is only about Location, and but the staff is rude which is only about Staff. As we can see in Table 2, the performance of X2C-A increased significantly and reached a performance score that is better even than the first classified. However, the performance of X2C-B slighted decreased in its X2C-B-s version. This means that the model of this latter system is not helped by this kind of ”noise” removal technique. This last result shows that such approach does not have a general applicability but it depends on the model; however, it shows to work very well on X2C-A.
In order to identify the categories where we perform better, we calculated the score of our systems on each category1, as shown in Table 3 and Table 4. In Table 3 X2C-A is the best of our systems on all the categories except Cleanliness and Wifi, where X2C-B has reached the higher score. In Table 4, X2C-A-s shows the best performance on all of the categories. By comparing the results across 1To obtain these scores, we modified the ABSITA evaluation script so that only one category is taken into account. In Table 5 we show the results of the Aspect-based Category Polarity task to which X2Check did not formally participate. In fact, after the evaluation deadline we had time to work on the ACP task.
In order to deal with the ACP task, we decided to take advantage of our ready-to-use, standard
X2Check sentiment API
The results shown in Table 5 show that our
assumption is valid. In fact, despite being a single
sentiment model for all of the categories, we reach
the fifth place in the official ranking with our
X2CA-s system, at a distance of just 0.057 from the
best system specifically trained on such training
set. Furthermore, the ACP performance depends
on the ACD results, in fact the former task
cannot reach a performance higher than the other. For
this reason, we decided to evaluate the sentiment
performance reached on the reviews whose
categories have been correctly predicted. Thus, we
created a score capturing the relationship between
the two results: it is the ratio between the micro
F1 score obtained in the ACP task and the one
obtained in the ACD task. This hand crafted score
shows the quality of the sentiment model, by
removing the influence of the performance on the
ACD task. The overall sentiment score obtained is
88.0% for X2C-B and 87.1% for X2C-A, showing
that even if a specific train has not been made, the
general purpose X2Check sentiment API shows
very good results (recall that, according to
1 2 3 4 X2C-A-s 5 X2C-B-s 6 7 8 baseline
Tables 6 and 7 show for each category the micro-F1 and the sentiment score of the ACP task, calculated like in Table 4, and the relationship between ACP and ACD scores per category. We can see that the sentiment model has reached a very good performance on Cleanliness, Comfort, Staff and Location since it is close or over the 90%. However, like noticed for the ACD results, it is difficult to handle reviews about the Value category. 4
In this paper we presented a description of two different implementations for dealing with the ACD and ACP tasks at ABSITA 2018. In particular, we described the models used to participate to the ACD competition together with some post deadline results, in which we had the opportunity to improve our ACD results and evaluate our systems also on the ACP task. The resuls show that our X2C-A system is top ranking in the official ACD competition and scores first, in its X2C-A-s version. Moreover, by testing our ACD models on the ACP tasks, with the help of our standard X2Check sentiment API, the X2C-A-s system scores fifth at a distance of just 0.057 from the best system, even if the other systems have a sentiment classifier specifically trained on the training set of the competition.