English. ABSITA is the Aspect-based Sentiment Analysis task at EVALITA 2018 (Caselli et al., 2018). This task aimed to foster research in the field of aspect-based sentiment analysis within the Italian language: the goal is to identify the aspects of given target entities and the sentiment expressed for each aspect. Two subtasks are defined, namely Aspect Category Detection (ACD) and Aspect Category Polarity (ACP). In total, 20 runs were submitted by 7 teams comprising 11 total individual participants. The best system achieved a micro F1-score of 0.810 for ACD and 0.767 for ACP.
Italiano. ABSITA e` l’esercizio di
valutazione di aspect-based sentiment
analysis di EVALITA 2018
In recent years, many websites started offering a high level interaction with users, who are no more a passive audience, but can actively produce new content. For instance, platforms like Amazon1 or TripAdvisor2 allow people to express their opinions on products, such as food, electronic items, clothes, and services, such as hotels and restaurants.
In such a social context, Sentiment Analysis
(SA) is the task of automatically extract subjective
opinions from a text. In its most basic form, a SA
system takes in input a text written in natural
language and assign it a label indicating whether the
text is expressing a positive or negative sentiment,
or neither (neutral, or objective, text). However,
reviews are often quite detailed in expressing the
reviewer’s opinion on several aspects of the target
entity. Aspect-based Sentiment Analysis (ABSA)
is an evolution of Sentiment Analysis that aims
at capturing the aspect-level opinions expressed in
natural language texts
At the international level, ABSA was
introduced as a shared task at SemEval, the most
prominent evaluation campaign in the
Natural Language Processing field, in 2014
(SEABSA14), providing a benchmark dataset of
reviews in English
The task was repeated in SemEval 2015
(SEABSA15) and 2016 (SE-ABSA16), aiming to
facilitate more in-depth research by providing a new
ABSA framework to investigate the relations
between the identified constituents of the expressed
opinions and growing up to include languages
other than English and different domains
ABSITA (Aspect-based Sentiment Analysis on Italian) aims at providing a similar evaluation with respect to texts in Italian. In a nutshell, partic
ipants are asked to detect within sentences (expressing opinions about accommodation services) some of the aspects considered by the writer. These aspects belongs to a close set ranging from the cleanliness of the room to the price of the accommodation. Moreover, for each detected aspect, participants are asked to detect a specific polarity class, expressing appreciation or criticism towards it.
During the organization of the task, we collected a dataset composed of more than 9,000 sentences and we annotated them with aspects and polarity labels. During the task, 20 runs were submitted by 7 teams comprising 11 individual participants.
In the rest of the paper Section 2 provides a detailed definition of the task. Section 3 describes the dataset made available in the evaluation campaign, while Section 4 reports the official evaluation measures. In Section 5 and 6, the results obtained by the participants are reported and discussed, respectively. Finally, Section 7 derives the conclusions. 2
In ABSITA, Aspect-based Sentiment Analysis is decomposed as a cascade of two subtasks: Aspect
Polarity (ACP). For example, let us consider the sentence describing an hotel:
I servizi igienici sono puliti e il personale cordiale e disponibile. (Toilets are clean but the staff is not friendly nor helpful.)
In the ACD task, one or more ”aspect categories” evoked in a sentence are identified, e.g. the pulizia (cleanliness) and staff categories in sentence 2. In the Aspect Category Polarity (ACP) task, the polarity of each expressed category is recognized, e.g. a positive category polarity is expressed concerning the pulizia category while it is negative if considering the staff category.
In our evaluation framework, the set of aspect categories is known and given to the participants, so the ACD task can be seen as a multi-class, nonexclusive classification task where each input text has to be classified as evoking or not each aspect category. The participant systems are asked to return a binary vector where each dimension corresponds to an aspect category and the values 0 (false) and 1 (true) indicate whether each aspect has been detected in the text. Table 1 shows examples of annotation for the ACD task.
For the ACP task, the input is the review text paired with the set of aspects identified in the text within the ACD subtask, and the goal is to assign polarity labels to each of the aspect category. Two binary polarity labels are expected for each aspect: POS an NEG, indicating a positive and negative sentiment expressed towards a specific aspect, respectively. Note that the two labels are not mutually exclusive: in addition to the annotation of positive aspects (POS:true, NEG:false) and negative aspects (POS:false, NEG:true), there can be aspects with no polarity, or neutral polarity (POS:false, NEG:false). This is also the default polarity annotation for the aspects that are not detected in a text. Finally, the polarity of an aspect can be mixed (POS:true, NEG:true), in cases where both sentiments are expressed towards a certain aspect in a text. Table 2 summarizes the possible annotations with examples. The participants could choose to submit only the results of the ACD subtask, or both tasks. In the latter case, the output of the ACD task is used as input for the ACP. As a constraint on the results submitted for the ACP task, the polarity of an aspect for a given sentence can be different than (POS:false, NEG:false) only if the aspect is detected in the ACD step. 3
The data source chosen for creating the ABSITA datasets is the popular website booking.com3. The platform allows users to share their opinions about hotels visited through a positive/negative textual review and a fine-grain rating system that can be used for assigning a score to each different aspect: cleanliness, comfort, facilities, staff, value for money, free/paid WiFi, location. Therefore, the website provides a large number of reviews in many languages.
We extracted the textual reviews in Italian, labeled on the website with one of the eighth considered aspects. The dataset contains reviews left by users for hotels situated in several main Italian cities such as Rome, Milan, Naples, Turin, Bari, and more. We split the reviews into groups of sentences which describe the positive and the negative characteristics of the selected hotel. The reviews have been collected between the 16th and
Sentence I servizi igienici sono puliti e il personale cordiale e disponibile La posizione e` molto comoda per il treno e la metro. Ottima la disponibilita´ del personale, e la struttura della stanza CLEANLINESS 1 0 0
STAFF 1 0 1
COMFORT 0 0 1 the 17th of April 2018 using Scrapy4, a Python web crawler. We collect in total 4,121 distinct reviews in Italian language.
The reviews have been manually checked to verify the annotation of the aspects provided by booking.com, and to add missing links between sentences and aspects. We started by annotating a small portion of the whole dataset split by sentences (250 randomly chosen sentences) using four annotators (the task organizers) in order to check the agreement of the annotation. For the ACD task, we asked the annotators to answer straightforward questions in the form of “Is aspect X mentioned in the sentence Y ?” (Tab. 1).
The set of italian aspects is the direct translation of those booking.com: PULIZIA (cleanliness), COMFORT, SERVIZI (amenities), STAFF, QUALITA-PREZZO (value), WIFI (wireless Internet connection) and POSIZIONE (location). Similarly, for the ACP subtask, the annotation is performed at sentence level, but with the set of aspects already provided by the ACD annotation, and checkboxes to indicate positive and negative polarity of each aspect (Tab. 2). The result of the pilot annotation has been used to compute an inter-annotator agreement measure, in order to understand if it was possible to allow annotators to work independently each other on a different set of sentences. We found agreement ranging from 82.8% to 100% with an average value of 94.4% obtained counting the number of sentences annotated with the same label by all the annotators.
In order to complete the annotation, we assigned different 1,000 reviews to each annotator (about 2,500 sentences on average). We split the dataset among the annotators so that each of them received a uniformly balanced distribution of positive and negative aspects, based on the
scores provided by the original review platform.
Incomplete, irrelevant, and incomprehensible sentences have been discarded from the dataset during the annotation. At the end of the annotation process, we obtained the gold standard dataset with the associations among sentence, sentiment and aspect. The entire annotation process took a few weeks to complete. The positive and negative polarities are annotated independently, thus for each aspect the four sentiment combination discussed in Section 2 are possible: positive, negative , neutral and mixed. The resulting classes are: cleanliness positive, cleanliness negative, comfort positive, comfort negative, amenities positive, amenities negative, staff positive, staff negative, value positive, value negative, wifi positive, wifi negative, location positive, location negative, other positive, other negative.
For each aspect, the sentiment is encoded in two classes: negative = (* positive = 0, * negative = 1) positive = (* positive = 1, * negative = 0) neutral = (* positive = 0, * negative = 0) mixed = (* positive = 1, * negative = 1) Please note that the special topic, OTHER has been added for completeness, to annotate sentences with opinions on aspects not among the seven considered by the task. The aspect OTHER is provided additionally and it is not part of the evaluation of results provided for the task.
We released the data in Comma-separated Value format (CSV) with UTF-8 encoding and semicolon as separator. The first attribute is the id of the review. Note that in booking.com the order of positive and negative sentences is strictly defined and this can make too easy the task. To overcome Trial set Training set Test set
Description Trial dataset containing a small set of features used for checking the format of the file format The dataset contains sentences provided for training. They have been selected using a random stratification of the whole dataset.
The dataset contains sentences provided for testing. They contains sentences without the annotations of aspects. this issue, we randomly assign for each sentence a new position in the review. As a consequence, the final positional id showed in the data file do not reflect the real order of the sentences in the review. The text of the sentence is provided at the end of the line and delimited by ". It is preceded by three binary values for each aspect indicating respectively: the presence in the sentence (aspectX presence:0/1), the positive polarity for that aspect (aspectX pos:0/1) and finally the negative polarity (aspectX neg:0/1). Fig. 1 shows an example of the annotated dataset in the proposed format.
The list of the datasets released for the task is provided in Tab. 3 and the distribution of the sentences among aspects and polarity is provided in Tab. 4. The subdivision adopted for it is respectively 0.34%, 69.75%, 29,91% for trial, training and test data. The datasets can be freely downloaded from http://sag.art. uniroma2.it/absita/ and reused in noncommercial projects and researches. After the submission deadline, we also distributed the gold standard test set and evaluation script. 4
We evaluate the ACD and ACP subtasks separately by comparing the classifications provided by the participant systems to the gold standard annotations of the test set. For the ACD task, we compute Precision, Recall and F1-score defined as: F 1a = P2aP+aRRaa , where Precision (Pa) and Recall (Ra) are defined as: Pa = jSa\Gaj ; Ra = jSaj jSa\Gaj . Here Sa is the set of aspect category
jGaj annotations that a system returned for all the test sentences, and Ga is the set of the gold (correct) aspect category annotations. For instance, if a review is labeled in the gold standard with the two aspects Ga = fCLEANLINESS; STAFFg, and the system predicts the two aspects Sa = fCLEANLINESS; COMFORTg, we have that jSa \ Gaj = 1, jGaj = 2 and jSaj = 2 so that Pa = 12 , Ra = 12 and F 1a = 21 . For the ACD task the baseline will be computed by considering a system which assigns the most frequent aspect category (estimated over the training set) to each sentence.
For the ACP task we evaluate the entire chain, thus considering both the aspect categories detected in the sentences together with their corresponding polarity, in the form of (aspect; polarity) pairs. We again compute Precision, Recall and F1-score now defined as F 1p = P2pP+pRRpp . Precision (Pp) and Recall (Rp) are defined as Pp = jSpjS\pGj pj ; Rp = jSpjG\pGj pj , where Sp is the set of (aspect; polarity) pairs that a system returned for all the test sentences, and Ga is the set of the gold (correct) pairs annotations. For instance, if a review is labeled in the gold standard with the pairs Gp = f(CLEANLINESS; P OS); (STAFF; P OS)g, and the system predicts the three pairs Sp = f(CLEANLINESS; P OS); (CLEANLINESS; N EG); (COMFORT; P OS)g, we have that jSp \ Gpj = 1, jGpj = 2 and jSpj = 3 so that Pa = 31 , Ra = 12 and F 1a = 0:28.
For the ACP task, the baseline is computed by considering a system which assigns the most fresentence_id; aspect1_presence; aspect1_pos; aspect1_neg; ...; sentence 201606240;0;0;0;0;0;0;0;0;0;0;0;0;1;1;0;0;0;0;1;1;0;"Considerato il prezzo e per una sola notte,va ..." 201606241;1;0;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"Almeno i servizi igienici andrebbero rivisti e ..." 201606242;0;0;0;1;0;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;"La struttura purtroppo \‘e vecchia e ci vorrebbero ..." quent (aspect; polarity) pair (estimated over the training set) to each sentence.
We produced separate rankings for the tasks, based on the F1 scores. Participants who submitted only the result of the ACD task appear in the first ranking only. 5
We received submissions from several teams that
participated in past editions of EVALITA, in
particular to the SENTIPOLC (Sentiment Polarity
Classification
We also provide the result of a baseline system that assigns to each instance the most frequent class in each task, i.e., the aspect (COMFORT) and polarity (positive) for that aspect, according to the frequency of classes in the training set. The results of the submissions for the two tasks, and the baseline (namely mfc baseline), are reported in Tab. 5 and Tab. 6. Of the seven teams who participated to the ACD task, five teams also participated to the ACP task.
The results obtained by the teams largely outSystem ItaliaNLP 1 UNIPV gw2017 2 gw2017 1 ItaliaNLP 2 SeleneBianco ilc 2 ilc 1 mfc baseline
Micro-P 0.8264 0.8612 0.7472 0.7387 0.8735 0.6869 0.4123 0.5452 0.2451 perform the baseline demonstrating the efficacy of the solutions proposed and the affordability of all two tasks. The results obtained for the ACD task (Tab. 5) show a small range of variability, at least in the first part of the ranking (the top results are concentrated around a F1 score value of 0.80). On the contrary, the values of precision and recall show higher variability, indicating significant difference among the proposed approaches. 6
The teams of the ABSITA challenge have been invited to describe their solution in a technical report and to fill in a questionnaire, in order to gain an insight on their approaches and to support their replicability. Five systems (ItaliaNLP, gw2017, X2Check, UNIPV, SeleneBianco) are based on supervised machine learning, that is, all the systems for which we have access to the implementation details, with the exception of VENSES, which is a rule-based unsupervised system. Among the system that use supervised approaches, three systems (ItaliaNLP, gw2017, UNIPV) employ deep learning (in particular LTSM networks, often in their bi-directional variant).
All runs submitted can be considered ”constrained runs”, that is, the systems were trained on the provided data set only.
Besides additional training data, some
systems employ different kind of external resources.
Among these, pre-trained word embeddings are
used as word representations by UNIPV
(Fasttext5) and gw2017 (word embeddings provided by
the SpaCy framework6). The system of ItaliaNLP
employs word embedding created from the ItWaC
corpus
Some of the systems are ABSA extensions built on top of custom or pre-existing NLP pipelines. This is the case for ItaliaNLP, VENSES and X2Check. Other systems make use of off-theshelf NLP tools for preprocessing the data, such as SpaCy (gw2017, UNIPV) and Freeling7 (SeleneBianco).
Finally, additional resources used by the
systems often include domain-specific or affective
lexicons. ItaliaNLP employed the MPQA
affective lexicon
In the ACD task, the precision of the second
ranked system (gw2017) is significantly higher
than that of the first system (ItaliaNLP), although
the latter ranks at the top because of a higher
recall. This unbalance between precision and recall
is mainly due to the high number of aspect that
can be assigned at the same time to a sentence: a
system returning too many aspects is exposed to
low precision but higher recall, while a more
conservative system would achieve the opposite
situation. Further details about the systems
developed for the task can be found in the technical
reports of the partecipants: ItaliaNLP
The large availability of user-generated contents over the Web that characterizes the current tendencies of virtually sharing opinions with others has promoted the diffusion of platforms able to analyze and reuse them for personalized services. A challenging task is the analysis of the users’ opinions about a product, service or topic of dis
fastText/blob/master/pretrained-vectors. md
VU-sentiment-lexicon cussion. In particular, the ABSA (Aspect-based Sentiment Analysis) task concerns the association of a polarity (positive, negative, neutral/objective) to the piece of the sentence that refers to an aspect of interest. In ABSITA, we proposes to automatically extract users’ opinions about aspects in hotel rewievs. The complexity of the task has been successfully faced by the solutions submitted to the task. Systems that used supervised machine learning approaches, based on semantic and morphosyntactic features representation of textual contents, demonstrate encouraging performances in the task. Good results have also been obtained using rule-based systems, even though they suffer from generalization issues and need to be tailored on the set of sentences to classify. The decision to use additional resources as additional lexicons in conjunction with semantic word embeddings have been demonstrated to be successful. More details about the implementation of the systems that participated in the task can be found in their specific reports. In conclusion, we consider the ABSITA 2018 task a success and an improvement of state of the art for the ABSA task in the Italian language.