In this work, we report the results of some experiments with Aspect Based Sentiment Analysis (ABSA) on a dataset consisting of user reviews of products of a manufacturing company operating in the beauty industry. We focus on one of the more challenging ABSA tasks, the Aspect Category Opinion Sentiment task, and compare the results obtained by using three diferent tools.
CEUR ceur-ws.org
sentence ”it’s very reasonably priced”, when the subject is not explicitly named, the value of the aspect term is ”NULL”. • opinion term (o): is the word, or the words, used by opinion users to convey their sentiments or feelings about the target entity or aspect. For example, in the sentence ”The pizza is delicious but the service is terrible”, ”delicious” and ”terrible” are opinion terms, expressing a positive and negative sentiment toward the pizza and the service, respectively. • polarity (p): characterizes the sentiment orientation expressed towards an aspect category or an aspect term. Sentiment polarity can be positive, negative, or neutral indicating that the sentiment is favorable, unfavorable, or neither, respectively.
Among the tasks of Aspect-based Sentiment Analysis that aim to predict a single sentiment element, there are: • Aspect Term Extraction (ATE); • Aspect Category Detection (ACD); • Opinion Term Extraction (OTE); • Aspect opinion co-extraction (AOCE); • Aspect Sentiment Classification (ASC). • Aspect-Opinion Pair Extraction (AOPE); • End-to-End ABSA (E2E-ABSA); • Aspect Category Sentiment Analysis (ACSA); • Aspect Sentiment Triplet Extraction (ASTE); • Aspect Category Sentiment Detection (ACSD); • Aspect Category Opinion Sentiment (ACOS).
ASC
Input sentence sentence sentence sentence, pizza sentence, service sentence sentence sentence sentence sentence sentence
Output pizza (a), service (a) food (c), service(c) delicious (o), terrible (o) positive(p) negative (p) {pizza (a), delicious (o)}, {service (a), terrible (o)} {pizza (a), positive p)}, {service (a), negative (p)} {food (c), positive (p)}, {service (c), negative (p)} {pizza (a), positive (p), delicious (o)}, {service (a), negative (p), terrible (o)}
{food (c), pizza (a), positive (p)}, {service (c), service (a), negative (p)} {pizza (a), food (c), delicious (o), positive (p)}, {service (a), service (c), terrible (o), negative (p)}
In this paper, we will focus our attention on the ACOS task which aims at predicting all the sentiment information at once, namely category (c), aspect term (a), opinion term (o), and polarity (p). For the ACOS task, a relatively limited body of research and literature exists. Our primary objective is to establish an integrated framework that leverages multiple tools for eficient ACOS task execution.
Concerning the annotations, there are diferences with the datasets available in the literature due to the many implicit aspects referring to packaging and opinion terms often composed from multiple words.
The dataset is composed of a training and test set, and each sentence can have multiple annotations.
623 881
133 157
756 1038
The composition appears balanced in terms of positive and negative polarity (p), with neutral sentiment not being of interest. As regards the categories, thirteen classes were identified, mostly balanced, except for the category belonging to ”general satisfaction of the final consumer”.
For this work, a custom template in Label Studio was built, which allows all elements of interest to be annotated for each review. In Figure 1 we show an example of a sentence annotated with this annotation tool: the explicitly mentioned aspect and opinion elements can be directly selected in the text, while the polarity and the category, which is not shown, can be chosen from the predefined ones. A translation module has been developed to convert the JSON encoding of the dataset exported from Label Studio to other formats.
In this section, we present the details of the experimental evaluation we performed on our
dataset using some tools that have been specifically built for the ACOS task. We have selected
three tools that have stemmed from significant studies in this field and for which the source
code is publicly available online. All the selected tools leverage the fine-tuning of pre-trained
models, specifically T5 [
PyABSA batch-size learning rate
16 3e-4 32{a, o}, 16(p), 8(c) 2e-5{a, o}, 3e5(p),(c) 16 5e-5 epochs 20 20 20
The second experiment is motivated by the fact that sentences in our domain often contain implicit opinions, frequently composed of multiple words rather than single terms. So we established a relaxed correctness criterion for considering a prediction correct when it matches the gold standard in terms of aspect, category, and polarity, and when the similarity between the predicted opinion term and the real one is at least 70%. For computing string similarity we used the SequenceMatcher4 Python function, which compares pairs of sequences by finding the longest common subsequence while excluding uninteresting elements, with a quadratic time 2AdamW optimizer: is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments with an added method to decay weights.
complexity for the worst case. In this way, for instance, the prediction of the opinion “super practical to slip into my bag” can be considered correct even if the real opinion is “practical to slip into my bag”. 3.1. Results The performance of the models was evaluated using precision, recall, and F1-Score. Precision is the ratio of relevant instances retrieved to all instances retrieved. Recall is the ratio of relevant instances retrieved to all relevant instances. F1-Score is the harmonic mean of precision and recall. The results are shown in Table 3. A quadruple prediction is deemed correct only if it matches the gold standard in all four components, except for the last tool in the table.
Among the tools with base pre-trained models (T5-base and BERT), the Paraphrase modeling tools seem to be the overall best, but the support for the implicit opinion, lacking from this tool, could be important for some application domains. The Extract Classify-ACOS tool seems to be slightly better than Paraphrase modeling in terms of precision but has a significantly lower value for recall. The last tool we considered, PyABSA, is not the best in terms of performance but it turned out to be very well designed, allowing us to customize it for performing further experiments using a larger pre-trained model (T5-Large) and employing a similarity criterion for one of the components. By using the larger model the precision increased from about 32% to 41% using the standard correctness criterion, and to 54% using the relaxed correctness criterion based on similarity.
We benchmarked three ACOS systems from existing literature on a new domain using our custom dataset.
The research aims to create a unified framework for executing various ABSA tasks using diferent tools on the same dataset. Adapters would handle data translation into the correct format. The framework should allow defining various experiments and exploring diferent scenarios through automatic and controlled selection of test and train data, based on data categories and polarities.
We foresee an integrated framework where these tools’ predictions are used to automatically or semi-automatically enhance and expand the training data, thereby improving the eficiency and overall quality of the sentiment analysis models.