Aspect category sentiment analysis (ACSA) is a sub-category of Aspect-Based Sentiment Analysis (ABSA) that aims at identifying the aspect categories and corresponding sentiments involved in a sentence, regardless of whether the aspect terms are explicitly mentioned or not [2].
This challenging task requires understanding context and linguistic nuances. For example, an aspect may be mentioned implicitly rather than explicitly, or a single piece of text may contain contrasting sentiments about different aspects [3]. ACSA provides detailed insights on various categories of products/services, helping in product development, marketing, and customer service. It automatizes the analysis of a large volume of customer feedbacks, identifying areas for improvement or differentiation, and informing strategic decisions and market responses.
In this paper we present PyACSA, a new ACSA tool that is based on a feature of PyABSA, used for the more complex task of Aspect Category Opinion Sentiment (ACOS) [4]. We developed a new dataset for the ACSA task on the Beauty and Personal Care domain. We created a framework with various functions: from converting datasets across various formats (SemEval2014 1 , SemEval2015 2 , SemEval2016 3 , JSON), to manual and semi-automatic data annotation for this task, up to visualizing prediction data on graphs by calculating a sentiment index for each category of the domain. In addition, the PyACSA tool is experimented on several datasets available in the literature, demonstrating excellent results, comparing with ACSA-Gen [5], a state-of-the-art tool for ACSA. two elements(category and sentiment polarity). In the table below we report an example that shows the extraction of all elements from a sentence. As mentioned in the introduction, we developed the PyACSA tool by specializing a new feature of PyABSA [4], which was designed for the more complex task of Aspect Category Opinion Sentiment, to the Aspect-Category Sentiment Analysis. It is important to highlight the fact that the task is carried with the same format as SemEval 2016 task 5, subtask 2 [8], which means at text-level. At text-level, given a customer review about a target entity, the goal is to identify a set of {category, polarity} pairs that summarize the opinions expressed in the review. The polarities can be "positive", "negative", "neutral" (when a category is mentioned without any sentiment), or "conflict" (when the same category is expressed in a positive and a negative inside the same text, but neither of the two is dominant).
The PyACSA tool uses the T5 (Text-to-Text Transfer Transformer) [9] model which is based on a standard encoder-decoder Transformer [10] capturing long-range dependencies in text. The T5 model, if trained on a vast corpus of text data, converts various natural language processing (NLP) tasks into a text-to-text format, achieving state-of-the-art performance on many benchmarks covering summarization, question answering, text classification, and more. The pre-trained model used in this work is flan-t5-xl [11], an extension of the original T5 model with improved performance.
The PyABSA tool, which is implemented in PyTorch [12], is built using this pre-trained model and fine-tuned on the dataset using several instructions, one for each element. We developed PyACSA by modifying the PyABSA code so that only one extraction is performed containing both categories and sentiment polarities. This method takes an input text, categories, and polarities, and returns a string that combines the instructions with the provided input. This formatted string is then fed to the model for training or prediction. A specific module facilitates the creation of a dataset for this task by preparing the data in the required format, creating the training and test datasets, and reading JSON data from a file. With this approach, we aim to simplify the implementation of the ACSA framework that exploits several utilities to facilitate this task. The tool's integration with PyABSA provides a robust and flexible system for our specific needs, allowing us to streamline the data preparation and model training processes exploring the application of a text-to-text model in a task where it is not typically used.
In this section, we present the framework we developed for using and evaluating the PyACSA tool. We developed a web application in Python using the Flask package to promote the use of this task, facilitate user interaction with the model, and make the entire process more accessible and efficient. The framework contains key components such as data format transformation modules, for converting various input formats into the JSON format required by PyACSA), and utilities for manual and semiautomatic data annotation, allowing users to annotate data directly within the web app or use the model's predictions to assist in the annotation process. This dual approach enhances flexibility and efficiency, catering to different user needs and preferences, while leveraging the model for annotations improves accuracy and accelerates the workflow.
Furthermore, the framework contains a module that generates a bar chart along with a sentiment index (see Figure 3), which evaluates the categories of the reviews entered into the system. This feature provides valuable insights into the sentiment distribution across different aspects of our domain, aiding in the analysis and interpretation of the data. We release our code at https://github.com/lorisdiquilio/ACSA-Framework-using-T2T-model.
In the following paragraphs, we describe the components of the framework in detail, highlighting their functionalities and expected benefits.
The data converter module provides various converters designed to facilitate the transformation of data across various formats, ensuring compatibility and ease of use for subsequent tasks. The module is composed of several functions written in Python to transform data across tasks, sub-tasks, and formats. Among the main converters we developed, there are those for conversion of data from SemEval 14, 15 and 16 (XML) format to the JSON format of PyACSA, which have been used to evaluate PyACSA on the SemEval datasets.
This module allows the user to annotate the dataset in the JSON format used by our tool (shown in Listing 1). Data can be annotated both manually and in a semi-automatic way.
This module allows to load a training file, in JSON format, that contains annotated text. After loading the file, it is possible to annotate each review, by adding new annotations selecting the categories and polarities, or by deleting annotations, if needed. Additionally, the module allows to add categories that are not already known to the system, offering flexibility and adaptability to various annotation needs. When the annotation process is finished, it is possible to export the annotations in JSON format.
This module is designed to leverage the initial trained model for further improvements. Once a trained model is available, through this module we can use the model itself to suggest annotations on new data, making the dataset creation process faster and more efficient.
Similar to the manual annotation module, upon accessing this tab, the reviews and model predictions will be displayed, with the initial prediction next to the text. Multiple predictions for the same sentence will appear below the text. After reviewing the model's predictions, necessary corrections can be made and saved back into the original training file (JSON). This feature was used to annotate more data and improve model performance during experiments. In Figure 2 we see that the model predicts two correct tuples. It is easy to understand that in this way the annotation process becomes more efficient.
This module can be used to generate a report providing a final assessment of the reviews for a product.
We compute a sentiment index, which measures the overall quality of the product through all its reviews within the relevant domain. In this case, the domain represents a Skin Care, Body Care and Hair Care products. The sentiment index is computed based on the aggregated sentiment scores across different categories, offering a comprehensive evaluation of the product's performance. The sentiment index is computed as follows:
where Positive and Negative denote the number of positive and negative reviews, respectively. Note that the sentiment index value ranges from −1 to 1.
In Figure 3, we show the bar chart generated by the module, which displays the review polarities for each category of the selected product. This bar chart provides a visual representation of how users perceive different aspects of the product, with each bar indicating the level of positive or negative sentiment associated with a specific category. Figure 4 presents the overall sentiment index and the sentiment index calculated for each category. The overall sentiment index gives a comprehensive view of the general perception of the product, while the category-specific sentiment indices allow us to delve deeper into particular aspects. This dual representation helps in understanding not only the general acceptance of the product but also the specific areas where it excels or falls short.
From the results, we can evaluate the aspects that perform positively and negatively for this specific product. For instance, we can conclude that this product is generally well-received because it is effective, has a pleasant texture and smell, and is conveniently sized for travel. However, there are some drawbacks noted by users, such as the small quantity (for some), poor delivery and packaging, and the high cost of the product.
We have built an ACSA dataset based on Beauty and Personal Care reviews. The dataset was annotated manually and semi-automatically by one of the authors and is available at the repository.
In addition to the new dataset, we used some publicly available datasets4 : SemEval Laptop and Restaurant, and MAMS (Multi-Aspect Multi-Sentiment). We used the Data conversion module of our framework to convert datasets in XML format to the JSON format used by PyACSA. In Table 2 we show some statistics about the datasets.
In the experiment, the following settings are used: pre-trained flan-t5-xl with 3 billion parameters, learning rate of 5 × 10 −5 . Epochs and batch size are set to 10 and 6, respectively. A regularization parameter (𝐿 2 Regularization) 5 that helps prevent overfitting by reducing model weights during training is set to 0.01. The Warmup-Ratio (the ratio of total training steps used for a linear warmup from 0 to learning_rate) is set to 0.1. The PyACSA tool is compared with ACSA-Gen [5], one of the state-of-the-art tools in this field, as stated in the survey [6], which also leverages the power of a pre-trained generative model. Note that the model used by PyACSA is more powerful than that used by ACSA-Gen, which is a BART-large-MNLI 6 , the best BART model for the classification task. The latter was used with the template "The sentiment polarity of <given_category> is <polarity_type>" for each label. The configurations for BART-large-MNLI are: learning rate of 4 × 10 −5 , 15 epochs, and batch size of 16.
The results reported in the table are based on the most common metrics: Precision, Recall, and Micro-F1 Score. The results show that PyACSA performs better than ACSA-Gen in all the considered datasets, likely due to the power of the pre-trained model, as the BART model has a smaller size (approximately 406 million parameters) compared to the T5-XL (3 billion). We notice that PyACSA performs quite well in the ACSA task at the text level, except on the Laptop dataset, probably because of the high number of categories it contains.
We adapted an existing tool from the literature to transition from an Aspect-Category-Opinion-Sentiment (ACOS) task to an Aspect-Category Sentiment Analysis (ACSA) task. We developed a comprehensive web application featuring various sections, including the transformation of different data formats for various Aspect-Based Sentiment Analysis tasks, using the model for data annotation, and creating a sentiment index to evaluate what are the performances in terms of topics in the reviews analyzed, entered into the tool. Additionally, we released a new dataset, which will be made available in the literature for future research in this domain, and we evaluate the tool by comparing it with a state-of-the-art tool. Our primary goal is to promote the use of this task by simplifying the entire underlying process, thereby facilitating broader adoption and application in the research community. However, there are some limitations to our current approach. One significant challenge is that loading the model is resource-intensive and requires a dedicated space for that. Additionally, the interface is currently tailored to the specific domain mentioned in the paper (Beauty dataset), and future work should aim to expand its applicability across several domains to ensure broader usability. While the web application is continuously improving, future efforts will focus on implementing new features, including sections for model uploads. For future work, we want to evaluate the web app interface developed in this study with human participants to ensure that the interface is intuitive and user-friendly, highlighting possible areas of improvement.