In this study, we developed an Aspect-Category Sentiment Analysis (ACSA) framework encompassing data conversion, semi-automatic annotation methods using predictions, and the creation of a prediction-based report. We aimed to adapt an Aspect-Category-Opinion Sentiment (ACOS) tool from the literature to the Aspect-Category Sentiment Analysis (ACSA) task. We developed a web application where the dataset released in this paper (beauty dataset) can be annotated manually or semi-automatically and incorporated into the training data to enhance the model. Additionally, we also evaluated our framework using various datasets available in the literature, comparing with a tool that follows a similar approach.
Aspect-Based Sentiment Analysis (ABSA) tasks. They difer in that ACOS extracts four elements from
the text (aspect terms, category, opinion terms, and sentiment polarity) [
ceur-ws.org two elements(category and sentiment polarity). In the table below we report an example that shows the extraction of all elements from a sentence.
‘Though the service might be a little slow, the waitresses are very friendly.’ ‘service’ ‘waitresses’ ‘service’ ‘staf’ ‘a little slow’ ‘very friendly’ ‘negative’ ‘positive’
As mentioned in the introduction, we developed the PyACSA tool by specializing a new feature of
PyABSA [
The PyACSA tool uses the T5 (Text-to-Text Transfer Transformer) [
The PyABSA tool, which is implemented in PyTorch [
In this section, we present the framework we developed for using and evaluating the PyACSA tool. We developed a web application in Python using the Flask package to promote the use of this task, facilitate user interaction with the model, and make the entire process more accessible and eficient. The framework contains key components such as data format transformation modules, for converting various input formats into the JSON format required by PyACSA), and utilities for manual and semiautomatic data annotation, allowing users to annotate data directly within the web app or use the model’s predictions to assist in the annotation process. This dual approach enhances flexibility and eficiency, catering to diferent user needs and preferences, while leveraging the model for annotations improves accuracy and accelerates the workflow.
Furthermore, the framework contains a module that generates a bar chart along with a sentiment index (see Figure 3), which evaluates the categories of the reviews entered into the system. This feature provides valuable insights into the sentiment distribution across diferent aspects of our domain, aiding in the analysis and interpretation of the data. We release our code at https://github.com/lorisdiquilio/ACSA-Framework-using-T2T-model.
In the following paragraphs, we describe the components of the framework in detail, highlighting their functionalities and expected benefits.
The data converter module provides various converters designed to facilitate the transformation of data across various formats, ensuring compatibility and ease of use for subsequent tasks. The module is composed of several functions written in Python to transform data across tasks, sub-tasks, and formats. Among the main converters we developed, there are those for conversion of data from SemEval 14, 15 and 16 (XML) format to the JSON format of PyACSA, which have been used to evaluate PyACSA on the SemEval datasets.
This module allows the user to annotate the dataset in the JSON format used by our tool (shown in Listing 1). Data can be annotated both manually and in a semi-automatic way.
This module allows to load a training file, in JSON format, that contains annotated text. After loading the file, it is possible to annotate each review, by adding new annotations selecting the categories and polarities, or by deleting annotations, if needed. Additionally, the module allows to add categories that are not already known to the system, ofering flexibility and adaptability to various annotation needs.
This module is designed to leverage the initial trained model for further improvements. Once a trained model is available, through this module we can use the model itself to suggest annotations on new data, making the dataset creation process faster and more eficient.
Similar to the manual annotation module, upon accessing this tab, the reviews and model predictions will be displayed, with the initial prediction next to the text. Multiple predictions for the same sentence will appear below the text. After reviewing the model’s predictions, necessary corrections can be made and saved back into the original training file (JSON). This feature was used to annotate more data and improve model performance during experiments.
This module can be used to generate a report providing a final assessment of the reviews for a product.
We compute a sentiment index, which measures the overall quality of the product through all its reviews within the relevant domain. In this case, the domain represents a Skin Care, Body Care and Hair Care products. The sentiment index is computed based on the aggregated sentiment scores across diferent categories, ofering a comprehensive evaluation of the product’s performance. The sentiment index is computed as follows:
Sentiment Index =
Positive − Negative Positive + Negative (1) where Positive and Negative denote the number of positive and negative reviews, respectively. Note that the sentiment index value ranges from −1 to 1.
In Figure 3, we show the bar chart generated by the module, which displays the review polarities for each category of the selected product. This bar chart provides a visual representation of how users perceive diferent aspects of the product, with each bar indicating the level of positive or negative sentiment associated with a specific category. Figure 4 presents the overall sentiment index and the sentiment index calculated for each category. The overall sentiment index gives a comprehensive view of the general perception of the product, while the category-specific sentiment indices allow us to delve deeper into particular aspects. This dual representation helps in understanding not only the general acceptance of the product but also the specific areas where it excels or falls short.
From the results, we can evaluate the aspects that perform positively and negatively for this specific product. For instance, we can conclude that this product is generally well-received because it is efective, has a pleasant texture and smell, and is conveniently sized for travel. However, there are some drawbacks noted by users, such as the small quantity (for some), poor delivery and packaging, and the high cost of the product.
We have built an ACSA dataset based on Beauty and Personal Care reviews. The dataset was annotated manually and semi-automatically by one of the authors and is available at the repository.
In addition to the new dataset, we used some publicly available datasets4: SemEval Laptop and Restaurant, and MAMS (Multi-Aspect Multi-Sentiment). We used the Data conversion module of our framework to convert datasets in XML format to the JSON format used by PyACSA. In Table 2 we show some statistics about the datasets.
MAMS Rest 16
Laptop # Train # Test # Categories # Positive annotations # Negative annotations # Neutral annotations # Conflict annotations
In the experiment, the following settings are used: pre-trained flan-t5-xl with 3 billion parameters, learning rate of 5 × 10−5. Epochs and batch size are set to 10 and 6, respectively. A regularization parameter ( 2 Regularization)5 that helps prevent overfitting by reducing model weights during training is set to 0.01. The Warmup-Ratio (the ratio of total training steps used for a linear warmup from 0 to learning_rate) is set to 0.1.
The PyACSA tool is compared with ACSA-Gen [
The results reported in the table are based on the most common metrics: Precision, Recall, and Micro-F1 Score.
Tool
Metrics ACSA-Gen
P R F1 P R F1
The results show that PyACSA performs better than ACSA-Gen in all the considered datasets, likely due to the power of the pre-trained model, as the BART model has a smaller size (approximately 406 million parameters) compared to the T5-XL (3 billion). We notice that PyACSA performs quite well in the ACSA task at the text level, except on the Laptop dataset, probably because of the high number of categories it contains.
We adapted an existing tool from the literature to transition from an Aspect-Category-OpinionSentiment (ACOS) task to an Aspect-Category Sentiment Analysis (ACSA) task. We developed a comprehensive web application featuring various sections, including the transformation of diferent data formats for various Aspect-Based Sentiment Analysis tasks, using the model for data annotation, and creating a sentiment index to evaluate what are the performances in terms of topics in the reviews analyzed, entered into the tool. Additionally, we released a new dataset, which will be made available in the literature for future research in this domain, and we evaluate the tool by comparing it with a state-of-the-art tool. Our primary goal is to promote the use of this task by simplifying the entire underlying process, thereby facilitating broader adoption and application in the research community. However, there are some limitations to our current approach. One significant challenge is that loading the model is resource-intensive and requires a dedicated space for that. Additionally, the interface is currently tailored to the specific domain mentioned in the paper (Beauty dataset), and future work should aim to expand its applicability across several domains to ensure broader usability. While the web application is continuously improving, future eforts will focus on implementing new features, including sections for model uploads. For future work, we want to evaluate the web app interface developed in this study with human participants to ensure that the interface is intuitive and user-friendly, highlighting possible areas of improvement.