English. In identifying the different emotions present in a review, it is necessary to distinguish the single entities present and the specific semantic relations. The number of reviews needed to have a complete dataset for every single possible option is not predictable. The approach described starts from the possibility to study the aspect and later the polarity and to create an ensemble of the two models to provide a better understanding of the dataset. Italiano. Nell'identificazione delle diverse emozioni presenti in una recensione è necessario distinguere le singole entità presenti e le singole relazioni semantiche. Il numero di recensioni necessarie per avere un dataset completo per ogni singola opzione possibile non è predicibile.
L'approccio descritto parte dalla possibilità di creare due modelli diversi, uno per la parte di categorizzazione, e l'altro per la parte di polarità. E di unire i due modelli per ottenere una maggiore comprensione del dataset.
Automating the correct recognition of the various
problems can lead to the timely addressing of the
same to the persons appointed to solve them.
The research was carried out with the dataset
provided within the task called ABSITA,
Aspectbased Sentiment Analysis at EVALITA 20181
The dataset is a selection of hotel reviews taken in Italian from the portal Booking.com. 2
Each review has been cleaned up by special characters, lemmatized and brought to lowercase with the SpaCy2 framework.
Generic Italian texts have been used, instead of reviews in the accommodation context to be sure that the model will be suitable for more business models, to generate vectors in fastText3. The best one has a dimension of 200, with character ngrams of length 5, a window of size 5 and 10 negatives.
The system is the ensemble of two different
models to improve the ability to discover hidden
properties
The first model is a bi-directional Long ShortTerm Memory (BI-LSTM).
This model is used for the discernment of the ASPECT. Layer (type) Output Shape Param # =================================== e (Embedding) (None, 100, 200) 1420400 _______________________________________ b (Bidirection) (None, 512) 935936 _______________________________________ d (Dense) (None, 7) 3591 =================================== A second BI-LSTM model is used for the discernment of POLARITY. _______________________________________ Layer (type) Output Shape Param # =================================== e (Embedding) (None, 100, 200) 1420400 _______________________________________ b (Bidirection) (None, 512) 935936 _______________________________________ d (Dense) (None, 14) 7182 =================================== A dropout and a recurrent_dropout of 0.1. The optimizer for both is the RMSProp. The loaded embedding is trainable.
Both the systems use Keras4 to create the RNN models.
The models were trained and tested with a 5-fold cross-validation with a ratio of 80% training and 20% testing. The best model was automatically saved at each iteration.
A threshold of 0.5 was used on the first model to activate the result of the last layer. In the second model, the threshold was of 0.43.
Aspect Category Detection (ACD)
The results show that the models are useful to understand the category of a review better than its polarity.
After that we ensemble the two models
The threshold of activation was a range between 0.45 and 0.55.
A third model, a LightGBM5
The number of leaves is 250, the learner set as ‘Feature’, and a the learning rate at 0.04. The result of the union between the three models could not be submitted to the final evaluation, due to the limit of 2 possible submissions, but reported results higher than 83% in the tests carried out after the release of the complete dataset for ASPECT and 75% for POLARITY.
Also, the inference is faster than the RNN models. https://keras.io https://github.com/Microsoft/LightGBM
In the evaluation phase, we can see how the results have given reason to the ensemble of the two results.
It is clear that the ACP task (table 4) is the beneficiary of this process, instead of the ACD one (table 3) that lost more than one point.
The study of the dataset is influenced by the little extension of the training dataset and by the specificity of some terms that could refer to different categories such as the comfort of the room and the quality/price ratio.
Various types of data preparation have also been used, including the preservation of special characters, the shape of words (to better identify cities or places written in capital letters), and some SMOTE functions to increase the number of entries but with poor results and noticeable overfitting. 4
Creating an ensemble of models to bring out various properties of a review gave better results than using a single model in the polarity identification.
The terms used in the review are sometimes misleading and can be used both positively or negatively, and to identify different categories of the hotel.
In the near future, we are ready to create a system to split the text of the review to categorize only a single sentence, or less a single subject or object. In this way, we will be ready to evaluate also the polarity of the single object or subject, and only the terms single related to it to improve the result of the ACP task.
The performance of the system will also be evaluated by replacing all the possible entities with variables known as: l l l l l
City Museum Panoramic Point Railway station
Street and with a pre-category knew a priori as Breakfast for words like Coffee, Cornetto, and Jam. The expected result is to reduce the variance of the dataset, to improve the ACD result, and to be able to use the system in production.
Finally, we will evaluate the speed and effectiveness of a CNN model in which the tasks, ASPECT, and POLARITY, can be studied separately and then merged.
Bennici, M. and Seijas Portocarrero, X. (2018). The validity of dictionaries over the time in Emoji prediction. In Tommaso Caselli, Nicole Novielli, Viviana Patti, and Paolo Rosso, editors, Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA’18), Turin, Italy. CEUR.org.