English. In this paper, we describe our
approach to the sentiment classification
challenge on Italian reviews in the healthcare
domain. Firstly, we followed the work of
Bacco et al. (2020) from which we
obtained the dataset. Then, we generated our
model called KERMITHC based on
KERMIT
Italiano. In questo lavoro, presentiamo il
nostro approccio al task di sentiment
analysis per le recensioni italiane in ambito
sanitario. Abbiamo seguito il lavoro di
Bacco et al. (2020) da cui abbiamo
ottenuto il dataset. Successivamente,
abbiamo usato KERMITHC basato su
KERMIT
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
People are practically reviewing anything in online sites and understanding the polarization of a comment through automatic sentiment classiifer is a tantalizing challenge. In recent years, the number of virtual reviewers has drastically increased and there are many products and services, which can be reviewed. Each person, before buying a product or a service, searches into reviews from people who have already had experienced the product or the service. Review portals are usually linked to the leisure or business activities such as the world of tourism, e-commerce or movies. However, there are topics where these reviews and the associated automatic computed sentiment may induce to select wrong services, which may dramatically affect personal life.
When dealing with health-related services, the
effect of positive or negative reviews on hospitals
and doctors can have a potential catastrophic
impact on the health of who is using this piece of
information. QSalute 1 is one of the most important
Italian portals of reviews about hospitals, nursing
homes and doctors. It is very important for
patients to seek the best hospital for their condition
based on the past experience of other patients.
Reviews in the world of health benefit both patients
and hospitals because they are a means to discover
problems and solve them
Automatic sentiment analyzer have then a big
responsibility in the context of health-related
services. In these sensitive areas, it is important to
design AI systems whose decisions are
transparent
model or a prediction, they will not use it
In this article, we investigate a model that can
mitigate the responsibility of sentiment
analyzers for health-related services. The model we are
using exploits syntactic information within
neural networks to provide a clear visualisation of
the internal decision mechanism of the model that
produced the decision. We propose KERMITHC
(KERMIT for HealthCare) based on KERMIT
In the rest of paper, Section 2 gives details about the dataset and methods, while Section 3 and 4 describe the experiments, the results obtained and their discussion. Finally, in Section 5 we present the final conclusions and future goals. 2
To explore our hunch that syntactic interpretation may help in Healthcare reviews recognition, we leverage: (1) a Healthcare training corpus (Sec. 2.1); (2) a KERMITHC , which is based on syntactic interpretation and it can explain its decisions; and finally, (3) some challenges solved due to KERMITHC (Sec. 2.2). 2.1
In order to investigate reviews in healthcare area, we selected the QSalute portal, one of the most important health websites in Italy. This portal can be defined as the TripAdvisor of hospital facilities, indeed it talks about: Expertise, Assistance, Cleaning and Services. In addition to the reviews, there are some associated metadata such as: user id, hospital name, review title and patient pathology. To ensure privacy we do not consider sensitive data such as user id and hospital name.
We used a free available scraper on GitHub 2 to download the dataset. Then, to model this data to a sentiment analysis task, we followed the indications provided by Bacco et al.(2020) - in detail, a review is: (1) negative if the average of its scores 2The scraper is available at https://github.com/l bacco/Italian-Healthcare-Reviews-4-Senti ment-Analysis is less than or equal to 2, (2) positive if the average of its scores is greater than or equal to 4 (3) neutral otherwise.
The resulting dataset is composed of 47,224 reviews consisting of: 40,641 reviews in the positive class, 3,898 in the neutral class and 2,685 in the negative class.
In this work, we solely consider positive and negative classes, so our final dataset is composed of 43,326 reviews. The dataset is heavily skewed (93,80% positive class - 6,20% negative class) favoring reviews labeled as positive. 2.2
KERMITHC (KERMIT for HealthCare) architecture is composed of 3 major parts: (1) a KERMIT model described in Zanzotto et al. (2020), (2) a Transformers model and (3) a decoder layer that combines the results obtained from the previous two sub-parts. In figure Fig.1 we show a graphical representation of the architecture of KERMITHC , pointing the parts that compose it.
The architecture of KERMITHC makes it a
particular model, because it combines the
syntax offered by KERMIT with the versatility of a
Transformer-model. We use KERMIT because it
allows the encoding of universal syntactic
interpretations in a neural network architecture.
KERMIT component is itself composed of two parts:
KERMIT encoder, which converts parse tree T
into embedding vectors and a multi-layer
perceptron that exploits these embedding vectors. The
second sub-part of our architecture is composed
of a Bidirectional Encoder Representations from
Transformers, - as known as BERT - to classify the
sentiment of the reviews. BERT is a pre-trained
language model developed by Devlin et al. (2019)
at Google AI Language. In particular, since the
task concerns sentences in the Italian language, we
have used a special BERT version pretrained on
that language called AlBERTo
We used KERMITHC architecture to examine if
it is possible to answer the research questions
showed in KERMIT
To provide a comprehensive answer to these questions, we tested the architecture in a completely universal setting where both KERMIT and AlBERTo are trained only in the last decision layer.
The rest of the Section describes the experimental set-up, the quantitative experimental results and discusses how we can use the KERMITviz to explain decisions of neural network inferences over examples. 3.1
This section describes the general experimental set-up of our experiments and the specific configurations adopted.
The parameters used for the KERMIT encoder are those proposed in Zanzotto et al., (2020) paper. The constituency parse trees used for KERMIT sub-part are obtained using our freely available script on GitHub3.
We tested several different BERT version
pretrained on Italian language in order to get the best
model for our task. In particular, we tested the
following transformers: (1) UmBERTo
Since our experiments are text classification
task, the decoder layer of our KERMITHC
architecture is a fully connected layer with the
softmax activation function applied to the
concatenation of the KERMIT sub-part output and the final
[CLS] token representation of the selected
transformer model. Finally, the optimizer used to train
the whole architecture is AdamW
3The code is available at https://github.com/L eonardRanaldi/Constituency-Parser-Italia n
4The code is available at https://github.com/A RT-Group-it/KERMIT-4-Sentiment-Analysison-Italian-Reviews-in-Healthcare
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
AlBERTo
Accuracy 0.71 (± 0.14) 0.66 (± 0.27) 0.78 (± 0.13) 0.74 (± 0.28) 0.87 (± 0.05)† 0.94 (± 0.01)† 0.93 (± 0.03)† 0.96 (± 0.01)† 0.81 (± 0.16) 0.70 (± 0.29) 0.79 (± 0.08) 0.87 (± 0.08) 0.58 (± 0.23) 0.68 (± 0.20) 0.68 (± 0.19) 0.57 (± 0.23) 0.64 (± 0.11) 0.63 (± 0.26) 0.79 (± 0.09)† 0.88 (± 0.06)† 0.75 (± 0.09) 0.80 (± 0.19) 0.70 (± 0.24) 0.72 (± 0.26) 0.91 (± 0.06) 0.89 (± 0.21) 0.56 (± 0.30) 0.41 (± 0.25) 0.71 (± 0.20) 0.73 (± 0.29) 0.82 (± 0.08) 0.85 (± 0.14) 0.84 (± 0.14) 0.80 (± 0.18)
As in the work proposed by Bacco et al.(2020), we chose to divide the dataset by “Site” and evaluate the models using accuracy and F1-score metrics. Despite this division, the dataset is still very unbalanced favoring the class 1 (positive reviews).
We reports results in terms of the accuracy, Macro
F1 and Weighed F1. Observing Table 2, we can
see that the performance obtained by KERMITHC
always exceeds the best configuration of BERT:
AlBERTo. Hence, trained on the Healthcare
review dataset
(a) S: Uno staff di grandissima competenza e professionalita`! (b) S:Pessima assistenza e servizi assenti tranne il primario di reparto di neurochirurgia eccellente professionista
Using the KERMIT-viz visualiser, we analysed how important the contribution of symbolic knowledge provided by KERMIT can be. In many cases it makes all the difference. Looking at the Figure 2, these are two sentences with a positive target. The first sentence (shown in Fig. 2a) is clearly positive while the sentence shown in the Fig. 2b could be ambiguous as the patient makes bad remarks about the service but praises the head of the department. We can observe how some words have been colored in red (therefore they have received a greater weight during the classification phase) emphasizing the positive aspects of the sentence and causing it to be labeled as “positive review”. In this way the explainability is guaranteed and in very delicate topics - like sentiment in health reviews - we can have more “trust” on sentiment analysers. 5
In this article, we investigated a model that
can mitigate the responsibility of sentiment
analyzers for health-related services. Our model
KERMITHC exploits syntactic information within
neural networks to provide a clear visualisation of
its internal decision mechanism. KERMITHC is
based on KERMIT
We studied several versions of pre-trained
BERT models on the Italian language and found
out that AlBERTo is, among them, the best model
for this task. However, KERMITHC , which is
composed of KERMIT+AlBERTo, outperforms
better than AlBERTo model alone. Additionally,
via KERMIT-viz, we visualized the reasons why
KERMITHC classifies the dataset. We observed
how KERMITHC captures relevant syntactic
information by catching the keywords in each
sentence giving them more weight in the decision
phase, mitigating and capturing possible errors of
the sentiment analysers. Our future goal is to be
able to have full control of the sentiment analysers
by injecting human rules