Since the beginning of the vaccination campaign against Covid-19 in our country, resistance to vaccination has emerged on the part of a not negligible portion of the Italian population. Emotions (such as sadness, fear, etc.) and the polarity (positive / negative) of an opinion published on social media are essential for analyzing people's position towards a topic. For this reason, we applied two Natural Language Processing tools, FEEL-IT and SentIta, to a few thousands of social networks posts against the COVID-19 vaccine or specifically the booster shot. We find out some significant insights about the prevalent emotions among users and propose to combine the outputs of the tools in order to increase the classification performance of an opinion according to three possible sentiments (positive/neutral/negative).
clearly identifiable, albeit incorrect, reasons, as is the case for vaccine refusal, which is based on cognitive bias, disinformation and conspiracy theories. The phenomenon of hesitancy instead depends on more nuanced opinions and arguments that have a certain individual or group variability. Emotions (such as sadness, fear, etc.) and the polarity (positive / negative) of an opinion published on social media are essential for analyzing people’s position towards a topic. The purpose of the article is to understand the emotions of that part of the Italian population that has resistance to vaccinating against the SARS-CoV-2 virus.
Despite the huge interest of the Natural Language Processing community, the majority of
benchmark datasets have been proposed for English [
This paper applies the FEEL-IT [
The paper is organized as follows: Section 2 introduces related work, Section 3 explains the methodology adopted for data collection and manual annotation, Section 4 describes the application of FEEL-IT and SentIta to the data and Section 5 concludes the paper.
FEEL-IT [
SentIta[
Considering works on the Italian language and about vaccination campaigns, Tavoschi et al. [13] developed an opinion mining system to monitor the Twitter posts; the system was targeted on vaccine hesitancy (in general) in Italy in the period from September 2016 to August 2017 (before the Covid pandemic). The authors manually labelled 693 training tweets into three categories (against vaccination, in favor or neutral) and trained several machine learning classification models; the best performing (according to a 10-fold cross validation analysis) was based on a Support Vector Machine and reached an average accuracy of 64.8%. Furini [14] performed a word frequency-based analysis to categorize posts by several dimensions (afective, biological, medical and social) distinguishing ProVax and NoVax posts from 2015 to 2017 (before the availability of Covid vaccines); concerning the afective class, he considers four categories, namely anxiety, anger, danger and rage. Gori et al. [15] labelled about 7000 tweets in Italian as pro-vax, no-vax or neutral to the Covid vaccines; their work could be used to develop sentiment analysis tools targeted to study the sentiment about the Covid vaccines.
We retrieved opinions about COVID-19 anti-vaccination in general and against the vaccine third shot by monitoring Telegram, Facebook and Twitter social networks in diferent time intervals. The Telegram groups monitored were ‘Io Non Mi Vaccino Chat’ (I won’t get vaccinated), ‘Vittime vaccino Covid in Italia’ (Italian Covid vaccine victims), ‘Singles italiani NON vaccinati’ (Italian not vaccinated singles), ‘No Vaccini Covid sui Bambini’ (No Covid vaccine for kids), ‘COMBATTENTI NO BOOSTER - NO TERZA DOSE - NO VAC - NO GREEN PASS’ (Fighters against the booster shot and green pass), ‘Personale Scuola - No Green Pass - No Booster Vax’ (School personnel against booster shot and green pass) between August 20th 2021 to February 27th 2022, a particularly relevant period in the evolution of the pandemic situation8. Posts were downloaded in JSON format, converted and grouped in a single CSV file. These groups collect opinions from people fully against the COVID-19 vaccine or against/hesitant to the vaccine booster shots.
From the Facebook group ‘NON FARÒ LA TERZA DOSE’ (I will not get the booster shot) posts were monitored and downloaded from December 8th 2021 to February 2nd 2022, as this group
became active later than the Telegram groups. For the download the exportcomments.com tool was used, which allows one to export social media comments in CSV format.
Finally, we retrieved Twitter data using the Twitter API and restricting the search to the following Italian hashtags: #vaccino, #secondadose, #terzadose, #booster. Every day, twice a day, 100 tweets were downloaded between January 26th 2022 and March 7th 2022.
All texts were preprocessed by removing duplicates and retweets; moreover, Telegram and Facebook comments were checked to find if the initial substrings ‘vac’, ‘terz’, ‘dos’ were present, which are the roots of the Italian words vaccino, vaccinazione, terza, dose (terza dose’ is the booster shot in English). This was done in order to check that they were indeed related to the vaccine, while for Twitter we trusted the presence of the hashtag.
Eventually, the number of collected Telegram posts was 4077, the number of Facebook posts was 84 and the number of tweets was 3767, for a total of 7928 texts so distributed in the groups/hashtags: #vaccino (2056), ‘Io Non Mi Vaccino Chat’ (1852), #terzadose (993), ‘Vittime vaccino Covid in Italia’ (780), ‘Singles italiani NON vaccinati’ (738), #booster (660), ‘COMBATTENTI NO BOOSTER - NO TERZA DOSE - NO VAC - NO GREEN PASS’ (449), ‘Personale Scuola - No Green Pass - No Booster Vax’ (211), ‘NON FARÒ LA TERZA DOSE’ (84), #secondadose (58), ‘No Vaccini Covid sui Bambini’ (47). Each of them is associated with its timestamp. Overall this search allowed us to collect opinions spanning a lot of weeks, from August 20th 2021 to the beginning of March 2022, and focusing both on the sentiment on the vaccine in general and on the booster shot. From the third week of August to the end of October 2021 we have between 30 and 100 opinions per week. From the second week of November 2021 more texts are available (from diferent social networks), ranging from 150 to 250 per week, with peaks between the third week of January and the third week of February 2022 (800-1100 posts per week). This is highlighted in Figure 1.
A subset of the posts, 1350, was manually labelled by all authors, who are native Italian speakers. Labelling was performed twice: 1) the first time we removed comments that did not contain any emotion, ending up with 884 posts: each of them was assigned one of the four emotions handled by FEEL-IT (joy, sadness, fear, anger); 2) the second time we kept all 1350 posts and each of them was assigned one out of three classes (positive, negative, neutral), meaning that a post has either a predominant positive sentiment, or a predominant negative sentiment, or does not express any emotion according to the reader, in order to evaluate SentIta performance (see Section 4.2). Table 1 shows the label distribution after the manual annotation for task 1). Table 2 shows the result of the manual annotation for task 2).
In our experimental evaluation we (i) perform emotion recognition with FEEL-IT and sentiment classification with SentIta, (ii) we produce some statistics from the results, (iii) we compare the automatic labelling with our manual labelling in order to test the performance of these tools. Both FEEL-IT9 and SentIta10 are open-source Python libraries. The SentIta model is written in Python 3.6 and is implemented in Keras 2.2.4 with Tensorflow 1.11 backend. The Sentita package contains the model and the necessary pre-processing functions.
We first experimented with emotion recognition with the FEEL-IT library applied over the dataset of 7928 posts. Results are shown in Table 3.
On the subset of opinions manually labelled we tested the performance of FEEL-IT using the open source ML library scikit-learn11. We considered the 884 posts expressing an emotion and computed the confusion matrix, accuracy, precision, recall and F1-score for multi class
classification, with the number of classes equal to 4. We obtained an accuracy of 0.542, other results are shown in Table 4.
Results show that joy is the most mistaken emotion, and FEEL-IT performance varies
depending on the class. It is quite precise in recognizing true joy and anger emotions in texts,
but not the other two. It is able to find correctly positive comments based on anger (recall
83.3%), but not based on the other emotions. Anger is the most recognized emotion in the
data both by humans and the tool. The varying results should take into account that the
FEEL-IT corpus was from a very diferent context than the COVID-19 texts, and [
Secondly, we experimented with sentiment analysis with the SentIta library over the dataset of
7928 posts. As SentIta provides two scores, each ranging in the [
Most of the posts had very low scores in both dimensions: the highest density is around the origin, and the median was 0.09 for the positive score and 0.16 for the negative. This shows that in the considered dataset many posts were classified as neutral. The average is 0.15 and 0.26, respectively for positive and negative, with higher values for negative, as many posts were anti-vaccination.
In order to evaluate the performance of SentIta on the subset of opinions manually labelled with 3 classes (1350 posts), we decided to discretize the two scores applied by SentIta to each text into 3 distinct classes: positive, neutral and negative. The conversion of the scores into classes was based on the fact that we know, as said above, that a low value of both scores indicates a neutral sentiment, a high value of the positive score and a low value of the negative one means a positive polarity, finally a high value of the negative score and a low value of the positive one indicates a negative polarity in an opinion. This excludes the case in which the scores are similar and high, but here we decided to consider a neutral sentiment again, since the positive and
Examples of the output of SentIta for 3 opinions from our dataset, representative of positive, negative Text Ragazzi io vi capisco! Vi voglio bene! Sono con voi! Io niente dosi zero! Vi appoggio e vi capisco!!! Spero veramente vi unirete a me.
Guys I understand you! I love you! I’m with you! Zero shots! I support you and I understand you !!! I truly hope you will join me.
Non lo voglio fare, nemmeno una di dose, ma mi stanno costringendo.
I don’t want to do it, not even a shot, but they are forcing me. "Questo studio ha mostrato che l’impatto della vaccinazione sulla trasmissione nella comunità delle varianti circolanti di SARS-CoV-2 non pare essere significativamente diverso dall’impatto tra le persone non vaccinate." "This study showed that the impact of vaccination on community transmission of circulating variants of SARS-CoV-2 does not appear to be significantly diferent from the impact among unvaccinated people." negative score (oneg)
positive score (opos) (1 + 0.05) * positive_score + 0.1 (1)
After this conversion we could compute accuracy, precision (P), recall (R) and F1-score for multi class classification, with the number of classes equal to 3.
We obtained an accuracy of 0.487, other results are shown in Table 6.
The best sentiment recognition is done about neutral opinions, while positive opinions are often misclassified. In order to improve SentIta and FEEL-IT performance on the manually labelled dataset, we tried to combine the two as described in the next section.
To compare the results of the application of the two systems on the same data, we performed several tests over the complete dataset (7928 posts).
Firstly, we colored the scatter plot according to the emotion recognition by FEEL-IT; the results are in Figure 4.
This plot confirms that anger, fear and sadness (yellow, red and green points resp.) characterize opinions to which SentIta assigns lower values for the positive score and higher values for the negative one, concentrating near the Y-axis; joy (blue) receives higher values for positive score. Red and blue points (fear and joy) concentrating near zero should represent neutral opinions which received two low scores by SentIta but could not be labelled with a proper emotion by FEEL-IT (as a neutral opinion does not contain an emotion).
Secondly, we interpreted the positive and negative scores provided by SentIta as the coordinates of a point in the plane (in a square of size 1 × 1). Note that all the points stand in a circle of radius 1 from the origin, so it might be interesting to identify each point by its polar coordinates (, ); in such a representation the angle ∈ ︀[ 0, 2 ]︀ is a measure of the negativity of the post, while the radius can represent the strength with which the argument is pushed forward. Figure 5 is a bubble plot that shows in a synoptic way the evolution of the sentiments for each emotion during time, and can be seen as a comparison of the outputs of the two considered systems. For each week in the considered time range (plotted in the -axis) and for each emotion (each emotion is plotted with a diferent color) there is a bubble having the -coordinate of the center equal to the average angle that was obtained in that week and for that emotion; the radius of the bubble is proportional to the average strength of the posts. Posts having strength less than 0.1 were removed from the computation of the average angle. The graph shows that joy, the only positive emotion provided by FEEL-IT, is associated to lower angles in SentIta, corresponding to points having a larger positive score and a smaller negative score; the three negative emotions (fear, anger and sadness), are associated to angles closer to 2 , i.e. closer to the -axis or having a negative score larger than the positive score in SentIta. The plot also highlights that negative emotions (fear, anger and sadness) prevail with respect to joy, confirming that people posting in the monitored groups are against COVID-19 vaccine or the booster shot.
Thirdly, we tried to combine the outputs provided by the two systems by formulating the rule given in Equation 2: positive and negative is the conversion of SentIta’s scores into discrete classes as described in Section 4.2 and joyFEEL-IT indicates that FEEL-IT predicted ‘joy’ as an emotion. These rules maintain the principle of discretization in 3 classes, positive, neutral, and negative. , p , 21 , 1, 21
Mon, Dec 20, 21
Tue, Feb 8, 22
Wed, Mar 30, 22
Time (week)
The idea is to classify as neutral those posts in which the two systems do not agree, i.e. when ‘joy’ is identified as an emotion (by FEEL-IT) but a negative polarity is found by SentIta, or vice versa when a positive polarity is identified together with one of the “negative emotions” (sadness, anger, fear).
After the conversion of the scores we computed accuracy, precision, recall and F1-score for multi class classification, with the number of classes equal to 3. We obtained an accuracy of 0.544, other results are shown in Table 7.
With respect to the performance obtained at the end of Section 4.2, precision increases significantly; a minor improvement can be seen also in recall. About 50% of the negatives are classified as neutral, and a lot of positives are classified as neutral (70.8%). In general, the combined system misclassifies with the “adjacent class” instead of the opposite class. This third experiment demonstrates that a simple yet efective combination of the outputs of the two NLP systems allows to increase the classification performance of SentIta alone, and to improve the understanding of its results by the user, by means of the discretization of the scores (real values) in 3 more intuitive classes by means of a simple conversion.
We applied two pre-trained neural network-based models for natural language processing, FEEL-IT and SentIta, to COVID-19 anti-vaccination posts downloaded from the major social networks between the end of 2021 and the beginning of 2022. We evaluated the performance of the two models separately on a subset of manually labelled data and then of the two models jointly by combining their outputs. We extracted both an insight of the most prevalent emotions during several months of the pandemic and proposed a method for increasing the performance of the two systems alone through a combination of their output.
In the future this work could benefit both from improvements to the experimental activity and from the application of new techniques, by: • deepening the analysis of the training sets of the two systems used, making it clear the diferences with the texts analyzed for this work. For example, by computing the diferent average length of sentences, the percentage of lexical overlap, etc. • when considering the combination of the systems, analyzing in what percentage the two systems are in agreement on the test set (both in cases of correct and incorrect classification) and how many times they are not (comparing the accuracy of the two systems in cases of disagreement); • applying a diferent interpretation to SentIta scores when both similar and high: it would be interesting to see in how many of those cases the systems guess or fail with respect to the manual labels; • applying Deep Learning techniques to the complete dataset, for instance for identifying classes of opinions by unsupervised learning in order to avoid manual labelling.
This work was partially supported by the project “NO Esitazione (’NOE’). Per una comunicazione eficace della vaccinazione anti COVID-19” funded by Fondo per l’Incentivazione alla Ricerca 2021 (FIR) of Ferrara University. Special thanks to Dr. Davide Civolani for his help in manually labelling the dataset. CEUR-WS.org, 2016. URL: http://ceur-ws.org/Vol-1749/paper_026.pdf. [12] P. Basile, V. Basile, D. Croce, M. Polignano, Overview of the EVALITA 2018 aspectbased sentiment analysis task (ABSITA), in: T. Caselli, N. Novielli, V. Patti, P. Rosso (Eds.), Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Turin, Italy, December 12-13, 2018, volume 2263 of CEUR Workshop Proceedings, CEUR-WS.org, 2018. URL: http: //ceur-ws.org/Vol-2263/paper003.pdf. [13] L. Tavoschi, F. Quattrone, E. D’Andrea, P. Ducange, M. Vabanesi, F. Marcelloni, P. L. Lopalco, Twitter as a sentinel tool to monitor public opinion on vaccination: an opinion mining analysis from september 2016 to august 2017 in Italy, Human Vaccines & Immunotherapeutics 16 (2020) 1062–1069. URL: h t tp s :/ / do i . or g / 1 0 . 1 0 8 0 / 2 1 6 4 5 5 1 5 . 2 0 2 0 . 1 7 1 4 3 1 1. doi:1 0 . 1 0 8 0 / 2 1 6 4 5 5 1 5 . 2 0 2 0 . 1 7 1 4 3 1 1. arXiv:https://doi.org/10.1080/21645515.2020.1714311, pMID: 32118519. [14] M. Furini, Identifying the features of ProVax and NoVax groups from social media conversations, Comput. Hum. Behav. 120 (2021) 106751. URL: https://doi.org/10.1016/j.chb.2021.1 06751. doi:10.1016/j.chb.2021.106751. [15] D. Gori, F. Durazzi, M. Montalti, Z. Di Valerio, C. Reno, M. P. Fantini, D. Remondini, Mis-tweeting communication: a vaccine hesitancy analysis among twitter users in italy, Acta Bio Medica: Atenei Parmensis 92 (2021).