Understanding public opinion on governmental decisions has always been crucial for assessing policies' effectiveness, especially when facing exceptional events requiring prompt decisions. Computational linguistics and social scientists have long observed modern social media platforms as they are a perfect stage for spreading opinions swiftly and transparently. Natural Language Processing (NLP) techniques have been widely used for analyzing public discussion [e.g., 1,2,3].
The COVID-19 pandemic, arguably the most prominent of such exceptional events, prompted the Italian government-and other European governments-to release multiple financial measures to cushion the impact on the population. These so-called "bonuses," issued pro bono, i.e., with no interest payments from recipients, aimed at increasing liquidity and reducing tax burdens. However, despite reaching varied recipients, comprehending the measures' reception and evaluating their effectiveness still needs to be explored.
To address this gap, we collect and release MoniCA, a new social media dataset for MONItoring Coverage and Attitudes of Italian measures to COVID-19. Mon-iCA comprises approximately 10,000 posts spanning ten months collected on X.com. These posts pertain to the Italian public's discussions on diverse financial measures introduced during the pandemic. Building on an extensive body of literature that examines public sentiment during the pandemic [e.g., 4,5,6,7,8], this work offers new insights into the limited research specifically addressing Italy. 1 This paper details the dataset's collection and release. It introduces the annotations we compiled for each post, including sentiment, emotion, irony, and discussion topics. Then, we conducted an analysis using traditional models and transformer-based language models to predict these aspects from textual data, demonstrating the dataset's potential usability. Moreover, using state-ofthe-art interpretability tools, we explained the models' decision processes. We found that explanations are faithful and plausible to human judgments.
MoniCA will allow a retrospective examination of the efficacy -and inefficacy -of governmental measures implemented in Italy during the COVID-19 pandemic, as perceived by the population. By doing so, we seek to provide insights that can inform policymakers about the strengths and weaknesses of such financial measures, ensuring better preparedness and response strategies for any future crises. the coverage and people's attitude towards Italy's government's financial aid to combat the COVID-19 crisis. We collect annotations of several aspects to allow for a finer-grained analysis. We used state-of-the-art NLP and interpretability tools and reported key insights on public sentiment.
To build a comprehensive resource, reflecting multiple facets of the phenomenon and usable for future policymakers, we prioritized 1) topic and time coverage in our collection process ( §2.1), and 2) relevance refinement and data annotation to enrich the initial pool with additional metadata ( §2.2).
We collected approximately 200,000 posts from X in late 2022. We then filtered each post to obtain data that was in Italian (per the platform-retrieved metadata), not a repost, dated between March 1, 2021, and December 31, 2021, and selected via hard keyword matching.
We chose search keywords and phrases that match the informal name of any of the measures -e.g., "bonus bicicletta" (eng: bike bonus) or "bonus babysitting. " -and download all matching posts. The keywords we used to identify relevant discussions in the posts were selected based on insights from an author who is native to Italy and was residing there during the pandemic period (2019-2022). Additional keyword refinement was supported by details from the National Social Security Institute (INPS) about COVID-19 measures. 2Below is the complete list of financial measures on which we focused (see Appendix for corresponding keywords):
• Bonus mobilità (Mobility bonus): contribution of 750 euros that could be used to purchase electric scooters, electric or traditional bicycles, for public transport subscriptions. To improve the initial pool quality, we removed duplicates (n=6543). Moreover, after manually inspecting the pool, we discarded posts related to the keywords "decreti" (eng: decree) and "credito d'imposta" (eng: tax credit) as they mainly pulled unrelated or too generic posts. The resulting collection counts approximately 100,000 posts relative to 12 different queries.
To balance annotation quantity and quality, we decided to collect extensive annotations for 10% of the initial pool.
3.2%
Subjectivity in MoniCA.
Table 2 Sentiment in MoniCA.
A critical issue with our initial pool was the presence of news posts, most frequently by media agencies and newspaper accounts. However, these posts are irrelevant to our goal of monitoring public perception of bonuses. Following previous work [7], we conducted a first round of annotation for relevance. We held round-table meetings to settle on a shared definition of relevance; then, we assigned 200 posts to each annotator and requested to choose whether each was relevant. We considered a tweet irrelevant if it mentions a bonus but focuses on another topic. 3 Next, we trained a supervised classifier to detect relevance and used it to select 10,400 additional posts from 7238 unique users. 4The annotation was conducted in three iterations. In the first two, we tasked annotators to annotate a shared set of 100 posts to compute agreement and tune annotation guidelines. Then, we assigned each annotator 3,333 posts, non-overlapping among them. In the next step we aggregated the labels. For subjectivity, sentiment, and irony we selected the annotations through majority voting, while for emotions and topics we used all the identified emotions from all the annotators. During this process, we identified some missing values in annotations that we addressed by removing them. The final set comprises 9,763 posts with one annotation each.
See Appendix B for full details on the annotation process, including pay rates, annotation platform and guidelines, inter-annotator agreement, intra-annotator consistency over time, and classifier performance.
Annotation Fields. To conduct the annotation, we provided annotators with i) the post's main text, ii) publication date, iii) at most two antecedent posts in the conversation tree, and iv) any multimedia content if present.
Irony Anger Sadness Joy Disgust Fear 66.7% 16.8% 5.8% 3.2% 2.2% 13.1%
Emotion and irony in MoniCA.
When available, the preceding posts and media are the conversational context and can help disambiguate the post's meaning. Each post was annotated for (1) subjectivity, (2) sentiment, (3) topic, and (4) emotion and ( 5) irony. Subjectivity was assessed as binary (subjective or not subjective); sentiment classification included negative, neutral, and positive categories; irony was annotated as ironic or not ironic; The topics were carefully pre-determined together with annotators, taking into account the aspects we aimed to extract from the data (see Table 4 for the list of topics); emotions included anger, sadness, joy, disgust, and fear categories; irony was assessed as binary. Annotators were given the possibility to select more than one emotion and topic per post. Moreover, we asked annotators to highlight the (6) span(s) of text that motivated their sentiment annotation. ( 1), ( 2), ( 3), ( 4) and ( 5) will serve to map the public opinion on the studied measures, and (6) will allow us to verify whether NLP models detect sentiment like a human would ( §5).
General Statistics. Tables 1,2 and 3 report the distribution of sentiment and emotions over the possible options.
Similar to related work [6,7,8], both sentiment and emotion are heavily skewed toward negative attitudes. The vast majority of posts (96.8%) are subjective; among them, 78% of the posts are negative, whereas 62% show anger. Irony notably appears in 5.4% of the posts. Table 4 shows the discussion topics and their proportion. Half of the posts are directed toward politicians, with even a higher spike in negative sentiment (93.4%).
These findings, taken together, convey a critical message: The majority of social media comments about financial aid in Italy in 2021 are from unhappy people. Such users posted on X with a negative sentiment, showing anger, sadness, disgust, or fear eight times out of ten. Some of our fine-grained annotations disclose some potential reasons: 8.5% of posts mention struggling to obtain a bonus, 1.4% not having the requisites, and 1.3% do not benefit from or get the bonus.
We are particularly interested in verifying whether stateof-the-art NLP tools can help us automatically model and detect the users' opinions. If models succeed at this task, they will serve as a digital barometer for monitoring issues and pitfalls of state-enacted financial aids. We designed four text classification tasks to train a model for automatic (1) Subjectivity, (2) Sentiment, (3) Emotion, (4) Irony, and (5) Topic detection. (1) and ( 5) are binary classification tasks; (2), (3), and (5) are three-, six-, and nine-way multi-class classification tasks.
We used Logistic Regression (LR), fine-tuned a pretrained Italian BERT model named UmBERTo [10], and tested an existing BERT model for emotion and sentiment detection in Italian named FEEL-IT [11] 5 .
LR has been trained on preprocessed texts: We converted all posts to lowercase and removed special characters and stopwords, replaced URLs and user handles with special tags, and performed stemming.
Given the significant class imbalance in our annotated data, we report both macro and weighted F1 scores. Macro F1 averages the performance across all classes, highlighting the model's effectiveness on minority classes. Weighted F1 adjusts for class distribution, reflecting overall performance in line with class prevalence. This dual reporting provides a balanced view of the model's performance.
Table 5 reports classification performance for every model-task pair in our setup. Our experiments revealed disparate performance across tasks.
We observed higher scores on the subjectivity detection task, probably due to the easier binary setup and the high unbalance. Emotion detection proved most challenging due to the subtle distinctions between classes. Interestingly, UmBERTo classified instances as either anger or joy, while LR defaulted to anger for all cases. FEEL-IT stood out by successfully identifying sadness and fear, highlighting the need for more data to capture the full spectrum of emotional nuances. None of the classifiers ever detected disgust.
Topic detection was also another difficult task. In addition to a higher number of unique topics, text content among topics might overlap (e.g., users who complain about struggling to get a bonus might use similar language to those who cannot see benefits from it).
UmBERTo demonstrated strong performance, excelling in three out of five tasks (avg. Macro F1: 43.18, Weighted F1: 74.8). Interestingly, simpler methods like logistic regression also performed reliably (avg. Macro F1: 35.68, Weighted F1: 71.88). These results are promising, showing that both straightforward models and advanced large-scale models-pretrained in the target language, Italian-can effectively serve as tools for automatic detection of subjectivity, sentiment, emotion, irony, and public attitudes. However, the natural imbalance in the data plays a significant role in these experiments, suggesting that further work is needed to address this issue more effectively.
Interpretability research in NLP has developed methods and tools to help explain the rationale behind a model prediction. These tools are beneficial to assess and debug models, e.g., by checking whether a model "is right for the right reason" or the cause of the error [12].
We conducted an additional interpretability analysis on UmBERTo, the best-performing model across our detection tasks (see §4). This study aims to verify whether the model's decision process aligns with those highlighted by humans. Transparency on model internals and human alignment promotes accountability and trust. 6Setup. Following [13,14], we use four common posthoc token-level attribution methods [15], i.e., LIME [16], SHAP [17], Integrated Gradient [18], and Gradient [19] across different configurations. Given a model and a model prediction (e.g., Sentiment: "Negative"), each XAI methods for explaining the sentiment analysis task (best values in bold, ↑: higher is better, ↓: lower is better).
method assigns an importance score to each input token for that prediction. Table 6 reports an explanation example in the first row and the human rationale annotated in the second row. We use faithfulness and plausibility [20] to evaluate explanations. Faithfulness evaluates how accurately the explanation reflects the inner workings of the model. Plausibility, on the other hand, assesses how well the explanations align with human reasoning. We use the human rationales provided by the three annotators during the annotation phase, and the UmBERTo model trained on the sentiment classification task, explaining the most likely class label for each test instance. We use three faithfulness (Comprehensiveness, Sufficiency, and Correlation with leave-out-out) and plausibility (Token IOU, Token F1, AUPRC) metrics as described in DeYoung et al. [21,ERASER] and leverage ferret [14] for explanation generation and evaluation.
Table 7 shows that LIME is, on average, the best model to explain predictions, indicating that LIME provides explanations that are both comprehensive and sufficient.
We documented the collection and release of MoniCA, the first large-scale dataset for monitoring the coverage and attitudes of financial aid enacted by the Italian government during the COVID-19 pandemic. It counts around 10,000 annotated posts for subjectivity, sentiment, emotion, irony, and topic. We conducted a first analysis and discovered that (1) most posts have a negative tone and (2) NLP and machine learning models can help detect it. Finally, we conducted a preliminary explainability study to understand how models predict sentiment from text. We found that explanation quality varies across methods and recommended LIME as a sensible starting choice.
Our dataset and study fill a critical research gap by examining Italian public sentiment towards COVID-19 measures. Future research will build on this groundwork to build more effective opinion monitoring and mining tools and ultimately inform prompt and targeted policy decisions. Additionally, to better understand the severity of negative attitude, future research may concentrate on examining hate speech in relation to public policies during the pandemic in Italy [22,23].
Our collection might not represent the opinions of the entire population. All posts included in our dataset were taken from X, which might have a specific user demographic that is skewed towards a specific demographic.
Additionally, a potential limitation might arise from the dependency of our data on keyword matching. This form of sampling might prevent some topics from being included in the dataset. However, we carried out keyword selection very carefully, including words and phrases that captured discussions around pro-bono government aid (see Section 2.2).
Another limitation is that our data covers a specific but quite broad temporal window from March 1 to December 31, 2021. This window corresponds to a phase of the pandemic, and changes in public opinion following this period are not captured.
Data for the MoniCA dataset was gathered using X's proprietary historical API, via an academic subscription.
Below is the complete list of f keywords used for data collection in the form of a tweepy 7 query:
• Bonus mobilità (Mobility bonus): "bonus mobilita" OR "bonus bici" OR "bonus monopattino" OR #bonusmobilita OR #bonusbici OR #bonusmonopattino. • Bonus 600 euro: "bonus 600 euro" OR "bonus 600euro" OR "bonus 600" OR #bonus600euro OR #bonus600
Profile and pay rate. For annotating the MoniCA dataset, three student research assistants with backgrounds in Machine Learning and Natural Language Processing were hired full-time. They were each compensated for 32 hours of work at a rate of about 18 euros per hour. We provided each annotator with an initial set of annotation guidelines, and we organized initial meetings to familiarize them with the task and refine the guidelines.
Platform. We used Label Studio8 using a custom labeling schema. We report the annotation schema and guidelines in the repository associated with the project. A screenshot of an annotated example is shown in Figure 1 for reference.
Agreement and consistency. The three annotators shared a pool of 100 posts. On these, we computed Krippendorff's alpha of 0.57 on subjectivity (i.e., is the post subjective or not), 0.60 on the post sentiment, and 0.51 on whether the contextual information was used. The agreement on sentiment increases to 0.61 when considering only posts that were considered subjective by everyone.
Moreover, we provided each annotator with a copy of 100 samples randomly shuffled later in the pool of posts to validate their consistency over time [24]. Annotators were highly consistent. On average, they annotated subjectivity consistently 95% of the time and sentiment 87% of the time.
This project has in part received funding from Fondazione Cariplo (grant No. 2020-4288, MONICA) and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 101116095, PER-SONAE). Debora Nozza and Fabio Pernisi are member of the MilaNLP group and the Data and Marketing Insights Unit of the Bocconi Institute for Data Science and Analysis. Giuseppe Attanasio conducted part of the work as a member of the MilaNLP group. Additionally, he was partially supported by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (Center for Responsible AI) and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020.