The study of institutional communication related to the pandemic, and to the population's response to it, is of great relevance today. The Italian spokesperson for communication regarding the pandemic has been, during the year 2020, the former Prime Minister Giuseppe Conte. We retrieved 4,860,395 comments from his Facebook official page and built the ConteCorpus, a new Italian resource annotated in CoNLL-U format. A first aim of the research was to evaluate the performance of the model used to annotate the corpus. Models trained on social media texts are usually not very generalizable. Nevertheless, the results of the evaluation were good, especially in parsing metrics, and showed that a parser trained on Twitter data can be successfully applied to Facebook data. A second aim of the research was to provide an overall view of the content of such a large corpus; for this purpose, topic modeling was conducted, training an LDA model. The model generated 5 topics that cover different aspects linked to the pandemic emergency, from economic to political issues. Through the topic modeling we investigated which topics are prevalent on particular days.
During the year 2020, the Prime Minister Giuseppe Conte has played a major role in institutional communication, particularly in communication regarding the policies undertaken to manage the health emergency. We assumed that interesting content from the point of view of the response of the population to institutional communications regarding the pandemic would have been found on his social media profiles. Therefore, we created ConteCorpus,1 retrieving more than 4 million comments from his Facebook page2 starting from January 2020 until December 2020, and we annotated it in CoNLL-U format3.
A first aim of the research was to evaluate the performance of the model used to annotate the dataset. Models trained on social media texts usually are poorly generalizable even on text retrieved from the same social media, therefore we wanted to test the performance on Facebook texts of a model trained on Twitter texts. In order to evaluate the model, we created a gold standard by extracting 1,000 sentences from the ConteCorpus and manually revising them.
A second aim of the research was to provide an overall view of this large corpus. For this purpose we performed a Topic Modeling. We trained a LDA model sampling 10% of the ConteCorpus. The LDA model generated 5 topics related to different aspects of the pandemic emergency. The model was used to see which topics were the most relevant before and after the announcement of the first and the second period of restrictions adopted to fight the pandemic in Italy.
The paper is structured as follows: we first review the relevant literature for our research (section 2), then we describe the data collection and the creation of the corpus (section 3). In section 4, we describe the evaluation we performed of the model we used to annotate the corpus in CoNLL-U format, and in section 5 we report the results of the topic modeling experiment. In section 6 we provide some concluding observations. Copyright ©️ 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 https://github.com/Viviana-dev/Conte_Corpus 2 https://www.facebook.com/GiuseppeConte64/ 3 https://universaldependencies.org/format.html 2
Since the beginning of the health emergency,
there has been a proliferation of computational
analyses that exploit data extracted from social
media. These data are considered relevant as they
allow us to generalize about human social and
linguistic behavior, especially regarding the
pandemic event. Among the tasks that have been
conducted on data drawn from social media in this
period, sentiment analysis, emotion profiling and
topic modeling are the most common
In particular, Topic Modeling aims at finding
hidden semantic structures within the texts and to
model them into concepts. The unsupervised
clustering technique LDA (Latent Dirichlet
Allocation), developed by
As is well known, the language of the web is
characterized by deviation from the standard
language that challenges the use of NLP tools.
Several classifications have been proposed to label
the nature of web and social media language. In
general, the labels aim to define a variety of
language that is diaphasically low and at an indefinite
point on the diamesic axis, e.g., “netspeak”
We have downloaded 4,860,395 comments and 534 posts published during the year 2020 on Giuseppe Conte’s Facebook official profile. We made call to any 2020 post ID of Giuseppe Conte’s official page to retrieve text, object id, and created time of comments. The calls to the Facebook API Graph4 were made month to month in the same fashion. Nevertheless, as Table 1 shows, a larger amount of data has been retrieved in the month of March, April, and October. In the same period in Italy the more restrictive measures to fight pandemic were taken by the government. 3.2
After the data collection, we processed the data
with the Neural Pipeline Stanza5 to enrich the
texts with some annotations. Stanza is an
opensource Python NLP toolkit, which “features a
language-agnostic fully neural pipeline for text
analysis, including tokenization, multiword token
expansion, lemmatization, part-of-speech and
morphological feature tagging, dependency parsing,
and named entity”
We processed the data divided in 12 packages; each correspond to one month data. We used every processor of the pipeline, besides the Named Entity Recognition module (TokenizeProcessor, POSProcessor, LemmaProcessor, DepparseProcessor). We personalized the model in or6 Universal Dependencies (UD) is a “framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages” (https://universaldependencies.org/). der not to split the sentences, 8 forcing the TokenizeProcessor to consider each comment as a sentence. Furthermore, we added two metadata to each sentence: one refers to the id of the post from which the comment was retrieved, and the other is the creation time of the comment. The aim is to make it easier to retrieve the comments from the corpus by their created time or post id if one needs to analyze a particular period of time or a particular post. 4 4.1
We built a gold standard with a dual purpose: to evaluate the performance of the model on this new collection of social media texts, and to create a standard that can be used for future training and testing. We randomly selected 83 sentences from each file of the corpus annotated automatically (one file is composed of one-month comments), and manually revised the 1,000 sentences collected. The manual revision has followed the principle that what is understandable by a human would be correct. 4.2
To perform the evaluation, we used CoNLL 2018
UD shared task official evaluation script.9 Table 2
shows the scores of evaluation metrics resulting
from the performance of Stanza model on the test
set of the ConteCorpus. Table 3 compares the
scores of evaluation metrics resulting from the
performance of Stanza model on the test set of
PoSTWITA-UD and the ConteCorpus. The first
two columns are the scores on metrics that
evaluate segmentation. The row called UPOS shows the
7
https://universaldependencies.org/treebanks/it_postwita/index.html
8 Sentence segmentation and tokenization are jointly
performed by the TokenizeProcessor
What we found most challenging during the
manual revision of the 1,000 sentences annotated
automatically was correcting the errors in
tokenization: many words that the tokenizer should have
splitted were joined together. This type of
tokenization error is often found when punctuation is
used with non standard function. For example: we
found that the token “oneste…volevo”
(“honest…I wanted to”) - an adjective, a punctuation
mark and a verb - are conflated in a single token.
In the manual revision, tokens like this have been
splitted in three different tokens and other missing
tags were added. The presence of such conflated
words mayhave caused a worse score in the metric
that evaluates the performance of segmentation,
and consequentially in the other scores. The
evaluation on the parser starts with aligning system
nodes and gold nodes; their respective parent
nodes are also considered; if the system parent is
not aligned with the gold parent or if the relation
label differs, the word is not counted as correctly
attached. Despite errors in segmentation seem
frequent in the corpus, this did not cause an excessive
lowering of the scores on the various metrics
reported in Table 2 and 3. Another error that appears
frequently regards the lemma assigned to the
abbreviations that are not present in
PoSTTWITAUD. Canonical abbreviations are tagged correctly,
for example “cmq” for “comunque” (“however”).
The abbreviations tagged incorrectly are those
which appeared few times: such as “ql” that stands
for “quelli” (those). An unexpected good result
has been achieved on parsing metrics. This result
could be due to the “preference of UD scheme in
assigning headedness to content words”
Overall, the model trained on PoSTWITAUD turned out to perform well on the test set of the ConteCorpus because PoSTWITA-UD tagset has been adapted with attention to some recurrent features of social media languages. Our evaluation showed that a model trained on texts retrieved by social media can adapt well to other social media texts if one pays attention to the neural architecture of the model and the annotation format being used. 5
To provide an overall view of the content of this large corpus we performed a Topic Modeling training and testing an LDA model on the ConteCorpus. To perform topic modelling, we sampled 10% of the sentences in our dataset and trained a LDA model. We treated each sentence as a document. We pre-processed lemmas removing stopwords, downloading Italian stopwords list from the NLTK (Natural Language Toolkit) library10 and manually inserting missing stopwords. We filtered out tokens that appear in less than 15 documents and tokens with less than three letters; additionally, we kept only the 100,000 most frequent words. We transformed the documents into vectors creating a bag-of-words representation of each document. Then, we performed the term frequency-inverse document frequency (TF-IDF) on the whole corpus to assign higher weights to the most important words. Gensim LDA model11 was applied first to the bags-of-words and secondly on the TF-IDF corpus to extract latent topics. Better performances were achieved with the LDA model applied to bags-of-words. We determined the optimal number of topics in LDA using the Coherence Value metric.12 The underlying idea is that a good model will generate topics with high topic Coherence Value score. We ran different LDA ex10 https://www.nltk.org/. 11 https://radimrehurek.com/gensim/models/ldamodel. html. periments varying the number of topics and selected the model with the highest medium topic Coherence Value score. Our final model generated 5 topics and has a topic medium Coherence Value score of 0.5. Table 4 illustrates the top ten most representative terms associated with each detected topic. 5.2
As expected, all the topics extracted from the
corpus are related to the concerns about the
emergency. The focus is on the economic aspect of the
emergency. The first ten most frequent words in
Economics topic (Table 4 and Figure 3) are
economic terms: “loan”, “company”, “to pay”
“money” etc. In all the other topics at least one of
the 10 most frequent words comes from the
economic sphere. Among the ten most frequent words
of each topic there are only two words regarding
the pandemic, found in Pandemic topic: "virus"
and "pandemic". It is no coincidence that the most
frequent word in this topic is “to go out”. The need
to face the emergency through the intervention of
the institutions is evident. This is shown espe
12 Coherence Value met