The availability of large annotated corpora from social media and the development of powerful classification approaches have contributed in an unprecedented way to tackle the challenge of monitoring users' opinions and sentiments in online social platforms across time but also arose the challenge of temporal robustness of such detection and monitoring systems. We used as case study a dataset of tweets in Italian related to the COVID-19 induced lockdown in Italy to measure how quickly the most debated topic online shifted in time. We concluded that it is a promising approach but dedicated corpora and fine tuning of algorithms are crucial for more insightful results.
The task of abusive message detection is a very
challenging one and from multiple perspective.
From the computational point of view, despite
the increasing interest and effort of the
community on developing automatic systems abusive
language detection and related tasks for different
languages
Copyright © 2021 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
goodness of hate speech prediction systems, and
of NLP algorithms in general, is rooted in how
well they capture and model all the relevant
characteristics of language in the context of a specific
phenomenon and its evolution over time
The recent availability of long-term and largescale digital corpora and the effectiveness of methods for representing words over time can play a crucial role in the recent advances in this field. In particular, social media have recently become one of the predominant sources of linguistic data, being the venue for noticeable phenomena in the domain of NLP tasks. They represent the ideal communication context to address the challenges we have outlined.
This paper aims to characterize how the
online conversation on the Italian Twitter around the
ifrst Covid-19 lockdown, imposed in Italy in 2020,
shifted very quickly from one heated debate to
another one, following the quick succession of news
reports on both news cases and institutional
advice and rules on how to navigate everyday life
as the crisis was unfolding in the entirety of the
world. At first we tried to identify the most
polarizing conversation by analyzing the presence
of hate speech using AlBERTo
In this work we use as case study a dataset of
tweets related to the COVID-19 induced lockdown
in Italy, as this was the perfect example of
government measures that deeply affected everyday life
of citizens and hence had the potential to spark
very heated debates online. We rely on a recently
developed resource, named 40wita 1
Even allowing for the influence of a different
context, this finding induced us to conclude that
an unknown but not negligible percentage of
hateful messages were left undetected. We believe
that increasing the training dataset size and
quality could lead to better results. For this
experiment the data were annotated using guidelines
developed for an hate speech detection task, while
a set of new guidelines developed specifically for
this context could be a significant improvement in
the quality of the labelled data. Another possible
adjustment relies on the number of annotators and
the exploration of the best metric to compute their
disagreement, following the latest published work
on annotating subjective tasks
The predominant categories of hate speech are represented by tweets containing derogatory words, abusive terms related to moral and behavioural defects, and words indicating cognitive disabilities and diversity. To gain a deeper insight on how this classification has unfolded we analysed which were the most common words that classified a tweet into a specific category. Quite often the words that determine whether a tweet falls or not into a category, and independently on the category, are very generic (e.g., “problema”=“problem”, “storia”=“history”) or can assume very different meaning depending on the context (e.g.: “cane”=“dog” can be used as a derogatory term or with a neutral meaning), and this contributes in creating a noisy tweets classiifcation. This insight is meaningful in showing why HurtLex presents some struggles in the accuracy of this task. For this reason, the division into pre-defined categories turned out to be not as informative as we were hoping at the beginning. An improvement on this would encompass a manual revision of the list of words for each category, in order to exclude the most generic ones and retain only those which can potentially improve the accuracy of the result. We also conducted a manual revision of all the tweets belonging to the categories with less than 30 tweets, while for the other categories we choose a random sample of 30 tweets, for consistency with the previous case. One of the most interesting findings was that in the category “rci - locations and demonyms”, in contrast to the global dimension of the pandemic, our data counter-intuitively showed that the debate was centered strictly around the measures taken in Italy and the differences between national and local rules.
This lexicon-based approach, even though it did not lead to the desired outcome, was nevertheless important to gain more information on our corpus and experience for future directions. In the next sections we will focus on the most powerful classiifcation tool that we employed on this dataset: two 2http://hatespeech.di.unito.it/resourc es.html different algorithms for unsupervised topic modeling. 3
We implemented two different classification algorithms. At first we run an exploratory topics analysis with a Latent Dirichlet Allocation (or LDA) and then a Dynamic Topic Modeling (or DTM) to better capture the temporal evolution of topics in the discourse.
Latent Dirichlet Allocation The first topic
model algorithm that we applied to our dataset is
the Latent Dirichlet Allocation, which was first
introduced by Blei
This model was able to correctly and precisely identify the conversations around the first relevant news around the incoming pandemic. Examples of this include the first restrictions on movements following the first Covid-19 outbreak in Lombardy and Veneto, the national lockdown issued in March and the consequent gradual shift of the conversation towards the difficulties of normal life in such a new context.
As powerful as this model is, it showed a fundamental limit for our perspective and purpose. The relevant topics were punctual but, as expected, not consistent over time because the model was completely re-trained on data from every single week, hence the results for each single time slice were agnostic of the result for every other time slices, and therefore not time-consistent, or comparable, by design. To overcome this issue we implemented a Dynamic Topic Modeling.
Modeling
At rfist we fine tuned the model by optimizing
the perplexity and the coherence score. The first
score captures the behaviour of the model towards
data which were previously unknown by means of
a normalised log-likelihood of a held-out test set.
However there are relevant studies (for example
Topic 0 Topic 1 Topic 2 Topic 3 Topic 4 quarantena altro lavoro governo sanita`
quarantine other work government healthcare but sometimes they even anti-correlate. For this reason a second metric was elaborated: the coherence score, to better model human judgement. This measure captures the degree of semantic similarity between the words related to each single topic ( i.e., a measure of the likeness of their meaning). We did not have an annotated corpus that can serve as a training set, hence we only explored the trend of the coherence score with reference to changes in the number of topics, chunksize of data, number of passes and evaluation score. We then concluded for 5 topics and 20 words per topics, as listed in the following Table 1. We chose to leave one topic undetermined (“Topic 1 - Other”) to label all the messages that the algorithm struggled to correctly assign to a specific topic.
The DTM outputs each unlabelled topic as a list of words with a relevance value. This value, between 0 and 1, represents the probability of a single word to be affiliated with a specific topic. The rationale behind the decision of choosing only 5 topics is that a higher number did not improve the understanding of the corpus as it led to a noisier classification. Each additional topic consisted of a list of words that were either very general in their meaning, or not very close semantically, or both, which made it very difficult to find a topic label that properly represented all the listed tokens.
The most powerful feature of the DTM is that, for each topic, it is possible to rank the most relevant words based on their attached probability value (of referring to the specific topic) and see how they evolve over time. In the following Figure 1, the change in ranking for all the 20 words involved is presented as a coloured heatmap, where the blue values represents words with higher ranking while the red ones are at the lower end of ranking.
There are two main insights we can gain from this visualization. The first one is that topics Topic 0 "Quarantine" Words Ranking are lists of pretty common words, which proves how hard of a task topic detection is, because of the complexity and versatility of human language, where general words can be used in different contexts with different meanings. The second insight is that the biggest changes in the word ranking happen within the first time slices. A possible explanation may be traced back to how this dataset was created. The list of hashtags and trends used to filter the tweets was compiled in February and was fixed in time. This means that potentially interesting tweets were left out because they contained hashtags that emerged as relevant later in time but hence were not captured by the keywords used for selecting relevant tweets.
In order to measure the temporal trend of predominance for each topics, we computed, for each of the 13 time slices, the ratio of documents labeled as predominantly referring to each of the topic.
We plotted in Figure 2 the normalized share of documents classified as containing each of the topics in each time slices, to highlight the relative trends over time.
Topics Distribution Sorted by Mean/Max Value topic 0 = quarantine topic 1 = other topic 2 = employment topic 3 = government topic 4 = healthcare 1.0 0.8 is that the discourse on Twitter does not only follow closely the most recent and relevant news but it quickly shifts from one topic to the other. In fact, all major peaks in Fig. 2 are followed by a sharply decreasing trend, indicating an immediate loss of predominance and hence an alternation of the dominant arguments of debates.
We explored in a similar way also the temporal evolution of the share of tweets labelled with the Hurtlex categories.
For each of the time slices we computed the relative frequency of tweets labeled with every categories and then created a stacked plot of their maximum values (shown in Figure 3) and the normalized mean values (shown in Figure 4) of their frequencies, to identify both peaks and categories that were consistently predominant through the time.
The relevance of the Hurtlex category related to derogatory words detected over the whole dataset, as described in Section 2, confirms its validity also at a weekly time granularity, as shown by Figure 3. Looking at the chart as a whole it is important to notice that, as we have already highlighted before, the peaks occur in time slices 3 and 5, which respectively correspond the the issue of the first red zones in Italy and two major public health news regarding Lombardy, the hardest hit region of Italy in the first phases of the pandemic (see Table 2 for details).
It is relevant to notice that these peaks occur exactly in the same time slices as the peaks in Figure 2 for the topics ”quarantine” and ”healthcare”, showing that the most heated debates happened around public measures that affected directly and immediately on both the collectivity (”healthcare”) and personal life (”quarantine”). Analysing the mean value of the frequencies, in Figure 4, we can see that categories rank differently from Figure 3. More specifically we see that for example ”ddf - physical disabilities and diversity” is by far the most consistent over time but it represents somehow a generic type of offensive language, not correlated with the pandemic, and to some extent this is as a noisy classification of tweets and it would be interesting to investigate further how to improve on this result. 4
In this work we tried to tackle the challenge of measuring and quantifying the topic shift in the public discourse on Social Media, using as a case study the online debate on Twitter following the Covid-19 related lockdown in Italy in 2020, by means of a dedicated dataset. By combining multiple classification methods we gathered insights into which governmental measures generated the most debated online conversation but we also concluded for the need of deeper investigation on how to build ad hoc corpora and methods to investigate specific linguistic phenomena as online conversation with rapid topic shift following the flow of news coming from both online and traditional media outlets. We also tried to inform AlBERTo with information extracted from topic modeling but the results were far from satisfying. This is a promising way to enhance the accuracy of hate speech prediction, but we concluded that a further investigation on size and characteristics of datasets is essential to gain better results.