This paper investigates stance and stereotypes within a dataset of Twitter conversational threads in Italian. The starting point of these conversations are tweets containing misinformation, in the form of racial hoaxes targeted at migrants, identified as untrustworthy by fake news debunking websites. The conversational structure of the dataset gives us the opportunity to observe and collect evidence about some linguistic and social phenomena at play in the propagation of stereotypes and the interactions between users which stem from them. We propose a theoretical background, as well as quantitative and qualitative analyses of our annotated data, at diferent levels of granularity, which can provide insights into the dynamics of Italian online discourses on the topic of migration.
CLiC-it 2023: 9th Italian Conference on Computational Linguistics, Nov 30 — Dec 02, 2023, Venice, Italy $ alessandrateresa.cignarella@unito.it (A. T. Cignarella); simona.frenda@unito.it (S. Frenda); tom.bourgeade@unito.it (T. Bourgeade); cristina.bosco@unito.it (C. Bosco); francesca.derrico@uniba.it (F. D’Errico)
© 2023 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
allowed for a more comprehensive evaluation of rumor detection systems, focusing on not only identifying rumors, but also understanding their stance.
Finally, in the context of the Italian language, the
SardiStance shared task was introduced in EVALITA
20201, ofering a pioneering challenge for Italian stance
detection [
In SemEval 2016 [
In a more general context, researchers started to study using a combination of psychology and natural language the development of stance in online conversational con- processing frameworks. We delved into the dynamics of texts and have employed a slightly diferent annotation racial stereotyping by extracting replies and comments scheme, to classify attitudes toward rumors or broader written below a controversial post written by the famous topics: Support, Deny, Query, Comment, often represented Italian singer Gianni Morandi, where he compared nowaas SDQC. This categorization has provided a versatile days migrants in the Mediterranean Sea to Italians imapproach especially classifying tweets belonging to the migrating to the USA in the 1920s. We explored how same conversational thread. these stereotypes manifest and spread, providing valu
Aker et al. [
Building upon this line of research, our team presented the tweets that mention the racial hoax, and 2,253 are our latest paper at the EACL 2023 conference. In the pa- replies-to-replies, that is, replies to direct replies. Thereper titled “A Multilingual Dataset of Racial Stereotypes in fore, the corpus preserves the conversational structure Social Media Conversational Threads” we introduced a of the Twitter threads, allowing a better analysis of the novel multilingual dataset called Multi-StereoHoax relations between these conversations’ participants. An [14]. Our study aimed at studying racial hoaxes and example of a conversational thread is reported in Figure 2. stereotypes in three diferent languages: Italian, Spanish In this work, we are interested in studying the stance and French. The dataset is labeled with a complex anno- expressed in the messages of the conversations towards tation scheme, based on the Stereotype Content Model the veracity of the hoax. Considering the purpose of (SCM) proposed by Fiske et al. [15]. It is a theoretical our analysis, we chose to adopt the SDQC schema of framework that provides a psychological understanding annotation adding a label called “Head” to identify the of how stereotypes are formed, maintained, and applied texts that spread the hoax or start the conversational in social contexts. thread (identified in Figure 2 as “Source Racial Hoax”).
These studies provide crucial insights into the origins, Inspired by Aker et al. [
As an example, in Figure 3 a source racial hoax, i.e., the Head of a Twitter conversation, and four replies (one per SDQC label): StereoHoax-It [14] is the Italian subset of a corpus of conversations collected on Twitter originated from hoaxes targeting migrants. We started from an initial list of hoaxes deemed racial as they tend to explicitly or implicitly attack immigrants, inciting to adopt a contestant stance to the phenomenon of immigration. This initial list was created by consulting debunking websites (bufale.net2 and BUTAC3).
Diferently from the standard schema used by Moham- In the remainder of the paper we provide analyses only mad et al. [16] where annotators determine if the author focusing on the 2,472 tweets that present agreement beof the message is in favor/against/neutral towards a spe- tween annotators, and leave the study of “complex cases” cific phenomenon, the SDQC+H gives us the possibility and incomplete annotations for future versions of the to identify, more precisely, the attitude of the author with corpus. respect to the hoax that targets immigrants. This annotation was applied to direct replies and replies-to-replies toward the hoax declared in the Head (or “Source Racial 4. Analyzing the Corpus Hoax” in Figure 2 as defined in Bourgeade et al. [14]).
3.1. Enriching the Corpus with Stance In this section, we provide quantitative and qualitative Labels analyses regarding annotation of stance. In Figure 4, the bar chart shows the distribution of labels annotated by Two diferent annotators, a male and a female Italian A1 and A2. We can observe how both annotators had native between 25 and 35 years old (one master student similar judgements when handling users’ stance towards in Linguistics and a PhD student in Digital Humanities) migrants. From the same figure, it can also be seen that have participated in the annotation campaign. They both for both annotators, Comment is the predominant laannotated all the tweets contained in StereoHoax-It bel (blue), followed by Support (green), Query (yellow) and later additionally annotated it for the dimension and Deny (red). The same percentages are respected in of stance as described above. The annotation was per- the final label distribution of stance calculated over the formed using Label Studio4 – an open-source annotation gold-labeled portion of the dataset, i.e. 2,472 tweets (see platform. Annotator 1 (A1) and Annotator 2 (A2), were Figure 5). both assigned 5,255 tweets in total, and they were asked to label them accordingly to the scheme presented in the previous section (i.e., SDQC+H).
Due to the complexity of the task, and to the fact that annotators could skip annotating a tweet in case of uncertainty, in this phase, we were able to collect only 3,123 complete annotations. Once the first round of labeling was completed, we performed an inter-annotator agreement test by calculating Cohen’s kappa coeficient, which resulted in = 0.3318 (fair agreement). The cases in which A1 and A2 provided two diferent labels were solved by a third experienced female annotator (A3), an Italian native, 25-35 years old post-doc researcher in NLP.
Thanks to this, some tweets with disagreements were Figure 4: Annotations of A1 and A2. adjudicated, thus increasing the size of the gold-labeled data. However, despite this efort, some disagreements remained for some instances, and we refer to them as “complex cases”. In Table 1 we report the numbers that are the outcome of the annotations and some more details regarding their nature.
n# tweets 2,132 449 202 2,472 5,255
details skipped tweets / of-topic incomplete annotation from either A1 or A2 agreement between A1, A2 + A3 (gold) complex cases
total
The results do not seem to show a particularly significant co-occurrence of one phenomenon with the other. Although, as expected, the majority of tweets annotated as (b) In percentages of labels Support, also contain racial stereotypes (34.68%), and the Table 3 majority of tweets annotated as Comment do not contain Confusion matrices showing the distribution of Support, Deny, forms of stereotyping towards migrants (63.60%). Query and Comment labels with respect to the depth of each
We could have expected a significant portion of the tweet in its conversation. *the minimum depth for conversaDeny label to co-occur with the absence of stereotypes, tions with missing links is 2. but the tweets annotated with that label are very sparse (they are only 6.58% in total), therefore it is not suficient for drawing meaningful conclusions.
In order to evaluate the influence of conversational structure on the distribution of stance labels within our dataset, we measured the “conversation depth” of each individual tweet. Specifically, each Head tweet was assigned a depth of 0 (however, these Head tweets were not considered for the rest of this analysis). The conversation depth of each subsequent tweet was then determined by calculating the length of the reply-chain leading back to the original Head of its conversation. Unfortunately, due to the nature of the phenomenon we are investigating here, numerous tweets (1, 947) presented gaps in their respective reply-chains, due to the deletion of content (either by their authors, or by moderation of the microblogging platform). In these cases, we assigned the minimum potential depth value of 2, given that all Heads are accounted for in our dataset. Table 3 thus depicts the distribution of varying stance labels according to depth within the dataset.
Although the label distribution across depths largely mirrors the overall dataset distribution, we can observe a higher proportion of Support messages in direct replies (depth 1). This might suggest that users who aim to challenge the veracity of a racial hoax might be more inclined to express themselves as replies to replies, rather than directly under the initial posts. To estimate the correlation of stance labels with depth, we perform a Chi-squared test and compute Cramér’s : we find a Chi 2 value of 130.762, with a p-value of 4.34× 10− 22, as well as a of 0.134, which thus only indicates a small association [17]. To investigate diferences among specific conversations, we compute two measures of “controversiality”: 1. Support-Deny Balance (SD-B) is simply derived from the proportion of Support minus Deny messages, as a positive or negative percentage of their sum: count() − count() count() + count() 2. max P-index implements the measure proposed by Akhtar et al. [18], the Polarization Index, where we consider each conversation as an instance with its Support and Deny replies as annotations. We then iterate over all possible = 2 partitions of these annotations (proponents and opponents) for each conversation, and find the maximum Pindex which we report here.
Table 4 presents these measures for the 10 largest conversations (in number of tweets) in the dataset, as well as their percentage of messages containing stereotypes. We only display the top-10 both for space reasons and because further conversations are too small to compute meaningful metrics (starting from the 27th largest conversation the number of messages is 4 or less, and many have no Support and Deny replies). Conversation #1 can be considered the most “controversial” in this dataset (SD-Balance closest to 0%, largest max P-index), and it also happens to be the largest. The Head of this conversation is the following tweet (adapted into English): “Now Matteo Salvini is in court in Catania for defending the borders, please also tweet #IstandWithSalvini, let’s make him feel our afection!” . As this is a call for support for a controversial figure in Italian politics, this explains the relative balance of Support and Deny replies, though the number of messages presenting stereotypes remains relatively low, possibly due to supporters’ intent not to have their messages moderated by the platform. Examples of more polarized conversations are Conversations #3 and #5: the former does not have a single Deny response, with the Head being a tweet criticizing the verdict for the 2017 Kobili Traoré murder trial in France, which attracted a significant number of replies containing stereotypes against immigrants and Muslims; whereas the latter provoked a larger proportion of Deny responses compared to Support, with its Head propagating a racial hoax about the Italian government supposedly secretly bringing in illegal immigrants by plane during the COVID-19 pandemic. Interestingly, Conversation #8 (also displayed in Figure 2) concerns the same subject as Conversation #3, but displays a greater proportion of Deny responses than the former, indicating that the same subject may be received wildly diferently, depending on the context it is introduced in.
To investigate the vocabulary employed by users supporting and denying the hoaxes expressed in the heads of conversational threads, we present: the most relevant n-grams (unigrams, bigrams, and trigrams) of the messages annotated with the presence of stereotypes. The n-grams are weighted using the TF-IDF measure on normalized texts after a specific phase of preprocessing that involves: the deletion of all user mentions, stop-words, punctuation and URLs, leaving only words that were lexically significant. For the tokenization and lemmatization, we employed the small model for the Italian language available in the SpaCy5 library.
By looking at the resulting lists of words and expressions, we noticed that texts labeled as Support are explicitly ofensive towards immigrants. On the contrary, the ones that are labeled as Deny tend to stress the condition of need and poverty of immigrants, and are more empathetic. In this second list, we also noticed some ofensive words but towards political parties leaning farright. Some of the most relevant n-grams of both lists with their TF-IDF values are reported in Table 5.
Support casa governo entrare clandestino bastardo cinese potere dare merde schifoso invasione risorsa peso
TF-IDF 5.357 3.889 3.750 3.323 2.901 2.677 2.599 2.521 1.822 1.605 1.574 1.338 1.314
Deny nave disperato pelle povero propaganda difeso minacciare povero cristo poveraccio andare cagare leghista afamato malato indifese raccogliere pomodoro approdo coraggio
In this article, we explored the expression of stance and stereotypes as occurring in a dataset of Twitter conversational threads in Italian, focused on the topic of migration. The dataset consists of dialogues originating from tweets containing misinformation marked as untrustworthy by experts.
The analysis of the dataset shed light on the distribution of stance labels and their relationship with stereotypes. The majority of tweets were annotated as Comment, followed by Support, Query, and finally Deny. While
there was no significant co-occurrence between stance and stereotypes, tweets annotated as Support were more likely to contain racial stereotypes. On the other hand, tweets annotated as Comment were less likely to exhibit forms of stereotyping.
The corpus analysis provided insights into how the structure and nature of conversations, and lexical choices in messages, afect the perceived stance of users towards racial hoaxes.
In conclusion, this work paves the way for further investigations about topics closely related to the social phenomenon of misinformation that should be countered to stimulate accurate information dissemination and create a more inclusive online environment. In future research, we may increase the size of the dataset, improve the annotation guidelines and consider the feedback provided by the annotators. We may moreover further investigate the relationship between stance and stereotypes, as well as explore interventions to mitigate the harmful efects of stereotypes in online conversations.
In line with the recent trend of the main NLP conferences, we add a brief section addressing the limitations of our work. In this work, we enrich our corpus previously introduced in Bourgeade et al. [14], using a similar annotation framework, and therefore the same limitations brought forward in this work still apply here: more specifically, regarding the practical reliability of the theoretical social-psychological framework used to derive the annotation guidelines. In addition, the Italian subset of the multilingual StereoHoax corpus has a very limited size and presents many unbalanced dimensions and high data sparsity. If in the future it will be used for computational tasks, as it is intended, it should be made more balanced and more inclusive in terms of data sources.
This work was partially funded by the International project STERHEOTYPES - Studying European Racial Hoaxes and sterEOTYPES, funded by the Compagnia di San Paolo and VolksWagen Stiftung under the ‘Challenges for Europe’ Call for Projects (CUP: B99C20000640007). The work of T. Bourgeade is funded by the project StereotypHate, funded by the Compagnia di San Paolo for the call ‘Progetti di Ateneo - Compagnia di San Paolo 2019/2021 - Mission 1.1 - Finanziamento ex-post’. Italian social media corpus where psychology meets NLP, Information Processing & Management 60 (2023) 103118. [13] F. D’Errico, C. Papapicco, M. T. Delor, ‘Immigrants, hell on board’: Stereotypes and prejudice emerging from racial hoaxes through a psycho-linguistic analysis., Journal of Language & Discrimination 6 (2022). [14] T. Bourgeade, A. T. Cignarella, S. Frenda, M. Laurent, W. Schmeisser-Nieto, F. Benamara, C. Bosco, V. Moriceau, V. Patti, M. Taulé, A Multilingual Dataset of Racial Stereotypes in Social Media Conversational Threads, in: Findings of the Association for Computational Linguistics: EACL 2023, 2023, pp. 674–684. [15] S. T. Fiske, A. J. Cuddy, P. Glick, Universal dimensions of social cognition: Warmth and competence, Trends in cognitive sciences 11 (2007) 77–83. [16] S. Mohammad, S. Kiritchenko, P. Sobhani, X. Zhu, C. Cherry, A dataset for detecting stance in tweets, in: N. C. C. Chair), K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), ELRA, Paris, France, 2016. [17] C. M. B. McLaughlin, The association between two categorical variables: A review of cramér’s v and alternative statistics, The Journal of the Royal Statistical Society. Series D (The Statistician) 26 (1977) 187–195. [18] S. Akhtar, V. Basile, V. Patti, A new measure of polarization in the annotation of hate speech, in: AI*IA 2019 – Advances in Artificial Intelligence, volume 11946, Springer International Publishing, Cham, 2019, pp. 588–603. URL: http://link.springer. com/10.1007/978-3-030-35166-3_41. doi:10.1007/ 978-3-030-35166-3_41.