Introduction and Background

Various recent NLP studies have focused on detecting stereotypes online, often in conjunction with forms of abusive language [1,2,3,4,5]. The importance of tackling this phenomenon is due to its impact on social structures and the power of individuals. Therefore, detecting stereotypes can prevent their emergence and spread, and thereby have a positive impact on our society.

In social psychology, a stereotype has been defined as a set of beliefs about others perceived as belonging to a different social group [6]. It oversimplifies the features of the group and generalizes a particular feature, applying it to all its members [6]. In contrast to the emotional component of prejudice and the behavioral component of discrimination, a stereotype is associated with the cognitive component of the triad [7]. In language, stereotypes can be expressed explicitly or implicitly [8]. Explicit stereotypes deliver a straightforward message, clearly revealing the associated traits, often using derogatory adjectives [9,10]. In contrast, implicit stereotypes are more nuanced and indirect, requiring the reader to infer their meaning [11]. These implicit stereotypes can be com-municated through linguistic devices such as metaphor and irony [9], negation [12], or entailments [13]. Recently, efforts have been made to formalize the strategies for expressing implicit stereotypes, with the goal of establishing standardized criteria for annotators [14]. An example of explicit stereotype is "[Gli immigrati] buttano via il cibo che gli danno per poi andare a mangiare i poveri cani, dove finiremo!" 1 (extracted from StereoHoax-IT corpus), in which the generalization of the target group and the association with an action is expressed in a present tense with a habitual aspect. On the other hand, in the example "Come noi rispettiamo loro e il colore della loro pelle, così loro che abitano nei nostri paesi dovrebbero portare rispetto nei nostri confronti." 2 (SterheoSchool corpus), the stereotype is not overtly manifested, but it must be inferred through the evaluation of the in-group and an exhortative sentence.

From a computational linguistics perspective, concerns have been raised about how to detect and process stereotypes, a task often considered closely related to the detection of abusive language or hate speech [15]. Alongside research on hate speech, the study of stereotype detection has increased, particularly within evaluation tasks [16,4,17,18,19]. However, the detection of implicit stereotypes remains a significant challenge [20]. There are several works that deal with stereotypes in more complex narratives, such as microportraits [21] and political debates [22]. The detection of implicitness has also been studied with reference to several other phenomena, in particular those characterized by subjectivity, such as irony [23]. In this paper, we analyze the implicit manifestation of stereotypes targeting immigrants, using a well-defined annotation schema proposed by Schmeisser-Nieto et al. [14] and tested on a subset of comments from Spanish newspapers (DETESTS [5]). This schema represents different criteria for determining the implicitness of stereotypes in an attempt to formalize the concept. Disentangling strategies of implicitness presents a significant challenge, often resulting in the identification of multiple categories within the same text.

Our main contributions consist of expanding the annotation with topics of stereotypes about immigrants [5] and the strategies to implicitness [14], as well as testing this schema on two existing Italian datasets. These datasets share the same domain as those used for Spanish, stereotypes about immigrants, and include data extracted from Twitter (now X) as reactions to specific hoaxes (StereoHoax-IT) and comments written by high school students to two examples of fake news artificially created within psychological experiments (SterheoSchool) as described in [24,25]. Analyzing the annotated texts, we noted that implicit stereotypes appear to be conveyed especially through logical linguistic relations like entailment and the behavioral evaluation of immigrants in both datasets. Moreover, in most cases, the annotators needed to use contextual information to determine the presence of stereotypes. For example, in this case "Che centra lui e Italiano!, può essere massacrato!"3 (StereoHoax-IT) the author of the message expresses a stereotype complaining that foreigners enjoy better treatment than Italians, who can indeed be "macellati" (slaughtered).

The rest of the paper is organized as follows: Sections 2 and 3 describe the datasets and the annotation applied; Sections 4 and 5 present quantitative and qualitative analyses of the annotated data; and Section 6 summarizes the results and provides guidance regarding future work.

Datasets

In this work, we focus on two annotated corpora containing implicit stereotypes developed within the STER-HEOTYPES project 4 and the SterotypHate project 5 . Their content is related to attitudes regarding immigrants and they share similar conversational structures and the same annotation scheme. Each message in these datasets is contextualized, i.e. collocated within a discourse thread or presented as a comment on a given news item. For the annotation scheme, each message is annotated for the presence or absence of anti-migrant stereotypes, and, if present, for other related categories such as whether the stereotype was expressed implicitly or explicitly and which forms of discredit the stereotype could be classified at. This category is inspired by the Stereotype Content Model (SCM) [7] and allowed us to observe the stereotype from a perspective that encompasses psychology and computational linguistics [26]. In section 3, we show how we extended this annotation to describe the dimension of implicitness 6 . StereoHoax-IT [27] is a contextualized multilingual dataset of tweets annotated primarily for the presence of anti-migrant stereotypes. The dataset consists of replies to tweets identified as containing racial hoaxes specifically targeting migrants and collected from debunking websites from French, Italian and Spanish Twitter, collected from 2019 to 2021. Each message is provided with its "conversation head" (the message containing the source racial hoax), and its direct parent message (if applicable). In this paper, we only use the Italian subset, which includes 3,123 instances. Due to the rarity of the phenomenon, there is a significant class imbalance: 472 instances (15%) contain a stereotype, 332 of which (70%) are implicit and 140 (30%) are explicit.

SterheoSchool [28] consists of a selection of data collected in Italian schools during experiments conducted by social psychologists [24,25]. More precisely, it includes the reactions of teenagers, who read two hoaxes artificially created and presented as news articles, recorded via a cell phone interface. The hoaxes were designed to elicit reactions to stereotypes in readers. For each news item, readers were asked to comment on the news and on the main character of the articles. These comments are also associated with metadata, such as the age and declared gender of the author. By collecting data generated by teenagers, this corpus aims to fill a gap in the literature in which teenagers are an underrepresented category in data annotated for text classification tasks. We applied the annotation scheme mentioned above to the news and comments. This corpus consists of 1,147 comments, of which 337 (33.8%) are annotated as containing stereotypes, of which 152 (45%) are expressed in an implicit form.

Annotation

The annotation scheme we applied on the two corpora is based on two different layers, topics of stereotypes and implicitness strategies, as well as the need for context.

The topics of stereotypes were firstly introduced within an evaluation task, DETESTS [5], in which the participants had to train models to decide whether a text contained stereotypes, and when they did, classify the stereotype into ten different categories:

• Xenophobia victims Immigrants are perceived as victims of xenophobia and discrimination. They enrich culture and diversity and should have the same rights as citizens. • Suffering victims Immigrants are portrayed as victims of poverty and violence in their places of origin and as having to face difficult situations in their host countries. Context and implicitness strategies were initially proposed as criteria that could help annotators to annotate implicitness, since their vagueness may decrease Inter-Annotator Agreement (IAA) [14]. By context, we refer to information contained in previous messages, which is considered necessary to understand the meaning of the message to be annotated, as in the following example: "Sempre assolti...sempre misure e pesi differenti". Context: "Uccide anziana ebrea al grido di Allah Akbar. Assolto perché drogato. "7 (StereoHoax-IT). Regarding the strategies and linguistic devices used to convey implicit stereotypes, we have revised the criteria proposed in [14] as follows:

• World knowledge World knowledge refers to the shared cultural, social and historical knowledge needed to interpret messages, e.g., "La scuola si inchina all'islam: l'aceto è bandito dalle mense." The annotation was carried out on the Label Studio platform by three native Italian speakers with a background in linguistics, some of whom specialized in NLP. They achieved an acceptable to good IAA in the majority of cases, as reported in Table 1, which varies across categories and corpora. By observing Table 2, we can see that only a few topics have been marked by the majority of annotators , while not all the implicit criteria have been identified in the texts (i.e., 'humor/jokes').

Quantitative Analysis

Table 2 shows the distribution of the disaggregated annotations across both datasets. Columns 0%, 33%, 67% and 100%, respectively, indicate the number of instances per label that were annotated by no annotator (0%), by one annotator (33%), by two annotators (67%) and by all three annotators (100%). Column % positive class shows the percentage of the label voted by the majority of annotators, and its total number of cases in parentheses.

Firstly, an inconsistency in the distribution of labels can be observed since SterheoSchool has a representation of labels of more than 10% on only four labels. This disparity is due to the extraction methods of each dataset: the topics of the racial hoaxes used to extract the dataset were more balanced in StereoHoax-IT than in SterheoSchool, with the latter focusing generally on security and cultural differences that are discussed in the two only contexts provided to the students for their comments. However, while in the former there is a representation of all the stereotypical topics that portray immigrants as threats, the security issue is highly prevalent in both datasets.

A common trend shows that the most frequent implicitness strategy in both datasets is 'entailment/evaluation', accounting for 64% in StereoHoax-IT and 80% in Ster-heoSchool. To a lesser degree, 'extrapolation' appears in both datasets, with 13% in the former and 19% in the latter, respectively. Other represented strategies that exceed 10% of instances are only found in StereoHoax-IT.

The label 'context' has a high prevalence in both datasets, accounting for 38% in StereoHoax-IT and 80% in SterheoSchool. This is expected, as it depends on the methodology to produce the comments-spontaneous versus controlled-and the variety of contexts: two fake news for StereoSchool and 50 racial hoaxes for StereoHoax-IT. The limited amount of data unfortunately does not allow us to reliably evaluate a correlation between 'context' and certain implicitness strategies, as shown in Table 3, except for the association between 'entailment/evaluation' and 'context' across both datasets. The correlation between 'implicitness' and 'context' is also shown in Bourgeade et al. [27], with significant associations of the aforementioned labels in three languages: French, Italian and Spanish. In StereoHoax-IT, the correlations between the 'context' and 'irony/sarcasm', 'extrapolation' and 'imperative/exhortative' are also significant, whereas the category of other implicitness strategies is also significantly correlated in SterheoSchool, which can be analyzed qualitatively to determine if there is a pattern among them. The other strategies do not have representative instances that allow for analyzing them comparatively, except for 'extrapolation', which is significantly correlated in StereoHoax-IT but not in SterheoSchool.

In terms of co-occurrences between topics and implicit strategies, we can observe from Table 4 that there is also a great disparity in both datasets. Focusing on the two topics with the highest representation in SterheoSchool (Culture & religion, 51%, and security, 35%), which account for the majority of the corpus, we can analyze some differences with StereoHoax-IT. Firstly, 'culture & religion' is expressed primarily through entailments or evaluations (65 co-occurrences) and secondarily through extrapolations in SterheoSchool. In contrast, the distribution of strategies used to represent 'culture & religion' stereotypes is more evenly spread in StereoHoax-IT. A similar pattern is observed with the topic of 'security', which, while concentrating strategies in 'entailment/evaluation, ' also utilizes a range of other strategies, particularly 'extrapolation' and 'imperative/exhortative'. With these co-occurrences, we can reaffirm that the different methods to extract the data have an impact on the characteristics of it, and therefore, its distribution of labels. For instance, the messages were written in a non-controlled environment, which gives the authors the freedom to express themselves without constrains. Moreover, the

Qualitative analysis

To deepen the analysis of implicitness strategies and their interaction with different topics, we explore some messages to uncover the linguistic structures that are characteristic of implicit communication.

Example 1 has been annotated with the topic 'public health' and 'figures of speech' and 'Irony/Sarcasm' for the strategy of implicitness; all labels achieved a 67% IAA.

1) Governo di involtini primavera!!! 18 (StereoHoax-IT) In the context given for this message, the author complains that the government did not use more restrictive measures against Chinese children during the early stages of COVID-19. First, an ironic reading, i.e., as stating A to mean not-A, is triggered by the metonymy "spring rolls" [29], identifying Chinese citizens through a traditional Chinese dish. Second, disapproval is conveyed showing a kind of favorable attitude of the Italian government toward Chinese children.

Example 2 was annotated as 'culture & religion' by all three annotators. In terms of the implicitness strategies, it was labeled as both 'extrapolation' and 'entailment/evaluation' by two out of the three annotators.

2) Venezia, donne velate sputano al crocifisso. 19 (StereoHoax-IT) In this case, the noun phrase "veiled women" is a case of lexical narrowing, i.e., a lexical item conveys a meaning that is more specific than the item's encoded meaning. The reader selects a more specific meaning on the basis of stereotypes and world knowledge [30] of the meaning of "veiled women", which denotes a set of women who wear a veil, narrowed to mean Muslim women. This equalization arises from the stereotype that posits that if a woman wears a veil, she is a Muslim. Furthermore, the absence of the determiner in the noun phrase, that usually indicates a generic reference, combined with the imperfective aspect and present tense of the verb, may suggest a habitual interpretation of the predicate "spit on the crucifix" [31]. 'Extrapolation' strategy here refers to the attribution of this action to the entire category.

Among the more frequently agreed implicitness strategies, there are 'imperative/exhortative' and 'figures of speech', which have linguistic and punctuation features closer to explicitness: the former is associated with a specific grammatical mood and the exclamation mark, while the latter is associated with a question mark (considering that rhetorical questions are frequently annotated as a figure of speech), see e.g.:

3) Se non fate niente Fra 10 anni l'italia sarà tutta musulmana! 20 (StereoHoax-IT) 4) Come ci si può sentir sicuri in una società che permette questo? meschina 21 (SterheoSchool) The high IAA for the category of 'irony/sarcasm' is 19 Trasl."Venice, veiled women spit on the crucifix. " 20 Trasl."If you do nothing In 10 years Italy will be completely Muslim" 21 Trasl."How can one feel secure in a society that allows this? mean" also interesting, and has been studied especially in social media [32,33], as a means to lower the negative social cost of what has been said. The two categories that most frequently co-occur with 'irony/sarcasm' in StereoHoax-IT are 'figures of speech' (out of 35 instances, six are also ironic) and 'humor/jokes' (out of three cases, two are ironic), as in the next example: 5) @Belle facce intelligenti! Viva Lombroso! 22 (67% Humor/Jokes, 67% Irony/Sarcasm, StereoHoax-IT) We found messages in which 'entailment/evaluation' cooccurs with 'irony/sarcasm', but this correlation should be analyzed in depth to be considered relevant, as 64% of instances were annotated as 'entailment/evaluation. '

Conclusions

In this paper, we applied an annotation scheme for analyzing the implicitness of stereotypes against immigrants according to two main dimensions (i.e., topics and strategies for making the content implicit) to the Italian StereoHoax-IT and SterheoSchool corpora. Adding these two layers of annotation allowed us to observe that annotators need to use contextual information to determine the presence of stereotypes especially, when specific strategies have been used by the author of the message (irony/sarcasm, extrapolation, entailment/evaluation, and imperative/exhortative). Moreover, implicit stereotypes appear to be conveyed mainly through logical linguistic relations such as the entailment and behavioral evaluation of immigrants and, in fewer cases, via 'imperative/exhortative', 'irony/sarcasm' and 'extrapolation. '

As future work, we plan to perform a comparative analysis with the datasets in Spanish, which have already been annotated with this schema, in order to understand cultural analogies and differences in portraying immigrants as threats, enemies or victims.