In recent years, hate speech detection models have achieved significantly improved results, largely due to advances in Large Language Models (LLMs). As a result, research has increasingly focused on more nuanced phenomena, such as the detection of implicit hate and stereotypes. Although the challenge of identifying implicit language has been largely explored, it remains an open issue for state-of-the-art models due to their limited ability to grasp contextual and culturally specific knowledge. In this work, we address the task of identifying stereotypes implicitly encoded in hate speech messages, and propose a method for generating them by leveraging the combined potential of LLMs and Knowledge Graphs (KGs). As a first step, we designed an ontology specifically tailored to represent implicit hate speech. We then populated the ontology using a subset of an Italian-language hate speech dataset, in which targets and implied stereotype statements were manually annotated. The remaining portion of the dataset was reserved as a test set to evaluate the impact of knowledge graph-derived information on LLM-generated stereotypes. For each input sentence, relevant knowledge was extracted from the ontology using SPARQL queries and used to enrich the prompt provided to various LLMs. We compared the results of the knowledge-enhanced approach against those of a baseline few-shot learning approach. Evaluation was conducted using BLEU, BERTScore and ROUGE metrics. Additionally, given the high subjectivity of the task, we performed a manual qualitative analysis on a subset of the model outputs to assess both the quality of the evaluation and the soundness of the generated stereotypes. Warning: This paper contains examples of explicitly ofensive content.
resources. We then populate the ontology using a subset Several approaches have been explored to identify
of an Italian dataset on implicit stereotypes, which com- subtle hate speech, including transformer-based
modprises manual annotations on HS targets, hateful chunks els [
In the following Section (2) we present relevant related English, resources for the Italian language remain limited,
works on detection and analysis of subtle hate speech with only a few datasets addressing the hate speech
phe(2.1), together with graph-based approaches (2.2) to the nomenon in depth. Notable studies [
StereoGraphUnveilingStereotypes ous tasks.
Zhao et al. [21] propose MetaTox, a RAG-based
approach that integrates a meta-toxic knowledge graph In this work, we aim to perform the task of implicit
with LLMs for hate speech detection. First, LLMs are used stereotype generation using LLMs, comparing a
baseto construct the KG by combining data from three En- line approach with a KG-enhanced alternative. Given a
glish datasets. Then Qwen and LLaMA3.1 are prompted sentence and its associated hate speech target, the model
to classify tweets as toxic or non-toxic. The authors is prompted to generate the subtle stereotype that
condemonstrate that the MetaTox method enables to re- tributes to the message’s hateful nature. In the
followduce false positives, leading to better generalization and ing sections, we briefly present the proposed pipeline
reduced hallucinations from LLMs. Lin [
In the context of implicit hate speech, Yadav et al. generate more accurate and transparent stereotypes.
[
Compared to the reviewed literature, our approach represents a step forward, particularly in the area of Italian language hate speech detection. While most prior work has focused on the detection of implicit hate speech, our study shifts the emphasis toward the explanatory capabilities of LLMs, specifically investigating how these can be enhanced through the integration of structured knowledge. Furthermore, by focusing on stereotypes and adopting and hybrid evaluation approach (automatic and human-based), our work also provides valuable insights into the ability of LLMs to uncover sound and coherent stereotypes from implicit language, as well as into the reliability of the evaluation metrics used.
leveraged the Open Stereotype Corpus2 [34] containing 3,578 Italian tweets collected between October 2018 and June 2019 from the Contro l’Odio dataset [35]. The dataset was annotated by five diferent annotators. For each message, the annotators identified the specific chunk (trigger) containing the hate content, the implicit stereotype (if present) and the stereotype cluster (a more general class aiming at creating a stereotype categorization). In the original dataset the authors automatically distinguished between agent and patient parsing each rationale, we chose to simplify this distinction aggregating the two columns under a unique class named "target". An example of the dataset structure along with a subset of annotations is presented in Figure 3. From the dataset
two subclasses: Group and Person. These subclasses represent diferent types of targets and are connected to specific situations via the hasTarget relation, which links a message to its corresponding target. The class Type is designed to provide a taxonomy for both targets (e.g., racial target, religious target) and stereotypes (e.g., ‘are dangerous’, ‘are unclean’). The ontology was subsequently populated using SPARQLAnything4 [37] leveraging the datasets described in the previous section as data source. After this process we obtained a knowledge graph containing triples as to the followings: ster:_803176483174780929 rdf:type dul:Situation ; rdfs:label "Forza ragazzi, 180mila clandestini all anno, rom da tutte le parti, illegalita totale, Coop rosse e bianche che lucrano. ora sapete cosa votare" ; dul:hasTarget ster:immigrati ; ster:hasStereoManifestation ster:180mila-clandestini
-allanno ; ster:hasStereotype ster:invadendo-italia .
For the ontology design process we adopted a fully manual approach to ensure the quality of the resulting re- ster:invadendo-italia source through several means: aligning it with foun- rdf:type ster:Stereotype ; dational ontologies and related semantic resources, en- srtdefrs::hlaasbTeylpe ""SionnvoaIdnevnadsooirtia"l.ia" ; suring the conceptual correctness of the defined classes, and minimizing the potential introduction of bias. The ster:immigrati rdf:type foaf:Group . ontology includes four top-level classes: Situation, Stereotype, Agent, and Type. The class Situation is aligned with the homonymous class from the foundational ontology DOLCE [36]. Its purpose is to link a given target and its associated stereotype to a specific occurrence, such as a Twitter post, in order to avoid the introduction of bias or overly generic statements about stereotypes. The class Stereotype captures the implicit assertions conveyed in a given sentence. The class Agent, aligned with the FOAF (Friend of a Friend) ontology3, has
ID ster:_803176483174780929, is an instance of the class Situation. It has a specific content, expressed trough the relation rdfs:label, and it is associated to a specific stereotype chunk trough the relation ster:hasStereoManifestation. The tweet is then associated with a particular target, ster:immigrati, as well as a stereotype, ster:invadendoitalia. The stereotype is then defined as an instance of the class
SELECT ?s ?stereotype
WHERE {{ ?s a dul:Situation ; dul:hasTarget <{target_uri}> ; ster:hasStereotype ?stereotype .
ster:Stereotype and linked to a specific cluster their ability to understand the subtle stereotype embedSonoInvasori through the relation ster:hasType. ded in the message. We selected these three distinct LLMs because they are state-of-the-art, multilingual, open-source models with comparable architecture and 4. Experiment Setting medium scale size.
The task is conducted in the Italian language. For the In the next sections, the experimental setting is presented. baseline, we used a few-shot learning approach and for The following approach consists of three main steps: the prompt construction we adopt a vanilla structure Knowledge retrieval, where relevant information is re- setup; the prompt is written in Italian. Additionally, it trieved from the KG (Section 4.1); Prompting, where three includes instructions on how to structure the output senmodels are prompted using both a few-shot baseline and tence, explicitly asking the models to generate output in a few-shot KG-enriched approach 4.2; and Evaluation (4.3), where the results are assessed using both automatic tkhneofworlemdagte[-esnuhbajneccetd]ap[parroea/chdoin]co[rpproerdaitecsaatep]ro. mThpet metrics and manual evaluation. containing information about the target entity from the KG. For each target, we associate the relevant retrieved 4.1. Knowledge Extraction stereotypes. The full prompt is presented in Appendix For every sentence of the test set we extracted relevant A. The output produced by the LLM was preprocessed knowledge from the Knowledge graph leveraging the before the evaluation, removing generic elements profollowing SPARQL query: vided by the LLM, such as the usual formulaic closing statements (e.g., asking if it can assist further). models
and explore
BERTScore [42] and ROUGE [43]. BLEU measures how }} many n-grams in the generated text appear in the reference text, focusing on precision and penalizing very short outputs. ROUGE focuses on recall, checking how much
Using this query we were able to retrieve all the stereo- of the n-grams or sequences of the reference text aptype associated with a certain tweet that has the specified pear in the generated text, often used for summarization. target. For example using "immigrati" as target we are BERTScore compares the generated and reference texts able to extract triples like the followings, in which the using deep contextual embeddings from BERT, capturing ifrst element is the ID, the second the gold stereotype semantic similarity beyond exact word matches. and the third hateful span: Since recent studies [44, 45] have highlighted the limister:_id sono-irregolari clandestini-musulmani tations of automated evaluation methods and some scholster:_id non-rispettano-legge nn-amano-subire-le ars [46, 47] are beginning to emphasize the potential -nostre-leggi-sti-migranti of hybrid approaches and aware of the fact that stereoster:_id spacciano immigrati-spacciatori-e- types are characterized by high subjectivity and culturestupratori specific variation, we conducted also a human-based evalSince our goal is to prove that this integrated informa- uation to better understand the reliability of the metrics tion could improve implicit stereotype generation, we used in such contexts. To this end, we designed a twofold rely on the gold-standard targets provided in the dataset. manual assessment.
This avoids the noise introduced by potential errors in First, aiming at replicating the automatic evaluation, target prediction. One limitation encountered is the over- we compared the generated (baseline and KG-enhanced) representation of certain targets, which appear with a outputs with the gold standard, assessing semantic and high number of samples. To reduce the impact of the “lost syntactic similarity on a scale from 1 to 5. The annotain the middle” phenomenon [38] and to balance the quan- tors were presented with the following questions: (Q1) tity of information, we randomly sample 20 stereotypes To what extent, on a scale from 1 to 5, is the baseline per target. output semantically and syntactically comparable to the gold standard? (Q2) To what extent, on a scale from 1 to 5, is the KG-enhanced output semantically and syntacti4.2. Prompt Construction cally comparable to the gold standard? Then, aware that to stereotypes are not universally shared, and that some are more prevalent or culturally dependent than others [48], we aimed to evaluate the extent to which the generated
LLaMA-3.1-8B, gemma-2-9b-it [39] Mistral-7B-Instruct-v0.2 [40] to
Gold I rom sono trufatori
Baseline I rom sono falsi invalidi per commuovere.
KG rom_sinti sono ladri i migranti non sono profughi gli avvocati pagano i migranti.
gli immigrati sono criminali i migranti sono criminali gli immigrati sono violenti gli immigrati delinquono stereotype might be culturally recognizable from our own perspective as white Italian researchers aged between 25 and 30. The evaluation of generated stereotypes was conducted only on content produced by the baseline model, as the KG-enhanced method provides the model with additional contextually relevant information. Annotators were asked to assess whether, in their own perspective, the generated stereotype reflects commonly held beliefs or societal biases (Q3). For example, the stereotype "gli avvocati pagano i migranti" ("Lawyers pay the migrants") was judged unrealistic by all three annotators. In contrast, "gli immigrati delinquono" ("Immigrants commit crimes") received two positive evaluations out of three, suggesting that this stereotype may reflect a commonly held bias in the Italian context. The human evaluation was conducted by three annotators on a subset of 50 sentences. An example of the conducted manual evaluation is presented in Table 1.
and BERT-based scores. Gemma2 benefits the most, with its BLEU score more than doubling and a big gain in Rouge. LLaMA3.1 and Mistral7B also show consistent, though smaller, improvements. The BERT-based scores 5. Results indicate better semantic relevance with KG. Overall, the KG helps the models produce more accurate and meanIn the next sections the experiment results are provided. ingful results.
While automated methods are eficient, they often lack precision. In contrast, human evaluation ofers greater 5.2. Human-based Analysis contextual understanding but is time-consuming and costly. To balance accuracy and eficiency, we applied an automatic method to the full dataset and selected a smaller subset for manual evaluation.
the baseline and the KG-enhanced method. Each answer was evaluated on the basis of its similarity to the gold standard, the normalized results are presented in Table 3. Furthermore, for the baseline generation only, annota5.1. Computer-Based Analysis tors were asked to assess whether the stereotypes reflect In the Table 2 are presented the result of the genera- commonly held beliefs or communal biases. LLaMA 3.1 tion task comparing the three models across the two the highest average scores for both baseline and KGapproaches, i.e. baseline versus knowledge graph en- enhanced outputs, demonstrating strong overall perforhanced. The Results shows that adding the information mances. Gemma 2 shows lower results across all metrics, from KG improves the performance of all three models, while Mistral7B performs the lowest on both baseline and KG averages. Human evaluation further confirms 5.4. Error Analysis that incorporating knowledge from the graph improves model performance across all models and annotators. In To gain deeper insight into the functionality and limitaaddition, the variation in annotators’ scores highlights tions of our approach, and to identify areas for potential the subjective nature of the task and the challenge of future improvements, we conducted an error analysis achieving consistent judgments. Annotator 2, for exam- on the tweets where the KG-enhanced method showed ple, generally rates outputs higher, particularly for KG- the lowest performance. Overall we observed that errors enhanced responses, while Annotator 3 is more critical. frequently occurred when the input contained named Human-evaluated results confirm the trends observed in entities or subjects that difered from the primary target. computer-based scores (for all the models and the annota- For example, in the tweet: tors the score are higher in the case of the KG-enhanced Finanzia l’invasione degli immigrati: ecco approach), demonstrating how our method improves the la prova. La vergogna di George Soros, model’s ability to explicitly address implicit hate speech "Epnagdlirsohn:"eH" edf’uItnadlisat.he immigrant invasion: here is and suggesting that automatic measures can be informa- the proof. The shame of George Soros, the ’master’ tive for this type of task. of Italy."
Regarding the assessment of the generated stereotype the human evaluation reveals divergence tendency: the KG-enhanced output was: "George Soros finanzia LLaMA shows the average highest scores across the three l’invasione degli immigrati" (English: "George Soros annotators, and the value appears to be high especially funds the immigrant invasion"), while not conceptually according to Annotators 1 and 3. Gemma2 shows a simi- incorrect, this difers from the gold standard:"i migranti lar tendency, especially regarding the annotators 2 and vogliono invadere l’Italia" (English : "The migrants want 3. Finally, Mistral tends to have an overall lower score to invade Italy."). A similar issue occurred in the tweet: about the stereotypes soundness, suggesting that it may produce less biased or not realistic content.
metrics and human judgment, we compared the results of BLEU, ROUGE and BERT Score with human evaluation over a sample of 50 sentences, as seen in Figure 3. The three plots help identify which metric aligns more closely with human evaluation.
From the plots, it is evident that the BERT Score metric (shown in the third plot) correlates more consistently with the annotators’ evaluation, suggesting it is a more reliable indicator of quality for this task. This is due to the nature of BERT score, which leverages contextual embeddings to measure similarity on a semantic level. Conversely, BLEU and ROUGE metrics (depicted in the ifrst and second plots, respectively), which operate more on the lexical-syntactic level, show more variability and several limitations in accurately matching human judgment.
Understanding the relationship between automatic and manual assessment is crucial for contextualizing the values obtained from each metric and evaluating model performance in a meaningful way. The comparison also helps to understand which metrics are more robust and reliable, especially for tasks requiring deep contextual and pragmatic understanding.
non rispetta noi? Che senso ha difendere la loro cultura o presunta cultura quando essi disprezzano la nostra? La ministra Trenta contro Salvini: sbagliato dire che l’Islam è terrorismo English: What’s the point, minister Trenta, of respecting thos who don’t respect us? What’s the point of defending their culture or so-called culture when they despise ours? Minister Trenta against Salvini: it’s wrong to say that Islam is terrorism" The KG-enhanced output was "la ministra Trenta disprezza la cultura italiana." (English: Minister Trenta despises Italian culture.) whereas the gold standard was: "i musulmani vanno contro i valori dell’Occidente" (English: Muslims go against Western values). In other cases, when the model encounters a target associated with a high number of stereotypes, it tends to concatenate many of them into a generic and incoherent output.
In some cases, both the baseline and the KG-enhanced approaches struggle to recognize irony and fail to produce a reliable underlying stereotype. For example, consider the following sentence: #Dimartedi Stasera indottrinamento pro Europa. Alla bisogna sono benvenuti anche gli stranieri. Bravo #Floris, vai a cager English: #dimartedi tonight: pro-Europe indoctrination. If needed, even foreigners are welcome. Well done #Floris, go to hell.
Both the baseline and the KG-enhanced approaches generate the "gli stranieri sono benvenuti" (English: Immigrants are welcome), failing to detect the subtle irony in the original message. (a) BLEU scores compared to all annotators (b) ROUGE-L scores compared to all annotators (c) BERTScore compared to all annotators
Finally, we observed challenges in tweets with com- identifying abusive language. Specifically, we explore plex hypotactic structures and multiple subjects. In such the role that additional information from a knowledge cases, models often fail to correctly identify the primary graph may play in the understanding and generation of target and to produce relevant output. Furthermore, the underlying stereotypes. We compare a baseline few-shot KG-enhanced method tends to generate overly long re- approach with a knowledge-enhanced method, leveragsponses in these situations, which can reduce the coher- ing diferent LLMs. We observed that prompts enhanced ence and precision of the generated content. In summary, with additional information outperformed the baseline the worst-performing examples often occur because the approach. To better assess the reliability of the automatic model misidentifies the target of the hate tweet, lead- evaluation metrics, we also conducted a manual evaluaing to reduced accuracy. However, in many cases, the tion, replicating the task performed by the automatic metmodel still manages to extract a correct implicit message, rics. The human evaluation confirmed the results, showwhich, while diferent from the gold standard, is present ing higher scores for the knowledge graph-enhanced in the tweet. In such cases, the prediction is valid, but approach. While the manual assessment was aligned the reference annotation fails to recognize it as correct. with the automated results, we observed a high degree of variability in the scores. This suggests that evaluating such generated content is inherently subjective and 6. Conclusion can vary based on the annotators’ culture, age, or beliefs. These findings highlight the importance of contextualIn this work, we aim to investigate whether large lan- izing evaluation metrics and recognizing that they may guage models are able to uncover implicit stereotypes carry biases or oversimplify complex phenomena. From embedded in hate speech messages. This task is impor- the error analysis, we observed that the KG-enhanced tant as it helps uncover the subtle content of hate speech approach occasionally struggles to manage the quantity messages and supports hate speech detection models in of information provided, suggesting that further studies are needed to better understand the extent to which such models can efectively integrate additional knowledge.
To sum up, the findings of this research suggest that knowledge graph-based approaches are highly promising, even in the hate speech domain, where they remain largely underexplored.
In this work we focused on the integration of stereotypes, retrieving targets from the gold standard. This allows us to concentrate the analysis on the knowledge insertion process within the LLM, minimizing the introduction of noise. As future work, we intend to test the approach using a state-of-the-art target detection model. Although this may introduce errors due to target misclassifications, it would enable full autonomy for the proposed method and enhance its applicability in real-world scenarios. Target detection methods can also return multiple potential targets in cases of uncertainty, providing a fuller stereotype context for posts that may involve more than one target. While we noticed that diferent stereotypes are associated to the same target, as a future work we may consider an approach based on semantic similarity to select the most contextually relevant stereotypes. This approach could ofer a more focused context for the prompt and reduce the likelihood of model misunderstandings. During the error analysis phase, we identified errors potentially caused by the ‘lost-in-the-middle’ phenomenon. Future work should explore in greater depth how models manage diferent quantities of input information. Finally, it is important to highlight that the manual evaluation we conducted—particularly regarding the cultural shareability of the generated stereotypes, is inherently biased and reflects the perspectives of the researchers involved in this study. As future work, it would be interesting to carry out a large-scale, prospectivist survey to explore the diversity of opinions on stereotypes and to investigate the dominant worldview conveyed by diferent large language models.
We acknowledge that when dealing with hate speech, particularly stereotypes targeting minorities, it is essential to be mindful of the potential of introducing bias or unintentionally amplifying hateful content. We made eforts to control and reduce the presence of bias and to remain aware of its potential introduction. During the experimental phase, we prompted LLMs to generate implied stereotypes, which in some cases resulted in the generation of hateful or ofensive content. The generated hateful content is intended solely to remain within the context of this experimental research. Its occurrence also provides additional insights into how LLMs can produce harmful language despite safety filters. lation, 2002, pp. 311–318. [42] T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger,
Y. Artzi, Bertscore: Evaluating text generation with bert, in: International Conference on Learning Representations, 2020. URL: https://openreview.net/ forum?id=SkeHuCVFDr. [43] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013/. [44] B. Peng, Y. Zhu, Y. Liu, X. Bo, H. Shi, C. Hong,
Y. Zhang, S. Tang, Graph retrieval-augmented generation: A survey, arXiv preprint arXiv:2408.08921 (2024). [45] H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, Z. Liu,
Evaluation of retrieval-augmented generation: A survey, in: CCF Conference on Big Data, Springer, 2024, pp. 102–120. [46] S. Tsaneva, D. Dessì, F. Osborne, M. Sabou, Knowledge graph validation by integrating llms and human-in-the-loop, Information Processing & Management 62 (2025) 104145. [47] G. Faggioli, L. Dietz, C. L. Clarke, G. Demartini,
M. Hagen, C. Hauf, N. Kando, E. Kanoulas, M. Potthast, B. Stein, et al., Perspectives on large language models for relevance judgment, in: Proceedings of the 2023 ACM SIGIR International Conference on
Theory of Information Retrieval, 2023, pp. 39–50. [48] S. T. Fiske, Prejudices in cultural contexts: Shared stereotypes (gender, age) versus variable stereotypes (race, ethnicity, religion), Perspectives on psychological science 12 (2017) 791–799. (baseline: context = ∅, KG-method: context = [[target, hasStereotype, stereotype],. . . ]) // Examples: post = "a roma nuovi campi nomadi e gli italiani abbandonano le abitazioni. e noi abbiamo gente come la boldrini che difende tutto questo." implicit_message = i rom sono pericolosi post = "Papampero a San Vittore e festa del Pd? Te credo! I detenuti votano tutti a sinistra, pure rom e islamici." implicit_message = la sinistra protegge i criminali stranieri // Task: Extract from the following post Declaration on Generative AI