Violence perpetrated to their own partner is a social issue that can take place in diferent forms and in diferent settings (i.e., in person, online). These diferent forms of violence can be circumscribed into two broad categories known as Intimate Partner Violence (IPV) and Cyber Intimate Partner Violence (C-IPV). Social Media and technologies can exacerbate these types of behaviors but some “digital footprints”, such as textual conversations, can be exploited by Artificial Intelligence models to detect and, in turn, prevent them. With this aim in mind, in this paper, we describe a scenario in which the Italian Language Model family LLAmAntino can be exploited to explain the presence of toxicity elements in conversations related to teenage relationships and then educate the interlocutor to recognize these elements in the messages received.
Studies so far have shown that one of the most common types of violence is the one committed towards their own partner, namely intimate partner violence. Due to the high rate of these behaviors in society, their early detection can be useful in reducing them. A fruitful way to reach this goal is by building AI models to discriminate against possible violence-related behaviors. Indeed, the identification of these behaviors can be problematic for victims due to the nature of the relationship with their perpetrator. In fact, people continue to hold disbelief concerning romantic engagement, which can turn into acceptance of harmful behaviors. Therefore, having a tool that can help in identifying possible violent behaviors could serve as a preventive measure for the exacerbation of harmful situations. In particular, we propose the adoption of Large Language Models (LLMs) to explain the presence of toxicity elements in a dataset of conversations related to teenage relationships. We are convinced that this novel approach, which provides the reasons why a message represents violence, can educate the interlocutors and promote partner violence prevention.
The paper is structured as follows: in Section 2, we provide a frame of what is intimate partner violence, the diferent forms, and the deleterious intra and interpersonal consequences.
In Section 3, we briefly describe the LLM we adopted in our scenario. Section 4 focuses on the task of explaining toxic language in the context of IPV. We describe the dataset and the diferent types of annotations provided by researchers in General Psychology, as well as the prompting strategy adopted to instruct the language model. Finally, in Section 5, we draw some conclusions and discuss directions for the continuation of the work.
In 2023, the World Health Organization’s (WHO) report underlined an increasing rate of women’s death due to intimate partner violence, almost 5% higher than the one detected in 2017. Indeed, intimate partner violence does not occur only in terms of physical violence (e.g., violence that exacerbates until victims’ death) but also in other multiple forms and it is not related only to women but can be perpetrated towards men as well. Intimate partner violence has been defined as all forms of abuse and/or aggression performed by a partner to their own partner[1]. Consequently, four patterns of categories can be identified (i.e., physical violence, sexual violence, psychological violence, stalking, monitoring, and control) [2]. Each of these categories corresponds to specific violent behaviors which have been shown to change in their duration and severity[3]: • Physical violence concerns the use of force to intentionally harm and injure the partner; • Sexual violence refers to sexual acts or advances carried out without the victim’s consent; • Psychological violence corresponds to communication with the aim of detrimentally impacting the partner’s mental and emotional well-being and exerting control over them; • Stalking, monitoring, and control consists of persistent and unpleasant attention and communication inducing fear or concern about personal safety. traits) correlated to the perpetration of both in-person and cyber IPV, and the detrimental consequences for victims [2, 5, 6]. In light of the detrimental consequences for victims of IPV and C-IPV, an imperative issue is trying to early detect these violent behaviors with the final goal of preventing their escalation. (C)-IPV detection can be problematic for victims because they are victims of their own romantic partner. In other words, being emotionally attached to the person who is committing violent acts towards themselves can reduce victims’ ability to recognize such violent behaviors. Consequently, automatic detection of IPV and C-IPV behaviors can greatly help people in objectively identifying toxic and violent relationships and disengaging from them. This is the main motivation for our work: we propose the adoption of an LLM as an "assistant" being able to explain why a message, in the context of an intimate relationship, can be toxic. The explanation makes partners aware of the fact that violence is being committed or sufered and describes the reasons for this happening, as well as the consequences (for example, emotional sufering), with the hope that it can act as a deterrent.
scenario. LLMs have proved their ability to excel in a large number of areas in the field of Natural Language Processing and also show good performance in solving tasks on which they have not explicitly been trained on [7, 8]. Notable examples of State-of-the-Art LLMs are Moreover, the rising use of technologies has facilitated surely represented by OpenAI’s ChatGPT [9], Meta’s the escalation of the above described violent behaviors LLaMA [10], BLOOM [11] and Mistral [12]. such that scholars have coined new forms of IPV as- However, training these models requires an outstandcribed to the so-called Cyber Intimate Partner Violence ing amount of computational resources and data for (C-IPV)[4]. C-IPV shares the same characteristics as the training phases. This last requirement is particuIPV but occurs through the use of technologies or in larly tricky in the case of languages other than English, cyberspace. Recurrent behaviors of C-IPV perpetrators which are known to be underrepresented. For the Italinclude cyber sexual violence, cyber psychological vio- ian language, there are other models in the literature, lence, and cyber stalking, monitoring and control. Pre- such as Camoscio [13] and Stambecco [14], both LLaMA cisely, cyber sexual violence includes pressuring partners instruction-tuned models, Fauno [15], a conversational to send sexual content, coercing partners into sexual Baize model and finally Cerbero [ 16], a Mistral-based acts, and sending unwanted sexual content. Cyber psy- model. All these models release few trained weights and chological violence involves using technology, such as do not exceed 13 billion in parameters. pictures, videos, and text messages, to cause emotional LLaMAntino [17] is a family of LLMs that, starting harm to partners, such as spreading rumours or insulting from the pre-trained weights of LLaMA 2, were further partners through text messages. Finally, cyber stalking, refined for comprehension and text generation in the Italmonitoring and control behaviors correspond to access- ian language. The LLaMAntino training pipeline follows ing electronic devices and accounts without permission two main steps: the first one is represented by language to monitor their partner or have information on them. adaptation, which allows a predominantly English model The majority of studies carried out so far provided useful like LLaMA to adapt to the Italian language. The second information on the characteristics of these phenomena, step consists of fine-tuning the model to further improve their prevalence, individual diferences (e.g., personality its capabilities on specific tasks. Currently, the models composing the LLaMAntino family are the following: • LLaMAntino-Chat models based on the
LLaMA 2-Chat versions1 with language adaptation for Italian and further fine-tuning (7B, 13B, 70B). • LLaMAntino models based on the LLaMA 2 versions2 with language adaptation for Italian and instruction-tuning (7B, 13B, 70B).
The idea is to create a dataset of toxic conversations annotated with information about the type of violence (e.g., physical, cyberstalking, cyber sexual violence), the presence of aggressive communication, the adoption of abusive language and, in general, with information that could be useful to provide a "technical" explanation, as if were given by a professional expert in the subject, such as a psychologist. The aim is to provide explanations, well grounded on relevant CIPV literature, that point out the elements of toxicity in the conversation. Therefore, we started from a dataset available on HuggingFace [18], which contains sentences classified as toxic or healthy, referring to teenage relationships. We extended the dataset by adding specific annotations related to CIPV to sentences classified as toxic. Then, we elaborated on the annotations to obtain an explanation that can be used for Few-shot prompting. The following subsections provide details on the dataset, annotation, and experiments.
The original dataset “toxic-teenage-relationships” was created to help in eforts to identify and curb instances of toxicity between teenagers[18]. It consists of 334 sentences collected by 8 teenagers (4 males and 4 females) of Spanish nationality aged between 15 and 19, who were appropriately instructed on interpersonal relationships to be classified as toxic or not. The group of teenagers had two weeks to collect Spanish language sentences that they spoke or heard in their environment either through interpersonal communication or via social media. Afterwards, the examples given by each student were discussed and evaluated by the others, using peer evaluation.
The classification (toxic or non-toxic) was also approved 1https://huggingface.co/meta-llama/Llama-2-7b-chat 2https://huggingface.co/meta-llama/Llama-2-7b by two specialists in the field. No personal or sensitive information has been recorded. As a general rule, if words associated with swearing, insults or profanity appear in a comment, it is likely to be classified as toxic, regardless of the author’s tone or intention, e.g. humorous/self-critical.
After classification, 165 sentences have been considered as toxic. With the aim of evaluating our Italian LLM, sentences have been translated into Italian by using two translation services (Google and DeepL). We added 5 of annotations: • the type of violence: physical or cyber; • the type of behavior that led to the physical vio
lence, e.g. sexual assault, stalking; • the type of cyber behavior that led to the violence,
e.g. cyber stalking; • the type of communication: aggressive or
non-aggressive; • the type of aggressive communication: e.g., use
of abusive language.
As for physical violence, the experts distinguished 4 annotations [2]: 1. physical violence: the voluntary use of force that
potentially causes harm and injury to the partner; 2. sexual violence: sexual acts without the partner’s
consent, even if only attempted; 3. psychological aggression: communicating with the intention of negatively influencing the mental and emotional state of the partner and wanting to control him or her; 4. stalking, monitoring and control: series of recurring and unwanted attentions and communications that create fear or apprehension and put the partner’s safety at risk.
As for cyber violence, the experts distinguished 3 annotations [6]: 1. cyber sexual violence: requesting or pressuring the partner to send sexual content against his or her will, pressuring the partner to engage in sexual acts; 2. cyber psychological violence, aggression: behavior to cause emotional distress to the partner; may include behaviors such as spreading gossip on social media, repeatedly insulting the partner via messages, even spreading videos or photos that cause emotional distress; 3. cyber stalking, monitoring, and control: using and accessing technological devices and accounts without the partner’s consent, use of technology to get information about your partner, in general any behaviours that aim at increasing control within the relationship). It includes fraping, that is the alteration of the partner’s information on social profiles.
As for aggressive communication, the experts distinguished 5 annotations [19]: "If you have nothing to hide, give me your phone" 1. curses; is: "The sentence is toxic because it is an example of cyber 2. ridiculousness or derision; violence. The behaviour falls in the category cyber 3. bad language; stalking, monitoring, and control since the 4. threat; aim is to obtain information on the partner’s life and estab5. attack on the person (on competence, character, lish a dynamic of control in the couple. Furthermore, the background, physical appearance).
At the end of the annotation phase, we had each toxic sentence annotated with information well-grounded in scientific literature about intimate partner violence. An example of a toxic sentence that reveals physical violence is: "Tu non sei niente senza di me" ("You are nothing without me", in English)
That sentence has been annotated in the dataset as follows: communication is aggressive because it reveals the intimidating intent of attacking the partner to violate his or her privacy." We built a 2-shot prompt by including: • the description of the task: "given a sentence from a conversation between partners in an intimate relationship, explain the reasons why the sentence expresses toxic language and represents a case of physical or cyber violence"; • 2 training toxic sentences with corresponding
explanations; • 1 test toxic sentence (without explanation) for which we want the model to generate an explanation. • type of violence: physical • type of behaviour: psychological aggres
sion • aggressive communication: yes • type of aggressive communication: derision, attack on the person
sentence were the canvas for writing the explanation included in the prompt. Therefore, we created 10 2shot prompts, as described before, by using the 30
An example of a toxic sentence that reveals cyber vio- sentences extracted from the dataset. The aim of lence is: the experiment was to assess whether the annotations actually help in explaining the reasons why a mes"Se non hai nulla da nascondere, dammi il sage is classified as toxic. The model evaluated in telefono" ("If you have nothing to hide, give our experiment was: LLaMAntino-2-Chat-13B-hfme your phone", in English) UltraChat, LLaMAntino-2-Chat for brevity3. Therewhich has been annotated in the dataset as follows: fore, we wanted to assess whether the model learns how to perform the task, by providing it with just two ex• type of violence: cyber amples. We compared qualitatively the explanations • type of behaviour: cyber stalking, given by LLaMAntino-2-Chat, when instructed by 2monitoring, and control shot prompts, with those generated when the model is • aggressive communication: yes prompted just with the task description and the toxic • type of aggressive communication: attack on sentence to be explained ("zero-shot prompting"). The the person experimental protocol was:
We randomly chose 30 annotated toxic sentences for a small, preliminary experiment with Few-Shot Prompting; 20 sentences were used for training, 10 for testing. For each training sentence, the annotations were turned into a natural language explanation used to build prompts for in-context learning. For instance, the explanation for the previous sentence 1. give LLaMAntino-2-Chat the task description and the first toxic sentence to be explained and record the explanation; 2. repeat prompting with the remaining 9 test toxic sentences and record the explanations; 3. give LLaMAntino-2-Chat the 10 2-shot prompts and record the explanations;
had 2 explanations: LLaMAntino-2-Chat 0-shot and LLaMAntino-2-Chat 2-shot. We asked 2 Psychology experts to evaluate independently the two explanations, by answering 3 questions:
UltraChat-ITA 1. Q1: Which explanation is most scientifically based? 2. Q2: Which explanation is more efective in making the partner who sufers aware of the violence? 3. Q3: Which explanation is most efective for educational purposes to make both partners aware that violent behavior is taking place?
city with an adequate level of efectiveness, according to the 2 experts, but annotating sentences with information Explanations were presented in pairs. To avoid bias, ex- useful for few-shot prompting does not bring benefits perts are not aware of which training provided the ex- on the explanations. This outcome might depend on the planation. Furthermore, the presentation order was ran- LLM used, as well as on the prompting strategy. Theredom: sometimes the LLaMAntino-2-Chat 0-shot was fore, we plan to extend the experiment, obviously by presented before LLaMAntino-2-Chat 2-shot, some- increasing the size of the test set, comparing the results times the order was reversed. For each question, we with another LLM, using Chain-of-Thought Prompting suggested 4 possible outcomes: LLaMAntino-2-Chat to improve the "reasoning" capabilities of the model. 0-shot (anonymized), LLaMAntino-2-Chat 2-shot (anonymized), both, none. For each test sentence, we consider the experts to be in agreement if they gave the 5. Conclusions and Future Work same answer to at least 2 of the 3 questions asked. In general, the expert were in agreement on 6 sentences, The prevalence of violent behaviors highlights the need showing the dificulty of the task of evaluating the quality for prompt intervention and preventive measures. We of explanations, given the sensitivity of the CIPV context. presented our proposal to utilize sophisticated Natural
Some interesting considerations have emerged from Language Processing techniques, including LLMs, to the results reported in Table 4.2, that can guide the next identify and describe toxic elements in discussions consteps of the investigation: cerning teenage relationships. By exploiting the proficiency of LLMs in processing and understanding human • no question has ever been answered "none". language, our approach seeks to go beyond just the detecTherefore, we can observe that the model never tion, aiming to grasp underlying motivations and factors showed hallucinations or gave inappropriate an- contributing to the emergence of harmful behaviours. swers. Of course, further testing will be necessary In future works, we intend to perform fine-tuning steps to generalize this statement; to better adapt LLMs to the specific task at hand. We also • on Q1, the results suggest that there is no prompt- plan to investigate how diferent pre-training techniques ing strategy that clearly emerges, thus revealing and architectures can be leveraged to enhance model perthat in general LLaMAntino-2-Chat explana- formance. To ensure the efectiveness of our approach, tions are properly based on scientific literature, we intend to confront our methodology with other modregardless of the prompting strategy; els and incorporate further annotations to enhance the • on Q2, the answers show some disagreement robustness and efectiveness of our methodology. This among the experts: one was clearly in favour of involves comparing the performance of our LLMs with LLaMAntino-2-Chat 0-shot, the other showed other state-of-the-art models. a slight preference for LLaMAntino-2-Chat 0- Moreover, we will explore the application of Chain-ofshot. We asked some motivations for the an- Thought prompting techniques, with the help of expert swers and it emerged that some explanations psychologists. This involves using prompts to guide the given by LLaMAntino-2-Chat 2-shot were neg- LLM’s decision-making process, with the goal of encouratively influenced by grammatical errors; aging the model to provide more detailed and grounded explanations for its choices. By working closely with etal impact of large language models, arXiv preprint experts in this area, we hope to gain valuable insights arXiv:2102.02503 (2021). into how these techniques can be best applied and re- [8] Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, ifned. We plan also to extend the datasets with further A. Li, M. He, Z. Liu, et al., Summary of chatgptannotations that provide more details about the language related research and perspective towards the future adopted (e.g. references to gender stereotypes or use of of large language models, Meta-Radiology (2023) particular linguistic structures), with the aim of building 100017. more complete prompts. [9] OpenAI, Gpt-4 technical report, 2023. arXiv:2303.08774. [10] H. Touvron, T. Lavril, G. Izacard, X. Martinet, Acknowledgments M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efiWe acknowledge the support of the PNRR project FAIR cient foundation language models, arXiv preprint - Future AI Research (PE00000013), Spoke 6 - Symbiotic arXiv:2302.13971 (2023).
AI (CUP H97G22000210007) under the NRRP MUR pro- [11] B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, gram funded by the NextGenerationEU. This Publica- S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, tion was produced with the co-funding of the European F. Yvon, et al., Bloom: A 176b-parameter openunion - Next Generation EU: NRRP Initiative, Mission 4, access multilingual language model, arXiv preprint Component 2, Investment 1.3 - Partnerships extended to arXiv:2211.05100 (2022). universities, research centres, companies and research [12] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. BamD.D. MUR n. 341 del 15.03.2022 – Next Generation EU ford, D. S. Chaplot, D. de Las Casas, F. Bres(PE0000014 - ”SEcurity and Rights In the CyberSpace - sand, G. Lengyel, G. Lample, L. Saulnier, L. R. SERICS” - CUP: H93C22000620001). Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, References CoRR abs/2310.06825 (2023). URL: https://doi.org/ 10.48550/arXiv.2310.06825. doi:10.48550/ARXIV. [1] M. E. Bagwell-Gray, J. T. Messing, A. Baldwin- 2310.06825. arXiv:2310.06825.
White, Intimate partner sexual violence: A review [13] A. Santilli, E. Rodolà, Camoscio: an of terms, definitions, and prevalence, Trauma, Vio- italian instruction-tuned llama, 2023. lence, and Abuse 16 (2015) 316–335. arXiv:2307.16456. [2] M. Breiding, K. C. Basile, S. G. Smith, M. C. Black, [14] Michael, Stambecco: Italian instruction-following R. R. Mahendra, Intimate partner violence surveil- llama model, https://github.com/mchl-labs/ lance : uniform definitions and recommended data stambecco, 2023. elements. version 2.0, 2015. URL: https://stacks.cdc. [15] A. Bacciu, G. Trappolini, A. Santilli, E. Rodolà, F. Silgov/view/cdc/31292. vestri, Fauno: The italian large language model [3] J. Spluska, L. Tanczer, Threat Modeling Intimate that will leave you senza parole!, arXiv preprint Partner Violence: Tech Abuse as a Cybersecurity arXiv:2306.14457 (2023).
Challenge in the Internet of Things, Emerald Pub- [16] F. A. Galatolo, M. G. Cimino, Cerbero-7b: A leap forlishing Limited, 2021, pp. 663–688. ward in language-specific llms through enhanced [4] L. Gilbert, X. Zhang, K. Basile, M. Breiding, M.- chat corpus generation and evaluation, arXiv j. Kresnow, Intimate partner violence and health preprint arXiv:2311.15698 (2023). conditions among u.s. adults —national intimate [17] P. Basile, E. Musacchio, M. Polignano, L. Siciliani, partner violence survey, 2010–2012, Journal of In- G. Fiameni, G. Semeraro, Llamantino: Llama 2 modterpersonal Violence 38 (2023) 237–261. els for efective text generation in italian language, [5] K. N. Duerksen, E. M. Woodin, Cyber dating abuse arXiv preprint arXiv:2312.09993 (2023). victimization: Links with psychosocial function- [18] Margarita Martínez Gabaldón, toxicing., Journal of Interpersonal Violence 36 (2021) teenage-relationships (revision 5ce5df0), NP10077–NP10105. 2023. URL: https://huggingface.co/datasets/ [6] L. Watkins, R. Benedicto, D. DiLillo, The cyber marmarg2/toxic-teenage-relationships. aggression in relationships scale: A new multidi- doi:10.57967/hf/0972. mensional measure of technology-based intimate [19] D. A. Infante, C. J. W. III, Verbal aggressiveness: partner aggression, Assessment 25 (2018) 608–626. An interpersonal model and measure, Communidoi:10.1177/1073191116665696. cation Monographs 53 (1986) 61–69. doi:10.1080/ [7] A. Tamkin, M. Brundage, J. Clark, D.-f. Ganguli, Un- 03637758609376126.
derstanding the capabilities, limitations, and soci