This paper investigates the application of various prompting strategies and Italian-language large language models (LLMs) to extract salient characteristics of gender-based crimes from judicial courtroom decisions. Recognizing the complex linguistic and legal structures inherent in such documents, we evaluate several types of prompting across multiple LLMs fine-tuned or pretrained on Italian corpora. Our approach focuses on identifying key elements such as crime typology, victim-perpetrator relationships, modus operandi, and main motivations behind the crimes against women. We present a comparative analysis of LLM performance on a small set of judicial courtrooms, highlighting the impact of prompt design on the extraction of legally and socially relevant information. The findings demonstrate the potential of prompt engineering to enhance the ability of LLMs to support socio-legal research and policy development in the context of gender-based violence.
rulings in the Italian judicial system. The study’s primary
objectives are firstly to explore the role of prompt
engiIn recent years, large language models (LLMs) have neering in guiding the model’s behaviour and improving
demonstrated remarkable capabilities in a variety of nat- output fidelity and secondly to evaluate the feasibility
ural language processing (NLP) tasks, showing potential of using these extracted outputs to generate statistical
for transforming domains that rely heavily on unstruc- analyses of juridical court rulings. A thorough evaluation
tured textual data [
Despite their general proficiency, LLMs remain largely legal domain. untested in such highly specialized applications where The contributions of this study can be summarised as linguistic nuances and factual accuracy are paramount. follows: The extraction of structured information from legal documents, such as the personal information of the accused, necessitates not only an advanced understanding of the language, but also strict adherence to domain-specific taxonomies and ethical considerations regarding data sensitivity. The anonymised and variable structure of legal texts further complicates this task, necessitating the development of tailored strategies for efective model deployment. Beyond their technical relevance, such advancements are of considerable societal value given their potential to underpin large-scale analyses of sociological and criminological trends.
This work investigates the use of LLMs to automate the extraction of key information from anonymised court • Prompt Evaluation – We performed a systematic evaluation and selection of prompts tailored to a legal taxonomy, identifying the linguistic and semantic limitations that afect model performance. • Empirical Assessment of LLM Outputs – We perform a detailed analysis of model behavior across multiple dimensions of a legal information extraction task, highlighting typical failure modes and model biases. • Data-Driven Legal Insights – We uncover statistical trends in italian criminal justice, while emphasizing the importance of post-extraction validation due to the inherent risks of misinterpretation or hallucination, especially on such anonymised data.
Information extraction Information Extraction (IE) Modern language models are typically trained on vast
is a foundational task in natural language processing amounts of data to capture various linguistic patterns.
that aims to automatically extract structured informa- However, especially in the case of smaller models, the
tion such as named entities, events, and relationships training data is often heavily skewed toward English,
refrom unstructured text. Traditional IE pipelines often sulting in reduced performance on other languages. As
rely on rules or shallow machine learning models [
challenges for information extraction due to its
specialized terminology, complex document structures, and
domain-specific entity types and relationships [ 6, 7, 8].
Recent studies have examined the potential of LLMs for
legal IE tasks [
Legal documents from diferent jurisdictions or legal systems introduce further complications, as they may follow distinct conventions, terminologies, and structural norms, making domain transfer particularly challenging [13]. Most current language models are primarily trained on English-language data, largely sourced from Western, English-speaking jurisdictions (e.g., the United States and the United Kingdom). Research has shown that LLM performance on legal IE tasks can vary significantly between in-domain and out-of-domain contexts, with performance degradation often linked to differences in document formality, legal drafting templates, and jurisdiction-specific clauses [ 14]. The intricate nature of legal texts adds another layer of complexity, as legal terminology and document structures can vary widely across legal systems and languages, necessitating specialized methods for handling non-English legal texts.
Most existing work has focused on English legal
documents. To the best of our knowledge, while some
attempts have been made in the Italian legal domain [
A campaign was designed to study several prompt engineering techniques to optimise the model’s responses to the extraction task. The following prompts types have been investigated: 1. Direct Instruction Prompt: This type of prompt directly asks for specific information or task completion, with clear, unambiguous instructions. It’s straightforward and expects a precise answer. For example: "What is the victim’s name?". 2. Socratic Prompt: This type of prompt encourages Socratic reasoning by asking consequent questions. The goal is to guide the model toward discovering information or coming to conclusions. For example: "What is the victim’s name?" followed by “What is <name>’s gender?”. 3. Structured Prompt: This type of prompt provides a specific framework or format in which the response should be structured. The adopted JSON-like format includes predefined fields into which the information should be extracted. This ensures consistency and organization in the answers. For example: “Extract the following details: {victim_name: ?, victim_gender: ?}”.
extract information out of italian criminal court rulings. 1meta-llama/Llama-3.1-8B-Instruct 2swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA
According to the selected types, 145 prompts have been identifying details, based on the surrounding textual condefined, both manually and by utilising Large Language text, with the goal of filling in the fields marked as “ OMISModels (LLMs)3. SIS”. However, an analysis of the model outputs revealed an overall unsatisfactory quality of de-anonymization. 3.3. Dataset While the models demonstrated certain inferential capabilities, the generated outputs frequently proved to To construct a suitable dataset for our study, 2,000 be inaccurate, incomplete, or contextually inconsistent. anonymized judicial court rulings were extracted from The most critical issues arose in the reconstruction of the DeJure corpus4 based on the presence of references personal names: models frequently suggested names that to specific norms related to gender-based crimes, i.e. Art. were inconsistent with the grammatical gender used in 609-quinquies, art. 572, art. 582, art. 609-bis, art. 609- the text, leading to uncoherent court rulings. For inoctis, art. 609-ter, art. 612-bis of the Italian Penal Code. stance, masculine names have been observed to be used We engaged 5 judicial experts to finally select only those in instances where feminine pronouns or adjectives were judicial court rulings efectively relevant for the consid- employed, thereby compromising the document’s natural ered case study. This targeted extraction strategy was lfow and readability. Furthermore, the models demonemployed to ensure the relevance of the selected court strated inconsistency in the attribution of names throughrulings to the legal domain under investigation. From out the document, frequently assigning diferent names this initial pool, a subset of 1,000 court rulings was sub- to the same individual across multiple mentions. The jected to manual evaluation by legal domain experts. The absence of global coherence indicated a restricted contexexperts assessed each sentence for its appropriateness tual awareness, thereby diminishing the dependability of and relevance, ultimately identifying 865 court rulings the automated procedure. In light of the aforementioned as suitable for inclusion in the final dataset. This process limitations, manual de-anonymization was ultimately ensured both the domain specificity and the quality of deemed the preferred approach in order to ensure both the data used in subsequent analyses. The dataset ob- accuracy and internal consistency. tained has been used for the identification of pertinent The manual de-anonymisation process enabled the information and for the extraction of statistics to finally introduction of specific cases, designed to provide a thormodel the gender-base violence phenomenon. ough and robust evaluation of the models.
Furthermore, in order to assess the ability of the se- Foreign names were introduced to assess the models’ lected models to extract salient information from the ability to handle information that deviates from convencourt rulings, we created a subset of de-anonymisation tional paradigms. The incorporation of such cases into judicial court rulings. This process was aimed at recon- the study was intended to assess the models’ capacity structing the removed/obscured information - such as to process unconventional information and to ensure proper names, places, entities or other identifying ref- consistency and accuracy, even in the presence of eleerences - by relying exclusively on the available textual ments that fall outside the more prevalent data categories content. The de-anonymization process was aimed at utilised during their training. creating a small benchmark for qualitative analysis to Additionally, complex cases involving multiple individcompare the performance of the Italian large language uals sharing the same surname were included to assess models. Specifically, the original anonymised court rul- the models’ ability to disambiguate identities, especially ings have been annotated to introduce pseudo-real infor- in cases where roles difer, such as a victim and defendant mation that the models could extract, in order to simulate with the same surname. This required the models to cora plausible context of application of the model itself. The rectly infer identities based on contextual details. Lastly, de-anonymised court rulings are utilised to evaluate the a case without any personal data was included with the capabilities of the selected models, as well as to identify objective of evaluating the eficacy of the selected modthe most efective prompts for the task of extracting the els in discerning instances wherein the requested data is information included in the taxonomy. notably absent. The inclusion of this particular type of input allows to assess the models’ ability to handle situaDe-anonymisation A subset of anonymized court rul- tions in which information is either completely missing ings was initially subjected to a de-anonymization pro- or deliberately omitted. cess using the considered language models. Each model The de-anonymisation procedure, enriched by was prompted to infer the missing information, such as these particular cases, results in a small dataset of 10 names of individuals, organizations, locations, and other judicial courtroom decisions that is well-suited for the evaluation of the models’ performance in challenging and incomplete scenarios.
definition of a few-shot instruction to ask Chat-GPT to generate new ones. 4www.dejure.it The first dataset (composed of 865 anonymized judicial court rulings) was used to extract statistical insights on gender-based violence in Italian court rulings, while the second one composed of 10 de-anonymized court rulings served to evaluate the models’ ability in the task of automatic information extraction and for the selection of the most promising prompts to adopt for the extraction task.
The understanding of crimes against woman starting from judicial courtroom decisions presents significant challenges, primarily due to the inherent complexity of legal language, which often involves dense, formal phrasing and domain-specific terminology. Additionally, judicial court rulings typically span between 3 to 15 pages (averaging about 21,000 characters, with the longest surpassing 137,000), resulting in lengthy and unstructured documents that demand robust document-level understanding. Compounding the dificulty is the frequent occurrence of multiple crimes described across diferent temporal contexts within a single sentence, requiring ifne-grained temporal reasoning and event disentanglement to accurately identify and extract relevant legal information.
relationships that are useful for the definition of the offence and the relevant entities. The objective is to obtain a complete and valid characterisation of the analysed court rulings. In order to achieve the desired taxonomy, the various classifications defined and proposed by the Istituto Nazionale di Statistica (ISTAT) were adopted and subsequently grouped into categories. Additional information about the identified categories, along with a schematic representation, are reported in Appendix A.
The proposed taxonomy has been adopted in the definition of the prompts for the extraction of salient characteristics of gender-based violence.
A key challenge in prompting models with court rulings is their length in tokens, which can significantly slow down the generation process. Since we query the same model multiple times on the same ruling using diferent prompts, we leverage the decode-only nature of language models by precomputing the key-value cache for each token in the ruling. At inference time, this allows us to avoid redundant computation of internal states during each forward pass.
Each prompt includes a predefined set of labels from which the model is expected to choose based on the extracted information. The model should output at least one label, optionally accompanied by an explanation or the relevant text span. For evaluation, we perform an exact string match between the stripped model output and the set of possible labels.
anonymized subset of court rulings focusing both on model performances and computational requirements.
Furthermore, results analysis allowed for the selection of the most promising prompts.
The selection of prompts played a pivotal role in determining the efectiveness of the selected language models in extracting structured information from juridical court rulings. This phase of experimentation revealed not only the variability in the interpretative capabilities of large language models (LLMs), but also several intrinsic limitations related to prompt design and the models’ generalization ability when confronted with legal language.
Preliminary analyses were conducted on the manually de-anonymised subset of court rulings, which permitted the empirical identification of prompt configurations that were optimally suited to the information extraction task. This experiment was able to shed light on a number of dificulties encountered by the models. In many cases, LLMs exhibited a fundamental misunderstanding of the semantic scope required by the prompt, often retrieving information that, while contextually related, diverged significantly from the specific data fields defined by the taxonomy (e.g., returning descriptive actions instead of categorical labels like profession or relationship type).
One of the primary limitations encountered was the ambiguity in natural language and its impact on the LLMs’ reasoning process. This was especially evident when models were asked to infer information indirectly stated or entirely absent from the text. Instead of indicating the lack of evidence, models frequently hallucinated responses, fabricating plausible but unfounded details. This behavior critically undermines the reliability of extracted data, particularly in legally sensitive contexts.
Another noteworthy limitation was the tendency of models to prioritize certain lexical or structural cues over deeper contextual understanding. This resulted in erroneous classification of attributes such as gender, age, and relationship roles, particularly in complex or nonstandardized sentence structures. Furthermore, despite clear instructions embedded in the prompt (e.g., limiting response length or choosing from a set of predefined options), the outputs regularly violated these constraints by (a) Pie Chart representing the victims’ gender distribution.
79% 19%
2% 52% 19% 29%
Male Female Not Specified Male Female Not Specified including a rationale that justifies the provided answer, revealing the models’ limited capacity for controlled generation. Nevertheless, such an explanation is not only not requested, but is also frequently illogical or based on spurious correlations, thereby accentuating the interpretability issue.
The comparison of the selected prompts demonstrated that the adoption of direct instruction prompts, which explicitly instructed the model to select from provided options or adhere to strict syntactic patterns5, resulted in a substantial enhancement in performance stability. Nevertheless, the more general limitations in comprehension and factual accuracy persist, particularly in circumstances where information is partial or ambiguous.
The statistical analysis was carried out on a set of 607 anonymized judicial rulings. This final number resulted from a filtering process that excluded rulings exceeding (b) Pie Chart representing the culprits’ gender disthe token limits of the models used, as well as those tribution. containing errors introduced during the OCR extraction of the original documents. After applying these cleaning Figure 1: Gender distribution of victims and culprits. steps, 607 out of the original 865 rulings were deemed suitable for analysis.
As discussed in Section 3.1, we focus on the results ob- A similar phenomenon was observed in the data pertained from the best-performing model, Phi-3-Mini (4B), taining to nationality. The majority of individuals idenwhich demonstrated strong performance while maintain- tified as both victims and culprits were of Italian origin ing low computational requirements. All generations are (89% and 90% respectively). A mere proportion of the produced using greedy decoding to allow reproducibility, subjects belonged to minority groups, with Nigerian, Chiwith the maximum number of tokens set to 512. The nese, and Albanian nationals being the most frequently extraction process was guided by the adoption of the mentioned among non-Italian individuals. In some cases prompts selected in the prompt evaluation phase, with (1,3% and 2,1% for culprits and victims), the nationality of the objective of capturing relevant characteristics and the subjects could not be established due to the absence extracting statistics and trends that would encompass of explicit references within the anonymised texts. the entire taxonomy area.
in the gender distribution of both victims and culprits. As shown in Figure 1a, the inferred victims were predominantly female, comprising approximately 79% of the identified cases. In contrast, as shown in figure 1b, the majority of culprits were male, accounting for 52% of the dataset. These figures align with established criminological patterns observed in domestic and gender-based violence cases. A notable proportion of records (19% for victims and 29% for perpetrators) lacked suficient information to determine gender, reflecting the limitations imposed by anonymization and the challenges in automatic extraction.
della vittima? Rispondi con "maschio", "femmina" o "non specificato" which translates to What is the victim gender? Reply with "male", "female" or "not specified" .
Nature of Relationships A thorough analysis of interpersonal relationships indicated that the majority of crimes occurred within familiar or intimate settings. As represented in Figure 2, conjugal relationships were the most frequently identified type of relationship (over 30% of cases), followed closely by cohabiting arrangements (over 21% of cases). These findings underscore the imperative for meticulous examination of domestic environments as pivotal contexts for violent ofences. A small yet noteworthy proportion of cases (around 2% of cases) exhibited ambiguous or non-identifiable relationships, thereby further emphasising the complexity involved in disambiguating personal information within anonymised legal documents, which frequently report such information in an indirect form.
Crime Scene and Modus Operandi The most frequent locations linked to criminal acts were private res21% 50% 15% 24.2%, and 23.9% respectively) emerge as most frequent.
ex-bospyofcruoiebshneoasdybsfir/tigaeinnrtldfsrsi/egniredlxsf-rcieonhdaascbqietuaxa-nsitnpstoaunsceecs/ofurlnlieeiadngeduneti/feimedoptlhoeyrerrelatsitvreangers Typologies of Crime and Motivation The most prevalent ofence detected within the corpus is homicide (around 36% of the cases), constituting over one-third of all analysed court rulings. Other prevalent categories included personal injury, physical assault, and threats (12%, 9% and 7% respectively), which often co-occur with domestic or interpersonal conflict. Finally, in terms of motive, quarrels/futile motives, insanity and grudges (38%,
MUR under the grant ReGAInS, Dipartimenti di Eccellenza 2023-2027 of the Department of Informatics, Systems and Communication at the University of Milano
Bicocca. Declaration on Generative AI During the preparation of this work, the author(s) used ChatGPT (OpenAI) and Grammarly in order to: Paraphrase and reword and Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.