Introduction

Artificial intelligence systems, now employed across a wide array of fields, can also serve as valuable aids for legal practitioners. Increasingly sophisticated tools enhance information search capabilities, automate the drafting or verification of legal documents, and facilitate technical evaluations, such as predictive justice. Utilizing such tools can yield significant benefits by enhancing the efficiency and quality of legal processes. In civil and common law systems, accessing legal judgments to retrieve legal decisions is essential for various legal tasks, including defending clients, constructing cases for prosecution, and issuing judicial decisions. In Italy, to ensure widespread information on the courts' decisions, for this purpose, a dedicated body, the Ufficio del Massimario, was established, whose purpose is to produce massime. In a concise yet detailed manner, these summaries (massime) encapsulate the legal principles articulated in judgments. Hence, legal professionals can consult these massime instead of delving into the entirety of legal decisions. The task of summarising legal texts and producing massime has been widely addressed in the last years [1], especially with the advent of the Generative AI [2,3]. Given the complexity of the task, the approach outlined in [1] focuses on handling the automatic production of a massima as an extractive summarization task. This involves extracting the most pertinent part of the judgement to assist in the drafting of the massima by the des-ignated office, utilizing a human-in-the-loop approach as discussed in [4]. The process of analyzing judgments and extracting relevant sentences can be significantly simplified through the use of pre-trained models [5,6]. These models function as versatile universal sentence/text encoders, capable of addressing a range of downstream tasks, including summarization [7]. These models consistently outperform other approaches, particularly after fine-tuning or domain-adaptation [8]. Despite the success of pre-trained transformers and LLMs in other summarization tasks [9], certain phenomena, such as hallucination in the generation of the text [10], the task of producing massime is still challenging for current extractive and abstractive summarization systems. Additionally, legal texts are often extensive, further increasing the summarization task's complexity. Identifying the portions of the text that contain the relevant information to be reported in the massime becomes challenging due to their length [11]. In this paper, we present an approach to producing an extractive summary by exploiting the ability of an LLM to generate abstract summaries from a document. Our approach selects, from the abstract, the sentences that best match the sentences in the source document. This approach, described in Sec. 2, reduces the hallucination phenomena, achieving results in a zero-shot setting, described in Sec. 3 comparable with a model trained with a domain dataset.

Extractive Projection

It is well known that generative models, particularly when used in summarization systems, are prone to hallucination phenomena (see [12] and references therein). In this case, new terms or, in worst scenarios, even informa- tion and facts not present in the original document are generated in the output summary. Several attempts have been made to try to tame such unwanted behaviour (for instance, see [13]), which may lead to serious problems in sensitive domains. Given its specific lexicon, the vast amount of fixed forms and judicial references, the legal domain is very delicate and unsuitable for a straightforward application of generative systems. To overcome such a problem, we introduce what we refer to as extractive projection, meaning, a transformation mapping a generated text into sentences of the original document.

Model

Defining the source documents 𝑑 ∈ 𝐷, the abstractive summary as 𝑎 ∈ 𝐴 with 𝐷, 𝐴 respectively the space of documents and abstractive summaries, the summarization prompt 𝑝 ∈ 𝑃 . The generative summarization transformation is defined as:

𝐺 : 𝐷 × 𝑃 → 𝐴 𝑎 = 𝐺(𝑑|𝑝) .(1)

We introduce the extractive summary as 𝑎 ′ ∈ 𝐴 ′ ⊂ 𝐴, and the extractive projection

Γ Γ : 𝐷 ˜× 𝐴 → 𝐴 ′ 𝑎 ′ = Γ(𝑑 ˜; 𝑎) ,(2)

where 𝑑 ˜∈ 𝐷 ˜, and 𝐷 ˜is the space of segmented documents (i.e., containing the same documents as 𝐷, but each one is split into a set of segments).

The projection Γ used in this work is a slightly modified version of the algorithm proposed in [7] to pre-process the data. As a main difference from [7] we allow the algorithm to select up to all the segments present in the document, without any parameter fixed a priori. Moreover, while in [7] this greedy selection algorithm is used to obtain an oracle summary for each document used as a reference to train the extractive model, here this algorithm is used to project the (abstractive) generated summary into the segments of the original document. Note that such procedure completely removes by construction any possibility of hallucination since the projection cuts off all possible novelties and generations. The greedy selection procedure employed is then simply a combinatorial optimization algorithm based on coverage metrics. In this respect, we tested several metrics, ranging from the average of ROUGE-1 and ROUGE-2 [14] as originally proposed in [7] to different linear combinations of Rouge-n and more sophisticated similarity metrics (e.g., BERTscore [15]. We observe that with the exception of very rare cases where the generated summary is produced in a different language with respect to the original document, all the coverage metrics produce accurate results (see Tab. 1).

In the multilingual setup, only a similarity metric based on multilingual embeddings, which is insensitive to language shifts, produces reasonable results, while ROUGE does not work correctly.

Results

As discussed in Sec. 1, we trained and tested the extractive summarization systems introduced in [1] on a dataset composed by judgments and massime from different courts 1 . Starting from a whole dataset of 1340 couple of (judgement, massima), we randomly selected a subset of 199 of them as a validation set, 940 as a train set, and the remnant 201 as test set. The latter has been further refined down to 61 "high quality" examples.

Table 2

Prompts used for generic summarization (𝑝 1 ) and domain tuned task summarization (𝑝 2 ).

selection, we first used the greedy algorithm proposed in Sec. 2 based on the average of ROUGE1 and ROUGE2 and then selected only data with that value larger or equal to 0.6 (see Fig. 2). The scores for the extractive model Ext, compared with the Oracle and those produced using generative models, are collected in Tab. 1. More specifically, we used two different prompts 𝑝1 and 𝑝2 (for details see Tab. 3) to estimate the effect of a "generic" summarization prompt, with a "task tuned" prompt specifically referring to the features of a massima [16]. As expected, we observe a small improvement in scores with all generative models using 𝑝2 over 𝑝1. Moreover, we compare the scores of a straightforward abstractive summarization Abs, with the setup proposed in this work, i.e., including the extractive projection called Gen-Ext in Tab. 1. For all the evaluations, we used a generative model of the gpt-turbo [17] family 2 . Interestingly, the scores obtained using zero-shots (no fine-tuning or contextual examples are involved) generative models, in both their types: abstractive (Abs) and extractive (Gen-Ext), seem to perform reasonably well when compared to the Ext model. An example of the summaries produced in all the setups are displayed in Tab. 3. It is worth noting that the scores obtained in this work should be interpreted only as a reference. They are affected by large statistical fluctuations, which make a direct comparison among the scores very tricky. Moreover, coverage scores are known to have a limited correlation with the effective quality of the summary produced, which requires some human evaluation by domain experts.

Conclusions

In this work we discussed the first results of a novel approach that can be used to obtain "hallucination"-free results out of a generative model. We applied such pro- cedure in a legal domain, where preserving factuality is mandatory.

While obtain only partial results, we find them to be reasonably promising but requiring some further investigation.

A comparison with different "open source" LLMs as generative models, estimating the parameter scaling effects on the performances and a complete or partial (see for instance [18] ) fine-tuning or domain-adaptation is dedicated to future studies. In conclusion, while discounting the difficulty of the task, given both by the inherent complexity of the structure of a massima that cannot be treated as a simple summary, and by the difficult evaluation of the results found, as well as the fact that automatic "token"-coverage metrics require some evaluation by human domain experts, we believe that LLMs, if appropriately applied, can offer a valuable tool even in domains where factuality is paramount.

Summary

Text Target L'inserzione automatica di clausole, prevista dall'art. 1339 cod. civ., costituisce una restrizione significativa del diritto di libertà economica consacrato dall'art. 41 Cost. di cui è espressione l'autonomia privata, e deve quindi trovare il suo fondamento in una legge formale o in un altro atto avente valore di legge in senso sostanziale o da esso richiamato tramite rinvio integrativo; pertanto, con riferimento al contratto avente ad oggetto la prestazione di servizi di vigilanza da parte di un istituto privato, tale integrazione non può aver luogo in base al decreto con cui il prefetto approva la relativa tariffa ai sensi dell'art. 135 del r.d. n. 773 del 1931, trattandosi di un mero atto amministrativo espressivo di un sindacato di congruità, avente natura di merito, del prezzo di una prestazione contrattuale, né l'ingerenza con efficacia reale sulla tariffa proposta dall'Istituto e liberamente accettata dal committente può trovare fondamento nella generale potestà del prefetto di impartire prescrizioni nell'interesse pubblico, in sede di rilascio di autorizzazioni di polizia, ai sensi dell'art. 9 del r.d. n. 773 cit.. L'autonomia privata non può essere derogata da atti amministrativi, ma solo da leggi formali. Il principio dell'autonomia privata è fondamentale e non può essere interpretato estensivamente. Le tariffe prefettizie non hanno efficacia normativa e non possono essere conferite da una presunta convergenza di vedute delle parti. Inoltre, il dovere di solidarietà sociale non può essere utilizzato come motivo di condanna al pagamento del maggior prezzo. La libertà economica può subire restrizioni solo in forza di una disposizione di legge. L'inserzione automatica di clausole ex art. 1339 c.c. riguarda solo i prezzi imposti con legge e non può essere derogata da atti amministrativi. La fedeltà alla decisione giuridica richiede il rispetto di tali principi.

Gen-Ext(p 1 ) -con atto di citazione notificato il 5 luglio 1995 l'istituto cooperativo di v. ed i. c. s.r.l. conveniva dinanzi al tribunale di lamezia terme l'istituto nazionale previdenza sociale per ottenerne la condanna al pagamento della somma di l. 639.590.300 a titolo di maggior corrispettivo per il servizio di vigilanza espletato in favore dell'ente, presso la sede di lamezia, dovuto in forza dell'inserzione automatica della tariffa inderogabile emanata dal prefetto di catanzaro in materia di vigilanza, superiore al prezzo pattuito con il contratto stipulato il 25 agosto 1983. dopo l'espletamento di consulenza tecnica d'ufficio il tribunale di lamezia terme con sentenza 23-27 novembre 1999 condannava l'inps al pagamento della somma di l. 1.148.787.862, oltre le spese di giudizio. avverso la sentenza, non notificata, proponeva ricorso per cassazione il fallimento dell'istituto cooperativo di v. e i. 157 del relativo regolamento di esecuzione, nonché l'art. 1175 e 1375 cod. civ. e dell'art. diritto -con il primo motivo il ricorrente deduce la violazione degli artt. 9, 134 e 135 del citato testo unico delle leggi di pubblica sicurezza e dell'art. 1339 cod. civile. con il secondo motivo ricorrente censura l'omessa motivazione nel discostarsi dalla concorde interpretazione delle parti. con l'ultimo motivo il fallimento deduce la violazione degli artt. 2 della costituzione.

Gen-Ext(p 2 ) 1339, cod. civ. 1339 cod. civ. l'inserzione automatica di clausole, prevista dall'art. 1339, cod. civ., costituisce una deroga incisiva al principio dell'autonomia privata e deve quindi trovare il suo fondamento in una legge formale -come testualmente previsto dalla norma -o in altro atto avente valore di legge in senso sostanziale o da esso richiamato tramite rinvio integrativo. il diritto di libertà economica consacrato dall'art. 41 cost., di cui è espressione l'autonomia negoziale delle parti nel modellare il contenuto di un contratto, può soffrire restrizioni solo in forza di una disposizione di legge, insuscettibile di interpretazioni estensive (ibidem, terzo comma). ne consegue la vigenza, in subiecta materia, di un principio di stretta interpretazione dell'art. civ. ; vieppiù giustificato da esigenze di tutela della concorrenza e del mercato, che verrebbero lese da una pratica di prezzi amministrati. l'asserita convergenza di vedute sull'efficacia cogente delle tariffe prefettizie non può, neanche in astratto, valere a conferire loro l'efficacia normativa di cui sono intrinsecamente prive. l'invocazione di un inderogabile dovere di solidarietà sociale che avrebbe imposto la maggiorazione del prezzo non ha, infatti, alcuna attinenza con l'operatività dell'eterointegrazione ex art.

Table 3

Example summary comparison among the various summarization systems proposed.