This paper provides an overview of the investigations being carried out at Scuola Superiore Sant'Anna on the use of Artificial Intelligence techniques for automated extraction of rhetorical roles and legal holdings from Italian case documents. These activities are framed within the ”Giustizia Agile” project funded by the Ministry of Justice, aiming at improvements to the eficiency of the Italian justice system, making use of advanced information technology means, among others. ∗Corresponding author.
1. Introduction
has an impact on the social and economic life of citizens. cal factors addressing adversely the duration of judicial
more eficient and in line with other European countries.
was reported as being among the countries with the least
eficient judicial system, with more than 500 days needed
for the first sentence, 800 days for the appeal and
reaching up to 1300 days for final judgments by the Supreme
Court [
nEvelop-O (D. Licari); 0000-XXXX-XXXX-XXXX (G. Marino); wider initiative to enhance the performance of judicial ofices, aiming at significant reductions of the backlog, by investigating on finding the major bottlenecks and critiprocesses; investigating the opportunities to add several innovations on the side of management and organization of the processes, as well as to embrace a wider adoption of digitalization of the processes through the use of ICTs.
reporting on some key experimentation being done with the use of Artificial Intelligence tools, and specifically Large Language Models (LLMs), in the area of automated summarization and Rhetorical Role Classification of Italian case documents. We focused on the extraction of legal holdings from Italian administrative justice documents. This activity is carried out by the highest exponents of Italian justice and is crucial to facilitate access to justice, create a ’precedent’ and ensure transparency in decisions. Furthermore, this is a delicate task because lawyers and judges rely on legal holdings to select caserelevant documents when searching for similar cases. Extracting this information from a judgment is a complex task that requires time-consuming eforts and specific combines a rhetorical role classifier, text summarization, and a scalable search engine to accurately and eficiently retrieve and analyze legal holdings from Italian case documents. Identifying the rhetorical roles of the diferent text segments allows for a better understanding of the structure and content of the document, which can help guide the summarization process. Irrelevant information (e.g. introduction) can be filtered out in pre-processing allowing the summary model to focus exclusively on the most important information in the document.
2004. For example, LetSum [
CEUR
Workshop Proceedings (CEUR-WS.org)
Workshop
Proce dings IhStpN:/c1e6u1r3-w-0s.o7r3g
1More information is available at https://www.unitus.it/it/unitus/ the sentences and uses TF-IDF to rank them. A
percentage of sentences for each rhetorical role is then selected which are those that contain the information on the legal
to be a part of the summary. The work [
In our approach, we focused on an extractive method, documents. The documents collected, generated roles,
due to two main reasons: (i) it highlights the most rel- and extracted summaries were given to the data store. A
evant sentences in a given document, constituting an web app will be developed for easier usage.
efective way to speed up a Judge’s work, and (ii)
summarizing long documents is extractive in nature [
In this paper, we propose a platform based on Italian
Legal BERT models [
for each sentence. We used a Hierarchical BERT model for this task. Each sentence is categorized into a single role. Overall, we categorized 5 diferent roles (INTRODUCTION, PARTIES, DEVELOPMENT, REASON, and CONCLUSION). The sentences of REASON are filtered
pronounced it; an indication of the parties and their lawyers; 2. CONCLUSION OF THE PARTIES: the conclusions of the prosecutor (if any) and those of the parties; 3.3. Italian-LEGAL-BERT Holding Extraction 3. DEVELOPMENT OF THE TRIAL: summary of the appealed judgment and reasons of appeal; 4. REASON: the concise statement of the factual and legal reasons for the decision (the statement of reasons); 5. CONCLUSION: the decisional content of the judgment.
Mean-BERT. This approach involves fine-tuning the Italian-LEGAL-BERT model to predict a score for each sentence in a document. The scores for training and evaluation were given by the harmonic mean of Rouge R-1
Both datasets were split 80% for training models and and Rouge R-2 scores (generated by ITA-ROUGE a mod20% for model testing. The data were obtained through ified version of Rouge metric for the Italian Language) scientific collaboration agreements between some Italian between the sentence and the corresponding document courts and the Scuola Superiore Sant’Anna. The ITA- holding. We generate these scores only for the training CASEHOLD dataset will be publicly released. and validation sets.
Since the sentences were already rhetorical role classi3.2. Rhetorical Roles Classification ifed, based on the scores generated, we then only chose the sentences which have the highest importance. The The identification of the roles that diferent text segments higher the score of a sentence, the higher the similarity play in a larger document has been done using a hierar- between the sentence and its corresponding holding. For chical BERT approach, in order to contextualize a single our experiment, REASON was the most important and sentence based on the content of the document. This DEVELOPMENT OF THE TRIAL (DEVELOPMENT) was model is based on a layered architecture whose bottom the second most important. We derived these two were layer is Italian-LEGAL-BERT and whose top layer is a 2 the most important by their scores, 75th percentile of layers transformer encoder. The sentences to classify are these sentences had score of more than 2.5 whereas other tokenized and given as inputs to Italian-LEGAL-BERT. roles were near zero.
The CLS output tokens are then retrieved and fed to the We made two datasets by removing sentences with transformer encoder, which extracts relevant features for other roles, (i) Only with REASON, (ii) with REASON, and each sentence. These features are then processed by a DEVELOPMENT. After getting the scores and choosing simple softmax-based classification layer to get the final only the important sentences of a document, we finepredictions. We will provide precise details about the tuned the Italian LEGAL BERT model with a regression training and performance of this model in further work. head to predict these scores.
1. R-1 and R-2 scores between each sentence and its respective document holding are computed for the training and validation sets. 2. To retrieve a single score out of the R-1 and R2, we computed their harmonic mean for each sentence. 3. Based on these scores and the previously predicted roles from the Rhetorical roles classifier, we chose the most important roles. The higher the score, the higher the importance. Two datasets were created based on this. 4. Italian-LEGAL-BERT was fine-tuned in the regression task of predicting the score for a given sentence. 5. The validation dataset was used to determine the optimal number of top k sentences to compose the final holding. We tried k = 3,5,7 and found that k = 5 yielded the best results.
Model
REASON REASON + DEVELOPMENT
R-1
for each document will also be indexed along with the documents. A tokenization layer on top of the Elasticsearch data store will be added to tokenize the input text. The search engine will be developed with a web app for the judicial people to be able to use it. This eficient retrieval of documents and their holdings might fasten the process of searching through documents.
Apart from search, Elasticsearch can also be used for analyzing data. This will be explored alongside the main search engine functionality while developing the final system.
In more detail, the following steps have been followed: Table 1 Comparison on ROUGE scores.
For testing, we followed the steps detailed below: 4. Preliminary Results 1. Two datasets were created based on roles similar to the training and validation sets. However, we don’t calculate the scores here beforehand. Instead, we use the trained model to predict them. 2. The sentences were then grouped into documents based on their document id. 3. The trained model was used to compute the score of each sentence. 4. The sentences were sorted by predicted scores. 5. The top 5 sentences were selected and sorted according to their index position in the original document to compose the final holding. 6. The ROUGE scores were evaluated between the extracted and the original holdings.
based on BERT and Transfomer improved the classification performance of rhetorical sentences by +12% in terms of Matthews Correlation Coeficient (from 0.81 to 0.91) compared to a model based only on BERT.
The experiments on holding extraction were on two datasets with diferent filters on the rhetorical roles: 1) only with the REASON and 2) with REASON + DEVELOPMENT OF THE TRIAL (REASON + DEVELOPMENT).
Their performance was evaluated with ITA-ROUGE, a modified version of the ROUGE metrics for the Italian language. The experiments were carried out on an NVIDIA
DGX system equipped with a 32GB TeslaV100 GPU. REA
Our software stack included PyTorch, Hugging Face SON outperforms REASON + DEVELOPMENT proving transformers, and Py-Rouge. We used Italian-LEGAL- that rhetorical roles and picking only the important senBERT as the encoder. This model has an embedding tences can yield better results. dimension of 768, an input token size of 512, 12 hidden layers with 12 attention heads, and an attention dropout of 0.1. A sequence regression head (i.e. a linear layer) 5. Conclusions and Future Work was added to the pooled output. The training was carried out with an AdamW optimizer and a linear scheduler. In this paper, we showed that the quality of extractive We trained both datasets for 4 epochs, using a batch size summarization can be increased by adding a Rhetorical of 16 and setting 256 as the maximum sequence length. Role layer and choosing only the most important parts of the document. This outperforms the HM-BERT model, which was trained on the same ITA-LEGAL-BERT with3.4. Legal Holding Search Engine out Rhetorical roles. Our future work involves in the future development of AI tools that can improve the performance of Judicial Ofices. This includes information retrieval, summarization, classification, question answering, and others. For our immediate future work, we will explore the possibilities of using a search engine paired
The documents, along with their generated rhetorical roles and extracted summaries, will be indexed into the with the summarization and role classification prototypes we already built.