The increasing backlog in processing legal cases in the Courts pushes toward the implementation of solutions that can reduce the workload and thus the time to complete cases. Searching for similar judgments is a relevant task that enables consultation of related legal proceedings. This paper proposes to automate the search process by developing a pipeline inspired by the modus operandi of legal domain experts. The pipeline includes an augmented reading of judgments for the purpose of semantically analyzing texts taking into account the context given by their classification, decisions and citations. The results are interpretable by legal domain experts, while the similarity case-matching output is underpinned by the information extracted from the relevant documents. The paper addresses a case study based on Italian national laws.
CEUR ceur-ws.org
The length of trials and the disposal of the backlog are two of the main challenges of many juridical systems. This is mainly due to the disproportion between the number of cases and the number of judges and resources assigned to juridical ofices.
3Corresponding author.
4These authors contributed equally. nEvelop-O CEUR Workshop Proceedings
In order to lighten the workload of the magistrate, our proposal is to shorten reading and researching times by providing a methodology to facilitate the reading of legal documents by eficiently extracting and exploiting the knowledge embedded in the judgments.
Nevertheless, the full exploitation of juridical documents via eficient automatic information
extraction allows us to perform Similar Case Matching and further speed up the magistrates’
research process [
This work describes a pipeline for the augmented reading of legal documents which results in a tool for the detection of the judgment’s relevant content, in order to facilitate the use of juridical databases, extracting information that goes far beyond the document’s metadata and keywords, allowing for a more in-depth understanding of the document and for advanced functionalities such as pairwise similarity comparison of juridical documents.
After a brief legal (Section 2) and technical (Section 3) contextualization, we present the specific case study by thoroughly analyzing the domain experts’ modus operandi when reading and comparing legal documents (Section 4). Subsequently, we propose a pipeline for the augmented reading of legal judgments (Section 5) followed by its evaluation in terms of time eficiency and Similar Case Matching (Section 6).
The idea of creating a computerized database of case law had been around since the 1960s [
In particular, the case-law databases, from a means of knowledge and information, are a tool to organize judicial ofices and implement their eficiency. The concept embraces the traditional collections of judgments, which are used as research and support tools; but it can also be referred to those statistical surveys aimed at recognizing the flows of litigation, e.g. the analysis of controversies, having a given object (and sub-object), defined by a specific ofice; the outcome of the judgment; to what extent appeals are successful or rejected (in relation to the matter of the proceedings).
Definitions We introduce some definitions for the understanding of the technical details contained in the article.
• Legal judgment: authoritative decision given by a court or tribunal, including a decree, order, decision, or writ of execution. • Judicial grade/ Instance: reference degree of the issuing judicial ofice (Court of First
Instance or Court of Second Instance - Appeal). • Section (i.e. Sezione): a distinct group of Judges, specialized in a given area of law (Labour Law, Family Law, Criminal Law, etc), within a larger body of them, which constitutes the Judicial Ofice taken into account. • Legal institute: set of rules dealing with a particular legal relationship, or facts. • Topic: the matter of a legal institute or of a section of a legal institute. • Citations: normative and case-law references. • Normative: Rules, especially rules of behavior.
• Case law: Judicial decisions resolved by courts using the concrete facts of a case.
Reading and comparing textual documents are cross-domain tasks. The task of Similar Case
Matching can be tackled with diferent technologies. Many early contributions focus on a
text-based similarity measure (i.e. TF-IDF-based similarity measures [
Many contributions refer to the analysis of scientific papers, which share a lot of
characteristics with legal documents. Firstly, both classes of documents can be described as semi-structured:
the structure of first- and second-instance judgments, as well as the IMRaD format of scientific
papers, suggest a standard that is often disregarded or misinterpreted. For this reason, both
domains present dificulties in dealing with discourse changes within the document. They
are conceptually segmented into ordered paragraphs describing the information (preamble,
introduction, specific case, reasons and procedures) leading to the decision in a legal judgment
or to the paper’s conclusions. Moreover, they contain citations that interconnect legislation
and case-law or scientific papers as appropriate. In particular, scientific papers have
interconnections that determine their contextualization in the literature. In the case of the smart
citation index scite [
Another important aspect is the transparency and interpretability of the results. The pipeline
framework [
In order to elaborate our methodology, we focus on a case study based on Italian judgments of
ifrst- and second-instance followed by generalization proposals to extend our pipeline to other
legal systems and to documents written in other languages.
4.1. Data
The dataset used for these experiments was retrieved from a public online platform [
A total of 5059 legal judgments were extracted, labeled with the 8 most populated Voci (Figure 1) and having the decisions paragraph well defined and separated from the facts paragraph of the document. Though the data sample is not totally random, the possible bias introduced doesn’t compromise the outcome of the experiments to the best of our knowledge. It has been observed that in first- and second-instance legal judgments the structure of the document is not standard. Moreover, facts and decisions paragraphs may be combined in a single paragraph of the judgments file and therefore these were discarded in order to treat the two parts of the document separately properly.
In order to draw inspiration from the legal domain experts’ modus operandi for reading and comparing legal judgments, we proceeded by conducting a process analysis performed by two lawyers. They read two judgments together and commented aloud on them by referring to relevant and irrelevant information in the document to determine whether the two judgments are similar or not similar. This step aimed to intercept the sequence of operations, reasoning, and weights that are given to the components of legal judgments in order to bring out the domain know-how and methodologies in the workload. Our assumption (and proposal) is that the process of reading and comparing can be generalized into an algorithmic form, in this way it can be automated to provide daily support. The manual process took about 1 hour, and we expect that the automated system can be faster, bringing organizational benefits. Reading of the judgment Having specified the context-based similarity criterion, the lawyers began reading one judgment at a time. The focus initially fell on Sezione, matter, and Court grade (first-instance, second-instance (or Appeal), or Supreme Court). Afterward, they read the parts of the legal judgment paying attention to the presence of institutions; careful attention should be paid to the fact that judgments are often anonymized in respect of privacy. Thereafter, they carefully read all paragraphs of the document (e.g. facts, decisions, PQM - i.e. an Italian acronym used to introduce the decision, etc.). The first part of the facts and decisions paragraphs was of particular relevance in defining, in a coarse-grained manner, the domain of the case on which the legal procedure was applied. Lawyers pointed out that the legislation and case law cited would require extensive knowledge that in some cases may require consulting the rules and maxims for a greater understanding.
Comparing two judgments Once the accurate reading of the judgments was completed, they proceeded by comparing the information found in the judgments so as to define whether or not they were similar. In particular, it was noted how Sezione and Voci are of paramount importance in constrainingly defining the context within which the judgments fall. If Sezione and matter are diferent, then the judgments are by their nature diferent from each other. Subsequently, the focus was on the first part of facts, which define the specific scope of the case, and decisions, which define the legal procedure applied to the specific situation described in the facts.
The discussion with the lawyers revealed that, depending on the search criteria, it is useful to analyze the facts, if one wants to look for similar situations, and/or the decisions if one wants to give more importance to the legislation and case law cited. Since the second-instance judgments contain information related to the first-instance, which would influence the comparison without introducing additional relevant information, it was preferred to consider only the first part of the decisions that contains information of greater importance to define the legislation and case law applied that allow defining the topic and field.
This section describes the steps that led the study from the analysis of the modus operandi of legal domain experts to the contextualization of legal judgments and the calculation of the Similarity Score. The output is accompanied by auxiliary information useful for understanding the system’s results.
The modus operandi of domain experts led us to the implementation of a pipeline (Figure 2) with the purpose of translating the process of reading and comparing judgments into a generalized algorithmic form. Taking into consideration the scalability of our solution, we propose a modular pipeline, whose steps are somewhat independent and can be computed separately, storing partial results in JSON or CSV format. This methodology allows a single calculation that results in a final matrix of similarity scores that allows the data to be queried with linear complexity in the number n of legal judgments in order to retrieve the m most similar documents given a judgment.
The pipeline consists of three modules (as shown in Table 1): classification comparison, decision-based similarity and citation overlap calculation, followed by the calculation of the similarity score according to the outputs of the previous steps. Given all the sentences in the dataset, each step produces in output a matrix that will contribute to the calculation of the last matrix containing the total similarity scores.
The target of the system includes experts in the legal domain such as magistrates and lawyers. Since computer systems are often dificult to understand, it is necessary to provide a way to make the results explainable and allow critical reasoning with respect to the efectiveness of the output. To do this, information is extracted during data processing that is useful in calculating individual scores in modules, which upon completion of reading and comparing judgments allows for an explanation of why the machine returned the results. Since trust in the computational system applied in the legal domain is fundamental, it is very important to be able to critically evaluate the results to come up against accountability and the efects on the parties in the legal case.
As revealed by the performed process analysis, the most important features in evaluating the relatedness of two judgments are their Sezione and Voci which determine their classification within each Court since they carry important information about the documents’ topic. In developing our work, these labels were available within the dataset.
Expanding this pipeline to a more generic scenario this information can be extracted through a process of automatic legal document classification. Domain experts claim that two judgments module classification comparison decision-based similarity citation overlap extracted information output
Classification Comparison Matrix where For each judgment, for each pair of judgments, we have Sezione and Voci with which they are labeled. 1 if the two documents are classified in
the same way and 0 otherwise.
For each judgment, Decision-based Similarity Matrix where the portion of the decisions paragraph for each pair of legal judgments we have analyzed in the module. a decisionsSimilarity score between -1 and 1.
For each pair of judgments, Citation-based Similarity Matrix where legislative and case-law citations in the overlap. for each pair of legal judgments we have a citationOverlap score between 0 and 1 having diferent Sezione or Voce are to be considered diferent and cannot be compared further. Conversely, once it is established that two documents cover the same topic, it is possible to continue investigating their level of similarity through the analysis of their content. This first step of the pipeline acts as a filter and consists of comparing the labels Sezione and Voci. If they are equal we proceed with the calculation of the similarity scores based on the document semantics performed in the next steps otherwise, the two legal judgments are diferent. A legal judgment can have multiple Voci, to be considered equal they must have at least one entry in common. Given two judgments j1 and j2, the classification comparison value was calculated as follows: = where (1 j2, respectively.
{ 1 if 1 == 2 0 otherwise and ( 1 ∩ 2) > 0 (1) , 1 ) and (2 , 2
) are the Sezioni and Voci of legal judgments j1 and
The second step is the computation of the judgments’ lexical similarity. This value is obtained
by comparing the first portion of the documents’ decisions paragraph, which, as suggested
by domain experts, contains the most information regarding the legal case. The text was
preprocessed through a pipeline that includes 1) conversion to lowercase, 2) removal of special
characters, 3) removal of URLs and HTML tags, 4) conversion of word numbers to their numeric
form, 5) removal of stopwords and 6) lemmatization through Morph-it! [
The choice for this kind of approach was dictated by the fact that very few Courts provide a classification taxonomy with a granularity finer than the documents’ Voci, making it dificult to ifnd a subtopic label at this stage.
Although this approach allows us to find a deeper level of similarity, it reduces the overall interpretability of the pipeline results since it relies on deep learning techniques. Given two judgments j1 and j2, the decision-based similarity score was calculated as follows: = ((1), (2))
Subsequently, we focused on the extraction of citations within the decisions’ part of the judgment as they can be used to define the juridical context in which the judgments are located. In this perspective, the document can be characterized by the citations it contains; hence documents with a higher citation overlap will have a higher similarity level as they appear in the same juridical proximity. In other words, as in topic modeling the words describe the topic of the document, in the case of legal judgments the citations define the context and domain of application of the legislation and procedures applied to the decisions.
Citations are extracted following the idea that two judgments dealing with similar decisions cite similar legislation and case-law. Citations in documents were extracted through the use of Linkoln [13], which allows, given a text, to extract legislative and case-law citations using a regex-based methodology. The extraction provides a list of citations described with text, context, identifiers, reference type (legislative or case-law), authority (Court of first- and second-instance or Supreme Court), document type (legal judgment, decree-law, legislation, etc.), normative link (where available), etc. The proximity between citation-base contexts was calculated through the overlap between citations within legal judgments as follows.
Given two judgments j1 and j2: = |(1) ∩ (2)| |(1) ∪ (2)| (2) (3) (4) (5)
The aforementioned modules can work in isolation and independently of each other, each providing in output a square matrix of order n, where n equals the number of judgments in the dataset. The matrices that contribute to the calculation of the Total Similarity Matrix (TSM) are as follows: Classification Comparison Matrix (CCM), Decision-based Similarity Matrix (DSM), and Citation-based Similarity Matrix (CSM). Each cell in the TSM matrix is populated with a value ≤ 2 calculated as below.
Given two judgments (j1, j2) and the equations 1, 2 and 3:
= ∗ ( ∗ + ∗ ) where and represent the weights to be assigned to the last two pipeline components. In matrix form on total judgments, we can rewrite the calculation as below: = ∗ ( ∗ + ∗ ) The and multipliers allow for diferent weighting of similarity in decision-based context and citation-based context. This is because, within a legal case decisions, legislation, and case-law can take on diferent impacts in setting the context. When comparing two judgments, legal domain experts attach diferent importance to the components just mentioned and in some ways, it can be subjective in nature. and have a default value of 1, the first approach was to keep this value in the similarity calculation, thus considering the two pipeline components equally.
Given a legal judgment, making a matrix with total similarity scores allows one to access the matrix with linear complexity and select the m judgments that are most similar from the perspective of the context described by decisions, legislation and case-law. In this way, reading the judgments and extracting the information occurs only once showing a potential advantage over reproducing the same operation manually.
For each sentence pair, the pipeline outputs a total similarity score and auxiliary information useful for understanding the result. The information extracted and maintained during the execution of the individual modules is broken down as in Table 1. Given two judgments j1 and j2, the extracted information supporting the score is composed by combining the information extracted from the individual modules. This kind of interpretable output format allows for a more transparent similarity measure, easy to motivate from a legal perspective.
Since we cannot provide an interpretable output for the decision-based similarity module, the numeric score is substituted with the text used to compute it, so that a legal actor can easily evaluate the accuracy of the comparison.
Due to the nature of the task and the domain of the documents, it was necessary to resort to domain experts in order to obtain an evaluation of the resulting similarity scores. The experts were presented with a set of triplets of judgments with shape (⋆,+,−) as evaluated by the pipeline where • ⋆ indicates the judgment taken into consideration, selected randomly from the analyzed data • + indicates a judgment evaluated as similar to ⋆ • − indicates a judgment evaluated as diferent from ⋆ They were then asked to binary rate the outputs as accurate or inaccurate and the time taken for the reading of each judgment.
Inter-Annotator Agreement The Inter-Annotator Agreement was evaluated using Cohen’s Kappa [14], obtaining = 0.769. The same metric was used to compute the average agreement between the annotators and the pipeline output, resulting in = 0.885. it is worth noting that the final evaluation was influenced by the subjective nature of Similar Case Matching. Time The average reading time (minutes per page) for legal judgments is calculated among the annotators and separated by types (time⋆ = 3, time+= 3, time−= 1.7). The time starts with the reading of the document in order to define whether two sentences are similar or diferent and ends with the closing of the document. The reading does not have to be complete. Notably, these measurements do not include the search for judgments that legal domain experts would have had to do in the absence of this system.
On the other hand, through the computational pipeline analysis, in a few seconds a judgment can be analyzed and compared to every other judgment in the dataset, in order to obtain a similarity score. Moreover, it is important to acknowledge that once the judgment is analyzed by the system, all the partial results are stored and accessible in O(n).
Further considerations From the evaluation process it emerged that judgments redacted by the same judge in a close time interval tend to be very similar to each other. This can be motivated by personal stylistic choices such as chosen lexicon and personal interpretation of the cited documents.
In this section, possible future developments of this work are presented by topic.
From the perspective of generalizing the pipeline to diferent judgment sets, its first step should not rely solely on the labels present in the dataset. A more generic approach can be found by automatically classifying the documents and storing this information together with the judgments’ metadata. Given the absence of a standard of classification among courts, this task presents diferent challenges and requires an alignment of the diferent classification taxonomies.
The extraction of the lexical embeddings as presented in this work can be improved in diferent ways. The first aspect to take into account is the extraction of the most significant part of the judgment on which to focus to obtain said embeddings. While it is true that the most relevant information is to be found in the first part of the decision paragraph, there has to be a more thorough analysis of the length of the portion to be selected. Furthermore, since juridical documents are unstructured by nature, the decision paragraph is not always present and can often be merged with other Sezioni such as Fatti. For this reason, it is necessary to find a method of identifying such a paragraph that does not rely solely on the formatting of the document. Looking at the module from another perspective, more investigation could be done on the model to be used to create said embeddings.
In order to define a more precise and meaningful citation overlap score, this measure could be weighed in diferent ways.
Citation stance A first suggestion would be to refine the calculation of overlap between citations by weighing each citation according to the stance of the context in which it appears. This is because a sentence may cite another document to accommodate its meaning or to distance itself from it. From this point of view, two judgments having the same citation but with diferent stances would indeed be in the same legal context but would have a diferent connotation that would allow a diferent nuance of meaning to be highlighted.
Naively, this diferentiation can be tackled through the computation of the polarity of the sentences in which the citations appear. Subsequently, said results can be further refined by a more precise stance detection analysis. Nevertheless, given the domain of the documents and the fact that the target of the analysis is known and always present in the sentences taken into consideration, the two tasks are in some way overlapping and the polarity of a sentence can coincide with its stance in regards to the citation.
Citation relevance The score computation could be further perfected by assigning a diferent degree of importance to each kind of citation. Although all references to other legal documents contribute in some way to defining the document’s context, some (such as Supreme Court decisions) have higher relevance than others.
Through a deeper investigation of the legal landscape, it would be possible to configure these weights so that each citation could be given the importance that it actually has, ulteriorly refining the overlap score.
On the same idea, some documents could be often referenced without a direct correlation to the judgment’s topic. In this scenario, it would be appropriate to consider a relevance measure to lower their contribution to the overall overlap measure. In particular, always-cited and background citations do not attribute additional information for defining the citation-based context, so it may be relevant to ignore them.
Improved extraction of caselaw citations The extraction of the Supreme Court’s rulings can be explored even to a greater extent since it can be useful to map principles of law stated and renewed. In the Italian legal system, it is considered a principle of law a generalization of the interpretation and application of the rule to a concrete case, given by the function of nomophilacy given by the Supreme Court (i.e. Corte di Cassazione). As discovered through the analysis conducted by our legal domain experts, there is a one-to-one correlation between principles of law and Supreme Court’s citations in the explicit form. With the proper preprocessing and structure of the data, the aid of proper algorithms, the pattern found by the regex used in our experiments can represent an accurate extraction methodology of both Supreme Court citations and principles of law.
Outcome extraction An additional step in our pipeline would be the extraction of the judgment’s outcome which would allow for a better understanding of the judgment as a whole. With legal research in mind, being able to automatically extract the output of a judgment would enable more sophisticated features such as filtering the documents given their final decision. Time evaluation It will be interesting to study the manual task again with the support of the developed computer system, so it may be possible to evaluate the usefulness, efectiveness and time gained. In addition, feedback given by domain experts about the actual potential of the search engine support is important.
The algorithmic reproduction of legal domain experts’ modus operandi can be implemented and of great interest for the optimization of legal proceedings, as it was shown to be far more efective and time-eficient than the current procedure.
The modularity of the pipeline allows for a scalable solution, which allows for a thorough analysis and comparison of legal documents in seconds and an indexing in O(n). By integrating such computation in an online platform for the consultation of legal documents, it would be possible to increase the productivity of magistrates by providing them with an interpretable and transparent resource targeted to legal domain experts. Consequently, by facilitating the reading and the retrieval of relevant information within the text, this approach would contribute to the disposal of the backlog inside Courts.
Furthermore, the reproduction of an existing procedure for reading and comparing legal documents, allows for the collection of relevant data, which can be used by legal domain experts to evaluate the pipeline output.
The dataset used for these experiments was retrieved from a public online platform aimed at
consulting Italian legal documents [
Furthermore, the availability of the same text in multiple languages would allow optimal results by computing every module of the pipeline using the language in which the technologies used have the best performances.
The research work has been funded in Next Generation UPP5 project supported from the European Union, National Operational Program Governance and Institutional Capacity 20142020, European Social Fund and European Regional Development Fund. The Next Generation UPP project is part of the ”Unitary project for the dissemination of the Ofice for Trial and the 5https://www.nextgenerationupp.unito.it/home implementation of innovative operating models in the judicial ofices for the disposal of the backlog”, promoted by the Italian Ministry of Justice and implemented in synergy with the interventions envisaged by the National Recovery and Resilience Plan (NRRP) in support to the justice reform.