In an era characterized by fast technological signs of progress that introduce new scenarios every day, working in the law field may appear very dificult if not supported by the right tools. In this paper, we discuss a recently submitted work that proposes a novel method, called PRILJ, that identifies paragraph regularities in legal case judgments to support legal experts during the redaction of legal documents. Methodologically, PRILJ adopts a two-step approach that first groups documents into clusters, according to their semantic content, and then identifies regularities in the paragraphs for each cluster. Embeddingbased methods are adopted to properly represent documents and paragraphs into a semantic numerical feature space, and an Approximated Nearest Neighbor Search method is adopted to eficiently retrieve the most similar paragraphs with respect to the paragraphs of a document under preparation. Our extensive experimental evaluation, performed on a real-world dataset, proves the efectiveness and the eficiency of the proposed method even if documents contain noisy data.
The legal sector is generally characterized by a slow response to the new scenarios that appear
every day in the modern society. In this context, Artificial Intelligence (AI) methods can support
the design of advanced (also automated) solutions to improve the eficiency of the processes
in this field. Among the attempts in this direction, we can mention the work presented in [
Following this line of research, in this discussion paper, we describe a novel method, called PRILJ, that identifies paragraph regularities in legal case judgements, to support legal experts during the redaction of legal documents. Methodologically, PRILJ adopts a two-step approach that first groups documents into clusters, according to their semantic content, and then identifies regularities in the paragraphs for each cluster. Embedding-based methods are adopted to properly represent documents and paragraphs into a semantic numerical feature space, and an Approximated Nearest Neighbor Search method is adopted to eficiently retrieve the most similar paragraphs. Therefore, given a (possibly incomplete or under preparation) document, henceforth called target document, PRILJ supports the retrieval of similar paragraphs appearing in a set of reference documents related to previous transcribed legal case judgments.
Document clustering has received a lot of attention by the research community, but together
with the design of advanced algorithms [
In this context, PRILJ has the main advantage of properly combining embedding methods, to catch the semantics, with a two-step approach, that consists in learning a diferent representation for each group of documents, rather than one single model. This aspect allows us to capture peculiarities of paragraphs according to the specific topic represented by each cluster of documents.
Our extensive experimental evaluation, performed on a real-world dataset, proves the efectiveness and the eficiency of the proposed method. In particular, its ability of modeling diferent topics of legal documents, as well as of capturing the semantics of the textual content, appear very beneficial for the considered task, and make PRILJ very robust to the possible presence of noise in the data.
Before describing PRILJ, in the following, we provide some useful definitions: • Training set : a collection of legal judgments, represented as textual documents, adopted to train our models; • Reference set : a collection of legal judgments, represented as textual documents, from which we are interested to identify paragraph regularities; • Target document : a legal judgment (possibly under preparation) about which we are interested to identify paragraph regularities from .
The training set and the reference set may fully (or partially) overlap i.e., = (or ∩ ̸= ∅), namely, the set of documents adopted to train our models may be the same as (or overlap with) the collection from which we want to identify paragraph regularities with respect to the target document. Note that PRILJ is fully unsupervised and the target document is never contained in either the training set or in the reference set (i.e., ∈/ ( ∪ )).
The three phases of PRILJ are detailed in the following subsections.
As shown in Fig. 1, PRILJ starts with the application of some pre-processing steps to the documents in . Specifically, the pre-processing consists of: i) lowercasing the text, ii) removing punctuation and digits, iii) applying lemmatization, and iv) removing rare words. The pre-processed documents are then used to train a document embedding model , that is subsequently exploited to represent each document of the training set in the latent feature space, obtaining the set of embedded training documents . Such documents are then partitioned into clusters [1, 2, ..., ] by adopting the -means clustering algorithm. Each cluster of documents becomes the input for a further learning step at the paragraph level: documents falling in the same cluster will contribute to the learning of a specific paragraph embedding model. Algorithmically, for each document cluster , 1 ≤ ≤ , we extract the paragraphs (i.e., sentences delimited by a full stop) from the documents falling into and train a paragraph embedding model . This approach allows us to learn more specific paragraph embedding models, according to the topic possibly represented by the identified clusters.
The embedding models, both at the document level and at the paragraph level, are learned
by PRILJ through neural network architectures based on Word2Vec Continuous-Bag-of-Words
(CBOW) [
In Fig. 2, we show the workflow followed by PRILJ to represent the paragraphs of the documents belonging to the reference set into a latent feature space. Analogously to the training phase, we pre-process the documents of the reference set . Then, each document of the reference set is embedded using the previously learned document embedding model . The embedded representation of the document is then used to identify the closest document cluster that corresponds the optimal paragraph embedding model (i.e., ). We stress the fact that PRILJ performs this step to identify the most proper paragraph embedding model to represent the paragraphs of a given document.
The set of all the embedded paragraphs is finally returned. Paragraph regularities for a given target document will be identified from such set .
The final phase, which workflow is represented in Fig. 3, starts by following the same steps mentioned in Sec. 2.2 to represent each paragraph of the target document in the paragraph embedding space. Specifically, the most proper paragraph embedding model is adopted to embed its paragraphs, selected by identifying the closest document cluster with respect to . For each embedded paragraph, we finally identify the top- most similar paragraphs from the set of embedded paragraphs belonging to the reference set.
It is noteworthy that their identification could straightforwardly be based on the computation
of vector-based similarity/distance measures (e.g., cosine similarity, Euclidean distance, etc.)
between the embedded paragraphs of the target document and all the embedded paragraphs
of the reference set . Such a pairwise comparison would be computational intensive and
would lead to ineficiencies during the adoption of the proposed system in a real-world scenario.
To overcome this issue, we adopt a more advanced method for the identification of the top-
most similar paragraphs, based on random projections. In particular, we propose an approach
based on Annoy [
All the experiments were performed using a real-world dataset consisting of 4,181 oficial public EU legal documents, provided by EUR-Lex (https://eur-lex.europa.eu/homepage.html), in a 10-fold cross-validation setting. All the documents of the testing set were considered as target documents, while the reference set was built by constructing 20 replicas of each paragraph of the documents in the testing set, perturbed by introducing a controlled amount of noise. In particular, noise was introduced by replacing a given percentage of words of each paragraph by random words selected from the Oxford dictionary (raw.githubusercontent.com/cduica/ Oxford-Dictionary-Json/master/dicts.json). In our experiments, we considered diferent levels of noise, namely, 10%, 20%, 30%, 40%, 50% and 60%, in order to evaluate the robustness of the proposed approach to diferent amounts of noise.
In order to assess the specific contribution of the adopted embedding strategies, we compared the results obtained through Word2Vec and Doc2Vec with those achieved using a baseline strategy, i.e., the classical TF-IDF. In all the cases, we adopted a 50-dimensional feature vector. Note that we use 50 features, since it is a commonly used dimensionality in other pre-trained embedding models. For TF-IDF, we selected the top-50 words showing the highest frequency across the set of legal judgments.
We specifically evaluated the contribution of the two-step model implemented in PRILJ with diferent numbers of clusters, i.e., ∈ {√︀| |/2, √︀| |, √︀| | · 2}, and compared the observed performance with that obtained without grouping training documents into clusters (henceforth denoted as one-step model).
Finally, we evaluated the efectiveness and the eficiency of the approach implemented in PRILJ for the identification of the -n most similar paragraphs based on ANNS (with = 100). Specifically, we performed an additional comparative analysis against a non-approximated solution based on the cosine similarity, on a subset of 100 documents randomly selected from the dataset. This analysis was performed considering the best number of clusters , and also focused on evaluating the advantages in terms of computational eficiency.
As evaluation measures, we collected precision@n, recall@n and f1-score@n, averaged over the paragraphs of target documents and over the 10 folds, with ∈ {5, 10, 15, 20, 50, 100}. Specifically, for each paragraph of a target document in the testing set, we considered as True Positives the number of correctly retrieved (perturbed) replicas from the reference set. Note that, in this discussion paper, for space constraints we only show the results in terms of f1-score@20.
In Fig. 4 we can observe that, although the baseline based on TF-IDF obtained acceptable
results, the adoption of the embedding methods implemented in PRILJ is significantly beneficial.
Moreover, although Doc2Vec is natively able to work with word sequences, Word2Vec always
obtains better results. This is possibly due to the fact that several paragraphs of diferent legal
documents may share a similar topic, and the adoption of the unique sequence ID to associate
the context with the document, as done by Doc2Vec (see [
In Fig. 5, it is possible to clearly observe the contribution of the two-step process we propose. Indeed, the results show that the proposed two-step model outperforms the one-step model, in all the situations. In particular, the two-step model is much more robust to the presence of noise: although we can still observe a decrease when the noise amount increases, its impact is much less evident. We can also observe that in general, the number of extracted cluster seems to not significantly afect the results, even if the best results are observed with = √︀| | · 2. This means that the documents are distributed among several topics and that learning a diferent (more specialized) paragraph embedding model for each of them is helpful to retrieve significant paragraph regularities.
Finally, the comparison between the adopted ANNS and the exact computation of the cosine similarity emphasized a diference of 0.6% in terms of f1-score@n, which can be considered negligible. On the other hand, the advantage in terms of eficiency is significant: the exact search required up to 1000x the time took by the ANNS implemented in PRILJ (see Table 1).
In this work, we discussed PRILJ, a novel approach to identify paragraph regularities in legal judgments. PRILJ represents documents and paragraphs thereof in a numerical feature space by exploiting embedding methods able to catch the context and the semantics. Moreover, PRILJ is based on a two-step model, that groups similar documents into clusters and, for each of them, learns a specific paragraph embedding model. This approach allows us to properly catch peculiarities exhibited by paragraphs and documents of similar topics and to handle the presence of noise in a robust manner. Finally, PRILJ is able to identify paragraph regularities very eficiently, thanks to an ANNS strategy.
Our extensive experimental evaluation has shown the accuracy and the eficiency of the developed approach on real data. This means that PRILJ can be considered a useful tool in real-world scenarios, also when large collections of legal documents have to be analyzed. GP acknowledges the support of Ministry of Universities and Research through the project “Big Data Analytics”, AIM 1852414-1 (line 1).