Artificial Intelligence techniques are already popular and important in the legal domain. We extract legal indicators from judicial judgments to decrease the asymmetry of information of the legal system and the access-to-justice gap. We use NLP methods to extract interesting entities/data from judgments to construct networks of lawyers and judgments. We propose metrics to rank lawyers based on their experience, wins/loss ratio and their importance in the network of lawyers. We also perform community detection in the network of judgments and propose metrics to represent the dificulty of cases capitalising on communities features.
• Applied computing → Law; • Computing methodologies → Information extraction.
Recent advances in Artificial Intelligence (AI) and Natural
Language Processing (NLP) allow the analysis of large numbers of
legal documents in aggregate in contrast to traditional methods. A
long-standing application of NLP to legal documents is
information extraction and retrieval from judicial decisions. The interest in
mining data from judgments can be explained by the critical role
they play in the administration of justice in both common and civil
law systems. The objective of our work is to analyze judgments by
French courts to gain insights about the operation of the French
judicial system, which could in turn help developing an interface
for laypersons. As explained in [
Numerous research have been carried out on case-law corpora
focusing on specific objectives. One of the long-time objectives is the
prediction of case outcomes. One of the first approaches in this field
was [
Our dataset consists of 40,000 rtf files that were crawled through Légifrance 1, a French legal publisher providing access to law codes and legal decisions. To navigate and crawl through Légifrance we used Selenium2, a python framework that simulates a real web browser. For our experiments, we used a sample of cases from the court of appeal consisting of 17,215 cases. We limit our first analysis to cases from the court of appeal due to the specificity of cases from trial courts and the Court of Cassation. For future works and analysis, a sample of cases from the Court of Cassation could also be used (more than 400,000 documents available on Légifrance). We decided to focus first on cases decided by civil courts and to exclude both criminal, administrative, and specialized courts. We also remove procedural judgments, such as court orders. Judgments analyzed here are solely final decisions called "arrêt de Cour d’appel." 3.2
Data preprocessing was the most challenging part of the project. The structure and wording of the legal documents, which vary between diferent courts and dockets, as well as the use of legal formal language, were challenging obstacles to conduct the text mining tasks. Below we analyze in detail how we approached each part, from segmenting the documents to extracting the persons taking part in each court case and their roles. 3.2.1 Segmentation.
Analysis of the macrostructure of cases. The decisions of courts of appeals in France follow an overall similar structure. First, the documents state practical information about the litigation such as dates, jurisdiction, and the diferent entities involved in the trial, listed in the following order:
• Appellant (appelant in French): The name of the party is
always anonymized, for example: "Monsieur Jean X."
• Appellant’s counsel: can be anonymized but always start
with the keyword "Me" or "Maître," for example, "Me Jean
Dupont."
• Appellee (intimée in French): this entity has the same
format as the appellant.
• Appellee’s counsel: Same format as the appellant’s
counsel.
• Court Entities (non-fixed order):
– Judge (magistrats, conseillers in French ): could be
anonymized but always start with the keyword "Président."
– Clerk (Grefier in French) : Can be anonymized but is
always expressed near the word "Grefier."
After listing the entities, decisions from French appeal courts
continue with the debate. The debate describes all the facts and
procedure leading to the appeal. It also states the diferent arguments
brought forth by the parties, and follows with the reasoning of
the court. Finally, it closes with the conclusion which states the
ifnal decision. The keywords separating the diferent parts vary
significantly, and are sometimes absent, which makes the
segmentation task complex. Keywords may vary from one appeal court to
another. We use graphs to compare the structure of judgments of
appeal courts in several territorial jurisdictions. Figure 1 represents
the flow of cases in two diferent jurisdictions. Each graph is built
by parsing judgments from the same jurisdiction into sentences
and then linking consecutive sentences by an edge. The name of a
node is the text of the sentence. When the sentence is more than
ifve words, the name of the node is "Long_Text_i_j" where i is
the index of the case in the whole dataset, and j is the index of the
sentence within the case. The size of the node is the occurrence
of its text in all the judgment of the considered jurisdiction. To
account for keywords that have small variations across documents,
we use the Jaro similarity to identify these variations, examples are
in table 1. The Jaro similarity is a similarity measure between two
strings 1 and 2 [
• | | is the length of string • m is the number of same characters not further than ⌊ ( |21 |, |2 |) ⌋− 1 • t is the number of transpositions.
The Jaro similarity is used to contract nodes of similar sentences, such that if two sentences have a Jaro similarity larger than 0.8, then they are considered belonging to the same node. Therefore the big nodes are common parts from all documents, and they represent the structure of these documents.
Figure 1 points out the diference in structure and flows that documents from diferent jurisdictions can have. For instance, Agen will use "ENTRE" to announce the appellant, and "ET" to announce the appellee, whereas Douai will use respectively "APPELANT" and "INTIMEE."
Nevertheless, we empirically observed that all the decisions, whatever the jurisdiction was, shared the same keyword "PAR CES faits et procedure procedure et pretentions des parties moyens et pretentions des parties faits procedure procedure et moyens des parties pretentions et moyens des parties
MOTIFS" to announce the final decision of the court (last big node in the two flows of figure 1).
Segmentation with keywords. We also sought to extract entities corresponding to the lawyers defending each party. As described above, legal entities are mentioned after the practical information in a fixed order. Moreover, domain experts confirmed these legal entities are mentioned in separate segments. These segments are often preceded by known keywords, as shown in figure 2. Once we have identified the beginning and end of each segment, we use them to extract lawyers’ names as described in the following subsection. 3.3
To detect lawyers’ names throughout the document, we discard,
ifrst, all segments except the appellant and the appellee segments.
Second, we segment them further into sentences using the sentence
tokenizer by Polyglot 3, which is a Python package providing
multilingual natural language processing tools. Third, we only keep
sentences containing honorifics used for lawyers such as "Me,"
"Maître," or lawyers’ keywords like: "representé par." We then use a
well-established Named Entity Recognition model by Polyglot [
From the initial segmentation, the final decision of the court is to be found in the conclusion segment of the judgement. Concerning judgments from appeal courts, the court will either confirm the first lower court decision (Tribunal judiciaire) or reverse it. However, the court can also partially confirm the judgment. In other words, the court can decide to accept one of the appellant’s requests, and therefore change the first decision partially. Empirically, we noticed that certain words are present in certain types of decisions, and after validation from the domain experts, we resorted to a keyword-based solution: • "Confirme", "Rejete", "Irrecevable": keep the first decision (Appellee "wins") • "Infirme", "Rectifier", "Réforme"": change the first decision (Appellant "wins") Out of a sample of 5832 cases, 570 conclusions (∼10%) include at least a keyword representing both outcomes, in which case we keep the outcome that has most keywords. This is a temporary solution that requires refinement in the future. 4
Once the entities’ recognition is complete, we extract all the instances (and their function) in every document. Since courts tend to have a limited number of lawyers, judges and court clerks, cases share the same entities. Therefore, all the cases can be considered a big graph where entities interact with each other. 4.1
We extracted the winning and losing lawyers in each decision. From this, we can define a directed weighted network. We draw an edge between lawyers if they have been opposed. The edge from lawyer i to lawyer j is weighted by the wins , of lawyer i to lawyer j: ↦→ = ↦→, + ↦→, Where ↦→, is the number of wins of lawyer i as an appellant and ↦→, is the number of wins of lawyer i as an appellee. Parameters ’a’ and ’b’ are used to weigh more winning as an appellant than winning as an appellee since it is known by legal experts that the event of winning an appeal is less frequent than losing it.We also confirm this intuition by counting the rate of appeals’ rejection from our dataset. We get a rejection rate of 0.9. We collapse both edges between two lawyers into one directed edge weighted by: |↦→ − ↦→ | log(↦→ + ↦→ + 1). In this case the edge direction is determined by the sign of ↦→ − ↦→ such that the edge target is the lawyer with most wins. To visualize the most important nodes, we remove lawyers with only one case (899 out of 2146), which leaves us with a network with 1247 nodes and 2182 edges. The resulting network appears in figure 3. The edge goes from a "losing" to a "winning" lawyer, and the width of the edge represents the diference in the number of wins. Node size is the number of appeal cases where the lawyer appears and the color captures the win-loss diference. 4.2
The collaboration network in figure 4 indicates lawyers that have been on the same side during an appeal case. The edges are weighted based on the wins minus the losses, so the network can capture which collaborations are the most successful. We have removed nodes with number of collaborations below a fixed threshold to obtain a decluttered visualization of the network. We obtain a network of 47 nodes and 94 edges out of 2182 nodes and 2950 edges. 4.3
In this section, we suggest three metrics to rank and compare between lawyers. First, we measure the experience of a lawyer by the number of judgments mentioning him as the appellant’s or appellee’s lawyer. Second, we compute the win-loss rate of lawyers. Third, we calculate the centrality of a lawyer in the opposing network.
In figure 5a, lawyers are ranked by their experience in going in front of the court of appeal. However, this measure alone does not indicate the performance of the lawyer. Thus we need to refer to the win/loss ratio to evaluate the performance. Lawyer 353 ranks ifrst in terms of the win/loss ratio instead of fifth in terms of the (a) Lawyers ordered by the total number of cases (b) Lawyers ordered by the win/loss ratio (c) Lawyers ordered by their importance using PageRank algorithm total number of cases. In this case, lawyer 353 performs better than lawyer 387, who is ranked first in figure 5a but ranks 9 in terms of win/loss ratio. A weakness of the win/loss ratio ranking is that it does not consider the experience of the opposing lawyer; while the opponent’s worth can be a measure of the win’s value. To this end, we compute the weighted directed PageRank of the opposing network 5c. As explained in section 4.1, the weights are the number of wins such that wins as an appellant’s lawyer counts more than wins as an appellee’s lawyer. So edges directed towards lawyers who win more representing an appellant have higher weights than edges directed towards a lawyer who wins more representing an appellee. Therefore top lawyers in figure 5c are lawyers who won against experienced lawyers and who win most as an appellant’s lawyer. Lawyer 387 is ranked best than lawyer 350 in terms of win/loss ratio, but worst in terms of PageRank measure. We could explain this diference in the ranking by the fact that the majority of wins of lawyer 350 wins as an appellant’s lawyer, while the majority of wins of lawyer 387 wins as an appellee’s lawyer. Thus it is recommended for an appellant to choose lawyer 350 rather than lawyer 387. 4.4
In this section, we develop a method to assess cases’ dificulty from the perspective of the appellant. More precisely, the aim is to compute the dificulty withing a group of cases dealing with the same legal issues. First, we built a network of cases to discover communities of cases about similar legal issues. Second, we use the win/loss rate of the appeal as a proxy to its dificulty .
Graphs encode knowledge and patterns more eficiently [
We built networks for diferent values of k from cases of the last three months of 2018, as shown in figure 7. The network naturally groups similar cases in communities. For example, in figure 7c cases against the same appellee and about the same issue. We also notice, ifgure 8 that cases with the same win/loss rate are grouped in the same communities.
We used NLP methods to extract information from judgments of the French court of appeal. We constructed indicators about the dificulty of lawyers’ performance and cases by using network analysis techniques on lawyers’ networks and cases’ networks. Our objective is to use these indicators to guide laypersons when confronted with the legal systems and contribute to the decrease of the access-to-justice gap by reducing the asymmetry of information characterizing the legal market. The lawyers’ ranking could serve to build a system that guides an appellee in choosing a lawyer. However, the lawyers’ ranking relies only on wins and losses of lawyers. In future work, we expect to produce a ranking that takes into account the legal area of the case and its dificulty, in such a way that the ranking could be more personalized to the needs of a layperson.
REFERENCES