In this paper, we propose a bootstrapping approach for semiautomated legal knowledge extraction. The approach is characterized by the use of a reference legal ontology that is progressively enriched with relevant concepts and related terms extracted from a corpus of legal documents (i.e., Court Decision documents). Supervised, multi-label classi cation techniques and black-box model explanation techniques are the core components of the bootstrapping approach i) to associate CD documents with appropriate concepts in the ontology and ii) to choose the terms that are decisive for determining the association between a document and a certain ontology concept, respectively. The goal of the proposed approach is to reduce the manual involvement of legal experts as much as possible and to improve the accuracy of document classi cation, by progressively enriching the term sets associated with ontology concepts. Preliminary experimental results are nally provided to show the contribution of the proposed approach on a corpus of real Court Decision documents.
In the legal domain, Court Decisions (CDs) are documents written in natural language where judges give concrete application of rules and concepts that constitute the law, by deciding whether the law has been violated in relation to the facts. Therefore, CDs are a core component of the legal system since a clear and exhaustive understanding of the judge decisions represents a useful support for the activities of all the actors involved in the legal system. However, quantity, complexity, and articulation of CDs are constantly growing. As a result, e ectively extracting the judge decisions about a given crime hypothesis from documents related to real trials is becoming increasingly di cult.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This volume is published and copyrighted by its editors. SEBD 2020, June 21-24, 2020, Villasimius, Italy.
In such a context, techniques and tools for automated extraction of legal
knowledge are strongly demanded, to support annotation, analysis, and
understanding of legal documents [
For these reasons, data science approaches are being proposed for automating
- as much as possible - the extraction of legal knowledge from textual documents
such as Court Decisions. Information retrieval techniques can be employed to
detect the occurrence of the terms associated with a concept throughout the
documents [
In this paper, we propose a bootstrapping approach for semi-automated extraction of both terminological and conceptual knowledge in the legal domain. The approach is characterized by the use of a reference legal ontology that is progressively enriched with relevant terms extracted from a corpus of CD documents. Multi-label classi cation techniques and black-box model explanation techniques are the core components of the bootstrapping approach i) to associate CD documents with appropriate concepts in the ontology and ii) to choose the terms that are decisive for determining the association between a document and a certain ontology concept, respectively. The goal of the proposed approach is twofold. On the one side, the approach aims to reduce the involvement of legal experts as much as possible so that document classi cation can scale to manage large CD corpora. On the other side, the use of iterative bootstrapping cycles aims to improve the accuracy of document classi cation, by progressively enriching the term sets associated with ontology concepts.
The paper is organized as follows. In section 2, the proposed bootstrapping approach for semi-automated extraction of terminological and conceptual knowledge in the legal domain is presented. In Section 3, technical details about the adopted machine learning techniques are provided. In Section 4, we present some preliminary results on a real corpus of Court Decision documents. Finally, in Section 5, we give our concluding remarks and we outline our future research issues. 2
Semi-automated legal knowledge extraction Our approach for semi-automated legal knowledge extraction is based on the iterative execution of a bootstrapping cycle articulated in a sequence of steps shown in Figure 1. The approach is based on a corpus of Court Decision (CD)
Corpus of Court Decisions
1 [t1, t2, …, tn] [t1, t2, …, tn] Legal Ontology
Annotation of Court Decisions through term retrieval 7 … [tn+1, …, tk] [tn+1, …, tk]
Knowledge enrichment 2 6
Text pre-processing t e s g n ii n a r T t e s t s e T Document-concept
Matrix [t1, t2, …, tn, tn+1, …, tk] [t1, t2, …, tn, tn+1, …, tk]
Terminological expansion
Supervised multi-label
classification 3 …
… Trained Model 5 … 4 Black box model explanation Term validation by legal experts documents and on a reference legal ontology where an initial version of knowledge is provided, both conceptual knowledge and terminological knowledge. We call conceptual knowledge the set of legal concepts that is formally represented in a reference legal ontology, where concepts are interlinked by semantic relations and associated with a corresponding terminological knowledge. We call terminological knowledge the set of natural language terms concretely used in a considered corpus of legal documents (i.e., Court Decisions) to refer to legal concepts. The initial ontology is manually de ned by domain experts and it is characterized by a set of legal concepts of interest (conceptual knowledge). A legal concept Ci in the ontology is associated with an initial term set Ti0 that represents the relevant terms featuring Ci that are extracted from the corpus documents since they have been recognized by the experts to be an instance of the concept Ci (terminological knowledge).
A bootstrapping cycle k is organized as follows:
Step (1). Term retrieval technique are employed to associate document with relevant ontology concepts. For each CD document d, the set of associated legal concepts Cd is determined as follows: (
" Cd =
Ci :
X w(t; d) t2Ti # th ) where w(t; d) is the weight of a term t in the document d according to standard information retrieval techniques based on tokenization, tf-idf, and PMI (Pointwise Mutual Information) for compound term detection. Moreover, th is a threshold used to set the minimum cumulative weight of all the terms t 2 Ti that is required for associating a corresponding concept Ci with the document d.
Step (2). For each document d in the corpus, a vector-based
representation d is generated to provide document embedding. In the literature, di erent
techniques can be employed to enforce vector-based document representation,
like for example bag-of-words, word2vec, and NVSM (Neural Vector Space Model).
In our approach, we choose to rely on doc2vec techniques [
Step (3). A multi-label classi er is employed to generate a model that is capable to predict the association of CD documents with legal concepts. In our approach, we employ a 1D Convolutional Neural Network (1D-CNN) with the goal to generalize the terminology of the documents and to enable the association of legal concepts with Court Decisions that actually contain terms other than those already included in the reference legal ontology. For each document d, the CNN receives the document vector representation d as input and it produces the corresponding concept vector representation cd as output. As a result, a classi cation model M is generated to map the correspondence between corpus The choice of CNN is due to the positive experimental results we observed in a number of considered case-studies. As a general remark, di erent kinds of multilabel classi er can be employed for enforcing document classi cation, like for example random forest and kNN. documents and legal concepts in the ontology. In particular, by Ci 2 M (d) we denote that the document d is associated with the legal concept Ci through the model M .
Step (4). We exploit black-box model explanation techniques in order to select the document features (i.e., terms) that play a major role in determining the decision of the multi-label classi er about the association of concepts with the corpus documents. As a result of Step (4), for a legal concept Ci, a set TCi is generated containing terms that mainly determine the decision of the CNN classi er to associate Ci with a considered document of the corpus.
Step (5). For each concept Ci, the terms in the set TCi n Tik are candidate to be exploited for terminological expansion. Legal experts are involved in a validation activity of candidate terms. As a result, for a concept Ci, the set Ri (TCi Tik) is de ned containing the terms that are relevant for Ci according to the expert evaluation Step (6). Finally, in Step (7), the terminological knowledge Tik+1 associated with each concept Ci is enriched as follows: T k+1 i
T k
i [ Ri
At the end of Step (7), a new bootstrapping cycle can be enforced. The goal of each bootstrapping cycle is twofold. On the one side, a bootstrapping cycle aims to improve the accuracy of document classi cation enforced in Step (3). In the rst bootstrapping cycle, the accuracy of classi cation can be low due to the fact that the training set is built by exploiting the terminological knowledge available in the initial version of the legal ontology. As long as the terminological knowledge of the ontology is enriched, the accuracy of the classi er is expected to increase. On the other side, a bootstrapping cycle aims to enrich the terminological knowledge of the legal ontology. The enforcement of new bootstrapping cycles is stopped when the enrichment of the terminological knowledge is terminated, namely when the expert validation (Step (5)) does not generate new terms to insert in the legal ontology.
In the following, more technical details about the black-box model explanation techniques are provided to better emphasize the original contribution of the proposed bootstrapping approach. 3
Knowledge enrichment: black-box model explanation and terminology expansion In a given bootstrapping cycle k, the goal of black-box model explanation and terminology expansion is to exploit the current version of the legal ontology Ok and to generate a new version Ok+1 where the term sets of THE ontology concepts are enriched with the discovered terminological knowledge. Terminology expansion is based on the multi-label classi cation model M k derived from the annotation of CD documents through Ok. During the training phase, M k learns the function that maps CD documents with terminology of Ok on the appropriate legal concepts. In addition, the model also learns to generalize such knowledge, to correctly associate legal concepts with CD documents that actually contain terms other than those included in Ok. This ability of M k depends on two main aspects of the training process. The rst one is that CD documents are encoded as vectors using doc2vec, thus documents that are semantically similar (but containing di erent terminology) are encoded as vectors which are \close" in the feature space (i.e., the space of terms). The consequence of this proximity is that the mapping function learned from the model M k tends to associate neighboring vectors (i.e., documents) with the same legal concepts. The second aspect is that documents that contain Ok terms often contain further terms that are also relevant to the legal concepts in the ontology, but which were not discovered/associated in previous bootstrapping cycles. In other terms, the model M k implicitly contains the relevant terminology required to map CD documents to legal concepts, even if this terminology is not included in Ok.
For each concept Ci, our goal is to detect the set of terms that play a crucial
role in determining the classi cation decision of M k, namely the terms that, if
deleted from the document, more likely may produce a di erent classi cation
result. Determining this set of terms is challenging due to the lack of an explicit
explanation capable of describing the behavior of M k. To this end, we exploit
black-box model explanation techniques. Recently, some approaches have been
proposed to provide a model explanation at least locally, which means to explain
why (i.e., due to which features/terms) a model decides to assign a given class to
a certain document [
TCi = Ci (t) = 8 < : t : t 2
9 [ d= d2DCi ; X
X
t2TCi d2DCi
(t; d)
Then, we associate each term t 2 TCi with a degree of relevance Ci (t) as follows:
Legal experts are then involved in the validation of terms in TCi . A
thresholdbased mechanism based on the degree of relevance Ci (t) can be enforced to
support the validation activity of experts. In particular, terms with value of
C (t) higher than the threshold are proposed to the expert for insertion in the
legal ontology, while terms with value of C (t) lower than the threshold are
proposed to be discarded. As a result of the expert validation, the set Ri is
de ned containing the terms that are relevant for the terminological expansion
of Ci so that the new version Ok+1 of the legal ontology can be de ned.
Example. In Figure 2, we show an example of two CD documents, d1 and d2
associated with the concept Drug in a legal ontology O1 about the drug criminal
legislation (see Figure 3). In our example, the ontology O1 is implemented by
using the Simple Knowledge Organization System (SKOS) [
d1: [...] Paragraph 14 of section 1 of the same act provides: \Narcotic Drugs means coca leaves, opium, cannabis, and every substance neither chemically nor physically distinguishable from them." [...] d2: [...]Defendant, who was charged by indictment with violation of 402 of the Illinois Controlled Substances Act" [...]
The association of d1 with the concept is due to the fact that it contains the terms Narcotic Drug and Cannabis that belong to the term set of the concept Drug in the legal ontology. The multi-label classi cation model M1 trained on d1 (and on the other documents contained in the training set) classi es d2 as a document related to the Drug concept. This decision is due to the similarity between the documents d1 and d2, which implies that the two vectors obtained by doc2vec are close in the feature/term space.
Through LIME, we detect the terms of d1 and d2 that mainly in uence the classi er decision. According to LIME, we obtain the following terms for the concept Drug: Narcotic Drug, Controlled Substances, Cannabis, Coca Leaves, Opium. In the list, Narcotic Drug and Cannabis are already present in the current ontology O1, while the others (underlined in Figure 2) are validated by the legal experts. In Figure 3, the validated terms are included in the ontology O1 to generate a new, enriched ontology O2 where the term set of the concept Drug is properly extended. In the subsequent bootstrapping cycle, the ontology O2 is exploited to automatically create the training set for the classi cation model M2. Such Criminal Procedure
Drug Trafficking
Verbs
Drug Narcotic Drug,
Cannabis
+ Controlled Substances, Coca Leaves,
Opium
Evidence
Unit of Measure
Drug Trafficking, Drug Sale, …
Gram, Grams, gr., …
Plastic bag, …
Legend related istance-of
LEGAL CONCEPT TERM-SET
Illinois Legislation 720 ILCS 570,
Illinois Controlled Substances
Act, …
Arrest, Arrested,
… a training set will include also d2 since the term Controlled Substances has been inserted in the term set of the concept Drug. A new round of classi cation and explanation can be executed to further improve the terminological expansion of ontology concepts and to generate a new ontology version O3. 4
The goal of our preliminary evaluation is to assess the i) the capability of discovering new relevant terms about the concepts in the reference legal ontology and ii) the improvement in terms of accuracy of the classi cation process across two bootstrapping cycles. The experimentation is based on a dataset of around 180,000 Court Decisions of the State of Illinois taken from the Caselaw Access Project (CAP) providing public access to U.S. law (https://case.law/bulk/ download) digitized from the collection of the Harvard Law Library. For the experiments, we select six concepts from our legal ontology, namely drug, drug tra cking verbs, unit of measure, illinois legislation, criminal procedure, and evidence. Document classi cation is enforced at the sentence level, which means that legal concepts are associated with each single sentence independently. This way, the 180,000 court decisions correspond to about 14,000,000 documents (i.e., sentences). In the rst bootstrapping cycle, the initial version of the legal ontology is characterized by concepts with small term sets (see Table 1). By relying on the term sets in the ontology, we select a subset of 115,993 CD sentences that constitutes the training set of the classi cation step. According to our annotation techniques, a sentence is associated with a concept Ci when at least one term belonging to the term set Ti is contained in the sentence. In Table 2, for each concept considered in the experimentation, we show the number of associated sentences resulting from the annotation step.
Each document is embedded in a 100-dimension vector using doc2vec to obtain a 115,993 100 corpus matrix. The model M1 used to train the classi er is a neural network organized in three layers. Between the input and the output layer, we use a convolution lter activated by ReLU. The M1 accuracy obtained by cross-validation is 0.77. The model M1 is then used to perform black-box model explanation and terminology expansion using LIME. For each concept Ci, we determine a new set of terms TCi . A term t 2 TCi is associated with the degree of relevance Ci (t). In the experimentation, a legal expert validated the top-20 terms in the set TCi of each concept Ci. In particular, the expert associated each term t with a numerical value in f 1; 0; 1g, where T 1 denotes the set of terms that were not in the ontology O1 and that are not relevant for the concept Ci; T 0 denotes the set of terms that were in O1 (and thus have been already validated as relevant); T 1 denotes the set of terms that were not in O1 but that are relevant for the concept Ci.
An overview of the results of terminological expansion is shown in Table 3.
The number of relevant terms retrieved in the terminological expansion (i.e., terms in T 0 or T 1) is equal to the 83% of the total number of new terms validated by the expert (TCi ). The 34% of those terms was not in the term sets of the initial ontology O1. As expected, the increment of new relevant terms is higher for the concepts that were associated with small term sets, such as illinois legislation, criminal procedure, and evidence. The number of irrelevant terms T 1 is limited with the exception of the concept evidence, because the criminal evidences usually consist in common objects that are used in a criminal context. These objects are thus associated with a generic terminology (e.g., garbage, suitcase) that cannot be associated per se to an evidence according to the legal expert. The new relevant terms are nally included in the new version O2 of the ontology that is used to automatically create a new training set for a second bootstrapping cycle. The new training set consists of 158,398 CD sentences (+37% with respect to the rst execution). In particular, the main increment of sentences is related to the concepts unit of measure (from 290 to 7,241 sentences) and evidence (from 2,830 to 33,417 sentences). These sentences are then used to train a new model M2 using the same neural network architecture of M1 and to enforce the execution of the knowledge enrichment steps. Finally, the accuracy of M2 obtained by cross-validation is 0.81 (+5.2%). 5
In this paper, we propose a bootstrapping approach for semi-automated legal knowledge extraction. Technical details about the use of multi-label classi cation techniques and black-box model explanation techniques are provided to show how we associate corpus documents with appropriate concepts in a reference ontology, and how we choose the terms that are decisive for determining the association between a document and a certain ontology concept, respectively. Preliminary results on a corpus of Court Decision documents are discussed to highlight the contribution of our proposed approach in real scenarios. Future work are about the extension of preliminary experiments on a larger corpus of Court Decision documents, and the comparison of obtained results by adopting di erent techniques for document annotation/embedding, document classi cation, and black-box model explanation.