There is a boom in digitizing the legal domain for use cases like classifying judgments [1,2,3], predictions [4,5], questions answering [6,7], finding similarities between judgments, and many more. Although much data is available in the legal domain, from court judgments to acts and deeds, its unstructured nature makes it inefficient and costly to process it to useful results. Accessibility and transparency of the key information have always been an issue. There is a requirement for a central place to keep this vast knowledge, and that too in an interoperable format.
With recent technological advances, AI applications can now process, understand and interpret the human language. Knowledge graphs [8] can represent large volumes of knowledge with their semantics providing easy access and structured querying abilities. Furthermore, when modeled following the Semantic Web standards, they can be used to reason, infer new information, or find any inconsistencies in data. Moreover, such knowledge graphs enable downstream applications such as question answering, dialogue, prediction, and classification systems.
In the literature, we find different models for creating a knowledge graph from unstructured documents. In recent years, neural networks such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Sequence-to-Sequence (Seq2Seq) models have shown promising results when used in the field of natural language processing (NLP) for sentiment analysis, information retrieval, and document classification. The state-of-the-art Information Extraction (IE) models use a supervised machine learning method that requires a huge amount of labeled training data. For general open-domain IE, publicly available labeled data is present, but domain-specific IE such as legal domain requires time-consuming data labeling tasks to create a large amount of training data. Although much work has been done in the legal domain for IE, the dataset available is not quite useful for every region. As the judicial system changes from country to country, so do changes in the legal terms, facts, artifacts, and law. There are many legal datasets available like in Chinese [9] and German [10] [1]. All these are specific to their region law; therefore, other regions beyond their boundaries cannot use it. For countries like India, where digitization of the judicial system is an ongoing process, it is challenging to find any proper dataset that will be useful for Information extraction. According to [11], creating the annotated data for a large corpus is time-consuming and too expensive. Moreover, while typing during the hearings in the courts, the court judgments get noisy due to misspellings, wrong placement of punctuation, and non-uniformity in the document structure. As for the linguistic part of the document, every judge has their vocabulary and interpretation; therefore, it becomes more difficult for the non-domain experts to comprehend legal text corpora and develop a dataset that satisfies all the system's requirements to process the documents with the uniform annotated data.
This work aims to explore a viable rule-based approach to systematically extract, annotate, and store the key entities of the legal domain as a knowledge base. The first step is to extract entities from the legal corpora using Information Extraction (IE) involving Named Entity Recognition (NER) and Relation Extraction (RE). The second step is to store, maintain, and update the retrieved knowledge in a graph database in a set of triples. As metadata, we have developed NyOn 1 (Nyaya Ontology) [12], a multilingual modular ontology for legal court judgments. NyOn has been developed taking entities from the Indian Supreme Court judgments taken from indiankanoon.org2 and is currently available in five languages (English, German, French, Spanish, and Hindi). NyOn is referred as the schema required for information extraction. It gives the broader aspect of the types of entities and the relations we need to identify from the legal court judgment documents. The knowledge graph thus constructed will be put to question answering to answer questions like:
• What are the different courts that are referred to in a specific case? • List of documents that fall under a specific type of jurisdiction? • Who were the parties involved in Union Of India vs Ex No. 3192684 W Sep. Virendra judgment on 7 January 2020? • How many judgments were passed in 2002?
• List all judgments ordered by CJI N V Ramana.
• What are the names of the witnesses in a specific case?
The main contributions of this paper are (1) An analysis of existing approaches for information extraction and knowledge graph construction, and (2) Construction of a knowledge graph for the Indian Supreme Court judgments and a quantitative evaluation of the proposed framework. The dataset and the source code are publicly available on GitHub3 . The rest of the paper is organized as follows. Section 2 provides a approach followed for constructing the knowledge graph from Indian Supreme Court Judgments, Section 3 provides the evaluation results, while Section 4 discusses the related work focusing on legal knowledge graph construction. Section 5 elaborates the conclusions derived from the work and presents future work ideas.
In this paper, we use the Indian Supreme Court Judgements to create the Knowledge Graphs, which will facilitate downstream applications such as question answering, judgment prediction, etc. Figure 1 demonstrates the general architecture of the knowledge graph construction approach we followed. It mainly consists of a three-step pipeline consisting of data prepossessing, entity extraction, relation extraction, and triple construction steps. Each of these steps will be described in the following subsections.
The input corpus to our Knowledge Graph construction pipeline contains 44,366 reportable and non-reportable court judgments published by the Supreme Court of India from 1947 to 2020 in plain text. The documents have been scraped using a Python-based scraper from IndianKanoon.org, a free-to-use case search engine for Indian legal documents sourced from the various court and tribunal repositories. The dataset comprises court judgments for cases from various legal subdomains (civil law, criminal law, property law, etc.). All text documents in the dataset are assumed to be in English language. Fig. 2 refers to a snapshot of one Indian Court Decision document. We call this data the caption data as no sentences are present in this part of the document. The text highlighted in orange represents the data, and the text highlighted in cyan color represents the entity type. Even though this data is not present in the sentence format, it represents most of the legal court decision documents useful for the knowledge graph creation. In the relation extraction part of the paper, we have discussed how we identify the relationship between the available entities in the caption format.
After selecting documents, we perform the data preprocessing on the corpus. In the GATE4 software (General Architectural of Text Engineering), we already have the Language Processing tools in ANNIE 5 (A Nearly-New Information Extraction System) such as POS Tagger, ANNIE Sentence Splitter, ANNIE English Tokenizer, etc. All these processing tools are preloaded into the software for data preprocessing.
Entity extraction is the step performed after data preprocessing for information retrieval. Here all the useful entities are identified from the unstructured data that forms relevant, useful information about the context of the document. After the data is scraped from indiankanoon.org 2 , the rules are written in JAPE (Java Annotation Patterns Engine) for the GATE software (General Architectural of Text Engineering) with the help of NyOn [12] Ontology for recognizing the entities in the documents. As discussed in the above subsection, these rules are given as a JAPE Transducer to ANNIE as a processing resource with other processing resources like POS tagger and Sentence Tokenizer. While annotating the entities using GATE API, we observed that it takes approximately one hour to annotate 100 documents as the length of the documents is quite long. One of the rules used for entity recognition is listed in Table 1.
After the data is annotated using the JAPE rules, the corpus is stored with the annotation state having the inline XML format of all the documents. We use a small Python script to convert all the inline XML documents into standard text documents. The converted standard text documents are then saved separately for the next phase of the system, which is relation extraction. The list of the entities identified in the court decision documents is presented in Table 2.
The entities extracted in the previous phase are passed as input in this phase. Here the relations are identified between the extracted entities. The list of the various relations that are identified between different entity types (Object Properties) are listed below in Table 3.
As discussed in the earlier section, in the caption data portion of the document, there are no sentences in between the entities; we use a small code to map the relation between such entities and the data with no text around. To refer to the type of relation, we refer to the NyOn {Token.kind == "word", Token.category != "CC"} | {Token.string == ". ", Token.category != "CC"} )+ :Name ({Token.kind == "punctuation"})? (( {Token.kind == "word", Token.string != "judgment"} | {Token.string == ". ", Token.sting != "judgment"} )+) :Name2 ) -> :Type.COURT_OFFICIALS = {kind = "BENCH", rule = "BENCH"}, :Name.JUDGE = {rule = "BENCH"}, :Name2.JUDGE = {rule = "BENCH"} Ontology, where the relation between the entities has been described. The NyOn ontology works as a base for us for entity extraction and relation extraction from the data. The list of relations between the entities and the values (Datatype Properties) is also provided in Table 4.
A knowledge graph consists of triples where every triple is represented in the form of subjectpredicate-object. After extracting the entities and the relations from the data, the next step is to create the triple to form a knowledge graph. The extracted entities and relations are stored corresponding to each other in the form of lists. 𝐸 = {𝑒 1 , 𝑒 2 , 𝑒 3 , ..., 𝑒 𝑛 } represents the entity set and 𝑅 = {𝑟 1 , 𝑟 2 , 𝑟 3 , ...𝑟 𝑚 } represents the relation set. To form a triple, we follow the given format 𝑇 = {𝑒 𝑎 , 𝑟 𝑘 , 𝑒 𝑏 } where {𝑒 𝑎 , 𝑒 𝑏 𝜖 𝐸, 𝑟 𝑘 𝜖 𝑅}.
Once the RDF model is constructed, it can be materialized into any RDF serializations such as RDF/XML, Turtle, or N Triples. The triples then can be loaded into an industry-standard triple store such as Blazegraph, Apache Jena, and Virtuoso and queried through a SPARQL endpoint. We are also currently working on exposing such triples as Linked Open Data with dereferenceable HTTP URIs.
This section will present the preliminary evaluation of the proposed approach. This evaluation focused on two aspects of the knowledge base construction process: named entity extraction and relation extraction of Indian Court Judgement documents.
We have evaluated both the rule-based Named Entity Recognition and Relation Extraction components of the proposed approach. As there was no academic benchmark for the type of documents in our use case, we have manually created a gold standard with a sample of documents with the help of human annotators who are domain experts.
There were no existing academic benchmarks for testing the entity and relation extraction on the Indian court judgment documents. Thus, we decided to create a gold standard for evaluating the performance of our proposed approach to entity extractions and relation extraction. We randomly selected five documents from the corpus and annotated them for gold entities and relations with the help of domain experts. The NyOn ontology [12] guided the gold entity types and gold relations that were annotated. Two annotators annotated each document to ensure the inter-annotator agreement, and any conflicts were resolved using a third domain expert. The annotated documents contained 363 named entities of 10 different class types and 154 triples of 6 relation types. We plan to increase the size of the gold standard for future work.
As evaluation metrics, we have used commonly used precision-recall measures. We have a set of gold annotations and machine-generated output from our rule-based pipeline for both entities and relations. Table 5 shows some example output of rule-based machine NER, including correctly and incorrectly identified entities. In this context, the precision was calculated as the percentage of the machine-generated entities/relations that were correct according to the gold annotations and recall as the percentage of total gold entities/relations identified in the machine output.
Table 6 shows the results of NER by individual documents level, and Table 7 shows the result for each class type level. We notice that our rules identify entities with high precision but suffer from recall issues. Furthermore, we have noticed that there is a wide variation of the recall values across different documents as well as different entity types. Some types, such as dates higher recall, while others, such as court decisions or law, have a lower recall. Based on these results, we are performing error analysis and working on improving the recall of our rules to capture most of the relevant entities.
Table 8 shows the results of relation extraction by individual documents level, and Table 9 shows the result by each relation type level. In contrast to entity extraction, here we see that both precision and recall have been affected by our rules. The results are expected because relation extraction is a more complex task than entity extraction, and entity extraction recall issues are propagated to relation extraction. Furthermore, there are several relation types, such as hasDecision and hasJudge, for which we have not implemented the rules.In future work, we plan to perform error analysis to improve the current set of rules and implement new rules to cover the relation types that are not currently covered.
Information extraction approaches are emerging and are making their space in various domains, whether biomedical, social-media, legal, etc. These approaches can play an auspicious role in building an enriched knowledge base, specifically for legal domains in maintaining the huge volume of legal information scattered on various portals. Information extraction approaches can also provide a way for developing various applications like question-answering systems for the legal domain, judgment prediction, dispute resolution, etc. The most commonly used approaches for information extraction are; first finding the named entities present in the unstructured data and then finding the relation between those entities that have been identified. Named Entity Recognition (NER) is somewhat challenging in the legal domain due to various legal terms, abbreviations, references made to the Acts and Laws, etc. Then after identifying these domainspecific entities, relation extraction (RE) is also of great importance as it identifies the relations between any two entities, which leads to triple construction and later knowledge graph creation.
Various review and survey articles on NER and RE focus on standard datasets, and even if most of the research work is done in the legal domain, the available dataset is not useful for the Indian context. We come across articles that focus on information extraction from unstructured text in the context of legal documents that do not cover the Indian context mostly. It is also discussed by Fernàndez-Cañellas et. al. [13] that Natural Language Processing (NLP) alone cannot guarantee the validity of the facts to be populated in the knowledge graph and data validation methods also need to be taken into account. In the legal context, this might mean the validity of the legal facts about judgments, cases, and stakeholders in the Knowledge Graphs (KG).
The relationship extraction depends highly on the syntactic and semantic analysis of sentences in a rule-based approach. Dragoni et al. [14] discussed how to combine the NLP approaches for rules extractions in the legal domain. The author uses the Deontic Lightweight Ontology called normonto, which represents and models the legal concepts and specifies the lexicons used for legal expressions like permission, prohibition, and obligation. Thomas and Sangeetha in [15] discuss rule-based entity extraction represented as regular expressions to detect entity mentions in Indian judicial texts using specific patterns or trigger words. The GATE tool and Java Annotation Patterns Engine (JAPE) grammar rules can support rule-based extraction; however, basic entities must be identified first. Andrew and Tannier in [16] take a hybrid approach by using a statistical Conditional Random Field (CRF) model and legal domain-specific JAPE rules in GATE gazetteer to annotate their dataset. Eftimov et. al. [17] proposes the rulebased approach for NER identification for evidence-based dieatry recommendations. Poudyal et. al. [18] also proposes a rule-based approach for NER and Relation Extraction. The author made use of the C programming language for rules extraction.
To sum up, as the paper focuses on the rule-based approach for named entity recognition and relation extraction for legal court judgment documents, we discussed various research work done in this area and how we can enhance the NER and Relation extraction for the legal domain by writing JAPE rules in context to India Judicial System. There exist many differences in the existing rule-based approaches, like [15] does not talk about the jurisdiction entity, and the paper focuses only on the Criminal Cases. [14] take the entity that expressions only legal expressions and not all the information related to the case document. [18] considers only entities related to the plaintiff, court, court staff, and decision. The authors did not consider other entities like petitioner, case jurisdiction, documents, evidence, etc. In this paper, we focus on extracting many such entities that are not considered in the works presented till now and making the information extraction comprehensive for the knowledge graph creation. Currently, we have focused only on the rule-based approach, but as the next step, we will use the Machine Learning and Deep Learning models for NER and Relation Extraction for the legal court decision documents.
In this paper, we have presented a pipeline for constructing Knowledge Graphs from the Indian Supreme Court Decisions corpus. The pipeline consists of data preprocessing, entity extraction, relation extraction, and triple construction. The generated triples are represented in RDF using the NyOn ontology and are stored in a triple store. The Knowledge Graph can be queried using SPARQL and can be used to build downstream applications such as Knowledge Base Question Answering with complex questions such as the ones that require aggregations or multi-hop reasoning (which are not possible with simple document retrieval or keyword search). In addition, the generated Knowledge Graph can become a useful resource for other AI tasks such as judgment predictions, case clustering, and classification.
The results are reasonably good but still have room for improvement. We plan to improve our rules to increase recall in entity recognition and both precision and recall in relation extraction in future work. We are also planning to explore neural approaches based on semi-supervised learning by using distant supervision data or using the output of the rule-based system as training data. Furthermore, we are planning to publish the generated data as Linked Data with public access and develop a semantic web portal to expose the knowledge base to various use cases.
This work is supported by the IHUB-ANUBHUTI-IIITD FOUNDATION set up under the NM-ICPS scheme of the Department of Science and Technology, India