Evidences -typically based on documents (e.g., letter, receipt, report, agreements, affidavits) and physical objects (e.g., knife, guns, photos, phone call data records) -are often used by lawyers in their arguments during a court case. The observations made through these evidences may have a significant effect on the judges' final decision. In order to develop a deeper understanding of the past court cases, it is valuable to identify various Evidences discussed in these cases and the observations which are made about them or through them. Such information about evidences has several applications such as understanding and representing legal arguments, determining strengths and weaknesses of those arguments, identifying relevant past cases in which similar evidences were discussed, etc.
In this paper, we discuss Natural Language Processing (NLP) based techniques for extracting information regarding Evidences mentioned in court judgement documents. We propose to represent this information in a rich semantic structure -Evidence Structure defined as an Evidence Information Model. Along with Evidences, we also identify and represent Witness Testimonies using the same Information Model. Initially, we discuss a twostep approach for identifying evidence and testimony sentences. In the first step, linguistic rules are used to determine whether a sentence contains any evidence or (ASAIL 2021), June 25, 2021, SΓ£o Paulo, Brazil. Envelope ali.basit@tcs.com (B. Ali); ravina.m@tcs.com (R. More); sachin7.p@tcs.com (S. Pawar); gk.palshikar@tcs.com (G. K. Palshikar) testimony information. Here, we use the rules proposed in Ghosh et al. [1] for identification of witness testimonies and design new rules for identification of evidence sentences. In the second step, we train a Weakly Supervised Sentence Classifier whose training data is automatically created using the sentences identified by the linguistic rules. It is a multi-label classifier which predicts whether any sentence contains an Evidence or Witness Testimony or both. Once all the Evidence and Testimony sentences are identified from the corpus of court judgements, we propose a Semantic Role Labelling (SRL) [2] based technique to automatically instantiate Evidence Structures for these sentences.
To demonstrate effectiveness of the proposed Evidence Structure, we discuss its use in the prior case retrieval application. We propose a matching algorithm for computing semantic similarity between a query and a sentence in a court judgement document. This algorithm makes use of the proposed Evidence Structure in which both the query and the sentence are represented, resulting in a semantically sound similarity score between them.
Previously, Ghosh et al. [1] identified witness testimonies from court case documents and used them for retrieving relevant prior cases. We propose that considering only witness testimonies leads to loss of key information regarding evidences mentioned in a case. Hence, we identify and use information about various evidences mentioned in the case documents leading to much better prior case retrieval performance as demonstrated in the experiments section. Moreover, Ghosh et al. [1] use a much limited semantic structure to represent information regarding events mentioned in witness testimonies. This structure does not capture important semantic information like whether event is negated, what are the causes behind the event, the manner in which event takes place etc. We propose a richer semantic structure addressing these limitations and design a suitable semantic matching algorithm for that structure. To the best of our knowledge, this is the first paper to apply NLP techniques for the extraction of evidence information from court judgements and demonstrate its use for retrieving relevant prior court cases.
The purpose of Evidence Information Model is to define a suitable structure to represent evidence information in court judgements. In this section, we describe Semantic Role Labelling in brief and how it is used to define our proposed Evidence Structure.
Semantic Role Labelling (SRL) is a technique in Natural Language Processing that identifies verbs/predicates in a sentence, finds phrases connected to every predicate and assigns an appropriate semantic role to every phrase. By doing so, SRL helps machines to understand the roles of important words within a sentence. Following are some key semantic roles identified for a verb/predicate (often corresponding to an action or event) by SRL techniques: ARG0 : proto-agent or someone who performs the action denoted by the verb ARG1 : proto-patient or someone on whom the action is performed on ARGM-TMP : the time when the event took place ARGM-CAU : the cause of the action ARGM-PRP : the purpose of the action ARGM-LOC : the location where the event took place ARGM-MNR : the manner in which the action took place ARGM-NEG : the word indicating that the action did not take place Consider the following example sentence:
O n A u g u s t 2 5 , 1 9 6 5 , t h e b a n k d i s h o n o u r e d t h e c h e q u e d u e t o i n s u f f i c i e n t b a l a n c e .
The various semantic roles to the verb d i s h o n o u r e d are annotated as follows:
We use the predicates and corresponding arguments obtained from the pre-trained AllenSRL model [3] to instantiate our Evidence Structure for the queries and candidate sentences.
The Evidence Information Model represents every Evidence Sentence giving information about one or more Evidence Objects in an Evidence Structure. We define an Evidence Object as one of the objects presented by the counsels to the judge along with the information and findings about the crime. It is thus, a physical entity that can furnish some degree of support, contradiction or opposition to some legal arguments. Some examples of Evidence Objects are:
β’ Documents (autopsy β’ action took place (e.g., a s p e r t h e c h a l l a n , f r a u d u l e n t l y , w i l f u l l y ) Table 1 shows examples of some Evidence Sentences along with the corresponding Evidence Structure Instances. In some cases, Observation Frame may be empty due to absence of ObservationVerb. In such cases, EvidenceObject may be present as a part of any argument in Evidence Frame. E.g., t h e c h e q u e is present as π΄ 1 in the Evidence Frame of the first sentence in Table 1.
Information about named entities and their types present in various arguments of Observation or Evidence frame is important. Hence, the Observation Frame and Evidence Frame are also enriched by annotating entities such as
Witness Information Model: Information in witness testimonies can also be represented using the same Evidence Structure. The statement verbs used in witness testimony sentences (e.g., s t a t e d , s a i d ) are treated similar to observation verbs and represented using Observation Frames. Similarly, other action/event verbs mentioned in witness testimony sentences are represented using Evidence Frames. Table 2 shows examples of some Witness Sentences along with the corresponding Evidence Structure Instances. The advantage of representing information about evidences and witness testimonies in the same structure is that we can make use of both these sources of information seamlessly, for prior case retrieval.
In this section, we describe our overall methodology which consists of two phases. In the first phase, we identify Evidence and Testimony Sentences using linguistic rules and weakly supervised sentence classifier. In the second phase, we instantiate the Evidence Structures for these identified sentences. For all our experiments, we use a corpus of 30,032 Indian Supreme Court judgements ranging from the year 1952 to 2012.
We identify Evidence and Testimony sentences using a two-step approach. In the first step, we use linguistic rules to obtain Evidence and Testimony sentences. In the second step, we use these sentences to train a sentence classifier.
x h i b i t s P -1 0 t o P -1 2 r e g a r d i n g h i s p r i v a t e a n d S t a t e p r o p e r t i e s , w e r e a p a r t t h e r e o f .
Step I: Linguistic Rules based Approach: As there are no publicly annotated datasets for identification of Evidence and Testimony sentences, we rely on linguistic rules to identify these sentences with high precision as our first step. The linguistic rules for identifying Evidence sentences are described in detail in Table 3. These rules identified 62,310 sentences as Evidences from our corpus. As there is no annotated dataset, in order to estimate the precision of the linguistic rules we use random sampling strategy. We selected a set 100 random sentences identified as Evidence by the linguistic rule, and got them verified by a human expert. The precision turned out to be 85%. Similarly, we use the linguistic rules proposed in Ghosh et al. [1] for identifying Testimony and non-Testimony sentences where the reported precision is around 85%. These rules identified 36,473 sentence as Testimony and 14,234 sentences as non-Testimony from the same corpus.
Step II: Weakly Supervised Sentence Classification:
We observed that although the linguistic rules identify Evidence and Testimony sentences with high precision, they may miss to identify some sentences which should have been identified as Evidence or Testimony (see examples in Table 4). Hence, we train a supervised sentence classifier to improve overall recall of identification of Evidence and Testimony sentences. The classifier used is a BiLSTM-based [4] multi-label sentence classifier whose architecture is depicted in Figure 1. This classifier is weakly supervised since its training data is automatically created using the sentences identified by the linguistic rules as follows:
β’ The classifier has two outputs -i) first output predicts a binary label indicating whether the sentence contains Evidence or not and ii) second output predicts a binary label indicating whether the sentence contains Testimony or not.
β’ 1824 sentences are labelled as Evidence and Testimony both. These sentences are identified as Evidence as well as Testimony by both the sets of linguistic rules.
β’ 60486 sentences are labelled as Evidence and non-Testimony. These sentences are identified as Evidence by the rules but not as Testimony.
β’ 34649 sentences are labelled as non-Evidence and Testimony. These sentences are identified as Testimony by the rules but not as Evidence.
β’ 14234 sentences are labelled as non-Evidence and non-Testimony. These sentences are identified as non-Testimony by the rules and not identified as Evidence.
After this classifier is trained, we use it to classify all the remaining sentences in the corpus. These sentences are neither identified Evidence by the Evidence rules nor as Testimony/non-Testimony by the Testimony rules. Using the prediction confidence, we selected top 10,000 sentences classified as Evidence and top 5,000 sentences classified as Testimony. Table 4 shows some examples of sentences identified as Evidence by the classifier but not by the linguistic rules. To estimate the precision, we again employed the random sampling strategy. We selected 100 random sentences each from these high confidence Evidence and Testimony sentences and a human expert verified them. The precision of 72% is observed for Evidence sentences and 68% for Testimony sentences. The precision of the sentence classifier is lower as compared
Linguistic Rules for identifying Evidence Sentences Any sentence π should satisfy the following conditions in order to be identified as an Evidence Sentence: E-R1 π should contain at least one Evidence Object as defined in Section 2.2. The list of words corresponding to evidence objects is created automatically by using WordNet hypernym structure. We create a list of all words for which the following WordNet synsets are ancestors in hypernym treea r t i f a c t (e.g., g u n , c l o t h e s ), d o c u m e n t (e.g. r e p o r t , l e t t e r ), s u b s t a n c e (e.g. k e r o s e n e , b l o o d ). This list is looked up to identify evidence objects in a sentence. E-R2 π should contain at least one action verb from a pre-defined set of verbs like t a m p e r , k i l l , s u s t a i n , f o r g e OR π should contain at least one observation verb from a pre-defined set of verbs like r e p o r t , s h o w , f i n d . Both the pre-defined sets of verbs are prepared by observing multiple example sentences containing evidence objects. E-R3 In the dependency tree of π, the evidence object (identified by E-R1) should occur within the subtree rooted at the action or observation verb (identified by E-R2) AND there should not be any other verb (except auxiliary verbs like h a s , b e e n , w a s , w e r e , i s ) occurring between the two. This ensures that the evidence object always lies within the verb phrase headed by the action or observation verb.
Example of Evidence Sentences Identified by the Classifier but not by the linguistic rules S 1 : R a j u P W 2 to the rules because, it is applied on a more difficult set of sentences for which the linguistic rules fail to identify any label. At the end of this two-step process (linguistic rules followed by the sentence classifier), we have 112,401 sentences identified either as Evidence or as Testimony.
In this phase, we discuss the technique of instantiating Evidence Structures for sentences identified as Evidence or Testimony in the previous phase. We used Semantic Role Labelling [3] to identify and fill the arguments of the Observation Frame and the Evidence Frame in the Evidence Structure Instance for every candidate sentence. This is demonstrated in Algorithm 1. We identify Observation Frames using Observation Cue Verbs. For each of these Observation Frames we identify the corresponding Evidence Objects and Evidence Frames. For identifying Evidence Objects, we first use Named Entity Recognition [5] and WordNet based Entity Identification [6] to identify the named entities in the sentence and annotate them in the Frames extracted. The Evidence Objects in a phrase are then obtained by selecting named entities annotated as one of the following types
In order to demonstrate effectiveness of the proposed Evidence Structure, we apply it for the task of prior case retrieval. This task is to create a relevance-based ranked list of court judgements (documents) in our corpus for a query. In order to retrieve prior cases for a query, we represent the query using an Evidence Structure Instance (πΈπ£ππ‘ππ’ππ‘ π ). We then compute the similarity of query instance πΈπ£ππ‘ππ’ππ‘ π against each document instance πΈπ£ππ‘ππ’ππ‘ π· obtained from every Evidence or Testimony sentence in the corpus. Algorithm 2 shows the steps for computing similarity. We refer to this algorithm as ππππππ‘πβ because of its semantic matching ability. We use cosine similarity between the phrase embeddings of corresponding arguments of the Evidence Structure Instances to compute similarity. For obtaining phrase embedding for any phrase (referred as πβπππ ππ ππ in Algorithm 2), we consider the average of GloVe word embeddings [7] of the words in that phrase excluding stop words. We compute the similarity scores within corresponding arguments of both the frames. These scores across different arguments are combined to get a final similarity score between πΈπ£ππ‘ππ’ππ‘ π and πΈπ£ππ‘ππ’ππ‘ π· . We multiply the final similarity score by a Sentence BERT [8] based similarity score between the query and the sentence containing πΈπ£ππ‘ππ’ππ‘ π· . This is necessary because errors in the automated SRL tool may lead to imperfect Evidence Structure instances in some cases. A sentence similarity score which is not dependent on any such structure within the sentences provides a complementary view of capturing sentence similarity. Finally, the overall relevance score of the query with a document is the maximum score corresponding to any Evidence Structure Instance πΈπ£ππ‘ππ’ππ‘ π· obtained from the document. Table 5 shows a running example of how a similarity score is computed between an Evidence Structure Instance (πΈπ£ππ‘ππ’ππ‘ π ) from a query and an Evidence Structure Instance (πΈπ£ππ‘ππ’ππ‘ π· ) from a document in the corpus.
While the task of evidence extraction from legal documents is related to several information retrieval and NLP tasks, there are no established baselines for the task. Bellot et al. [9] and Cartright et al. [10] have worked on Evidence Retrieval that identifies whole documents that contain an evidence. On the other hand, Rinott et al. [11] use Context Dependent Evidence Detection to find evidence information present in a sentence on a phrase level. As compared to this, we identify both Evidence and Testimony sentences, represent them in a rich structure and also use that for prior case retrieval. This is a challenging task due to the inherently complex nature of legal texts and the finer granularity of matching involved. Ji et al. [12] propose an Evidence Information Extraction system which captures evidence production paragraph, evidence cross-examination paragraph, evidence provider, evidence name, evidence content, crossexamination party and cross-examination opinion relating to an evidence presented in the court. While this technique may suit well for Chinese court records that follow a relatively structured representation, it does not suit well to the Indian Court Records that contain descriptive and varied formats of the court proceedings.
Gomes and Ladeira [13] and Landthaler et al. [14] performs full text search for legal document collection by obtaining word2vec word embeddings and then taking their average for computing similarity. However, computing the average of the embeddings gives a lossy representation where relative order of the words is lost. In contrast, we represent the sentences using the Evidence Structure Instances, where the structure itself takes care of the relative ordering. Gomes and Ladeira [13] demonstrate BM25 and TF-IDF for Prior Case retrieval. In our results section, we demonstrate the comparative poor performance of BM25 and TF-IDF in handling corner cases.
In this section, we discuss our experiments including the dataset, baseline techniques, evaluation metrics and analysis of results.
We use the Indian Supreme Court judgements from years 1952 to 2012 freely available at http://liiofindia.org/in/ cases/cen/INSC/. There are 30032 court (documents) containing 4,111,091 sentences where average sentence length is 31 words and standard deviation of 24.
For the task of prior case retrieval, we implement two baseline techniques:
β’ BM25: It is a popular TF-IDF based relevance computation technique. We use the BM25+ variant1 as described in Trotman et al. [15]. This technique uses a bag-ofwords approach that ignores the sentence structure. We input : π (sentence), ππ πΏ_π (set of semantic frames in π as per any semantic role labeller, each frame π consists of a predicate π.π and corresponding arguments π.π΄π πΊ 0 , π.π΄π πΊ 1 , π.π΄π πΊ 2 , π.π΄π πΊπ-πΏππΆ, etc.) output : πΈπ£ππ‘ππ’ππ‘π = Evidence Structure Instances of the input sentence consisting of πππ πππ£ππ‘ππππΉ ππππ (ππΉ) and πΈπ£πππππππΉ ππππ (πΈπΉ) parameter : ππ΅π_π πΈπ π΅π = {accept, a d d , a d m i t , a g r e e , a l l e g e , a l l o w , a l t e r , a p p r i s e , 5). As we are not resolving co-references, we are missing a few relevant documents. E.g., ππ π πΈ does not assign a high score for the following document for query π 3 (see Table 6) - . This is because t h e m in the Evidence Structure instance for s h o t is not explicitly known to correspond to t h e p o l i c e in the previous sentence.
In this paper, we discussed several NLP techniques for identifying evidence sentences, representing them in the semantically rich Evidence Structure and retrieving relevant prior cases by exploiting it. The proposed techniques are weakly supervised as they do not rely on any manually annotated training data, except for the human expertise in designing the linguistic rules. Keeping in mind the importance of witness testimonies in addition to evidences, we also extracted and represented the witness testimonies using the same Evidence Structure. For the application of prior case retrieval, we evaluated our proposed technique along with several competent baselines, on a dataset of 10 diverse queries. We demonstrated that our technique performs comparably for most of the queries and is the best considering the overall performance across all 10 queries. The results highlight the contribution of evidence and testimony information in improving prior case retrieval performance.
In future, we plan to apply advanced representation learning techniques for learning dense or embedded representation of an entire Evidence Structure instance. Also, we plan to automatically determine the best suited retrieval technique (BM25, Sentence-BERT or SemMatch) for any query based on its nature. We plan to explore ensemble of multiple retrieval techniques for improving prior case retrieval performance further.
Evaluation of various techniques for the task of prior case retrieval. All entries are of the form (R-Prec; Avg. Precision). (Note: Our proposed approach ππππππ‘πβ is referred as ππ. Underlines indicate the best performing results for each query across multiple techniques) W h a t a r e t h e c a s e s w h e r e β― π Siamese-BERT networks to obtain more meaningful sentence embeddings as compared to vanilla BERT [16]. We used the pre-trained model b e r t -b a s e -n l i -s t s b -m e a nt o k e n s to obtain sentence embeddings for sentences. Following Ghosh et al. [1], we use the pre-trained model as it is and did not fine-tune it further. This is because such fine-tuning needs annotated sentence pairs with labels indicating whether the sentences in the pair are semantically similar or not. Such annotated dataset is expensive to create and our aim is to avoid any dependence on manually annotated training data. Similar to Ghosh et al. [1], we used sentence embeddings obtained by Sentence-BERT to compute cosine similarity between a query sentence and a candidate sentence in a document. The overall similarity of a document with a query is the maximum cosine similarity obtained for any of its sentences with the query sentence. We use 3 settings considering different sentences in each document:
β’ ππ΅ π πΈ : Only Testimony or Evidence sentences β’ ππ΅ π : Only Testimony sentences β’ ππ΅ πΈ : Only Evidence sentences
All the baseline techniques and our proposed technique are evaluated using a set of queries and using certain evaluation metrics to evaluate and compare the ranked lists produced by each of these techniques.
Queries: We chose 10 queries (shown in Table 6) which represent cases and evidence objects of diverse nature (domestic violence, financial fraud etc.).
We created a set of gold-standard relevant documents for each query using the standard pooling technique [17]. We ran the following techniques to produce a ranked list of documents for each query -π΅π25 πππ , π΅π25 π πΈ , ππ΅ π πΈ , and our proposed technique ππππππ‘πβ π πΈ . We chose top 10 documents from the ranked list produced by each technique. Human experts verified the relevance of each document for the query. Finally, after discarding all the irrelevant documents, we got a set of gold-standard relevant documents for each query 2 . Metrics: We used R-Precision and Average Precision as our evaluation metrics [17].
This calculates the the number of relevant documents observed at π .
This captures the joint effect of Precision and Recall. It computes precision at each rank of the predicted ranked list and then computes mean of these precision values.
2 This dataset can be obtained from the authors on request
Table 6 shows comparative evaluation results for various baselines and our proposed technique. Average performance of π΅π25 π πΈ is better than π΅π25 πππ indicating that considering only Evidence and Testimony sentences for representing any document, results in better prior case retrieval performance. Other two baselines ππ΅ (Sentence-BERT) and ππ (our proposed technique ππππππ‘πβ) also consider only Evidence and Testimony sentences rather than considering all the sentences in a document. All the baselines which consider only Testimony sentences, perform poorly as compared to the corresponding techniques using both Testimony and Evidence sentences. This highlights the importance of evidence information as compared to using only witness testimony information for prior case retrieval as done in Ghosh et al. [1].
Considering the average performance across all the 10 queries, our proposed technique ππ π πΈ is the best performing technique in terms of both R-Prec and AP. The performance of ππ π πΈ is the most consistent across the diverse queries. It achieves minimum R-Prec of 0.24 (for π 1 ) as compared to other baselines like π΅π25 πππ , π΅π25 π πΈ and ππ΅ π πΈ which have minimum R-Prec of 0 for some queries. As described in Algorithm 2, ππ uses Sentence-BERT based similarity within sentences for producing an enhanced matching score. We experimented with a variant of ππ which does not rely on Sentence-BERT based similarity. This variant resulted in average R-Prec of 0.36 and MAP of 0.30 across all the 10 queries. Although this is lower than ππ π πΈ performance, the R-Prec is still comparable with π΅π25 π πΈ (avg R-Prec of 0.36) and better than that of ππ΅ π πΈ (avg R-Prec of 0.28).
For some queries, it is important to have some semantic understanding at sentence-level. For example, π 4 , which contains "negation", ππ΅ and ππ can capture the query's meaning in a better way. ππ such negations in a more principled manner as the Evidence Structure Instance captures negation as one of its arguments.
For ππ, the maximum matching score achieved for any Evidence Structure Instance in a document, is considered as the overall matching score with the whole document. In contrast, π΅π25 based techniques directly compute matching score for the whole document as they do not rely on sentence structure. This is one limitation of ππ which we plan to address as a future work. However, as ππ computes matching scores for individual Evidence Structure instances, it is able to provide better interpretation for each relevant document in terms of the actual sentences which provided the maximum matching score. Analysis of errors: We analyzed cases where ππ π πΈ was assigned a lower score to a relevant document or a higher score to a non-relevant document. We discovered 3 main reasons -missing or incorrect arguments within Evidence Structure instances, misleading high similarity between argument phrases and presence of co-references. Consider the following sentence for which ππ π πΈ incorrectly assigns a high score for query π 5 (see Table 6) -T h e p o l i c e r e p o r t a l s o r e v e a l s t h a t