In this paper, we present the solution of the team TRDDC Pune for the Artificial Intelligence in Legal Assistance(AILA) track 1 task on Precedent Retrieval in FIRE 2019. The task was to identify relevant legal prior cases for a legal query from a dataset of about 2,914 documents of cases that were judged in the Supreme Court of India. We used Named Entity Recognition to preprocess the case documents and the input query. We then ranked the preceding case documents using TF-IDF and BM25 algorithms. The results of our approach are comparable to the top ranked run on the task leaderboard.
In the past, substantial work has been done on designing and constructing the corpora
of legal cases for legal retrieval. Ontologies and Natural Language Processing are being
used to extract case factors and participant roles[
Artificial Intelligence for Legal Assistance (AILA) track challenge had 2 subtasks. Sub-task 1 was about identifying relevant prior cases. The participants were provided with 2,914 case documents that were judged in the Supreme Court of India. The participants were provided with 50 legal queries, each describing a situation. The task was to retrieve the most relevant Precedents among the 2,914 case documents for a given query.
A set of 2-3 relevant case documents was provided per query for the first 10 queries
as test data. The participants had to perform relevance ranking for the remaining 40
queries. Refer [
To find the relevant Precedents for a given query we followed the following steps: Step 1: Pre-process all the case documents to build a search corpus (Section 4.2) Step 2: Pre-process the query (Section 4.3)
Step 3: Rank the Precedents from the corpus using the query (Section 4.4) 4.1
The queries and the case documents contained substantial information about names, places, organizations, currencies, time, etc. that are specific to the case (E.g. ‘Gov. of Tamil Nadu’, ‘Indian Oil Corporation’, ‘13 Rs.’, ‘January afternoon’, etc.). Such information can be ignored to focus on events such as ‘murder’, ‘bribery’, ‘stole’, etc. that give primary information about the situation to perform relevance ranking.
As the first step, we prepared the corpus according to our intuition for query extraction.
We used spaCy[
A case document can contain 10-40 paragraphs on an average. These paragraphs give information about the background of the case, the situation and the judgements. We were interested to compare the results of performing a match on the whole case document versus on these individual paragraphs. So, we split every case document into individual paragraphs. Improvement after submission: We decided to take the entire case document without splitting it into paragraphs
We used spaCy’s tokenization to break down the paragraphs into individual words called tokens. We performed NER on the tokenized sentences to find named entities such as places, things, person, currency, time, etc.
Using the Named Entities identified in the previous step and a predefined list of stop words by spaCy, we removed the Named Entities and the Stop Words from the case documents (Fig. 1). 1. These appeals are filed ['1', '', 'these', 'appeals', 'are', against the order dated 'filed', 'against', 'the', 'order', 'dat29.3.2001 passed by the ed', '29.3.2001', 'passed', 'by', 'the', Madras High Court allowing 'madras', 'high', 'court', 'allowing', Crl.O.P. Nos.2418 of 1999. 'crl.o.p.', 'nos.2418', 'of', '1999'] 2. The appellant (Indian Oil Corporation, for short 'IOC') entered into two contracts, one with the first respondent (NEPC India Ltd.) and the other with its sister company Skyline NEPC Limited ('Skyline' for short). According to the appellant, in respect of the aircraft fuel supplied under the said contracts, the first respondent became due in a sum of Rs.5,28,23,501 and Skyline became due in a sum of Rs.13,12,76,421 as on 29.4.1997.
['2', 'the', 'appellant', 'indian', 'oil', 'corporation,', 'for', 'short', 'ioc', 'entered', 'into', 'two', 'contracts,', 'one', 'with', 'the', 'first', 'respondent', 'nepc', 'india', 'ltd', 'and', 'the', 'other', 'with', 'its', 'sister', 'company', 'skyline', 'nepc', 'limited', 'skyline', 'for', 'short', 'agreeing', 'to', 'supply', 'to', 'them', 'according', 'to', 'the', 'appellant,', 'in', 'respect', 'of', 'the', 'aircraft', 'fuel', 'supplied', 'under', 'the', 'said', 'contracts,', 'the', 'first', 'respondent', 'became', 'due', 'in', 'a', 'sum', 'of', 'rs.5,28,23,501', 'and', 'skyline', 'became', 'due', 'in', 'a', 'sum', 'of', 'rs.13,12,76,421', 'as', 'on', '29.4.1997.']
On reading the queries, we found out that the queries contained information such as background about the situation, the situation itself, subject of the appeal and participants. We define Appeal Context as the set of sentences in the query that describe the appeal. As we were interested in the information of the appeal only, we extracted this information from the query by finding the Appeal Context and then preprocessing this context.
1.
Extract the Appeal Context: We observed that most of the queries contained some key-words that help us to identify the context. We used the following list of appeal related keywords:['appeal', 'appeals', 'trial', 'hearing', 'plead', 'pleaded', 'appealing', 'cross-appeal', 'quash'].
We selected 15 sentences per query containing and surrounding these key words. For queries that did not contain any of these key words or were shorter than 15 sentences, we selected the entire query as the Appeal Context. Improvement after submission: We decided to take all the sentences in the query as the Appeal Context.
We performed tokenization of the selected sentences, remove Named Entities and Stop Words of the Appeal Context(similar to pre-processing of case documents). 4.4
BM25[
Using BM25, TF-IDF and an ensemble of BM25-TF-IDF, we found the score for every paragraph in every case document for a given query. The final score of a case document for a given query is the mean of the scores of the top 3 paragraphs of the case document. We ranked the case documents on a scale of 0 (least relevant) to 1 (most relevant) based on these scores.
Improvement after submission: We cleaned and used the whole case documents (without paragraph splitting) and the entire query (without selecting the Appeal Context) for relevance ranking using BM25 and TF-IDF. 4.3
Run ID HLJIT2019-AILA_task1_2 (1st)
TFIDF (5th) Ensemble (7th)
BM25 (8th) 0.07 0.05 0.04 0.0375
MAP After the organizers made the test data public, we performed ablation analysis and realized that the splitting of case documents to paragraphs and selection of the Appeal Context were not improving the results, and were in fact deteriorating it. This could be because narrowing down the query and restricting the query search to just the paragraphs led to missing out some key information for comparison. In fact, the simple removal of Named Entities (NE) in both case documents as well as queries improved the ranking results substantially. Table 2 shows the results.
F D I F T 5 2 M B
TRUE TRUE FALSE FALSE TRUE TRUE FALSE FALSE
TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE P_10 MAP BPREF 0.07 0.07 0.0575 0.0625 0.0575 0.0575 0.05 0.05 0.1743 0.1723 0.1319 0.1644 0.128 0.1261 0.1274 0.1487 0.1535 0.1504 0.1204 0.1468 0.1163 0.1123 0.11 0.1362
Rank 0.2771 0.2738 0.1949 0.2449 0.2424 0.238 0.2545 0.2679 Removal of Named Entities from both query and case helped in making the comparison more generic. For example, this resulted in all the bribery cases whether they happened in a police station, bank or some private company to be treated equally. According to Table 2. TF-IDF as well as BM25 performed the best when the named entities were removed from the query as well as the case documents. At the same time, TF-IDF performed better than BM25 in all the cases.
We have presented our approach for finding the relevant Precedents in the Task 1 in AILA track in FIRE 2019. After improvements, we found out that simply removing the named entities gave the best results. These results are comparable to the highest ranked approach on the leaderboard.
The BM25 and TF-IDF algorithms used in this approach are both word-matching based algorithms for relevance ranking. As a result, a query containing ‘kill’ does not get matched to a case document containing ‘murder’. The lack of exact matches prevented some of the case documents from getting a higher rank in spite of the situation being the same. In the future, we plan to further improve our technique by considering the meaning of the words using word vectors while performing relevance rankings. 7
We would like to thank Girish Palshikar, Sachin Pawar, Dr. Kripabandhu Ghosh and Nitin Ramrakhiyani from TRDDC, Pune for their guidance during our brainstorming sessions. We also thank Dr. Vahida Attar, HOD, Department of Computer and IT, COEP, Pune for her support.