Overview of the FIRE 2019 AILA Track: Arti cial Intelligence for Legal Assistance

Abstract

The FIRE 2019 AILA track focused on creating a framework for evaluating di erent methods of retrieving relevant prior/precedent cases and statutes given a factual scenario. There were two tasks for this track: (i) Identifying relevant prior cases for a given situation (Precedent Retrieval), and (ii) Identifying most relevant statutes for a given situation (Statute Retrieval). Given a situation that can lead to ling a case, the precedent retrieval task aims at nding case documents where similar legal situations were addressed. The statute retrieval task aims at nding relevant statutes that are applicable to the situation. The factual scenarios, statutes and prior case documents used in the tasks were from the Indian Supreme Court judiciary.

Keywords

• Legal data analytics
• Prior case retrieval
• Statute retrieval
• Legal facts

-

In countries following the Common Law system (e.g. UK, USA, Canada, Australia, India), there are two primary sources of law { Statutes (established laws, such as the Constitution of a country) and Precedents (prior cases decided in courts of law). Statutes deal with applying legal principles to a situation (facts / scenario / circumstances which lead to ling the case). Precedents or prior cases help a lawyer understand how the Court has dealt with similar scenarios in the past, and prepare the legal reasoning accordingly.

When a lawyer is presented with a situation (that will potentially lead to ling of a case), it will be very bene cial to him/her if there is an automatic system that identi es a set of related prior cases involving similar situations as well as statutes/acts that can be most suited to the purpose in the given situation.

Such a system shall not only help a lawyer but also bene t a common man, in a way of getting a preliminary understanding of the legal aspects pertaining to a situation, even before he/she approaches a lawyer. The system can assist him/her in identifying where his/her legal problem ts, what legal actions he/she can proceed with (through statutes) and what were the outcomes of similar cases (through precedents).

Motivated by the above scenario, we proposed two tasks in the FIRE 2019 track on `Arti cial Intelligence for Legal Assistance' (AILA): (1) Identifying relevant prior cases for a given situation (Precedent Retrieval), and (2) Identifying most relevant statutes for a given situation (Statute Retrieval). These tasks are described below. 1.1

Task 1: Identifying Relevant Prior Cases

The participants are given a set of 50 queries, each of which describes (in natural English language) a situation that had led to ling a case in an Indian court of law. We also provided 2; 914 prior case documents that were judged in the Supreme Court of India. For each query, the task was to retrieve the most similar / relevant case documents with respect to the situation in the given query. Here, the concept of `relevance' may be understood as follows { a prior case is considered relevant to a query if the case discusses a situation similar to that in the query, as judged by law experts. 1.2

Task 2: Identifying Relevant Statutes

We identi ed a set of 197 statutes (Sections of Acts) from Indian law, that are relevant to some of the queries stated above. We provided the participants with the title and description of these statutes. For each query, the task is to identify the most relevant statutes (from among the 197 statutes).

Both tasks consider Indian legal documents, i.e., Indian statutes and prior cases decided by Indian courts of Law (the dataset is detailed in the next section). Note that some similar research has been done on Chinese legal case judgments [ 12, 13 ], where state-of-the-art deep learning models have been applied to identify statutes given the facts of a case. It is to be noted that Chinese legal documents are very well-structured and segmented into section titles (similar to research papers). So the facts of the case can be easily extracted. On the other hand, Indian legal case judgments are not written in a structured way, and there are no section titles either. Hence it becomes a challenging task to extract the facts of the case [ 1 ] and create the datasets for these tasks. 2

Dataset

We collected all Indian Supreme Court case documents from 1952 to February 2018. We also collected 10; 685 Acts (e.g., Constitution of India 1950, Indian Penal Code 1860, Dowry Prohibition Act, 1961). Each Act contains articles or sections (e.g. Article 15 of the Constitution of India 1950, Section 302 of the Indian Penal Code 1860, Section 4 of the Dowry Prohibition Act, 1961 etc.). We call these articles/sections as statutes. All these documents were collected from Thomson Reuters Westlaw India.6 2.1

Creating Pool of Statutes

References to statutes is common in legal case documents. We extract the statutes (section or article number) cited from any of the case documents collected. Speci cally, the cited statutes are identi ed using the fact that statute citations usually follow one of (or a combination of) the following template patterns: { [section or article number] of the [Act]

e.g., Section 47 of the Code of Criminal Procedure, 1973 { [section or article numbers xx and yy] of the [Act]

e.g., Articles 15 and 21 of the Constitution { [section or article numbers xx, yy and zz] of the [Act]

e.g., Section 23, 27 and 39 of the Income-tax Act, 1961 { [section or article numbers xx to yy] of the [Act] e.g., Sections 56 to 60 of the Customs Act, 1962

In some case documents, Acts were cited without a year; such citations brought in an ambiguity if the said Act had multiple versions in di erent years, e.g., Finance Act 1980 , Finance Act 1983 , Finance Act 1984, etc. We discard all mentions of Acts that occur in multiple years and has been cited by a document without the particular year information.

We did not attempt to handle co-reference. For instance, a case document may mention at the beginning: \the bene t of probation under Section 4 of the Probation of O enders Act, 1958 (herein referred to as the Act)". In some later part of the document, it may cite \Section 6 of the said Act". Our heuristic based approach could not capture Section 6 of the Probation of O enders Act, 1958. However, the citation `Section 4 of the Probation of O enders Act' was captured.

To judge the performance of this heuristic method of extracting citations, we conducted a manual evaluation on a small set of 20 documents. Our method could achieve a precision of 1:0 and recall of 0:9 on this set. In other words, all the cited statutes identi ed by our method were correctly identi ed, and the method could correctly identify 90% of all cited statutes.

We then identi ed the top 200 most frequently cited statutes. Out of these, three (03) statutes were removed since they are repealed now. The resulting set contained 197 statutes. We anonymized the name of these statutes (e.g. Section 302 of the Indian Penal Code, 1860 was replaced with the identi er `S43'). Each statute has a title and description, e.g., the title of the Section 302 of the Indian 6 http://www.westlawindia.com/. Note that we use only the publicly available full text judgement. All other proprietary information had been removed.

Penal Code, 1860 is `Punishment for murder', and its corresponding description is `Whoever commits murder shall be punished with death, or [imprisonment for life], and shall also be liable to ne'. For each statute, the identi er, title and description were provided to the participants (for Task 2). 2.2

Query Formulation

We then identi ed the case documents which cite any of these 197 statutes (described above). From this set of case documents, we randomly selected 50 documents and extracted the facts manually. By `facts' we mean, the chronology of events that led to ling the case. Next, we anonymize these facts to form the queries. Names, locations, etc. were replaced with generic abbreviations like `P' (for the name of a person), `L' (for the name of a place) etc. Dates and statute mentions (if any) were also removed. This was done to make the query look as generic as possible, while preserving the legal fact of the matter for proper statute and precedent retrieval. The queries thus obtained were used for both Task 1 and Task 2. 2.3

Creating Pool of Prior Case Documents

We next extract the prior cases cited in the 50 documents (from where the queries are formulated, as described above). If a case title along with its unique ID is mentioned in the current case document, then the case is considered as a prior case. The set of all prior cases for all the 50 documents were given as the pool of documents from where the relevant prior case with respect to a particular query had to be retrieved. The total number of prior cases (out of which relevant cases had to be retrieved in Task 1) is 2; 914. 2.4

Training and Test Data

Note that the tasks can be modeled either as unsupervised retrieval tasks (where one searches for relevant statues/prior cases) or as a supervised classi cation task (where one tries to predict for each statute/prior case whether it is relevant to the given query).

To facilitate considering the tasks as supervised learning tasks, we provided the gold standard sets for 10 queries to the participants as training data. The evaluations were done on the remaining 40 queries.

In summary, the queries (for both tasks) were derived from the facts stated in certain Supreme Court cases, and the gold standard results consisted of prior cases (for Task 1) and statutes (for Task 2) that were actually cited by the lawyers arguing those cases. Hence, the gold standard can be thought of as curated by law experts. We followed this automated methodology in creating the dataset, since actually involving legal experts (e.g., to nd relevant prior cases / statutes) would have required a signi cant amount of nancial resources and time.

Methodologies for Task 1: Precedent Retrieval

For the rst task of retrieving relevant prior / precedent cases, we received a total of 23 runs from nine (09) participating teams. We brie y describe below the methodologies used by each team in each of their runs. The comparative results are in Table 1.

{ IITP[ 2 ]: The team has its members from Government College Of Engineering and Textile Technology, Berhampore and Indian Institute of Technology Patna (both from India). They submitted two runs IITP BM25 case and IITP doc2Vec case. Initially, every word from all the query and candidate case documents was converted to lowercase, lemmatized and stopwords were removed.

In IITP BM25 case, BM25 score between each query and each candidate were calculated. Top 2 documents retrieved based on this score were considered relevant.

In IITP doc2Vec case, document vectors were learned using Doc2Vec. Then a normalized score between each query and each candidate were calculated and top 2 documents were retrieved based on this score. { HLJIT2019-AILA[ 14 ]: This team was from Heilongjiang Institute of Technology, China. They submitted three runs that are described below. Their basic approach was to create an index of the case documents and try to extract keywords as query using di erent methods (runs). In the rst run HLJIT2019-AILA task1 1, top 50% of the IDF as the search key for each query was extracted. Then BM25 was used as the search score. In the second run HLJIT2019-AILA task1 2, for each query, top 50% of the rst IDF and the entire query as the search keywords was extracted. Then BM25 was used as the retrieval model. The search results were weighted and reordered. In the third method HLJIT2019-AILA task1 3, the rst 50% of the term and case documents of the TF-IDF for each query was extracted. Then word2vec was used to represent the vector. Euclidean distance was used to rank the documents. { HGC[ 3 ]: This team had its members from Harbin Engineering University, Harbin, China and Heilongjiang Institute of Technology, Harbin, China. Their evaluation method was mainly based on document keywords. Firstly, they used TF-IDF and textRank to extract keywords from queries. Then they use space vector model, language model and BM25 model to retrieve keywords. Among them, the number of topic words was selected experimentally. They used the data of FIRE 2017 IRLeD Track [ 6 ] to conduct parametric experiments. By designing the keyword extraction method and the corresponding retrieval model for the experimental control group, they used the space vector model combined with TF-IDF method to retrieve the results, and selected the top 70 keywords in the query documents { this was their rst run HGC 1. For their second run HGC 2, they used TF-IDF combined with BM25 model to retrieve, and the number of keywords is set to 70. For their third run HGC 3, they used language model combined with textRank keyword extraction method to retrieve the rst 60 words as query keywords. { Kavya S - Ganesh[ 9 ]: This team was from Vellore Institute of Technology, Chennai, India. Their approach was based on computing TF-IDF vectors on the queries, statutes and case documents. The case documents were ranked based on the cosine similarity between the query and the case documents. { SSN NLP[ 4 ]: This team was from SSN College of Engineering, India. They used di erent query and document embeddings for the di erent runs. In the rst run (SSN NLP 1), they used Word2Vec embeddings. In the second run (SSN NLP 2), they concatenated Word2vec and Glove vectors. In the third run (SSN NLP 3), they used the TF-IDF measure. { TRDDC Pune[ 8 ]: This team had its members for College of Engineering Pune and TRDDC Pune, India. From each query, they extracted sentences containing the `appeal' and `appeal'-related part. From the extracted part, they removed named entities, stop-words and punctuation. Next they preprocessed the data. First they split each document into paragraphs. From each paragraph, they removed named entities, stop-words and punctuation marks. For retrieval, they use TF-IDF, BM-25 and an Ensemble (of TF-IDF and BM-25). For each run they use all the case documents as corpus. They calculated the score of fact query vs every paragraph in case document. The nal score of each document is calculated by taking average of score of top 3 paragraphs of that document. { CUSAT-NLP[ 10 ]: This team was from Cochin University of Science and Technology (CUSAT), India. In their rst run (Task1 CUSAT NLP 1), the corpus of case documents were trained using Doc2Vec. Each case and query document are represented as a 100-dimensional vector. Cosine similarities between each query and case documents are calculated and sorted to obtain the most similar prior cases. In the second run (Task1 CUSAT NLP 2), the corpus used for training Doc2Vec is the complete list of case documents, query documents and statutes documents. Rest of the steps were the same as run1. { JU-SRM[ 7 ]: This team was from SRM University, Chennai and Jadavpur University, Kolkata (both from India). They used sentence level (Sent2Vec) and word level (Word2Vec) representations of the documents and queries. In the rst run, they used pre-trained Sent2Vec model to represent the case documents and queries. In the second run, a FastText model was trained on 2914 case documents, and this model was used for representation. For both runs, for each query-case pair, cosine-similarity was calculated. Using the cosine-similarity score, the rankings were calculated. { thuir legal[ 11 ]: This team was from Tsinghua University, Beijing, China.

In the rst run (thuir legal 1), they used the language model based retrieval. They have used unigram and bigram with linear interpolation. In the second run (thuir legal 2), they use vector space model (TF-IDF) for retrieval. In the third run (thuir legal 3), they combined the vector space model and mixed size bigram model using min-max normalization. 4

Methodologies for Task 2: Statute Retrieval

For the second task of retrieving relevant statutes , we received a total of nineteen (19) runs from eight (08) participating teams. We brie y describe below the methodologies used by each team in each of their runs. The comparative results are in Table 2.

{ IITP[ 2 ]: The team has its members from Government College Of Engineering And Textile Technology, Berhampore and Indian Institute of Technology Patna. In their run, all query and statutes were taken and from every document, every word converted into lowercase, stopwords were removed and words were lemmatized. Then BM25 score between each query and each statute was calculated, and the statutes sorted as per decreasing order of the score. Then the score was normalized and top 3 statutes were considered relevant for each query. { HLJIT2019-AILA[ 14 ]: This team was from Heilongjiang Institute of Technology, China. They created an index on the statutes and try to reorder the results using di erent methods. In the rst run, they extracted the top 50% of the IDF as the search key for each query of Query doc, and used BM25 as the search score. In the second method, for each query, the top 10 of the search result of the rst method in the rst task is used as the candidate query, and each case is extracted as the search keyword of the rst 50% of the IDF. Searched in the statutes index, the document frequency of each case retrieval result is used as the sorting method. In the third method, weighting and reordering the search results of each case in the second method was performed. { SSN NLP[ 4 ]: This team was from SSN College of Engineering, and submitted three runs. They considered a case and a statute document and then used (i) TF-IDF, (ii) Jaccard Similarity and (iii) count of common words in their three di erent runs respectively. { Kavya S - Ganesh[ 9 ]: This team was from Vellore Institute of Technology, Chennai. Their approach was based on computing TF-IDF vectors on the queries, statutes and case documents. The statutes were ranked based on the cosine similarity between the query and the statute documents. { CUSAT-NLP[ 10 ]: This team was from Cochin University of Science and Technology (CUSAT). In run 1, the corpus of case documents were trained using Doc2Vec. Each case and query document were represented as a 100dimensional vector. Cosine similarities between each query and case documents are calculated and sorted to obtain the most similar prior cases. In the second run,the corpus used for training Doc2Vec is the complete list of case documents, query documents and statutes documents. Rest of the steps were the same as the rst run. { JU-SRM[ 7 ]: From queries, key-phrases were extracted using rake-nltk library. From statutes, key-phrases were extracted using the same library. Then manual editing was done (addition and removal). Each query and statute key-phrases were encoded using pre-trained BERT model (with bertembedding package).

In the rst run, for each query statute pair, cosine-similarity scores of each query key phrase and statute key phrase was calculated. Then highest and second highest cosine similarity scores for each query-statute keyphrase pair were multiplied. This value was used as the similarity score between query statute pair. Using these values, rankings were calculated.

In the second run, for each query statute pair, cosine-similarity scores of each query key phrase and statute key phrase was calculated. Then highest query-statute keyphrase cosine-similarity was calculated (suppose H). Sum(all cosine similarity scores of query-statute keyphrase )/(no. of keyphrase in query no. of keyphrase in statute) was calculated (suppose AV G). Based on this similarity score the rankings were decided.

In the third run, along with highest (H) the second highest (SH) was also considered. The similarity score was then calculated as H SH AV G, which was used for ranking. { UBLTM[ 5 ]: This team was from the University of Botswana. They investigated the e ect of di erent retrieval models using Terrier 4.2. For the rst run they transform both the documents and statutes into TREC style format and use IFB2 weighting model of Terrier. For the second run they used only the title eld of statute on IFB2. In the third run, they used only the description eld of statute to retrieve using IFB2. The idea was to test the retrieval e ectiveness of the both the elds and each of the two elds. { thuir legal[ 11 ]: This team was from Tsinghua University, Beijing, China.

They rst generate summary of the query automatically instead of extracting the key and context sentences. The summarization tool is PKUSUMSUM, and they used the LexPageRank algorithm. They apply the restriction that the generated summary of one query contains no more than 200 words. As for statutes, they use both the title and description contents. In the rst run, they used the language model based retrieval. They have used unigram and bigram with linear interpolation. In the second run, they used vector space model (TF-IDF) for retrieval. In the third run, they combined the vector space model and BM25 using weighted averaging their scores after min-max normalization. The nal weights were the ones that performed best on the training data.

Evaluation

In both tasks, a submitted method generated a ranked list of documents (prior cases for Task 1 and statutes for Task 2) relevant to each query. One set of ranked lists (one for each query) generated by a certain method is called a `run'. We evaluate the submitted runs, based on their performance over all 40 test queries. To this end, we used the following performance measures: { Mean Average Precision (MAP): mean of the average precision scores for each query. This is our primary measure based on which the di erent runs are ranked. { Precision at 10 (P10): Number of relevant documents in the top 10 ranked results, averaged over all queries. Since each query contains 10 precedents and statutes on average, we report this score for the runs. { BPREF 7: a summation-based measure of how many relevant documents are ranked before irrelevant documents, averaged over all queries. This measure is especially useful when the set of relevant documents may not be completely known (as can be the case here). { recip rank 8: inverse of the position of the rst relevant document in the ranked list of a query, averaged over all queries.

We used the trec eval tool 9 for computing the metrics stated above. We choose MAP as the primary measure since it incorporates both Precision and Recall. 6

Results

Task 1: Results for Task 1 (retrieving prior case documents given a factual situation as query) is presented in Table 1. The runs are sorted in decreasing order of the MAP scores. The best performing run achieves MAP of 0:1492. This relatively low MAP score re ects that the task is challenging.

Note that the methods are all unsupervised. The training data was used mainly to understand how an approach was performing and for tuning parameters (e.g., assigning weights to di erent models when an ensemble of models were considered). Embedding methods like FastText, Sent2Vec were also used but the performance was not good. This is probably because these methods require a huge amount of data to be trained on. Additionally, the Legal domain being very specialized, pre-trained open-domain models do not perform well.

Task 2: Results for the task of retrieving relevant statutes given the facts is presented in Table 2. The runs are sorted in decreasing order of MAP scores. The best performing method achieves 0:1566, which again re ects the challenging 7 https://trec.nist.gov/pubs/trec16/appendices/measures.pdf 8 https://www-nlpir.nist.gov/projects/trecvid/trecvid.tools/trec eval video/A.README 9 https://trec.nist.gov/trec eval/ nature of the task. Similar to Task 1, the methods were mostly unsupervised except one where keyphrases were extracted manually (JU SRM 6).

Note that, the statute descriptions are usually smaller in length than the factual queries supplied in this task. Hence the idea of summarizing the query (as done by the best performing run) seems to be promising.

Although MAP is considered as the primary metric here, it can be noted that recip rank also plays an important role. The tasks being very challenging, it is useful to understand at what position in the ranked list the rst relevant document comes up. For both Tasks 1 and 2, we see that this score is 0:28 approximately. 7

Concluding Discussions

The FIRE 2019 AILA track has successfully created a benchmark collection of factual queries and its relevant statutes and prior cases. As evident from the result tables, we nd that the tasks of understanding what statutes and precedent cases can be relevant to a given situation, is indeed a challenging task. The best performing methods achieve a MAP score of 0:1492 and 0:1566 respectively on the tasks, which shows that there is lot of scope of improvement.

In future, we plan to extend the gold standard citations of statutes and precedents. It is possible that there are certain statutes and prior case documents that are actually relevant to the situation but has not been cited / made use of in the judgements. However, for a legal practitioner/layman, knowing the whole set of relevant documents is important, and not only the ones speci cally mentioned in the Supreme Court case judgment. Hence we plan to extend the gold standard, e.g., through pooling from the top results returned by the runs. However, this extension will require involvement of domain experts for judging relevance of the pooled documents, which will require signi cant amount of time (due to which we could not do the pooling in the time frame of this FIRE track). Acknowledgements: The track organizers thank all the participants for their interest in this track. We also thank the FIRE 2019 organizers for their support in organizing the track. Comparative Study of Summarization Algorithms Applied to Legal Case Judgments. In: Advances in Information Retrieval { Proceedings of European Conference on Information Retrieval (ECIR). pp. 413{428 (2019)

System Report for Arti cial Intelligence for Legal Assistance Shared Task. In:

References

1. Bhattacharya , P. , Hiware , K. , Rajgaria , S. , Pochhi , N. , Ghosh , K. , Ghosh , S. : A

2. Gain , B. , Bandyopadhyay , D. , De , A. , Saikh , T. , Ekbal , A. : IITP at AILA 2019: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

3. Gao , J. , Ning , H. , Sun , H. , Liu, R., Han, Z. , Kong , L. , Qi , H.: Fire2019@aila: Legal retrieval based on information retrieval model . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

4. Kayalvizhi , S. , Thenmozhi , D. , Aravindan , C. : Legal assistance using word embeddings . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

5. Lefoane , M. , Koboyatshwene , T. , Rammidi , G. , Narasimham , V.L. : Legal statutes retrieval: A comparative approach on performance of title and statutes descriptive text . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

6. Mandal , A. , Ghosh , K. , Bhattacharya , A. , Pal , A. , Ghosh , S. : Overview of the re 2017 irled track: Information retrieval from legal documents . In: Working notes of Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) { CEUR workshop proceedings , Volume 2036 . pp. 63 { 68 ( 2017 )

7. Mandal , S. , Das , S.D. : Unsupervised identi cation of relevant cases & statutes using word embeddings . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

8. More , R. , Patil , J. , Palaskar , A. , Pawde , A. : Removing Named Entities to Find Precedent Legal Cases . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

9. Rameshkannan , R. , Rajalakshmi , R.: Dlrg@aila 2019 : context - aware legal assistance system . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

10. Renjit , S. , Idicula , S.M.: Cusat nlp@aila- re2019: Similarity in legal texts using document level embeddings . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

11. Shao , Y. , Ye , Z. : THUIR@ AILA 2019: Information Retrieval Approaches for Identifying Relevant Precedents and Statutes . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )

12. Wang , P. , Fan , Y. , Niu , S. , Yang , Z. , Zhang, Y. , Guo , J.: Hierarchical matching network for crime classi cation . In: Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval . pp. 325 { 334 . SIGIR' 19 ( 2019 )

13. Wang , P. , Yang , Z. , Niu , S. , Zhang, Y. , Zhang , L. , Niu , S. : Modeling dynamic pairwise attention for crime classi cation over legal articles . In: Proceedings of ACM SIGIR Conference on Research & Development in Information Retrieval . pp. 485 { 494 . SIGIR ' 18 ( 2018 )

14. Zhao , Z. , Ning , H. , Huang , C. , Kong , L., Han , Y ., Han , Z .: Fire2019@aila: Legal information retrieval using improved bm25 . In: Proceedings of FIRE 2019 - Forum for Information Retrieval Evaluation (December 2019 )