In this article, we present a description of our systems as a part of our participation in the shared task namely Arti cial Intelligence for Legal Assistance (AILA 2019). This is an integral event of Forum for Information Retrieval Evaluation - 2019. The outcomes of this track would be helpful for the automation of the working process of the Indian Judiciary System. The manual working procedures and documentation at any level (from lower to higher court) of the judiciary system are very complex in nature. The systems produced as a part of this track would assist the law practitioners'. It would be helpful for common men too. This kind of track also opens the path of research of Natural Language Processing (NLP) in the judicial domain. This track de ned two problems such as Task 1 : Identifying relevant prior cases for a given situation and Task 2 : Identifying the most relevant statutes for a given situation. We tackled both of them. Our proposed approaches are based on BM25 and Doc2Vec. As per the results declared by the task organizers', we are in 3rd and a modest position in Task 1 and Task 2 respectively.
The working procedure of any judiciary system over the world is very complex
in nature. 3 The countries like India are having two primary sources of law
namely: Statutes (established laws) and Precedents (prior cases). Statutes are
applied legal principles to a situation like facts/scenario/circumstances which
lead to ling the case. Whereas, the precedents/prior cases help a lawyer to
understand how the court has dealt with similar scenarios in the past, and make
the reasoning accordingly. These kinds of tasks are very cumbersome and hard
to do manually. Automated solutions to these problems would be bene cial to
the lawyers. The system will do the task of nding relevant prior cases given
3 Copyright 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). FIRE 2019, 12-15
December 2019, Kolkata, India.
a current case and statutes/acts that will be more suited to a given situation.
These kinds of systems will be helpful to common men too. It will be able to
provide a preliminary understanding to common men on a particular case even
before going to the lawyers. It shall assist human beings in identifying where
his/her legal problem ts, what legal actions he/she can proceed with (through
statutes), and what were the outcomes of similar cases (through precedents).
The Arti cial Intelligence for Legal Assistance (AILA 2019) [
Earlier this type of work was introduced by [
We participated in both the tasks de ned. The rst task is to identify relevant
prior cases for a given situation and the other one is identifying the most relevant
statutes for a given situation. They provided datasets for both the tasks. The
dataset contains 50 queries. The queries are a description of legal situations. For
Task 1, there are 2,914 prior case documents and for Task 2, 197 statutes are
there which are basically the title and the corresponding textual description of
the statutes. We foster BM25 [
In this part, we discuss the preprocessing of the datasets for this task followed by the description of the proposed approaches and experimental procedures. Preprocessing: We perform a few preprocessing rst. For every query and candidate document, we extract all the words. Then, remove all the numbers, that are used to indicate paragraph number. We also remove the stop words from the documents by using NLTK English stop words list. Then we perform stemming and lemmatization of the words using Porter Stemmer and WordNet Lemmatizer respectively.
BM25: BM25 is a bag-of-words based retrieval function that ranks a set of
documents based on the query terms appearing in each document, regardless of the
inter-relationship between the query terms within a document. The objective of
the task is to retrieve the most similar/relevant case documents concerning the
situation of a given query document. The dataset contains 2914 candidates and
50 queries, among which 10 queries are used as train cases and 40 queries are
used for the testing purpose as directed by the task organizers'.
Doc2Vec: Doc2Vec is an unsupervised algorithm that learns xed-length
feature representations from variable-length pieces of texts, such as sentences,
paragraphs, and documents. This algorithm represents each document by a dense
vector that is trained to predict words in the document. More precisely, we
concatenate the paragraph vector with several word vectors from a paragraph and
predict the following word for the given context. Both word vectors and
paragraph vectors are trained by the stochastic gradient descent and backpropagation
[
The objective of this task is to identify the most relevant statues for a given query document. The dataset consisting of 197 statutes and 50 queries among which 10 queries are used for training purpose and 40 queries for testing purpose. Preprocessing: For every query and statute, we extract all the words. We perform removal of stop words, stemming and lemmatization of words using NLTK English stop words list, Porter Stemmer and WordNet Lemmatizer respectively. We submitted one run for this task: IIT P BM 25 statute: This is our only approach to this task. We compute the BM25 similarity score between a query document and every statute and then the candidates are returned in decreasing order of similarity score. 4
This section describes the results obtained in the two tasks by our proposed approaches. The results obtained in Task 1 are shown in the Table 1. In the Table, MAP is mean average precision, BPREF is a measure used when the relevance assessments are not enough and it is suspected that there are many documents (usually the relevant ones) which are not considered for assessment. In this Table, the last column i.e. column 5 indicates 1 / rank of the rst relevant document as provided by the task organizers'. It is basically, Mean Reciprocal Rank used for exact item search where we are interested in nding *one* (the rst one) correct answer and not *all* correct ones. The rst run, i.e. IITP BM25 case model achieved a MAP of 0.0984 and BPREF of 0.0869, which are the 5th among all 23 runs submitted for Task 1 by various teams. In terms of runs submitted by unique team, this run is in 3rd position in terms of MAP and BFREF. The results obtained in Task 2 are shown in Table 2. In this task (i.e. Task 2), we obtain a secure position in the leaderboard.
This paper presents the working note of our participation in Arti cial Intelligence for Legal Assistance (AILA 2019) track, which is an associated event of Forum for Information Retrieval Evaluation (FIRE)- 2019. The shared task de ned two tasks as follows: viz. i. Identifying relevant prior cases for a given situation ii. Identifying the most relevant statutes for a given situation. We participated in both the tasks. We submitted two runs for task1 and one for the Task 2. In Task 1, our models are based on Doc2Vec and BM25. We proposed the BM25 based model for Task 2. We stood 3rd in Task 1 and secured a moderate position in the scoreboard of the second task. Our future work including: { Development of deep learning based supervised approach. As this kind of approach itself is very data hungry, we need to increase the volume of the data. { We could make use of several other unsupervised similarity metrics for these tasks. { Contextual embedding followed by attention mechanism might be a good approach to tackle these tasks. But for all the cases data is the main bottleneck. We should put our focus on preparing more judicial domain annotated data for these problems. { We could enlarge this dataset by merging the dataset which was released in Information Retrieval from Legal Documents (IRLeD) - FIRE 2017. That dataset is in the same line of this track i.e. Arti cial Intelligence for Legal Assistance (AILA) 2019 dataset. 6
Mr. Tanik Saikh acknowledges all the co-authors for their contributions. We also acknowledge the funding agency for the commitment to providing funds.