hTe FIRE 2017 IRLeD Track focused on creating a framework for evaluating diferent methods of Information Retrieval from legal documents. There were two tasks for this track: (i) Catchphrase Extraction task, and (ii) Precedence Retrieval task. In the catchphrase extraction task, the participants had to extract catchphrases (legal
•Information systems →Information retrieval;
In a Common Law System1, great importance is given to prior cases. A prior case (also called a precedent) is an older court case related to the current case, which discusses similar issue(s) and which can be used as reference in the current case. A prior case is treated as important as any law writen in the law book (called statutes). hTis is to ensure that a similar situation is treated similarly in every case. If an ongoing case has any related/relevant legal issue(s) that has already been decided, then the court is expected to follow the interpretations made in the prior case. For this purpose, it is critical for legal practitioners to find and study previous court cases, so as to examine how the ongoing issues were interpreted in the older cases.
With the recent developments in information technology, the number of digitally available legal documents has rapidly increased. It is, hence, imperative for legal practitioners to have an automatic precedent retrieval system. The task of precedence retrieval can be modeled as a task of information retrieval, where the current case document (or a description of the current situation) will be used as the query, and the system should return relevant prior cases as results.
Additionally, legal texts (e.g., court case descriptions) are generally long and have complex structures [4]. This nature makes their thorough reading time-consuming and strenuous, even after 1https://en.wikipedia.org/wiki/Common_law/ as seen on 6th November, 2017.
Saptarshi Ghosh IIT Kharagpur; IIEST Shibpur
India relevant cases are retrieved. So, apart from a precedence retrieval system, it is also essential for legal practitioners to have a concise representation of the core legal issues described in a legal text [10]. One way to list the core legal issues is by keywords or key phrases, which are known as ‘catchphrases’ in the legal domain [6].
Motivated by the requirements described above, The IRLeD track focused on the following two tasks: Catchphrase extraction, Precedence retrieval. 1.1
Catchphrases are short phrases from within the text of the document. Catchphrases can be extracted by selecting certain portions from the text of the document.
In this task, a set of 400 legal documents (Indian Supreme Court case documents) was provided to the participants. For 100 of these documents (training set), gold standard catchphrases were provided — these gold standard catchphrases were obtained from a wellknown legal search system Manupatra (https://www.manupatra. com/), which employs legal experts to manually annotate case documents with catchphrases. The rest 300 documents were used as the test set. The participants were expected to extract the catchphrases for the documents in the test set. 1.2
For this task, two sets of documents were provided:
(
For each document d in the first set (current cases), the participants were to return a ranked list of documents from the second set (prior cases), in a way that the cases that were actually cited from d are ranked higher than the other documents (that were not cited from d). 2
We have developed two datasets corresponding to the two tasks:
(
hTe collection provided for the track consisted of 400 Indian
Supreme Court case documents. Out of these, 100 documents were
provided along with their gold standard catchphrases (training set)
while the participants were expected to find the catchphrases for
the rest 300 documents (test set).
(
hTe dataset for the task contained 1000 current (query) cases that were judged after the year 2000, and 2000 prior cases that were judged prior to the year 2000 (as described in the Introduction). All filenames were anonymized, and all citation markers from the current/prior cases were replaced with a special marker. 3
For the first task of Catchphrase extraction, we received a total of ten runs from seven participating teams. All the runs were supervised in nature except the run UBIRLeD_1, as described in Table 2. We briefly describe below the methodologies used by each team in each of their runs.
rightstepspune: This team participated from Right Steps Consultancy, Pune. In the method in their only run, the problem of catchphrase detection was modeled as sequential probabilistic labeling problem rather than a simple linear classification problem. Conditional Random Fields (CRF) 2LIIofIndia is a website hosted by the Legal Information Institute of India. algorithm was chosen with primary features such as POS (part-of-speech) and custom NER (Named Entity Recognition) tags and numerous secondary features representing the context. They first tokensied the texts into tokens using NLTK3 tokenizer. Then they applied POS (part-ofspeech) tags to each of the tokens again using the NLTK toolkit. These features along with several other features were used to train a model of CRF, which was then used to predict the catchphrases.
UBIRLED: This team participated from the University of
Botswana, Computer Science Department. They
submitted two runs. For this they have used two recently
developed catchphrase extraction tools:
(
AMRITA_CEN_NLP: This team participated from Amrita Vishwa Vidhyappetham, India and submited a supervised and fully automatic run. For this they have first determined a set of candidate catchphrases and hence represented the documents and candidate catchphrases as vectors. They used Doc2Vec[5] for representing the texts as vectors. Hence the scoring of candidate catchphrases was simply done by measuring the cosine similarity of their vector with the document vector.
HLJIT2017: This team participated from the Heilongjiang Institute of Technology, China. They have submited three runs in total. In all the three methods, they have approached the task as a classification problem and have used supervised fully automatic techniques.
For the first two runs they used bagging techniques. Here, the training set is divided into diferent sampling sets. Then these sampling sets are hence used to train a base classifier. They considered the base classifier as Decision tree[11] in one run and Random forest[3] in another run. In the third run they have used RankSVM4 which 3http://www.nltk.org/ 4urlhtp://www.cs.cornell.edu/People/tj/svm_light/svm_rank.html
uses Support Vector Machines (SVM) to solve the ranking problem of ranking the catchphrases. bphc_withPOS_1: This team participated from Birla Institute of Technology & Science, Pilani, India. They mainly concentrated on the preprocessing part and term scoring methods rather than phrase scoring methods. Their method extracts words rather than phrases. After a series of basic pre-processing, for scoring diferent unigrams they considered the frequency of occurrence within the document. Also, they have given a POS based weightage by checking which POS tags were more likely to be present within a catchphrase.
BITS_Pilani: This team has participated from Birla Institute of Technology and Science, Pilani, India. They have submited a supervised and fully automatic approach for extracting catchphrases.
hTe problem is formulated as a classification task and the objective is to learn a classifier using LSTM network. hTe proposed methodology involves a pipelined approach and is divided into four phases: – Pre-processing – Candidate phrase generation – Creating vector representations for the phrases – Training a LSTM network FIRE_2017_SR: This team has participated from Indian Institute of Engineering Science and Technology, Shibpur, India. They have submited one fully automatic supervised run. They used a deep neural network to train on a number of diferent features of the actual phrases. For extraction of catchphrases, a set of candidate phrases are ifrst selected using POS (part-of-speech) tags of the known catchwords. Once the candidate phrases are obtained. These candidate phrases are then classified using the deep neural network already trained. 4
In the second task of Precedence Retrieval, we received twenty one runs in total from nine participating teams. All of these runs were fully automatic in nature and their performance is as shown in Table 3. Described below are the methodologies used by each team in each of their runs.
lft_ielab: hTis team participated from Queensland
University of Technology, Australia. They submited a total
of three runs each of which use fully automatic methods.
For each of the query documents they have formed a set of
queries from the positions where the actual citations were
present5. Now the query formation was diferently done
in the three runs as described below:
(
Before applying the above filters to get the query terms, all terms were cleaned by removing stopwords and punctuation marks. Once the query terms were ready, they were used to retrieve prior cases using BM25[12] algorithm.
As, a single query document has multiple citation markers. So, the final set of retrieved documents was chosen to be the top-scored 1000 documents from the union of retrieved documents by all these queries.
HLJIT2017_IRLeD_Task2: This team from Heilongjiang Institute of Technology, China submited three runs. Al of the runs were fully automatic in nature. The runs are described as follows: – run_1: In this run they have used a language model based on Dirichlet Prior Smoothing[14]. – run_2: For the second search model they chose BM25 algorithm[12], which is a well-known probability based model. – run_3: In the third run they used lucene[7] which implements a vector space model to estimate the relevance of query and document. 5Note that the positions of the actual citations were marked using a marker in all the text documents.
SSN_NLP: This team participated from SSN College of Engineering, India. They submited three fully automatic runs as described below: – run_1: They considered the TF-IDF vectors of each document by using the TF-IDF vectorizer tool implemented in scikit-learn6. While considering the TFIDF vectors they have considered only the nouns in the document. Now, cosine similarity between the query document and the set of prior cases are calculated and hence sorted to present the top scored documents. – run_2: This is very similar to the first run except that while calculating the TF-IDF vectors, verbs were also considered in addition to nouns. – run_3: This run considers Word2Vec vectors for each document in addition to the TF-IDF vectors as described in the second run. rightstepspune_1_task2: This team participated from RightSteps Consultancy, India. They submited one fully automatic run. For measuring the similarity score between a pair of cases, they have used a weighed average of three diferent methods: – Regular Expression based: Here, diferent legal statutes (such as Articles) referred within the text were captured by using patern matching. Once the list of statutes have been obtained for a given query document, the same is atempted for every prior cases.
All prior cases that has any statutes in common are
retrieved.
– Topic Modeling based: In this method they employ
the implementation of Latent Dirichlet Allocation (LDA)
as in the gensim package.7 Hence, score of similarity
is calculated based on ratio of matching topic-words
to the total.
– Using Document Vector: To generate the document
vectors the following steps were followed:
(
text, giving filename as tag.
(
each cases were generated and saved for reuse.
(
A similarity matrix was generated where current cases are rows and prior cases as columns with values as cosine similarity between document vectors of the current-prior case pair (row-column). The values act as score for this particular approach.
UB_Botswana_Legal_Task2: This team participated from University of Botswana, Botswana. They submited three fully automatic runs. One common part in all the runs 6A open source library in python available at: http://scikit-learn.org/stable/ 7gensim is a python package available at https://radimrehurek.com/gensim/. was the basic query formulation. For this, they have tokenized the text and removed all stopwords and stemmed them using Porter Stemmer. The nest steps for each run is described below: – run_1: Using the formulated queries, they have deployed the parameter-free DPH term weighting model from the Divergence from Randomness (DFR) framework[2] IR platform as our baseline system to score and rank the prior cases. – run_2: They used the first run as the baseline system. In addition, they deployed the Sequential Dependence (SD) variant of the Markov Random Fields for term dependence. Sequential Dependence only assumes a dependence between neighbouring query terms [9, 15]. In this work, they used a default window size of 2 as provided in Terrier-4.2.8 – run_3: They used the first run as the baseline system.
In addition, they deployed a simple pseudo-relevance feedback on the local collection. They used the Bo1 model [1] for query expansion to select the 10 most informative terms from the top 3 ranked documents after the first pass retrieval (on the local collection). hTey performed a second pass retrieval on this local collection with the new expanded query.
UBIRLeD: This is another team participating from University of Botswana, Botswana. They have submited three runs all of them being fully automatic in nature. For each of the runs they have retrieved 1000 ranked prior case judgments.
For the second and third runs they have parsed the
prior case documents into two parts. To identify the most
informative terms they have used topic modeling,
specifically Latent Dirichlet Allocation (LDA). The terms
identiifed using LDA were then used to parse prior cases into
documents with two fields:
(
proach where the weight of LDA_TEXT was set to be 8Terrier is an open source Information Retrieval platform available at http://terrier. org/. far bigger than the weight of OTHER_TEXT, specifically they have used BM25F weighting model, parameter setings for the weight assigned to LDA_TEXT and OTHER_TEXT is 2.0 : 1.0 in terrier respectively, all other parameters were left as default. bphcTask2IRLeD: This team has participated from Birla Institute of Technology & Sciences, Pilani, India. They have submited one run that is fully Automatic in nature. Here, they have considered a minimized set of words by considering only 5000 such words whose combined score of POS (part-of-speech) occurrence probability and IDF (inverse document frequency) score is higher than the rest of the words. Using this focused subset of words they have formed document vectors for each of the documents (both prior cases and current cases). Each vector is of size 5000, where each field corresponds to each word in the focused set. Now a vector for a document is so formed that if a word in the focused set is present then its value in the corresponding field is the combination of its TF (term frequency), IDF, and POS occurrence probability. Now, the similarity score between two document vectors are measured by simply finding the dot product of the two. For each Query Case, the similarity is calculated for between this and all prior cases. Then the top ranked prior cases are reported.
AMRITA_CEN_NLP_RBG: This team has participated from Amrita Vishwa Vidhyappetham, India and have submited a fully automatic run. For this they have first represented the set of prior and current cases as vectors. To do so, they have used the Doc2Vec algorithm as implemented in the gensim package of python. Once the vectors are obtained the similarity between a query case document and a prior case is simply calculated as the cosine similarity between the two vectors. The top ranked prior cases are reported for each of the current cases. christfire_2017: This team participated from Christ University, Bangalore, India. They submited three runs in total and all were fully automatic in nature. The three runs are as described below: – run_1: The following steps are followed.
(
ment Term Matrix
(
ment term Matrix
(
to get semantic relationships of nouns
(
Table 2 compares the diferent runs for Task 1. RAKE being the only unsupervised methods has scored significantly lower than other supervised methods. Although CRF with POS and NER performs well it is to be noted that their overall recall is not very good. Whereas,the method using Doc2Vec gives beter overall recall.
In Table 3, we have diferent runs of Task 2 and their evaluation scores. It is to be noted that, using citation context(text around the citation markers in the query case), greatly improves performance for the top three methods. Other mentionable well performers would be Dirichlet Prior Smoothing and, TF-IDF vectors over nouns and verbs. These, if used in conjunction with citation context, might as well perform beter.
Although, the runs are sorted according to their MAP scores, it is to be noted that, in the legal context the Overall Recall is of special importance. As, in real-life, legal practitioners might even consider going through a hundred documents rather than going through just ten of them while missing out some important potential citations. So, a good evaluation technique would be a combination of MAP and Overall Recall. 6
CONCLUDING DISCUSSION hTe FIRE 2017 IRLeD track has successfully created a benchmark collection of Legal Case Statements and their Catchphrases by which we can compare the performances of various Catchphrase extraction methods over the legal domain. Also it has created a benchmark citation graph which can be used to evaluate methods for the prior case retrieval tasks. It can be noted that the highest MAP score is 0.390 in Table 3, which reveals the challenge in prior case retrieval.
In future, we plan to conduct other tracks as well, where the following can be considered: (i) Adding supervision to the precedence retrieval task, e.g., by providing a citation network for the documents in the set of prior cases, and (ii) adding new tasks such as document clustering/classification.
hTe track organizers thank all the participants for their interest in this track. We also thank the FIRE 2017 organizers for their support in organizing the track. lft_ielab_plm SSN_NLP_2
rightstepspune_1_task2
bphcTASK2IRLeD
0.386 Method Summary IDF, citation context
Dirichlet Prior Smoothing [14]
lucene RegEx, LDA, Doc2Vec BM25 DPH-DFR [2], BoI model[1] DPH-DFR [2] DPH-DFR [2], Sequential DepenLDA LDA BM25