=Paper=
{{Paper
|id=Vol-2036/T3-1
|storemode=property
|title=Overview of the FIRE 2017 IRLeD Track: Information Retrieval from Legal Documents
|pdfUrl=https://ceur-ws.org/Vol-2036/T3-1.pdf
|volume=Vol-2036
|authors=Arpan Mandal,Kripabandhu Ghosh,Arnab Bhattacharya,Arindam Pal,Saptarshi Ghosh
|dblpUrl=https://dblp.org/rec/conf/fire/MandalGBPG17
}}
==Overview of the FIRE 2017 IRLeD Track: Information Retrieval from Legal Documents==
https://ceur-ws.org/Vol-2036/T3-1.pdf
Overview of the FIRE 2017 IRLeD Track: Information Retrieval
from Legal Documents
Arpan Mandal Kripabandhu Ghosh Arnab Bhattacharya
IIEST Shibpur IIT Kanpur IIT Kanpur
India India India
Arindam Pal Saptarshi Ghosh
TCS Research IIT Kharagpur; IIEST Shibpur
India India
ABSTRACT relevant cases are retrieved. So, apart from a precedence retrieval
The FIRE 2017 IRLeD Track focused on creating a framework for system, it is also essential for legal practitioners to have a concise
evaluating different methods of Information Retrieval from legal representation of the core legal issues described in a legal text [10].
documents. There were two tasks for this track: (i) Catchphrase Ex- One way to list the core legal issues is by keywords or key phrases,
traction task, and (ii) Precedence Retrieval task. In the catchphrase which are known as ‘catchphrases’ in the legal domain [6].
extraction task, the participants had to extract catchphrases (legal Motivated by the requirements described above, The IRLeD track
keywords) from Indian Supreme Court case documents. In the sec- focused on the following two tasks: Catchphrase extraction, Prece-
ond task of Precedence Retrieval, the participants were to retrieve dence retrieval.
relevant or cite-able documents for particular Indian Supreme Court
cases from a set of prior case documents. 1.1 Task 1: Catchphrase Extraction
Catchphrases are short phrases from within the text of the docu-
CCS CONCEPTS ment. Catchphrases can be extracted by selecting certain portions
•Information systems →Information retrieval; from the text of the document.
In this task, a set of 400 legal documents (Indian Supreme Court
KEYWORDS case documents) was provided to the participants. For 100 of these
Legal Information Retrieval, Prior Case Retrieval, Legal Catchphrase documents (training set), gold standard catchphrases were provided
Extraction — these gold standard catchphrases were obtained from a well-
known legal search system Manupatra (https://www.manupatra.
com/), which employs legal experts to manually annotate case doc-
1 INTRODUCTION
uments with catchphrases. The rest 300 documents were used as
In a Common Law System1 , great importance is given to prior cases. the test set. The participants were expected to extract the catch-
A prior case (also called a precedent) is an older court case related phrases for the documents in the test set.
to the current case, which discusses similar issue(s) and which can
be used as reference in the current case. A prior case is treated
as important as any law written in the law book (called statutes). 1.2 Task 2: Precedence Retrieval
This is to ensure that a similar situation is treated similarly in every For this task, two sets of documents were provided:
case. If an ongoing case has any related/relevant legal issue(s) that (1) Current cases: A set of 200 Indian Supreme Court cases, for
has already been decided, then the court is expected to follow the which the prior cases were to be retrieved.
interpretations made in the prior case. For this purpose, it is critical (2) Prior cases: For each current case, we obtained a set of prior
for legal practitioners to find and study previous court cases, so as cases that were actually cited in the case decision. 1000 such cited
to examine how the ongoing issues were interpreted in the older prior cases were present in the second set of documents, along with
cases. other 1000 documents which were not cited from any document in
With the recent developments in information technology, the the ‘current cases’ set.
number of digitally available legal documents has rapidly increased. For each document d in the first set (current cases), the partic-
It is, hence, imperative for legal practitioners to have an automatic ipants were to return a ranked list of documents from the second
precedent retrieval system. The task of precedence retrieval can be set (prior cases), in a way that the cases that were actually cited
modeled as a task of information retrieval, where the current case from d are ranked higher than the other documents (that were not
document (or a description of the current situation) will be used cited from d).
as the query, and the system should return relevant prior cases as
results. 2 DATASET
Additionally, legal texts (e.g., court case descriptions) are gen-
We have developed two datasets corresponding to the two tasks:
erally long and have complex structures [4]. This nature makes
their thorough reading time-consuming and strenuous, even after
(1) Data for Task 1: A collection of legal case documents with
1 https://en.wikipedia.org/wiki/Common_law/ as seen on 6th November, 2017. their catchphrases: We built a dataset containing 400 court case
Case Id Catchphrases
1953.INSC.24 Actual Delivery, Advocate-General, Alternative Remedy, Appropriate, Assessment, Car-
http://liiofindia.org/in/cases/cen/
rier, Carrying on Business, Cause of Action, Commencement of the Constitution, Com-
INSC/1953/24.html petent Legislature, Consignment, Constitution of India, Constitutional Validity, Consump-
tion, Contract, Contract of Sale, Contravention, Cost, Dealer, Declared by Parliament, De-
duction, Definition, Delegate, Demand Notice, Despatch, Discrimination, Discriminatory,
Double Taxation, Existing Law, Export, Federal Court, Freedom of Trade
1991.INSC.12 Advisory Board, Allowance, Appropriate Government, Arrest, Constitutional Obligation,
http://liiofindia.org/in/cases/cen/ Constitutional Question, Constitutional Safeguard, Detaining Authority, Detention, De-
INSC/1991/12.html tenu, Duty of the State, Earliest Opportunity, General Clauses Act, Grounds of Detention,
Guarantee, Legal Obligation, Liberty, Order of Detention
1983.INSC.27 Commutation, Confinement, Conspiracy, Constitution of India, Death Sentence, Funda-
http://liiofindia.org/in/cases/cen/ mental Right, Imposition of Death Sentence, Judicial Proceeding, Life Imprisonment, Soli-
INSC/1983/37.html tary Confinement, Speedy Trial, Transportation for Life
Table 1: Examples of Indian Supreme Court cases and catchphrases taken from the Manupatra legal expert system (reproduced
from [6])
documents of the Indian Supreme Court, along with their catch- algorithm was chosen with primary features such as POS
phrases. The texts and their catchphrases were obtained from a (part-of-speech) and custom NER (Named Entity Recog-
well-known legal search system Manupatra which uses human le- nition) tags and numerous secondary features represent-
gal experts to annotate court case documents with catchphrases. ing the context. They first tokensied the texts into tokens
All decisions and the corresponding Catchphrases are available in using NLTK3 tokenizer. Then they applied POS (part-of-
text format. A few example Catchphrases are shown in Table 1 speech) tags to each of the tokens again using the NLTK
(reproduced from [6]). toolkit. These features along with several other features
The collection provided for the track consisted of 400 Indian were used to train a model of CRF, which was then used
Supreme Court case documents. Out of these, 100 documents were to predict the catchphrases.
provided along with their gold standard catchphrases (training set) • UBIRLED: This team participated from the University of
while the participants were expected to find the catchphrases for Botswana, Computer Science Department. They submit-
the rest 300 documents (test set). ted two runs. For this they have used two recently devel-
oped catchphrase extraction tools:
(1) Data for Task 2: A collection of legal case documents, (1) RAKE (Rapid Automatic Keyword Extraction):
and prior cases cited from them: We crawled a large number an unsupervised algorithm for keyword extraction [13].
of case documents of cases judged at the Supreme Court of In- (2) MAUI: a supervised algorithm for keyword extrac-
dia, from the site LIIofIndia (www.liiofindia.org/).2 The documents tion [8].
were downloaded in HTML, and the downloaded HTML files were • AMRITA_CEN_NLP: This team participated from Am-
then parsed to get the final texts. rita Vishwa Vidhyappetham, India and submitted a super-
The dataset for the task contained 1000 current (query) cases vised and fully automatic run. For this they have first de-
that were judged after the year 2000, and 2000 prior cases that were termined a set of candidate catchphrases and hence repre-
judged prior to the year 2000 (as described in the Introduction). sented the documents and candidate catchphrases as vec-
All filenames were anonymized, and all citation markers from the tors. They used Doc2Vec[5] for representing the texts as
current/prior cases were replaced with a special marker. vectors. Hence the scoring of candidate catchphrases was
simply done by measuring the cosine similarity of their
3 METHODOLOGIES FOR TASK 1: vector with the document vector.
CATCHPHRASE EXTRACTION • HLJIT2017: This team participated from the Heilongjiang
Institute of Technology, China. They have submitted three
For the first task of Catchphrase extraction, we received a total of
runs in total. In all the three methods, they have approached
ten runs from seven participating teams. All the runs were super-
the task as a classification problem and have used super-
vised in nature except the run UBIRLeD_1, as described in Table 2.
vised fully automatic techniques.
We briefly describe below the methodologies used by each team in
For the first two runs they used bagging techniques.
each of their runs.
Here, the training set is divided into different sampling
• rightstepspune: This team participated from Right Steps sets. Then these sampling sets are hence used to train a
Consultancy, Pune. In the method in their only run, the base classifier. They considered the base classifier as Deci-
problem of catchphrase detection was modeled as sequen- sion tree[11] in one run and Random forest[3] in another
tial probabilistic labeling problem rather than a simple lin- run. In the third run they have used RankSVM 4 which
ear classification problem. Conditional Random Fields (CRF)
3 http://www.nltk.org/
2 4 urlhttp://www.cs.cornell.edu/People/tj/svm_light/svm_rank.html
LIIofIndia is a website hosted by the Legal Information Institute of India.
2
Run_ID R-Prec Prec@10 Recall@100MAP
Overall Method Summary
Recall
rightstepspune_1_task1 0.215 0.281 0.248 0.479 0.248 CRF, POS, NER
UBIRLeD_2 0.190 0.254 0.305 0.370 0.326 MAUI[8]
AMRITA_CEN_NLP_RBG1_1 0.168 0.144 0.535 0.200 0.652 Doc2Vec
HLJIT2017_IRLeD_Task1_3 0.086 0.122 0.151 0.165 0.152 RankSVM
bphc_withPOS_1 0.066 0.102 0.137 0.161 0.165 TF, POS tags
HLJIT2017_IRLeD_Task1_1 0.030 0.058 0.033 0.140 0.033 Decision Tree
HLJIT2017_IRLeD_Task1_2 0.034 0.060 0.044 0.124 0.044 Random Forest
Bits_Pilani_1 0.030 0.049 0.080 0.093 0.100 LSTM network
FIRE_2017_SR 0.026 0.025 0.087 0.062 0.161 POS, Deep Neural Network
UBIRLeD_1 0.023 0.014 0.172 0.046 0.499 RAKE[13]
Table 2: Evaluation of runs for Task 1: Catchphrase Extraction. Runs are sorted in descending order of the Mean Average
Precision.
uses Support Vector Machines (SVM) to solve the ranking fully automatic in nature and their performance is as shown in
problem of ranking the catchphrases. Table 3. Described below are the methodologies used by each team
• bphc_withPOS_1: This team participated from Birla In- in each of their runs.
stitute of Technology & Science, Pilani, India. They mainly • flt_ielab: This team participated from Queensland Uni-
concentrated on the preprocessing part and term scoring versity of Technology, Australia. They submitted a total
methods rather than phrase scoring methods. Their method of three runs each of which use fully automatic methods.
extracts words rather than phrases. After a series of basic For each of the query documents they have formed a set of
pre-processing, for scoring different unigrams they con- queries from the positions where the actual citations were
sidered the frequency of occurrence within the document. present5 . Now the query formation was differently done
Also, they have given a POS based weightage by checking in the three runs as described below:
which POS tags were more likely to be present within a (1) flt_ielab_para: Here, the query was formed by con-
catchphrase. sidering a paragraph around the citation marker.
• BITS_Pilani: This team has participated from Birla Insti- (2) flt_ielab_idf: Here, only 50% of the words were con-
tute of Technology and Science, Pilani, India. They have sidered after weighing the terms by their idf (inverse
submitted a supervised and fully automatic approach for document frequency).
extracting catchphrases. (3) flt_ielab_plm: Here, only 50% of the words given by
The problem is formulated as a classification task and flt_ielab_idf were considered by its probability from
the objective is to learn a classifier using LSTM network. a parsimonious language model.
The proposed methodology involves a pipelined approach Before applying the above filters to get the query terms,
and is divided into four phases: all terms were cleaned by removing stopwords and punc-
– Pre-processing tuation marks. Once the query terms were ready, they
– Candidate phrase generation were used to retrieve prior cases using BM25[12] algorithm.
– Creating vector representations for the phrases As, a single query document has multiple citation mark-
– Training a LSTM network ers. So, the final set of retrieved documents was chosen to
• FIRE_2017_SR: This team has participated from Indian be the top-scored 1000 documents from the union of re-
Institute of Engineering Science and Technology, Shibpur, trieved documents by all these queries.
India. They have submitted one fully automatic super- • HLJIT2017_IRLeD_Task2: This team from Heilongjiang
vised run. They used a deep neural network to train on Institute of Technology, China submitted three runs. Al
a number of different features of the actual phrases. For of the runs were fully automatic in nature. The runs are
extraction of catchphrases, a set of candidate phrases are described as follows:
first selected using POS (part-of-speech) tags of the known – run_1: In this run they have used a language model
catchwords. Once the candidate phrases are obtained. These based on Dirichlet Prior Smoothing[14].
candidate phrases are then classified using the deep neural – run_2: For the second search model they chose BM25
network already trained. algorithm[12], which is a well-known probability based
model.
4 METHODOLOGIES FOR TASK 2: PRIOR – run_3: In the third run they used lucene[7] which
CASE RETRIEVAL implements a vector space model to estimate the rel-
In the second task of Precedence Retrieval, we received twenty one evance of query and document.
runs in total from nine participating teams. All of these runs were 5 Note that the positions of the actual citations were marked using a marker in all the
text documents.
3
• SSN_NLP: This team participated from SSN College of En- was the basic query formulation. For this, they have tok-
gineering, India. They submitted three fully automatic enized the text and removed all stopwords and stemmed
runs as described below: them using Porter Stemmer. The nest steps for each run
– run_1: They considered the TF-IDF vectors of each is described below:
document by using the TF-IDF vectorizer tool imple- – run_1: Using the formulated queries, they have de-
mented in scikit-learn6 . While considering the TF- ployed the parameter-free DPH term weighting model
IDF vectors they have considered only the nouns in from the Divergence from Randomness (DFR) framework[2]
the document. Now, cosine similarity between the IR platform as our baseline system to score and rank
query document and the set of prior cases are calcu- the prior cases.
lated and hence sorted to present the top scored doc- – run_2: They used the first run as the baseline sys-
uments. tem. In addition, they deployed the Sequential De-
– run_2: This is very similar to the first run except that pendence (SD) variant of the Markov Random Fields
while calculating the TF-IDF vectors, verbs were also for term dependence. Sequential Dependence only
considered in addition to nouns. assumes a dependence between neighbouring query
– run_3: This run considers Word2Vec vectors for each terms [9, 15]. In this work, they used a default win-
document in addition to the TF-IDF vectors as de- dow size of 2 as provided in Terrier-4.2.8
scribed in the second run. – run_3: They used the first run as the baseline system.
• rightstepspune_1_task2: This team participated from In addition, they deployed a simple pseudo-relevance
RightSteps Consultancy, India. They submitted one fully feedback on the local collection. They used the Bo1
automatic run. For measuring the similarity score between model [1] for query expansion to select the 10 most
a pair of cases, they have used a weighed average of three informative terms from the top 3 ranked documents
different methods: after the first pass retrieval (on the local collection).
– Regular Expression based: Here, different legal statutes They performed a second pass retrieval on this local
(such as Articles) referred within the text were cap- collection with the new expanded query.
tured by using pattern matching. Once the list of • UBIRLeD: This is another team participating from Uni-
statutes have been obtained for a given query doc- versity of Botswana, Botswana. They have submitted three
ument, the same is attempted for every prior cases. runs all of them being fully automatic in nature. For each
All prior cases that has any statutes in common are of the runs they have retrieved 1000 ranked prior case
retrieved. judgments.
– Topic Modeling based: In this method they employ For the second and third runs they have parsed the
the implementation of Latent Dirichlet Allocation (LDA) prior case documents into two parts. To identify the most
as in the gensim package.7 Hence, score of similarity informative terms they have used topic modeling, specifi-
is calculated based on ratio of matching topic-words cally Latent Dirichlet Allocation (LDA). The terms identi-
to the total. fied using LDA were then used to parse prior cases into
– Using Document Vector: To generate the document documents with two fields:
vectors the following steps were followed: (1) LDA_TEXT - A field containing words that have been
(1) Got every case as cleaned text, split it to form identified as most informative words for the collec-
list of words/tokens, for both, current and prior tion of prior case judgments.
cases. (2) OTHER_TEXT - A field containing other words that
(2) Created gensim TaggedDocument for each case have not been identified as most informative words.
text, giving filename as tag. The runs are as described below:
(3) A Map of tag to the content i.e. word-list for – run_1: A Baseline run where they have used the orig-
each cases were generated and saved for reuse. inal dataset, only parsing it to TREC format, the runs
(4) LDA model was built and saved. It was used were obtained using BM25 with default settings.
to generate document vectors for both current – run_2: This is the run for a field based retrieval ap-
and prior cases. proach where the weight of LDA_TEXT was set to be
A similarity matrix was generated where current cases far lower than the weight of OTHER_TEXT, specifi-
are rows and prior cases as columns with values as cally they have used BM25F weighting model, param-
cosine similarity between document vectors of the eter settings for the weight assigned to LDA_TEXT
current-prior case pair (row-column). The values act and OTHER_TEXT is 0.2 : 1.0 in Terrier respectively,
as score for this particular approach. all other parameters were left as default.
• UB_Botswana_Legal_Task2: This team participated from – run_3: This is the run for a field based retrieval ap-
University of Botswana, Botswana. They submitted three proach where the weight of LDA_TEXT was set to be
fully automatic runs. One common part in all the runs 8 Terrier is an open source Information Retrieval platform available at http://terrier.
6 A open source library in python available at: http://scikit-learn.org/stable/ org/.
7 gensim is a python package available at https://radimrehurek.com/gensim/.
4
far bigger than the weight of OTHER_TEXT, specifi- (4) Application of Latent Semantic Analysis, LSA
cally they have used BM25F weighting model, param- to get semantic relationships of nouns
eter settings for the weight assigned to LDA_TEXT (5) Similarity Calculation
and OTHER_TEXT is 2.0 : 1.0 in terrier respectively, In the preprocessing part the following steps were fol-
all other parameters were left as default. lowed:
• bphcTask2IRLeD: This team has participated from Birla Case Conversion, Special Character Removal, Num-
Institute of Technology & Sciences, Pilani, India. They ber Removal, Stopword Removal, Legal Stopword Re-
have submitted one run that is fully Automatic in nature. moval (Words that appear commonly in all judgments),
Here, they have considered a minimized set of words by and Document Stemming.
considering only 5000 such words whose combined score The citation context retrieval deals with retaining only
of POS (part-of-speech) occurrence probability and IDF those parts of the document that are around the cita-
(inverse document frequency) score is higher than the rest tion markers. The similarity calculation is done by
of the words. Using this focused subset of words they have measuring the cosine similarity among the two docu-
formed document vectors for each of the documents (both ment vectors (one of the current case another of the
prior cases and current cases). Each vector is of size 5000, prior case). Only the top 50 of the prior cases are re-
where each field corresponds to each word in the focused ported.
set. Now a vector for a document is so formed that if a
word in the focused set is present then its value in the 5 RESULTS
corresponding field is the combination of its TF (term fre- Table 2 compares the different runs for Task 1. RAKE being the
quency), IDF, and POS occurrence probability. Now, the only unsupervised methods has scored significantly lower than
similarity score between two document vectors are mea- other supervised methods. Although CRF with POS and NER per-
sured by simply finding the dot product of the two. For forms well it is to be noted that their overall recall is not very good.
each Query Case, the similarity is calculated for between Whereas,the method using Doc2Vec gives better overall recall.
this and all prior cases. Then the top ranked prior cases In Table 3, we have different runs of Task 2 and their evaluation
are reported. scores. It is to be noted that, using citation context(text around
• AMRITA_CEN_NLP_RBG: This team has participated the citation markers in the query case), greatly improves perfor-
from Amrita Vishwa Vidhyappetham, India and have sub- mance for the top three methods. Other mentionable well perform-
mitted a fully automatic run. For this they have first repre- ers would be Dirichlet Prior Smoothing and, TF-IDF vectors over
sented the set of prior and current cases as vectors. To do nouns and verbs. These, if used in conjunction with citation con-
so, they have used the Doc2Vec algorithm as implemented text, might as well perform better.
in the gensim package of python. Once the vectors are ob- Although, the runs are sorted according to their MAP scores,
tained the similarity between a query case document and it is to be noted that, in the legal context the Overall Recall is of
a prior case is simply calculated as the cosine similarity special importance. As, in real-life, legal practitioners might even
between the two vectors. The top ranked prior cases are consider going through a hundred documents rather than going
reported for each of the current cases. through just ten of them while missing out some important poten-
• christfire_2017: This team participated from Christ Uni- tial citations. So, a good evaluation technique would be a combi-
versity, Bangalore, India. They submitted three runs in to- nation of MAP and Overall Recall.
tal and all were fully automatic in nature. The three runs
are as described below: 6 CONCLUDING DISCUSSION
– run_1: The following steps are followed. The FIRE 2017 IRLeD track has successfully created a benchmark
(1) Data cleaning and citation context retrieval collection of Legal Case Statements and their Catchphrases by which
(2) Linguistic Preprocessing and creation of Docu- we can compare the performances of various Catchphrase extrac-
ment Term Matrix tion methods over the legal domain. Also it has created a bench-
(3) Application of Latent Dirichlet Allocation,LDA mark citation graph which can be used to evaluate methods for
(4) Similarity Calculation the prior case retrieval tasks. It can be noted that the highest MAP
– run_2: The following steps are followed: score is 0.390 in Table 3, which reveals the challenge in prior case
(1) Data cleaning and citation context retrieval retrieval.
(2) Linguistic Preprocessing and creation of Docu- In future, we plan to conduct other tracks as well, where the
ment term Matrix following can be considered: (i) Adding supervision to the prece-
(3) Application of Latent Semantic Analysis, LSA dence retrieval task, e.g., by providing a citation network for the
(4) Similarity Calculation documents in the set of prior cases, and (ii) adding new tasks such
– run_3: The following steps were followed: as document clustering/classification.
(1) Data cleaning and citation context retrieval
(2) Retaining only nouns from the data ACKNOWLEDGEMENTS
(3) Linguistic Preprocessing and creation of Docu-
The track organizers thank all the participants for their interest in
ment term Matrix
this track. We also thank the FIRE 2017 organizers for their support
in organizing the track.
5
Run_id MAP MRR Prec@10
Rec@100 Method Summary
flt_ielab_idf 0.390 0.719
0.781 0.236
IDF, citation context
flt_ielab_plm 0.386 0.710
0.771 0.237
Parsimonious language model, ci-
tation context
flt_ielab_para 0.364 0.702 0.221 0.749 Citation context
HLJIT2017_IRLeD_Task2_1 0.329 0.633 0.218 0.681 Dirichlet Prior Smoothing [14]
SSN_NLP_2 0.268 0.546 0.178 0.669 TF-IDF(nouns+verbs)
SSN_NLP_1 0.263 0.518 0.180 0.681 TF-IDF(nouns)
HLJIT2017_IRLeD_Task2_3 0.248 0.525 0.167 0.671 lucene
rightstepspune_1_task2 0.202 0.451 0.135 0.564 RegEx, LDA, Doc2Vec
HLJIT2017_IRLeD_Task2_2 0.178 0.407 0.129 0.595 BM25
UB_Botswana_Legal_Task2_R3 0.167 0.348 0.123 0.559 DPH-DFR [2] , BoI model[1]
UB_Botswana_Legal_Task2_R1 0.149 0.351 0.112 0.546 DPH-DFR [2]
UB_Botswana_Legal_Task2_R2 0.108 0.302 0.079 0.43 DPH-DFR [2] , Sequential Depen-
dence[9, 15]
SSN_NLP_3 0.101 0.277 0.076 0.435 Word2Vec(nouns+verbs)
UBIRLeD_2 0.098 0.190 0.069 0.380 LDA
UBIRLeD_3 0.090 0.170 0.062 0.373 LDA
UBIRLeD_1 0.072 0.142 0.049 0.299 BM25
bphcTASK2IRLeD 0.071 0.198 0.060 0.280 POS tags, TF, IDF
AMRITA_CEN_NLP_RBG1_1 0.006 0.015 0.003 0.058 Doc2Vec
christfire_2017_3 0.005 0.011 0.003 0.033 LSA(nouns only)
christfire_2017_2 0.003 0.010 0.002 0.044 LSA
christfire_2017_1 0.002 0.006 0.003 0.016 LDA
Table 3: Evaluation of runs for Task 2: Precedence Retrieval. Runs are sorted in descending order of the MAP or Mean average
Precision.
REFERENCES [8] Olena Medelyan. 2009. Human-competitive automatic topic indexing. (2009).
[1] G. Amati. 2003. Probabilistic Models for Information Retrieval based on Diver- http://cds.cern.ch/record/1198029 Presented on July 2009.
gence from Randomness. University of Glasgow,UK, PhD Thesis (June 2003), 1 – [9] Donald Metzler and W. Bruce Croft. 2005. A Markov Random Field Model for
198. Term Dependencies. In Proceedings of the 28th Annual International ACM SIGIR
[2] G. Amati, E. Ambrosi, M. Bianchi, C. Gaibisso, and G. Gambosi. 2007. FUB, IASI- Conference on Research and Development in Information Retrieval (SIGIR ’05).
CNR and University of Tor Vergata at TREC 2007 Blog Track. In Proceedings ACM, New York, NY, USA, 472–479. DOI:http://dx.doi.org/10.1145/1076034.
of the 16th Text REtrieval Conference (TREC-2007). Text REtrieval Conference 1076115
(TREC), Gaithersburg, Md., USA., 1–10. [10] J.L.T. Olsson. 1999. Guide To Uniform Production of Judgments, 2nd edn. Aus-
[3] Leo Breiman. 2001. Random Forests. Mach. Learn. 45, 1 (Oct. 2001), 5–32. DOI: tralian Institute of Judicial Administration, Carlton South (1999).
http://dx.doi.org/10.1023/A:1010933404324 [11] J. R. Quinlan. 1986. Induction of decision trees. Machine Learning 1, 1 (01 Mar
[4] Stefanie Brüninghaus and Kevin D. Ashley. 2001. Improving the Representation 1986), 81–106. DOI:http://dx.doi.org/10.1007/BF00116251
of Legal Case Texts with Information Extraction Methods. In Proceedings of the [12] Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance
8th International Conference on Artificial Intelligence and Law (ICAIL ’01). ACM, Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval
3, 4 (April 2009), 333–389.
New York, NY, USA, 42–51. DOI:http://dx.doi.org/10.1145/383535.383540
[13] Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic
[5] Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences
Keyword Extraction from Individual Documents. John Wiley and Sons, Ltd, 1–20.
and Documents. In Proc. International Conference on Machine Learning (ICML),
DOI:http://dx.doi.org/10.1002/9780470689646.ch1
Tony Jebara and Eric P. Xing (Eds.). JMLR Workshop and Conference Proceed-
[14] Fei Song and W. Bruce Croft. 1999. A General Language Model for Information
ings, 1188–1196.
Retrieval. In Proceedings of the Eighth International Conference on Information
[6] Arpan Mandal, Kripabandhu Ghosh, Arindam Pal, and Saptarshi Ghosh. 2017.
and Knowledge Management (CIKM ’99). ACM, New York, NY, USA, 316–321.
Automatic Catchphrase Identification from Legal Court Case Documents. In
DOI:http://dx.doi.org/10.1145/319950.320022
Proc. ACM Conference on Information and Knowledge Management (CIKM). 2267–
[15] Edwin Thuma, Nkwebi Peace Motlogelwa, and Tebo Leburu-Dingalo. 2017. UB-
2270.
Botswana Participation to CLEF eHealth IR Challenge 2017: Task 3 (IRTask1
[7] Michael McCandless, Erik Hatcher, and Otis Gospodnetic. 2010. Lucene in Ac-
: Ad-hoc Search). In Working Notes of CLEF 2017 - Conference and Labs of the
tion, Second Edition: Covers Apache Lucene 3.0. Manning Publications Co., Green-
Evaluation Forum, Dublin, Ireland, September 11-14, 2017. http://ceur-ws.org/
wich, CT, USA.
Vol-1866/paper_73.pdf
6