This paper presents our working notes for FIRE 2017, Information Retrieval from Legal documents -Task 2 (Precedence retrieval). Common Law Systems around the world recognize the importance of precedence in Law. In making decisions, Judges are obliged to consult prior cases that had already been decided to ensure that there is no divergence in treatment of similar situations in diferent cases. Our approach was to investigate the efectiveness of using legal catchphrases in precedence retrieval. To improve retrieval performance, we incorporated term dependency in our retrieval. In addition, we investigate the efects of deploying query expansion on the retrieval performance. Our results show an improvement in the retrieval performance when we incorporate term dependence in scoring and ranking prior cases. However, we see a degradation in the retrieval performance when we deploy query expansion.
Common Law Systems around the world recognize the importance
of precedence in Law. In making decisions, Judges are obliged to
align their decisions to relevant prior cases. Thus, when lawyers
prepare for cases, they research extensively on prior cases. In
addition, Judges also consult prior cases that had already been
decided to ensure that a similar situation is treated similarly in every
case [
In this work we investigate the importance of legal catchphrases as queries in precedent retrieval. These legal catchphrases are extracted from current cases. To achieve this, we used a training set of documents provided for Task 1 (catchphrase extraction) where case documents have corresponding gold standard catchphrase. We used Term Frequency-Inverse Document Frequency (TF-IDF) term weighting model to identify similarity between documents in the ∗On the importance of legal catchphrases in precedence retrieval †Lecturer, Department of Computer Science, University of Botswana ‡Lecturer, Department of Computer Science, University of Botswana training set and current cases. Queries were formulated using legal catchphrases from the most relevant documents in the training set.
For retrieval, we deployed the parameter-free DPH term
weighting model to score and rank prior cases. Moreover investigate whether
taking the dependence of query terms in to consideration when
ranking and scoring prior cases could improve thr retrieval
performance.Previous work has shown that incorporating term
dependency in scoring and ranking documents could significantly
improve the retrieval performance [
This paper is structured as follows. Section 2 contains a background on algorithms used. Section 3 describes the experimental setup. In Section 4, we describe the methodologies used for the 3 runs submitted by team UB_Botswana_Legal for Task 2. Section 5 presents results and discussions. 2
In this section, we begin by presenting a brief but essential background on the diferent algorithms used in our experimental investigation and evaluation. We start by describing the TF-IDF term weighting model, in Section 2.1. We then describe DPH term weighting model in Section 2.2, Lastly we describe the Bose-Einstein 1 (Bo1) model for query expansion in Section 2.3. 2.1
In our experimental setup, we used T F -I DF [
scoreT F -I D F (d; Q ) = ∑ 1 + log(t f ) log N t 2Q d ft (1) t f is the term frequency of term t in document d. d ft is the document frequency of term t - the number of documents in the collection that the term t occurs in. id f = log dNft is the inverse document frequency of term t in a collection of N documents 2.2
Our baseline system used the parameter-free DPH term
weighting model from the Divergence from Randomness (DFR)
framework [
scoreDPH (d;Q) = ∑t 2Q qt f norm (t f log((t f avlд_l ) ( tNfc )) + 0:5 log(2 π t f (1 tMLE ))) (2) where qt f , t f and t f c are the frequencies of the term t in the query Q , in the document d and in the collection C respectively. N is number of documents in the collection C, avд_l is the average length of documents in the collection C and l is the length of the document d. tM LE = tlf and norm = (1 tMLE )2 .
t f +1 2.3
In our experimental investigation and evaluation, we used the
Terrier4.0 Divergence from Randomness (DFR) Bose-Einstein 1 (Bo1) model
to select the most informative terms from the topmost documents
after a first pass document ranking. The DFR Bo1 model calculates
the information content of a term t in the top-ranked documents
as follows [
Pn (t ) w (t ) = t f x log2 + log2 (1 + Pn (t ))
(3) t f c Pn (t ) = (4)
N where Pn (t ) is the probability of t in the whole collection, t f x is the frequency of the query term in the top x ranked documents, t f c is the frequency of the term t in the collection, and N is the number of documents in the collection. 3 3.1
In this work we use the document collection provided by the
Information Retrieval in Legal Documents track organizers. It
comprised 200 documents representing current cases and 2000
documents representing prior cases [
For all our experimental evaluation, we used Terrier-4.2, an open source Information Retrieval (IR) platform. Documents were preprocessed before indexing: tokenising text, stemming each token using the full Potter stemming algorithm, and stopword removal using terrier stopword list. 4 4.1
For all the runs in this task, we indexed the 100 case documents provided in task1, which had the corresponding catchphrases using Terrier-4.2 IR platform. During indexing, each case document was first tokenised and stopwords were removed using the Terrier stopword list. Each token was then stemmed using the full Porter stemming algorithm.
For each current case provided in task 2, We used the TF-IDF term weighting model in Terrier 4.2 to score and rank the indexed case documents. Each case document was first pre-processed using the same pre-processing steps undertaken during indexing. After retrieving the top 40 case documents, we formulated queries for each current case using the gold standard catchphrases that appear in these ranked case documents and also in the current case document used for retrieval. 4.2
Using the formulated queries, we deployed the parameter-free DPH Divergence from Randomness term weighting model in Terrier4.2 IR platform as our baseline system to score and rank the prior cases. 4.3
We used UB_Botswana_Legal_Task2_R1 as the baseline system. In
addition, we deployed the Sequential Dependence (SD) variant of
the Markov Random Fields for term dependence. Sequential
Dependence only assumes a dependence between neighbouring query
terms [
We used UB_Botswana_Legal_Task2_R1 as the baseline system. In
addition, we deployed a simple pseudo-relevance feedback on the
local collection. We used the Bo1 model for query expansion to
select the 10 most informative terms from the top 3 ranked
documents after the first pass retrieval (on the local collection) [
This work set out to investigate the importance of legal catchphrases in precedence retrieval. The results of our submission in Table 1 were evaluated by the organizing committee of this task. Since most of the catchphrases were bi-grams and tri-grams, our exploitation of sequential term dependency variant for the Markov Random Fields for term dependence led to improvements in retrieval performance in terms of Mean Average Precision and Precision @ 10. Our attempt to improve retrieval performance using query expansion resulted in degradation in the retrieval performance. We suspect this might have been to due to query drift.
Run ID
0.3478 0.3506