This paper details the approach of implementing the Catchphrase Extraction and Precedence Retrieval tasks to be presented at Information Retrieval from Legal Documents by Forum of Information Retrieval Evaluation in 2017(Fire2017 IRLeD). For the task of Catchphrase Extraction, the classification-based and Rank-based methods were exploited, and various types of features were attempted. With respect to the task of Precedence Retrieval, the language model, the BM25 and the vector space model were employed. Comparisons to other submissions for the same tasks, show the presented methods to be one of the top performers.
(
Bagging is based on bootstrap sampling. First, we use the M-round bootstrap sampling method to obtain M samples containing N training samples. Then, based on these sampling sets, a base learner is trained. Finally, the M-based learners are combined. The problem of multiclassification is resolved by a simple voting method[2]. Decision tree and random forest[3] are adopted as our base classifiers. Denoted as Bagging(DTC) and Bagging(RFC) respectively.
RankSVM is a pair-wise learning to rank method that uses the SVM model to solve the ranking problem on document pairs. To rank the candidate catchphrase, we trained a ranking function on a training corpus using the Ranking SVM2.
num We construct the features from five aspects: statistical features, position features, syntactic features, mutual information and prior probabilities.
Statistical Features We use the term frequency(TF), inverse document frequency(IDF), TF*IDF, and BM25 score of a word in document di as the statistical features.
1) The first time of the word appears.
num
FirstOccur (w, d )
FirstOccur _score(w) i1 (
FirstOccur (w, d ) precleedne(d( w), d ) (
num
InFirstLas t(w, d )
InFirstLast_ score (w) i1 (
FirstOccur(w, d ) precleend(ed( w), d ) (
POS features We also choose the part-of-speech of a word as the features. For each sentence, we get the POS of each word using the Stanford POS Tagger[6]. We choose a subset of POS (i.e. NNS, NNPS, NNP, NN, VBZ, VBP, VBN, VBG, VBD, VB, TO, JJ, RB) as our features.
num is the number of legal documents, countFL(w,d) represents the number of occurrences of the word w in the first of sentence or the end of the sentence in the legal document d.
Mutual Information High quality keywords should be semantically related. If the relevance between a word w and a keywords k is low, then w may not be suitable as a keyword[7]. The degree of correlation between words is measured by the average mutual information.
1
MI (w1, w2 ,...wk )
(
I (w1, w2 ) p(w1, w2 ) log( pp(w(w1)1p,w(w2)2 ) ) (
priori keyword set P. P contains the keywords with their idf is greater than two.
1, w P
0, w P
(
The task of Precedence Retrieval is to retrieve the prior cases for a given a current case. We view the current case as the query, while the prior cases as the documents, and use three classical information retrieval models to resolve the problem of precedence retrieval.
For the query model and the document model, we use the language model based on Dirichlet Prior Smoothing[8]. The relevance between query and document is computed as follows.
score(q, d ) log p(q | d )
qwiidq
c(w, q) log[1 c(wi , d )
p(wi | C)
] n log d
(
The second search model we chose is BM25 model[9]. The relevance score is computed as follows.
BM 25 winqlog(tf ( wtfi (,wdi),d k)1id(1f (wbi ) b(k1len1()d ))) (
We also used lucene[10] which implemented the vector space model to estimate the relevance of query and document. The Formula is shown as follows: score(q, d ) coord (q, d ) queryNorm(q) (tf (t in d ) idf (t)2 t.getBoost() norm(t, d )) (11) t in q
Here, t represents the term containing the domain information; coord (q, d) means that when a document contains more search terms, the higher score of the document. tf(t in d) represents the word frequency that appears in document d; idf(t) word reverse document frequency; norm (t, d) represents the normalization factor;
In this task, a set of legal documents (Indian Supreme Court decisions) are provided. For a few of these documents (training set), the catchphrases (gold standard) are provided. Catchphrases are short phrases from within the text of the document. These catchphrases have been obtained from a well-known legal search system Manupatra (www.manupatra.co.in), which employs legal experts to annotate case documents with catchphrases. The rest of the documents will be used as the test set. The dataset of Catchphrase Extraction contains 100 training examples and 300 testing examples.
Firstly, we get a candidate catchphrase set. The statistics on the training corpus of catchphrase extraction show that, the nouns accounted for 68.85%, the verbs accounted for 9.13%, adjectives accounted for 12.49%, prepositions accounted for 5.90%, adverbs accounted for 1.43%, other types accounted for 2.20%. According to the above distribution, we choose noun, verbs, adjectives and adverbs.
On the candidate set, We have different combinations of features in the classification and ranking methods. The bagging model uses TF*IDF, BM25, Position Features, POS, Mutual Information and Prior Probabilities as feature set. The RankSVM uses TF*IDF, POS and Prior Probabilities as feature set.
A detailed description of the method parameter settings is shown in the following Table 1:
We submitted the results of the three groups, bagging (DTC)(denoted as HLJIT2017_IRLeD_Task1_1), bagging (RFC)(denoted as HLJIT2017_IRLeD_Tas-k1_2) and RankSVM(denoted as HLJIT2017_IRLeD_Task-1_3). The experimental results are shown in Table 2.
Note that our three submitted results are closed on the evaluation metrics MAP but different on the other evaluation metrics. We surmise that it is mainly because of the sequence of submitted catchphrases. The two results of the classification methods are sorted by alphabetical order, while the results of ranking are submitted in descending order of sorted scores.
In addition, we only choose the words not the phrases as the catchphrases. We tried to use some rule-based methods to construct phrases according to the word we extracted from the legal documents, but we have not achieved the improvement on performance. The low scores on MRP, MP@10, MR@100 and Overall Recall maybe caused by this reason.
Task 2 provides two data sets, the 200 current cases (Query_docs) which formed by removing the links to the 2000 prior cases and the prior cases which have been cited by the cases in Query_docs along with some random cases (not among Query_docs).
We do the same pre-processing for query extraction and index building. In order to discard some of the interference information in document, we filter the document through the lexical information only the nouns, verbs, adjectives, and Poter stemming, lower case and removing stop words are also implemented. For Language Model, we set the parameter mu=10000 and lambda=0.5, and for BM25, we set k1=1.8 and b=0.7.
In Task 2, We have submitted three group of results Language Model(HLJIT2017_IRLeD_Task2_1), Vector Space Model(HLJIT2017_IRLeD_Task2_2) and BM25(HLJIT2017_IRLeD_Task2_3). The results are shown in Table 3. MAP 0.1784 Mean reciprocal 0.4074 Rank Precision@10 0.2180 0.1290 0.1665 Recall@100 0.6810 0.5950 0.6710
From the experimental results, the language model is much better than the other two models. The MAP of the language model is 0.3291, the MAP of the vector space model is 0.2479, and the lowest of the probability model is 0.1784.
Some methods which can improve the performance of language model, such query extension and document extension, have not yet been applied in this evaluation. It may be our further work. In addition, to do the smoothing of the document D, we only apply the given set of legal documents; Too small collection of documents, resulting in the sparse words, And documents and query models also failed to adjust to the optimal. We believe that these methods will improve the performance of the search model and will be tried in future research.
We described the approach to resolve the problems of Catchphrase Extraction and Precedence Retrieval in Fire2017 IRLeD task.
For the task of Catchphrase Extraction, we tried to the classification-based and ranking-based methods. Various type of features is integrated into our models. The experiments show that the ranking-based model achieved better performance.
For the task of Precedence Retrieval, we have only tried several basic language models, such as language model, probability model and vector space model. Experiments show that the language model is more excellent than vector space model and BM25.
This work is supported by the National Natural Science Foundation of China (No.61772177) and the Special subject of State Key Laboratory of Digital Publishing Technology (The research on Plagiarism Detection-From Heuristic to Machine Learning).