FIRE2019@AILA: Legal Retrieval Based on Information Retrieval Model

Abstract

This paper describes our evaluation methods in the task 1 of AILA (Artificial Intelligence for Legal Assistance) tasks in FIRE 2019. The task 1 is to identify relevant prior cases for a given situation. We deem the task as an information retrieval task. We first extract the topic words from the given situation and use the topic words as a query to identify the relevant prior cases by using the information retrieval model. The best result gets the second place on the MAP.

Keywords

• Artificial Intelligence for Legal Assistance
• Legal Retrieval
• Information Retrieval

-

During the judge's trial of the case, prior cases help a lawyer understand how the Court has dealt with similar cases in the past and prepare the legal reasoning accordingly. Therefore, it is important to help judges find similar prior cases for a given situation. In this evaluation, the organizer provides 3000 cases of documents of cases that were judged in the Supreme Court of India. The task is to identify the most relevant cases related to the given situation [ 1 ]. In this paper, we introduced the methods in this task. Our method is divided into two steps. First, the topic words are extracted from the given situation. And then, the topic words are deemed as a query to search the relevant prior cases by using the retrieval model. We use experimental data provided by the Fire2017 legal retrieval task, and select the number of keywords according to the change of the MAP value.

Model Framework

The Framework is shown as following figure 1. First, the topic extraction module uses TF-IDF and TextRank methods to extract the topic words from a given situation. And then the top-k topic words, which are deemed to be a query, are input to the retrieval model such as the VSM(Vector Space Model), the BM25 model, and the LM(Language Model). The top-n cases in the retrieval result are deemed the relevant prior cases. 3 3.1

Topic Extraction Method

TF-IDF-based topic extraction method TF-IDF (term frequency-inverse document frequency) [ 2 ] is an important indicator of how important a word is in a document. The weight assigned by TF-IDF to the word t in document d can be expressed as: − = , × ， , = sents the frequency of the term in the document d, = log10 +1 inverse document frequency. Where indicates the number of all documents in which the term t appears, plus 1 is to prevent the denominator from being zero, and N

, ∑ , indicates the repreis the number of all documents. 3.2

TextRank-based topic extraction method

TextRank [ 3 ] is a graph-based topic extraction algorithm that can extract topic words from a single document. Formally, let = ( , ) is a directed graph consisting of a series of vertices V and edges E. represents the weight between two vertices and . In( ) represents a collection of points pointing to that point. Out( ) indicates that vertex points to a collection of points. Where d is the damping coefficient, which is between 0 and 1, and is generally set to 0.85. The score of a vertex is defined as follows, according to the formula for calculating the score of each word.

Identifying the Relevant Cases via Information Retrieval

Model

In our approach, we consider the extracted topic as a query and the prior cases as a document collection. The task of this evaluation can be formulated as a retrieval problem. We used three search models, as follows: 4.1

Vector Space Model The Vector Space Model [ 4 ] uses cosine similarity to calculate the similarity between a query and a document. Suppose the vector corresponding to the query q is represented by ̅( ) = ( 11, 12, ⋯ , 1 ), and the vector corresponding to the document d is represented by ̅( ) = ( 21, 22, ⋯ , 2 ), each of the vectors The components correspond to one term.

( , ) = = ̅( )∙̅( ) |̅( )||̅( )| = ∑ =1 1 × 2 √∑ =1 12 ∑ =1 22 (1) (2) (3)

During the retrieval process, the similarity scores of the query and each document are calculated and sorted according to the similarity score. The higher the ranking, the more relevant the document is to the query. 4.2

Probability Model BM25 [ 5 ] is a method for establishing a probability model based on factors such as word frequency and document length. For a query q is a collection containing n terms t. , represents the word frequency of the term t in the document d, N represents the number of all documents, represents the number of all documents in which the term t appears, represents the length of the document d, and represents the whole The average document length of the document collection. 1 is a tuning parameter with a positive value, and is another tuning parameter. The value ranges from 0 to 1. 3 represents a tuning parameter with a positive value, which is used to scale and control the frequency of the term in the query. n represents the number of terms in the query. The relevant line scores of each query word and document are calculated in the calculation process, and finally, the sum is obtained to obtain the correlation score between the query and the document. ( , ) = ∑ ∈ log [ ] ∙

, ∙( 1+1) , + 1∙[(1− )+ ∙ ] ∙ ( 3+1)∙ , 3+ ,

Language Model We build language models θQ and θD for queries and documents. According to the risk minimization model, the KL (Kullback-Leibler) distance metric is applied to the correlation between the query language model θQ and the document language model θD[ 6 ]. The scores for query q and document d become: ( , ) = KL( Q | D )   P(w | Q ) log wV

P(w | Q ) P(w | D ) (4)

Where V is the entire vocabulary, w is the term in V, and is the language model of the query and document, respectively. P(w|θQ) and P(w|θD) are the probabilities of w in the query language model θQ and the document language model θD, respectively. P(w|θQ) uses the maximum likelihood estimation, and P(w|θD) uses the Dirichlet smoothing[ 7 ] document language model. 5 5.1

Experiment

Experiment Setting In the process of extracting the topic words using TextRank, we use the TextRank method in the open-source toolkit HanLP† to extract the topic words. The VSM retrieval model was implemented using the Lucene [ 8 ] toolkit‡, and the BM25 retrieval model and language model were implemented using the Lemur[ 9 ] toolkit3§. 5.2

Parameter Settings We conducted six experiments according to topic extracting methods and retrieval models. The value of TopK is between 40 and 120, and the experimental results are as shown in Fig.2. And select the number of topic words according to MAP. 0.3 0.2 0.1 0 40

50 TF-IDF+VSM TextRank+LM 60 70 80 TextRank+VSM TF-IDF+BM25 90 † https://github.com/hankcs/HanLP ‡ http://lucene.apache.org/ §http://www.lemurproject.org/

It can be seen from the figure 2 that TF-IDF+VSM, TF-IDF+BM25, TextRank+LM are relatively stable, so we submitted the search results of these three models, labeled HGC_1, HGC_2, HGC_3, and the model parameter settings are as following Table 1 :

BPREF 0.1286 This paper describes the evaluation method we used in the FIRE2019 AILA. The TFIDF and TextRank based topic extraction methods are used in the topic extraction stage. And the VSM, BM25, and LM are used in the retrieval stage. Through the final evaluation results, it can be seen that the TF-IDF-based topic extraction method combined with the VSM model is superior to our other submission results.

Acknowledgment

This work is supported by the National Social Science Fund of China (No. 18BYY125).

References

1. P. Bhattacharya , K. Ghosh , S. Ghosh , A. Pal , P. Mehta , A. Bhattacharya. , P. Majumder , Overview of the Fire 2019 AILA track: Artificial Intelligence for Legal Assistance . In Proc. of FIRE 2019 - Forum for Information Retrieval Evaluation , Kolkata, India, December 12-15 , 2019 .

2. Singhal , Amit , et al. Document length normalization . Information Processing & Management 32.5 ( 1996 ): 619 - 633 .

3. Mihalcea , Rada, and Paul Tarau . Textrank: Bringing order into text . Proceedings of the 2004 conference on empirical methods in natural language processing . 2004 : 404 - 411 .

4. Salton , Gerard, Anita Wong , and Chung-Shu Yang . A vector space model for automatic indexing . Communications of the ACM 18.11 ( 1975 ): 613 - 620 .

5. Robertson , Stephen, and Hugo Zaragoza . The probabilistic relevance framework: BM25 and beyond . Foundations and Trends® in Information Retrieval 3 .4 ( 2009 ): 333 - 389 .

6. Song , Fei, and W. Bruce Croft . A general language model for information retrieval . Proceedings of the eighth international conference on Information and knowledge management. ACM , 1999 : 316 - 321 .

7. Zhai Chengxiang , and John Lafferty . A study of smoothing methods for language models applied to ad hoc information retrieval . ACM SIGIR Forum . Vol. 51 . No. 2 . ACM, 2017 : 268 - 276 .

8. McCandless , Michael, Erik Hatcher , and Otis Gospodnetic . Lucene in action: covers Apache Lucene 3 .0. Manning Publications Co., 2010 .

9. Ogilvie , Paul, and Jamie Callan . Experiments using the Lemur toolkit . TREC . Vol. 10 . 2001 : 103 - 108 .