This paper details the approaches of implementing the tasks of identifying relevant precedents and identifying relevant statues in the evaluation of Artificial Intelligence for Legal Assistance proposed by Forum of Information Retrieval Evaluation in 2019(AILA@Fire2019). We formalize the two tasks as the issue of information retrieval, and present the improved BM25 models to retrieve the prior cases and identify the relevant statues. For the task of identifying relevant precedents, the proposed improved BM25 model integrates the relevance scores of the original current case and the filtered current case. For the task of identifying relevant statues, the proposed improved BM25 models exploit the search results as the reference documents of the current case and integrate the ranking information of search results into the BM25 model. Comparisons to the other submissions for the same tasks, our improved BM25 model achieves the top performers for the task of identifying relevant precedents on all evaluation measures. For the task of identifying relevant statues, the improved BM25 model wins the second place on 1/rank of first relevant document and the third place on BPREF.
In a Common Law System1, great importance is given to prior cases. A prior case (also
called a precedent) is an older court case related to the current case, which discusses
similar issues and which can be used as reference in the current case [
With the recent developments in information technology, the number of digitally
available legal documents has rapidly increased. It is, hence, imperative for legal
practitioners to have an automatic precedent retrieval system. The task of identifying
relevant prior case can be modeled as a task of information retrieval, where the current
case document (or a description of the current situation) will be used as the query, and
the system should return relevant prior cases as results [
Usually, legal document retrieval considered as a rank task. Early approaches for
handling term dependencies in IR considered extensions of the bag of word
representation of texts, by including bi-grams to the vocabulary. Such an approach was
taken by Fagan [
For the task of Identifying Relevant Precedents, we apply the approach based on the information retrieval to obtain the prior cases of the current case.
Given a current case, denoted as , and a collection of prior cases denoted as , the goal of Identifying Relevant Precedents is to retrieve the relevant document d in collection when given the current case q.
For the submission HLJIT2019-AILA_task1_1, we choose the BM25 model as the retrieval model, defined in Eq. (1) tf (wi , d ) idf (wi ) (k1 1) rel(q, d ) BM 25(q, d ) log( ) wiq tf (wi , d ) k1 (1 b b LEN (d )) (1) avdl where w is the word in , avdl is the average length of the document, k1 and are the parameters of BM25.
In the evaluation, the and is preprocessed firstly as follows: Porter stemmer
[
For the submission HLJIT2019-AILA_task1_2, we modify the relevance computation to get an improved BM25 as follows: 2 http://lucene.apache.org/
rel(q, d) BM 25(q, d) BM 25(q, d) (2) where ’ is the same as HLJIT2019-AILA_task1_1, and ’’ is the original current case without IDF filtering. All the other settings are followed HLJIT2019-AILA_task1_1.
In addition, we also experimented with Word2vec method as the submission HLJIT2019-AILA_task1_3. After choosing the top m% words with high TF-IDF to represent the query and document , we represent and as vectors using Word2vec method, shown in Eq.(3):
1 n V (x) ti (3)
n i1 where denotes word vector of i-th term, is the top m% words. In experiments, we set the parameter = 50.
Then, we use the Euclidean distance to calculate the similarity between and . 3
(4) (5) For the task of Identifying Relevant Statues, we apply the approach based on the information retrieval to obtain the relevant statues.
Given a current case , and a collection of statues, denoted as , the goal of Identifying Relevant Statues is to retrieve the relevant statues s in collection when given the current case .
For the submission HLJIT2019-AILA_task2_1, we also choose the BM25 model as the retrieval model. We use the description part in statues to construct the document collection . And all the parameters setting and pre-processing are followed HLJIT2019-AILA_task1_1.
For the submission HLJIT2019-AILA_task2_2, we use the top-n relevant prior cases associated with the current case as the reference document , and integrate the relevant scores obtained by into the original relevant score, shown as Eq.(4): BM 25( pi , s j ) rel(q, s j ) BM 25(q, s j ) where s denotes the j-th statue in , is the i-th relevant prior case associated with , and ( , ) is the rank of in the search results of . In the evaluation, we select the top-10 search results as the reference documents.
Furthermore, considering the ranking information of relevant statues, we modify the Eq.(4) as follows rank( p, pi ) rel(q, s j ) BM 25(q, s j )
BM 25( pi , s j ) rank( p, pi ) rank(q, s j ) where ( , ) is the rank of s in the search results of . In the evaluation, the number of reference documents is 10, and the number of re-ranking relevant statues is 197.
document and the fourth place on BPREF. According to the results, the relevant prior case information is helpful to guide the judgment of current case. We describe an approach to Precedent Retrieval and Statute Retrieval that makes use of the improved BM25. Comparisons to the other submissions for the same tasks, our improved BM25 model achieves the top performers for the task of identifying relevant precedents on all evaluation measures. For the task of identifying relevant statues, the improved BM25 model wins the second place on 1/rank of first relevant document and the third place on BPREF.
This research was supported by the National Natural Science Foundation of China (No.61806075, No.61772177), the National Social Science Fund of China (No.18BYY125), the Natural Science Foundation of Heilongjiang province of China (F2018029) and the Social Science Fund of Heilongjiang Province of China (No.18TQB103).