This paper mainly introduces our approaches for the task provided by the FIRE2020(forum for information retrieval evaluation). Task 1 has 2 sub-tasks. The first part of it is to match similar legal cases, and the second part is to match legal cases with relevant statues. We tried the language model on the first task. Language models with different hyper-parameters are used to rank the queries. And the second task is handled as a multi-classification task. We applied a pre-training language model(BERT model developed by Google) on it, with different training parameters.
50 documents are given as the train set.
Fig 1 shows the procedure of which the data and queries are processed.
First the data will be pre-processed. In this step, a stop word list will be applied to the raw text to get rid of punctuation marks and the words that has so high frequency that it carries less information
Then, we use Lucene to create an index of the given text. We use language model with Jelinek Mercer smoothing to determine the similarity within the index. The λ we use is 0.7. The next step is a query with two steps. First, we use the original queries and store it’s results. Then, we apply an IDF screening on the queries. The terms that ranked top 50% are used to form a new query. At Last we accumulate the score of these two queries as the final results.
The preprocessing and creating index process of the task 1b is the same as task 1a.However, in this case, matching the keywords is much more difficult than the task1a since the document is much smaller. At the last ranking part, instead of using the IDF screening, we process the query by converting it into a bag-of-word vector, and calculate the Jaccard similarity of the vectors. Then, rank the results according to the similarity. Different learning rate, epoch , and random seed are tested.When it come to the train data, we randomly chose 80% of the given documents and use them as the train data,10% of the documents are used as validation data and 10% of the document are used as test data. The BERT model results a better result than the other models we tried, such as logistic regression model with the feature of TFIDF provided by the sklearn python library.
The settings of task1 is describe as below in Table 1： Table 1
Λ=0.7,μ=2000 Λ=0.7
Parameter Explanation: Λ is the parameter of the Luence built-in language model with Jelinek Mercer Similarity.It is said that Λ=0.1 is optimal for titles, and Λ=0.7 is optimal for long query. The μ stands for the default parameter in the Lucene language model with Dirichlet smoothing.
The settings of task2 is shown as below in Table2:
3 5 3 3 3 3 3 3 1e-5 1e-5 1e-5 1e-5 1e-5 1e-6 1e-5 1e-5 1e-5 1e-5 128 128 128 128 128 128 64 128 16 128
Random integers 1 1 1 1 1 1 1 1 1
In the experiment process, different epoch, learning rate and random seed is applied. We discovered that the batch size only affect the memory usage and train time cost.What’s more, the prime epoch setting we found is 3.
Beyond that ,we observe a significant over fitting.In other word, when epoch is over 3, the model is recording the data set.Through the loss is dropping continuously, the result on the test set is not improving.Therefore , we use the epoch value of 3.Learning rate will result in longer training time and no significant improvement on result.
For the choice of the BERT pretrained model, a bigger scale pretrained model results in a better result. However, memory usage and processing power consumption. In theory, we will have better result if we run our fine-tune on the bigger scale of BERT model. However, due to limitation of computation power and memory capacity, a rather small BERT model is chosen(In our experiment, the BERT-Base).
The random seed also have an impact on the final result, and rely on the specific test data set. The impact of the random seeds is caused by the dividing process applied to the document. Document
divided in different ways will affect the final fine-tuned model. We submitted two submissions with different random seeds, and it seems to make a dramatic difference.
The evaluation results of task 1a submit shows that our method of task 1a is rather efficient, compared to the methods we used before without the last step of processs with IDF feature. Table 4
This paper describes the methods we used in FIRE2020 and shows the potential improvement approaches. In conclusion IDF feature with language model will result better in long text tasks such as task 1a. In the contrast, tasks with smaller documents will gain more benefits from TF feature.
In the multi-classfication tasks, such as task2, Google Bert out performed many prior models, such as out-of-box logistic regression classifiers. We’re considering if we will achieve better results if we had other classifier to cooperate with the BERT model, instead of one.
This work is supported by National Social Science Fund of China (No.18BYY125) [1] Robertson, S., Steve Walker, and H. BM. "GM Okapi at trec-3." Proceedings of the Third Text
REtrieval Conference (TREC 1994). 1994. [2] Zhai, Chengxiang, and John Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. ACM SIGIR Forum. New York, NY, USA, 2017. [3] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [4] Bhattacharya, Paheli and Mehta, Parth and Ghosh, Kripabandhu and Ghosh, Saptarshi and Pal, Arindam and Bhattacharya, Arnab and Majumder, Prasenjit, Overview of the FIRE 2020 AILA track: Artificial Intelligence for Legal Assistance. Proceedings of FIRE 2020 - Forum for Information Retrieval Evaluation. Hyderabad, India, December, 2020.