In this paper, we combine lexical and neural ranking models for case law retrieval. In this task, the query is a full case document, and the candidate documents are prior cases that are potentially relevant to the current case. Most documents are longer than 1024 tokens, which makes retrieval and classification with Transformer-based models problematic. We create shorter query documents with diferent methods: term extraction, noun phrase extraction, entity extraction, and automatic summarization using Longformer-Encoder-Decoder (LED). We then combine the summaries with five diferent ranking models: a BM25 ranker, statistical language modelling, the Deep Relevance Matching Model (DRMM), a Vanilla BERT ranker, and a Longformer ranker. We optimised all models and combined the best lexical ranker with neural retrieval models using diferent ensemble classifiers. We evaluate our methods on the retrieval benchmarks from COLIEE'20 and COLIEE'21. We beat state-of-the-art models for case law retrieval with both benchmark sets. Our experiments show the importance of tuning lexical retrieval methods, summarizing query documents, and combining lexical and neural models into one ranker for efective case law retrieval. In addition, training and optimizing our rankers is much faster than passage-level retrieval models (a few hours compared to several days for training).
One of the challenges of case law retrieval in COLIn countries with common law systems, finding support- IEE’20 is that the input is a long case document with a ing precedents to a new case, is vital for a lawyer to median length of 2,815 words instead of a keyword query. fulfill their responsibilities to the court. However, with Four approaches to this problem exist. The first is the use the large amount of digital legal records – the number of unsupervised keyword extraction methods to create of filings in the U.S. district courts for total cases and short queries from the long query document [3]. A varicriminal defendants was 544,460 in 20201 – it takes a sig- ant is the unsupervised extraction of phrases or entities nificant amount of time for legal professionals to scan for as query terms [4]. The second approach, proposed by specific cases and retrieve the relevant sections manually. Tran et al. [5], is to train a supervised phrase scoring Studies have shown that attorneys spend approximately model for -gram phrases to select the phrases that are 15 hours in a week seeking case law [1]. semantically closest to an expert-written summary. The
This workload necessitates the need for information third approach, proposed by Rossi and Kanoulas [6], is retrieval (IR) systems specifically designed for the legal to use document summarization methods for creating domain. The Competition on Legal Information Extrac- shorter query documents. The fourth approach, successtion/Entailment (COLIEE) is a workshop that has been fully employed by Shao et al. [4] and Westermann et al. organized since 2014 as a series of evaluation competi- [7], is to analyze the documents on the level of individual tions related to case law [2]. COLIEE defines four tasks. paragraphs and then aggregate the paragraph scores in a document ranking.
DESIRES 2021 – 2nd International Conference on Design of In this paper, we use automatic summarization for Experimental Search Information REtrieval Systems, September creating query documents. We experiment with term 15–18, 2021, Padua, Italy extraction, noun phrase extraction, and supervised text s".vear.basekrnarei@@lliiaaccss..lleeiiddeennuunniivv..nnll((SA. .VAersbkearrni)e;) summarizers. As opposed to prior work, we approach ~ https://www.universiteitleiden.nl/en/stafmembers/arian-askari the task as an abstractive summarization problem. The (A. Askari); https://liacs.leidenuniv.nl/~verbernes/ (S. Verberne) current state of the art in abstractive summarization is 0000-0003-4712-832X (A. Askari); 0000-0002-9609-9505 the use of Transformer models [8, 9]. However, the input (S. Verber©n2e0)21 Copyright for this paper by its authors. Use permitted under Creative of pre-trained available models of these architectures CPWrEooUrckReshdoinpgs 1IhStphN:/c1e6u1r3-w-0st.o7r3gtpsCC:o/Em/mUwoRnwsLWwice.onusreskcAsotthruiborupttison.Pg4r.o0ovInc/teesretnaadttiioinnsagtlis(cCCs(C-BrYEe4Up.0)oR.r-tWs/jSu.odricgi)al-business- idsolciummiteendtstoin1o02u4r ctooklleenctsi,oannids ltohnegmeratjhoarintythoaft.caBseeltlaagwy 2020 et al. [10] proposed Longformer-Encoder-Decoder (LED), which is a Transformer variant that supports much longer ment summarization. inputs. In this paper, we evaluate the efectiveness of LED In COLIEE’20, the two best-performing teams use for case law retrieval. paragraph-level analyses to cope with the challenge of
We evaluate summarization with multiple ranking long documents. Shao et al. [4] (‘TLIR’) participate with models: probabilistic lexical ranking (BM25), statistical their method BERT-PLI [15], which models paragraphlanguage modelling, the Deep Relevance Matching Model level interactions. They combine BERT-PLI with lexical (DRMM), and Transformer-based architectures including matching features using word-entity duet model. The the Vanilla BERT model from Contextualized Embed- features are diferent lexical rankers (BM25, probabilistic dings for Document Ranking (CEDR) [11]. Although we language modelling) – without optimization – on the full summarize the query documents, the lengths of the doc- document content, and entities extracted by NLTK. They uments in the retrieval collection still causes problems reach competitive results. The method by Westermann for Transformer-based ranking architectures, which are et al. [7] (‘cyber’) selects the top-30 candidate documents limited to 512 tokens as the input length. To solve that using a paragraph similarity score based on the universal problem, we implemented a Vanilla Longformer ranker sentence encoder, and then applies an SVM model to the [10], with 4096 tokens as the input length, in a pair-wise TF-IDF representations of the query document and the ranking setting similar to Vanilla BERT. After we have candidate documents. optimized each individual ranker, we experiment with en- We use the supervised phrase-based summarization of semble methods that combine lexical rankers with neural Tran et al. [14, 5] as comparison for our summarization rankers. task, and the highest F-score obtained in COLIEE’20 by
Our contributions are four-fold: (1) We deliver a fine- Westermann et al. [7] as comparison for our retrieval tuned Longformer-Encoder-Decoder (LED) for abstrac- task. tive summarization of legal documents; (2) we deliver a pairwise Longformer ranker for long documents; (3) 2.2. Summarization of long documents We show that summarizing query documents with LED improves all ranking models; (4) We show that the com- Kanapala et al. [16] and Van de Luijtgaarden [17] give bination of a lexical ranker and a Vanilla BERT ranker in an extensive overview of research on legal document a simple ensemble classifier outperforms all baselines on summarization up to 2019. Here we focus on the recent COLIEE’20 and an optimized BM25 ranker with keyword abstractive summarization models. queries beats the state-of-the-art models on COLIEE’21. Pre-trained encoder-decoder Transformer models (e.g. BART [8] and T5 [9]) have achieved strong results in abstractive summarization tasks. However, pre-trained 2. Related Work models of these architectures are limited to texts that are shorter than 1024 tokens. Legal documents are com2.1. Case law retrieval monly longer than that; 72% of the query documents in COLIEE’20 are. Recently, Beltagy et al. [10] proposed Longformer-Encoder-Decoder (LED), which is a Transformer variant that supports sequence-to-sequence tasks for longer documents (up to 16k tokens). The authors show that LED outperforms the state-of-the-art models on the arXiv dataset. In this paper, we evaluate the efectiveness of LED for case law retrieval.
decisions using unsupervised keyword extraction models.
They find that the best performing model is KullbackLeibler divergence for informativeness (KLI) [12] and that the automatically generated queries were more efective than the average Boolean queries from experts.
The majority of the work addressing case law retrieval takes place in the context of COLIEE, the Competition on Legal Information Extraction and Entailment [13, 2]. 2.3. Transformer-based document Rossi and Kanoulas [6] proposed a pairwise ranking ranking model based on BERT. They apply automatic summarization using TextRank, to make the input document The typical approach in neural IR is two-step retrieval, length suitable for use in the BERT-based ranker. The where a first set of documents is retrieved using a tramost successful team in COLIEE 2019 is Tran et al. [14, 5] ditional ranker (e.g. BM25) and those document are re(‘JNLP’). They train a phrase scoring model that extracts ranked by a neural model that is trained on the relevance -gram phrases ( ≥ 5) to summarize the documents. assessments [18, 11, 19].
The weights are calculated based on the phrase score MacAvaney et al. [11] propose a joint approach that framework that was trained on COLIEE’18 summaries. integrates the classification vector of BERT into existing The authors achieved the state-of-the-art result on COL- neural models like the Deep Relevance Matching Model IEE’19, thereby showing the importance of query docu- (DRMM), resulting in Contextualized Embeddings for Document Ranking (CEDR). They fine-tune a pretrained
BERT model with a linear combination layer stacked atop Entity and phrase extraction Inspired by Shao et al. the [CLS] token as the Vanilla BERT ranker with pairwise [4], we used NLTK3 to extract noun phrases and named cross-entropy loss and the Adam optimizer. The authors entities from the content of query documents and candidemonstrate that Vanilla BERT and CEDR outperform date documents and used the extracted strings entities the state-of-the-art baselines in ad-hoc rankings. How- as the representation documents (see Section 4.4 for the ever, local-interaction neural ranking architectures like combinations we experimented with). DRMM are not scalable to long documents or need heavy interaction between word pairs in query and document. Abstractive summarization We experimented with Therefore, we will compare our method to the Vanilla the pre-trained LED model4 of Beltagy et al. [10] which BERT ranker. can process documents up to 16k tokens as input. Also,
More recent work has addressed the challenges of rank- we fine-tuned LED on the COLIEE’18 dataset, in which ing for long documents. Hofstätter et al. [20] propose a more than 80% of documents have summaries.5 After local attention Transformer model that uses a moving removing duplicates, 6, 257 unique documents are left window over the document terms and for each term at- for which a summary is available. For evaluation purtends only to other terms in the same window. They poses, we trained the model in a k-fold cross-validation obtain results significantly better than other state-of-the- setting ( = 10) for one epoch per fold, each with batch art models on the TREC Deep learning track. Sekulić size 1. We kept the other hyperparameters (optimizer, et al. [19] take a full-document approach by training a dropout, weight decay) identical to [10] and set the global Longformer model for ranking in ad-hoc retrieval. The attention on the first <s> token. We only summarized results they report on the MS MARCO set are low com- query documents with LED for the lexical rankers since pared to the leaderboard results. Our Longformer ranker they do not have limitation for input length. On other is similar to [19], but instead of implementing the ranker hand, since Transformer-based neural models are limited as a one-versus-all classifier, we train and evaluate it in a in input length for the candidate document content, we pair-wise setting and we evaluate it for case law retrieval. also experiment with summarizing candidate documents besides query documents for the best Transformer-based neural model in our experiments.
documents for each query document in COLIEE’20 with multiple retrieval models. We optimise the hyperparameters of each method on a validation set (see Section 4.3).
In the following, we will introduce each neural ranker that we use for legal case retrieval.
Summarization through term extraction We adopted Kullback-Leibler divergence for Informativeness (KLI), similar to Locke et al. [3] in the implementation of Verberne et al. [21]. For each term in a query document, 3.2.1. Lexical rankers we computed the KLI score:
BM25 We indexed the COLIEE’20 collection with Elas (|) ticSearch. The collection has 200 candidate documents () = (|) × log (1) for each query that need to be ranked. We used BM25 (|) with the default parameter values = 1.2 and = 0.75, where (|) is the probability of in the query doc- as well as with optimized hyperparameter values. ument and (|) is the probability of in a background language model. We use all candidate documents Language Modelling We used the built-in similaras the background collection to compute (|). We ity functions of Elasticsearch for the implementation of only consider unigrams as terms and we lowecase them.2 Language Modelling (LM) with two diferent smoothing: We then selected the top-10% of the total number of Dirichlet smoothing and Jelinek Mercer (JM) smoothing. terms in the document with the highest KLI score as a We only report the results for JM smoothing since we got the query.
2We tried to extract n-gram phrases (2<<=5) with = 0.8 but obtained the best results with unigrams.
3https://www.nltk.org/api/nltk.chunk.html?#nltk.chunk.util. tree2conlltags and https://www.nltk.org/api/nltk.chunk.html# module-nltk.chunk.named_entity 4https://huggingface.co/allenai/led-large-16384-arxiv 5In COLIEE’19 and COLIEE’20 the large majority (82%) of the candidate cases do not have a summary. similar results by these two smoothing methods. We also optimised the hyperparameter value ( ) for Language Modelling with Jelinek Mercer smoothing (LM JM).
[11], and is available for future work.6 3.2.3. Ensemble models 3.2.2. Neural rankers
Deep Relevance Matching Model (DRMM) As the ical rankers in one integrated system, we train ensemble DRMM architecture [18] is based on a local interaction models that take the scores of multiple rankers as feainput matrix, only a limited query length is possible. We tures. We experiment with three diferent classifiers for took the top 10% of query terms sorted by the KLI score this purpose: SVM with a linear kernel, SVM with an RBF for each query document. Then, we selected the average kernel, Naive Bayes, and Multi Layer Perceptron (MLP). resulting number of terms, 70, as the maximum length We experiment with diferent combinations of rankers’ of query. Thus, we used the top-70 query terms as query scores to find the best combination of rankers for having in DRMM. For calculating cosine similarity in the local an efective ranking. interaction matrix, we trained word2vec on the query candidates’ texts as suggested by the DRMM authors.
Furthermore, we optimize the network configuration of 4. Experiments and results DRMM to find the best combination of layers and neurons in legal case retrieval on COLIEE (see details in 4.1. Data Section 4.3).
the COLIEE competitions in 2018, 2020, and 2021.
Vanilla BERT ranker For Vanilla BERT, we fine-tuned The COLIEE’18 data contains human-written suma pre-trained BERT model (BERT-Base, Uncased) with maries of the case documents, which we use for the traina linear combination layer stacked atop the classifier ing evaluation of our summarization models (Section 4.2). [CLS] token on the COLIEE dataset in pairwise cross- For our retrieval experiments (Section 4.3), we use data entropy loss setting using the Adam optimizer. We used from COLIEE’20 and ’21.7 The Federal Court of Canada the implementation of MacAvaney et al. [11] (CEDR). We provided case laws with metadata for task 1. The metarepresent the query as sentence A and the document as data contains references to the noticed cases that are sentence B in the BERT input: the golden relevance labels for the query document. In COLIEE’20, there is a pool of 200 candidates for each “[CLS] query document [SEP] candidate document [SEP]" query document and the competitors should re-rank a limited number of documents per query. The pool of We truncate the query and candidate document text since candidates includes the noticed cases and non-relevant the BERT tokenizer is limited to 512 tokens. candidates, which are selected randomly. In contrast, in COLIEE’21, the whole collection should be considered Vanilla Legal BERT ranker For Vanilla Legal BERT, per query without having a pool of candidates. This difwe used LEGAL-BERT [22] which was pre-trained on ference makes task 1 in COLIEE’21 more dificult than in legal data. COLIEE’20.
We use the COLIEE’20 data to evaluate all ranking models and ensembles described in Section 3. In the COLIEE’20 data, there are 520 query documents in the train set and 130 in the test set; with 104,000 candidate documents in the train set and 26, 000 in the test set. The average length of the documents in the test set is 3,232 words, with outliers upto 10, 827. After we have found the best-performing rankers and ensemble, we evaluate those on the COLIEE’21 data and compare the results to the best results reported in the competition. In the COLIEE’21 data, there are 650 query documents in the train set and 250 in the test set, with 4, 415 documents as candidate documents for both train set and test set.
Vanilla Longformer ranker Since the input length is limited in Vanilla BERT, we implemented the Vanilla Longformer in CEDR [11] as a ranker which can receive 4096 tokens instead of 512 and has more chances to work efectively than Vanilla BERT. In Longformer, the [CLS] and [SEP] tokens are replaced by the tokens <s>, and </s> respectively. As suggested in the Longformer paper [10] we calculate the loss based on the <s> token with the addition of global attention to the <s> token. Inspired by Sekulić et al. [19] we feed the query document as sequence A and the candidate document as sequence B to the tokenizer, which yields the following input to the model:
“<s> query document </s> candidate document </cls>". longformer-D552/README.md 7https://sites.ualberta.ca/~rabelo/COLIEE2020/ https://sites.ualberta.ca/~rabelo/COLIEE2021/ and
We set the local attention window size to 512 tokens. To tune the network configuration (e.g., numbers of laylimit memory use, we use gradient checkpointing and set ers and hidden nodes) on the validation set. = 6 is the input size in training to 8,192 tokens which covers the optimal cut-of value. We trained six word2vec and more than 86% of COLIEE’18 documents completely fasttext models for three configurations from the litera(the longer documents are truncated at 8,912 tokens). We ture [18, 23, 24]. We also used the pre-trained word2vec set the maximum length to generate a summary for an on google-news and pre-trained fasttext on Wikipeda. unseen document as 10% of the length of the original We found that word2vec which was pre-trained accordtext. ing to the DRMM configuration gave the best results. For
We evaluate our fine-tuned LED to the pre-trained the network configuration, we use a four-layer architecLED [10], and compare it to the summarizer of Tran et al. ture throughout all experiments, i.e., one histogram input [5] (JNLP) on the COLIEE’18 data. Table 1 shows that layer (30 nodes), two hidden layers in the feed-forward our fine-tuned summarizer outperforms the baseline in matching network (128 nodes for both layers), and one terms of F-measure for ROUGE-1, ROUGE-2 and ROUGE- output layer (1 node) with the term gating network for SU6 on the COLIEE’18 dataset. The pre-trained LED the final matching score. We set the maximum query summarizer obtains the highest precision-ROUGE scores, length to 70 tokens as explained in Section 3.2.2. and the JNLP baseline the highest recall-ROUGE scores.
held-out subset (10% of the training set). We optimize the rankers for each document representation. In the text below, we refer to noun phrase representations as Q NP and C NP for the query/candidate documents respectively, and to entity representations as Q Entities and C Entities. We use Precision@k, Recall@k and F-measure@k as evaluation metrics, following the COLIEE evaluation mechanism.8 We select the best cut-of for each method based on the validation set and use that on test set. BM25 We found = 6 as the optimal cut-of for ranking with the whole text, and = 4 for summarized input. For the optimization we searched the following grid: ∈ {0, 0.1, 0.2, · , 1} and 1 ∈ {0, 0.1, 0.2, · , 3}. For BM25 with KLI the best parameters were = 0.9, and 1 = 2.8.9
such that the concatenated query document (truncated at 100 words), candidate document, and the separator tokens do not exceed 512 tokens. We re-rank the top-30 BM25 results. Since the @30 is about 95% for BM25, our ranker has the possibility to achieve 95% recall while still having a reasonable runtime. = 6 was the optimal cut-of for the arXiv-LED summarizer queries, and = 4 for the fine-tuned LED summarizer queries. We train each model for 100 epochs, each with 32 batches of 16 training pairs, with the initial learning rate of 3 * 10− 5, followed by a power 3 polynomial decay.
Vanilla Longformer ranker The local window size is set to 512. We fine-tune the pre-trained Longformer using pairwise hinge loss. Positive and negative training documents are selected from the relevance judgments. We truncate the document such that the sequence of the concatenated query document (summary), candidate document, and the separator tokens do not exceed 4,096
9We found = 0.6, 0.8, 0.8 and 1 = 1.8, 1.4, 1.4 as the best parameters for BM25 (Q NP C NP, Q words C NP, and Q NP C words) respectively.
10We found = 0.1 as the best parameters for LM JM with KLI. We found = 0.1, 0.6, 0.4 as the best parameters for LM JM (Q entities D entities, Q words D entities, and Q entities D words) respectively. tokens. We again re-rank the top-30 BM25 results. for the arXiv-LED summarizer queries, and = 4 for the ifne-tuned LED summarizer queries. We use the same training configuration as for Vanilla BERT.
and neural ranking models are shown in Table 2 and Table 3, respectively. The best lexical ranker is BM25, but LM JM with KLI is very close. The best neural ranker in terms of recall and F1 is DRMM. The best single ranker overall is BM25 with an F1 score of 63.95%. The highest precision is obtained by BM25 with KLI queries and the highest recall with BM25 with arXiv-LED. This indicates that lexical matching is important for this dataset. Table 2 COLIEE’21 results For evaluating the generalizability also shows the (mostly positive) efect of optimizing the of our results, we evaluated the best methods on the COLparameters of BM25, an efort that is not taken by most IEE’21 data. As explained in Section 4.1, the COLIEE’21 of the COLIEE participants. task is more dificult than the COLIEE’20 task because it The results of the ensemble models are shown in Ta- requires retrieval from a full document collection instead ble 4. With our ensemble models we improve over the best benchmark result (the Cyber team) by a large margin.
The best ensemble model in terms of F1 on the validation set is a combination of the best BM25 ranker (BM25optimised + KLI) and the second-best neural ranker Vanilla BERT (Vanilla BERT + SummaryQ (fine-tuned LED)). This indicates that BERT can add more to the combination with BM25 than the best neural model DRMM can. of re-ranking 200 documents. We optimised the BM25 parameters and the cut-of value on the COLIEE’21 validation set. As shown in Table 5, we beat the stateof-the-art result (TLIR team) with the optimized BM25 ranker. The poor result of Vanilla BERT on COLIEE’21 shows us that neural retrieval has more challenges for ranking the whole collection than with re-ranking the top-200. We also applied ensemble classifiers combining BM25 and Vanilla BERT but they could not improve the efectiveness; we suppose that it is caused by the lower performance of Vanilla BERT on COLIEE’21.
Analysis weights of classifiers As suggested in [29], we interpret the importance of Vanilla BERT feature in the ensemble classifiers based on the coeficient value in iftted SVM (linear). For Vanilla BERT the coeficient is higher (0.56) than BM25’s coeficient ( 0.43).
Analysis hyperparameters of BM25 The optimal value for in the literature is between 0.3 − 0.9 [30, 31, 32] and we found = 0.9 for the optimised BM25 with KLI. There are documents in COLIEE that, because of their length, contain multiple topics, and it was suggested before that documents that includes a variety of topics benefit from using a larger so that unrelated topics to a user’s search are penalized.11 The normal range for 1 is between 0 and 3 and for long documents that contain diverse information the 1 should tend to larger numbers [32]. We found the optimal value for 1 = 2.8 for BM25 with KLI which makes sense because of the long documents in COLIEE.
The efect of summarization The results show that summarizing the query document improves all rankers.
This holds for the KLI term extraction, noun phrase extraction, and the LED summarizer. This shows the importance of summarization for making the query documents shorter. Our results show that the best summarizing methods for neural ranking and lexical ranking are LED (fine-tuned) and keyword extraction (KLI) respectively.
However, summarizing the candidate documents does not improve the neural ranking performance. Another observation is that fine-tuning the summarizer improves the ranking in terms of F-measure for all rankers. We see the largest efect from summarization for Vanilla BERT.
Using noun phrases or named entities We experimented with noun phrases and named entities as document representation, inspired by Shao et al. [4]. Our experiments show that the efectiveness of using named entities instead of original content is much lower than of using noun phrases: The F1 score for BM25 optimised + Q entities C entities is 22.71% while F1 for BM25 optimised + Q NP C NP is 56.88%. Although the use of Analysis of unexpected results One unexpected re- named entities was suggested in prior work, they do not sult is that our Longformer ranker does not outperform play an important role in case law documents, at least not the Vanilla BERT ranker. We speculate that this is because so much that using them as document representations longformer receives more tokens as the input learning, leads to efective retrieval. which makes it more dificult to estimate the relevance between document and query, while the size of the train- Future work Some prior work has obtained good reing set is relatively small: In COLIEE’20, we have only sults with passage-level analysis for long document rank2, 680 relevant labels and Longformer could not converge ing [24, 33]. We think it is a promising direction to comduring training – it did not find the optimal loss after bine these passage-level with document-level methods 100 epochs. For future work, we will further pre-train to design a more efective legal case retrieval system Longformer on legal documents. This requires a GPU for future work. One challenge is that this approach with 32GB Ram which is not easily accessible. is computationally expensive since each paragraph of
Our results also show that DRMM could not beat BM25 the query document needs to be compared with each for case law retrieval. Some prior work has also indicated paragraph of the candidate documents. We have query that for some datasets, DRMM is close to BM25 in quality documents with up to 1,139 paragraphs in COLIEE. For and that sometimes BM25 works better than DRMM [26, future work we will focus on combining lexical document 27, 28], especially when BM25 is properly optimized. retrieval with eficient paragraph-level retrieval. We will
The third unexpected result is that the best Vanilla also evaluate how we can use COGLTX [5] to recognize Legal BERT (summaryQ) does not outperform the best the important sentences from the query cases and docuVanilla BERT model. We suppose this can be related to ment cases without having the limitation in length as a language similarity between COLIEE cases and part of pre-processing step. Inspired by [33], another direction the data that BERT Base was trained on (Wikipedia, and is working on neural models for legal case retrieval on more than thousand thousands books) because cases in ranking instead of re-ranking because in coliee’21 there COLIEE contain stories of applicant’s lives and this is are 4, 415 candidates per query (whole collection). almost the bigger part of each document.
pp. 55–64. [31] A. Trotman, A. Puurula, B. Burgess, Improvements [19] I. Sekulić, A. Soleimani, M. Aliannejadi, F. Crestani, to bm25 and language models examined, in: ProLongformer for ms marco document re-ranking ceedings of the 2014 Australasian Document Comtask, arXiv preprint arXiv:2009.09392 (2020). puting Symposium, 2014, pp. 58–65. [20] S. Hofstätter, H. Zamani, B. Mitra, N. Craswell, [32] A. Lipani, M. Lupu, A. Hanbury, A. Aizawa, VerA. Hanbury, Local self-attention over long text for boseness fission for bm25 document length normaleficient document retrieval, in: Proceedings of the ization, in: Proceedings of the 2015 International 43rd International ACM SIGIR Conference on Re- Conference on the Theory of Information Retrieval, search and Development in Information Retrieval, 2015, pp. 385–388.
2020, pp. 2021–2024. [33] H. Zamani, M. Dehghani, W. B. Croft, E. Learned[21] S. Verberne, M. Sappelli, D. Hiemstra, W. Kraaij, Miller, J. Kamps, From neural re-ranking to neural Evaluation and analysis of term scoring methods ranking: Learning a sparse representation for infor term extraction, Information Retrieval Journal verted indexing, in: Proceedings of the 27th ACM 19 (2016) 510–545. international conference on information and knowl[22] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Ale- edge management, 2018, pp. 497–506. tras, I. Androutsopoulos, Legal-bert: The muppets straight out of law school, arXiv preprint arXiv:2010.02559 (2020). [23] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, Transactions of the Association for Computational
Linguistics 5 (2017) 135–146. [24] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013). [25] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. [26] J. Frej, P. Mulhem, D. Schwab, J.-P. Chevallet, Learning term discrimination, in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020, pp. 1993–1996. [27] I. Chios, S. Verberne, Helping results assessment by adding explainable elements to the deep relevance matching model, 2020. URL: https://ears2020.github.
io/accept_papers/2.pdf . [28] J. Frej, D. Schwab, J.-P. Chevallet, Mlwikir: A python toolkit for building large-scale wikipediabased information retrieval datasets in chinese, english, french, italian, japanese, spanish and more (????). [29] I. Guyon, A. Elisseef, An introduction to variable and feature selection, Journal of machine learning research 3 (2003) 1157–1182. [30] M. Taylor, H. Zaragoza, N. Craswell, S. Robertson,
C. Burges, Optimisation methods for ranking functions with multiple parameters, in: Proceedings of the 15th ACM international conference on Information and knowledge management, 2006, pp. 585–593.