Researchers produce thousands of scholarly documents containing valuable technical knowledge. The community faces the laborious task of reading these documents to identify, extract, and synthesize information. To automate information gathering, document-level question answering (QA) ofers a flexible framework where human-posed questions can be adapted to extract diverse knowledge. Finetuning QA systems requires access to labeled data (tuples of context, question and answer). However, data curation for document QA is uniquely challenging because the context (i.e., text passage containing evidence to answer the question) needs to beretrieved from potentially long, ill-formatted documents. Existing QA datasets sidestep this challenge by providing short, well-defined contexts that are unrealistic in real-world applications. We present a three-stage document QA approach: (1) text extraction from PDF; (2) evidence retrieval from extracted texts to form well-posed contexts; (3) QA to extract knowledge from contexts to return high-quality answers - extractive, abstractive, or Boolean. Using the QASPER dataset for evaluation, ourDetect-Retrieve-Comprehend (DRC) system achieves a +7.19 improvement in Answer -1 over existing baselines due to superior context selection. Our results demonstrate thaDtRC holds tremendous promise as a lfexible framework for practical scientific document QA.
Growth in new machine learning publications has
exploded in recent years, with much of this activity
occurring outside traditional publication venues. For
example, arXiv hosts researchers’ manuscripts detailing
the latest progress and burgeon
CEUR
CEUR CEUR
ceur-ws.org Question Evidence What is the seed lexicon? three types.
The seed lexicon consists of positive and negative predicates. If the predicate of an extracted event is in the seed lexicon and does not involve complex phenomena like negation, we assign the corresponding polarity score (+1 for positive events and -1 for negative events) to the event.
We expect the model to automatically learn complex phenomena through label propagation. Based on the availability of scores and the types of discourse relations, we classify the extracted event pairs into the following Answer a vocabulary of positive and negative predicates that helps determine the polarity score of an event to keep pace with the latest developments in the field. evidence retrieval to generate ananswer. ral information retrieval, to find relevant literature. To tion of document-derived contents, particularly titles and date, scientific search engines [1, 2, 3] have focused on K Answers (K=3) no answer matches neither the former nor the latter event positive tive prediacnadtesnegaPDF pdf2image
DiT pdfminer.six PDF File
Page Images
Paragraph Bounding Boxes Question Text Wlexhiactoni?s the seed
ELECTRA_CE
Top-K Paragraph Texts (K=3)
CO (CONCESSION Pairs) r=0.07 The seed lexicon matches neiCthAer(CtAhUeSEformPaeir s)noTrhe r=0.10 itdshisettsCceheolreOeaudrtNrettsevClheereeExneifStvcor,SoeerInlmaaOtnte,Nmirdoa.anntotdWchrhteteihhyrseapeisendrl-aeisti--- r=0.93 sumcoeuTrthshee sterweoldateliveoexnictostnyhpacevoensis ts theCrAoeUfveSrpEsoe.sdiWtpiveoelaarasintsiudems.neegtahteive twporevdeicnattsesh.avIef ththeesapmredipoclartietioefs.an extracted
UnifiedQA
found in the details of the methodology, experimental 2. Dataset setup, and results sections. Furthermore, questions may require synthesis of document passages to produce an The Question Answering on Scientific Research Papers abstractive answer rather than simply extracting a con- (QASPER) dataset consists of 1,585 NLP papers sourced tiguous span. Reading and manually cross-referencing from arXiv, and is accompanied by 5,049 questions from the results of several papers is a labor-intensive approach NLP readers and corresponding answers from NLP practito glean specific knowledge from scientific documents. tioners. Papers inQASPER are cited by their arXiv DOIs, Therefore, efective tools to help automate knowledge which we used to fetch the original PDF documents as discovery are sorely needed. input to our system, as our work is focused on knowledge
A promising approach to extracting knowledge from extraction at the PDF level. scientific publications is document-level question answer- QASPER contains 7,993 answers categorized by aning (QA): using an open set of questions to comprehend swer type: Extractive (4142), Abstractive (1931), Yes/No ifgure captions, tables, and accompanying text [ 6]. Tradi- (1110), and Unanswerable (810). Using only theExtractive, tionally, the NLP community has focused on usingclean Abstractive and Yes/No answers, we match our model texts as context to their QA systems. However, this is prediction to the most similar answer when a question not representative of the vast majority of scholarly infor-has more than one answer, and report our performance mation found in structured documents. As QA garners accordingly. interest from the computer vision community, DocVQA QASPER is ideal for evaluating our proposed frame[7] and VisualMRC [8] have extended document QA to work because it provides: (1) paragraph text and table extracting evidence from single images, paving the way information to evaluate our layout-analysis model (in to extend contexts from text to visual sources. its ability to cleanly extract document regions); (2) ev
A foundational challenge in building robust document idence paragraphs to validate, and optionally finetune, QA systems is ensuring well-formed contexts, which en- our evidence retrieval model (in its ability to retrieve tails accurate text extraction and requires adaptation to good context paragraphs); and (3) ground-truth answers new document layouts. Nonetheless, even when text can to assess the accuracy of our QA model (in its ability to be cleanly extracted, there still remains the crucial task answer the question given the context). of identifying question-relevant paragraphs for answer prediction. 3. Methodology
Our contribution is ageneral-purpose system for QA on full documents in their original PDF form, Document QA on raw PDFs is necessary towards authat addresses the key challenges of scientific document tomating knowledge extraction from scientific corpora QA: (1) accurate text extraction from unseen layouts, (2) and has remained an unaddressed problem. To address evidence retrieval (i.e., context selection), and (3) robust this, we propose a flexible information extraction tool QA. to alleviate laboriously searching for answers grounded in evidence. Our system combines: (1) a robust text detext extraction (§ 3.2), evidence retrieval (§ 3.3), and QA weights: tion. We decompose this problem into three subtasks: tween and passage ∈ Our work addresses evidence retrieval at the PDF level. that BM25 is a strong baseline across many datasets
[14, 15]. Given a question containing tokens 1, … , tion and a PDF document, predict the answer to the ques- and a set of passages , the BM25 retrieval score beis defined using TF-IDF token (§ 3.4). images, has its semantic regions identified and their cor
First, the PDF document, represented as a series of 25 , responding text content extracted as passages. Second, =1 = ∑ log(
| | ( , ) )
( , )( 1 + 1) 1(1 − + || ) + ( , ) correspond to the respective tasks ofDetect, Retrieve and all architecture is shown in Figure2. These components are constants.
Irrelevant passages are filtered out so only the most relethe passages are ranked by their relevance to the question. where | | is the number of passages in the corpus;|| a context and question, the answer is predicted. The overvant passages are used as contexts for QA. Finally, given passages with token ; ( , ) is the term frequency of in passage ;
is the average passage length. 1 and is the length of the passage; ( , ) is the number of posed system. 3.2. Detect
TesseractOCR [10] extract text from documents indiscrim- the passage ∈ inately, including unwanted page numbers and footnotes,retrieval score is defined as the dot product of the two tector for visually rich documents, (2) explicit passage 3.3. Retrieve retrieval for evidence selection, and (3) multi-format answer prediction. We used pretrained open-source ma- Evidence retrieval identifies relevant passages by ranking chine learning models that are efective in a zero-shot setting. We also finetuned these models to improve our system’s end-to-end performance. 3.1. Problem Description them according to their similarity to the question. We considered several architectures.
Lexical Retriever
Document layout analysis models are trained to seg- which has been trained on additional data, as Karpukhin ment a document into its constituent components (e.g., et al. [16] has shown that the additional data improves ing boxes and segmentation masks. Predicted regions are a retrieval score where (, ; Φ) labels. It supports prediction of semantic region bound- and passage separately, cross-encoders [19] compute [CLS] encodes both which would need to be filtered out before the extracted text can be used as context paragraphs. Thus, prior to text extraction, document layout analysis should be performed to detect targeted regions (from which text is to be extracted). paragraphs, figures, and tables). The Document Image
and text detection.DiT uses a masked image modeling objective to pretrain a Vision Transformer1[2] without then passed to OCR tools for text extraction.
converts each page of the document to images. For each image, DiT detects the bounding boxes for paragraphs. The text within each bounding box is then extracted using where Dual Encoder
[16] learns via a contrastive training objective with inbatch negatives and hard negatives chosen byBM25.
For question and a set of passages , DPR measures question-passage similarity with a dual-encoder architecture, where encodes the question and encodes
to the same latent space [17, 18]. The resulting embeddings: ,;Φ
= (; Φ ) (; Φ ) Φ = [Φ , Φ ] denotes the retriever question and passage encoder parameters. We used the DPRmulti variant, are candidates for question evidence. pdfminer.six. The extracted texts form passages which that classifies whether is relevant to . Many crossretrieval generalizability.
Cross-Encoder Instead of embedding the questio n question and passage using the CLS token representation of their concatenation: ,;Φ
= softmax ( (, ; Φ) [CLS] + ) and are the weight and bias in the final layer encoders have since been proposed and a comparative analysis was performed 2[0], where the ELECTRA-base [21] cross-encoder (ELECTRA_CE) was declared as the
best cross-encoder due to its stability and efectiveness across datasets. Thus, the ELECTRA_CE model (trained on MS MARCO [22]) was selected as our starting cross- 4.1. QASPER Baselines encoder.
Once the passages have been ranked, the top - most relevant passages are used as contexts in the QA stage.
we use the Longformer-Encoder-Decoder L(ED) [25] as the baseline model for evidence retrieval and QA.
This model uses a modification of self-attention from 3.4. Comprehend the Transformer architecture 2[6] to encode longer seThe final stage of DRC is comprehending a document’s quences more eficiently. To jointly answer questions contents viamulti-format question answering. These for- and decide whether a context is relevant in providing mats correspond to answer types, which can be extrac- answer evidence,LED optimizes a multi-task objective. tive, abstractive, or Boolean. (An extractive answer is a In addition to answer generationL, ED adds a classificaspan of text taken verbatim from an evidence passage.tion head (termedevidence scafold ) that operates over An abstractive answer is a generated span not quoted each paragraph to predict binary labels (evidence or nonverbatim from the evidence. A Boolean answer is a binary evidence). Since we discarded unanswerable questions prediction: yes or no.) from QASPER, we retrain LED on the remaining ques
For comprehension, we useUnifiedQA [23], a genera- tions and evaluate with and without evidence scafoldtive question-answering model that has been pretrained ing. The retrainedLED serves as a fairer competitor to on 20 datasets and can predict all answer formats with UnifiedQA, which was not pretrained on unanswerable a single architecture. The answer type returned byUni- questions. fiedQA depends on the way the question is phrased.
For each of the relevant passages (from theRetrieve 4.2. Text Extraction with Layout Analysis stage), we pair the passage with the question as input to UnifiedQA to predict an answer. At the end, we hav e answers – one for each of th e passages.
First, a pretrainedDiT model predicts the bounding boxes of paragraphs on each page. Then,pdfminer.six extracts text within the bounding boxes. We denote this twostep procedure asDiT+pdfminer.six, and compare against TesseractOCR, which takes an image as input and returns the text found within the image, as well aspdfminer.six’s high-level extractor (pdfminer.six*), which takes a PDF as input and exploits PDF metadata to extract texts within sured by the 1 score between the predicted outputs and 4.3. Retriever-QA Implementation Details For BM25, we create an inverted index onQASPER validation and test sets using Pyserini2[7] with default parameters ( 1=0.9, =0.4). For DPR and ELECTRA_CE, we the target labeled inQASPER. Adopting the same notawe also evaluate its precision and recall. tion as Dasigi et al.[24], we name the 1 scores for our evidence retrieval and QA as Evidenc e-1 and Answer - 1, respectively. For text extraction, in addition to it s1 score,
4.4. Evaluation Metrics We evaluate DRC’s text extraction, retrieval, and QA stages separately. For each stage, performance is meastart with pretrained models from Hugging Face, then uate both our QA and evidence retrieval stages using this ifnetune them per hyperparameters shown in Table 1. In ifnetuning DPR and ELECTRA_CE, we sample batches single dataset. For QA, Answer- 1 is calculated between the tokens in the predicted answer and the tokens in the containing a 1:4 ratio of positive to negative evidence target answer. For our evidence retrieval stage, which passages.
1363200 model from Hugging Face. We finetune it in ranks passages by their relevance to a given question, Evidence - 1 is calculated between a fixed percentage of the top ranking passages and the set of passages labeled a weakly supervised manner using evidence passages as evidence inQASPER. ranked by ELECTRA_CE but with the original questions and answers fromQASPER. The choice to use retrieved
QASPER also contains the plain text for each of its documents, organized so that text from paragraphs and passages (instead of the human-labeled evidence passages tables are separated. We use this plain text to evaluate from QASPER) should make our system more robust to the eficacy of our text extraction to extract only the noisy context paragraphs. We show that a pretrained text primary content of PDF document. The precision, recall, extractor and evidence retriever can adapUtnifiedQA to the domain ofQASPER papers without labeled evidence. the tokens in a document’s extracted text and its tokens in and 1 score for text extraction are calculated between
Table 6 ELECTRA_CE and UnifiedQA models are not fineComparison between ELECTRA cross-encoders against LED tuned. DRC achieves an overall +2.31 improvement baselines in terms of Evidence- 1. in Answer - 1 over LED-base without scafolding on Model VEavli.denceT-es1t aQpApSrPoEacRh’s, wteestasppplilty. Twoeiamkpsruopveeruvpisoinonthteo ffinuelltyunzeeroU-snhio-t fiedQA: we sample extracted passages according to their ELECTRA_CE 31.75 36.37 retrieval scores from the pretrainedELECTRA_CE model, ELECTRA_CE-ft 31.58 36.12 assuming that higher ranked passages are correlated with LLEEDD--bbaassee-InfoNCE 2234..9940 2390..8650 selection probability for answer prediction. Thus, we are LED-large 31.25 39.37 able to finetune UnifiedQA without access to humanlabeled contexts, since labeled question-answer pairs are generally unavailable for large technical corpora. This 5. Results ifnetuning approach yields a +6.25 improvement to LEDbase in overall Answer - 1 on the test dataset.
We demonstrateDRC’s efectiveness on document QA by To analyze Answer - 1 performance when groundmeasuring its end-to-end performance. We also evaluate truth question-passage pairs are available, we consider an its constituent components on text detection, evidence re-ELECTRA_CE retriever finetuned on QASPER’s training trieval, and QA tasks against existingQASPER baselines. set. We then finetune UnifiedQA through weak superFor evidence retrieval, we study the benefits of having a vision using the now improved retrieverD.RC with a separate retrieval process, in contrast to the evidence se- finetuned ELECTRA_CE shows modest gains over the lection scafold for LED. Furthermore, we explore DRC’s zero-shot system but still lesser performance compared to performance in both zero-shot and finetuned settings, to the pretrainedELECTRA_CE with a weakly-supervised assess its performance under varying degrees of access UnifiedQA. This suggests that downstream QA perforto labeled data. mance is better improved by adapting to the target domain QASPER documents, than by receiving more relevant passages. 5.1. End-to-End QA System Combining a finetuned ELECTRA_CE retriever with a Table 2 shows DRC’s performance in terms of Answer-1. weakly supervisedUnifiedQA model shows the greatest In these experiments, we extract text from documents improvements overLED-base without scafolding, +7.19 using eitherTesseractOCR or DiT and rank passages us- in Answer - 1 on the and test dataset for all answer types. ing ELECTRA_CE. We then pass highly-ranked passages We observe that using finetuned ELECTRA_CE for weak to UnifiedQA for answer prediction. First, we study the supervision shows worse performance on Boolean quesinfluence of the text detection model on Answer- 1 per- tions than using the pretrainedELECTRA_CE to weakly formance by comparingTesseractOCR to DiT. While we supervise UnifiedQA. This discrepancy is likely due to observe that using DiT reports higher Answer - 1 than the small proportion of Boolean samples in the validation TesseractOCR across all answer types, the diference is and test datasets compared to other formats, 13% and 15% negligible. respectively.
Next, we examineDRC in the zero-shot setting where Across all experiments, DRC demonstrates superior performance toLED while solving a more dificult task: DPR’s inner product between question and passage or DRC starts from PDFs whileLED starts from clean texts. BM25’s weighted term matching.
DRC bridges an essential gap in real-world applications To analyze how retrievers perform with ground-truth for scientific knowledge extraction because PDFs are question-passage pairs, we also evaluate passage retrieval directly processed as input. In the following discussion,with DPR and ELECTRA cross-encoders finetuned on we validate our individual system components. QASPER. Here, ELECTRA_CE outperformsDPR on the test data for = 5%, 10% and 20% by an average recall 5.2. Text Detection of +3.95, +6.58, and +6.69, respectively. Notably,DPR has higher recall for =1%. We conjecture that this may be We evaluate three diferent methods for text extraction due to DPR’s contrastive objective utilizing hard negafrom PDF files: (1) DiT+pdfminer.six, (2) pdfminer.six*, tive sampling, but further analysis on the relationship and (3) TesseractOCR. Table 3 reports the average pre- between training objective and ranking is needed. cision, recall, and 1 between the extracted tokens and those in the ground-truth text. 5.3.1. Comparison to Evidence Selection Scafold
For paragraph extraction, DiT+pdfminer.six has better precision than pdfminer.six* (+19.33) and Tesserac- To compare against LED’s evidence scafold, we now tOCR (+19.01). We attribute this improvement to ex- treat ELECTRA_CE as a binary classifier. Akin to LED’s tracting fewer unwanted artifacts (e.g., page numbers, evidence scafold, we use the [CLS] representation of the headers, footers, and footnotes). For text within ta- question-passage pair as input to a single layer neural bles, only DiT+pdfminer.six is efective of-the-shelf. network to estimate the probability that the passage is relpdfminer.six* and TesseractOCR do not disambiguate be- evant as evidence to the question and use a classification tween text in and outside of tablesp.dfminer.six* and threshold of 0.5 [19]. Table 6 illustrates the evidence clasTesseractOCR would sufice if text is contained only in sification performance of zero-shot and finetuned ELECtables, or only in paragraphs, but not a mixture of the TRA_CE models against LED variations. Evidence- 1 two because the text from tables and paragraphs will be scores are computed using the extracted passages classiinterspersed. ifed as evidence with respect to the ground-truth set labeled inQASPER. We observe that the diference between 5.3. Evidence Passage Retrieval the zero-shot and finetuned ELECTRA_CE models is negligible. On the test split, zero-shotELECTRA_CE shows We compare DPR, ELECTRA_CE, and BM25 by their a notable Evidence- 1 improvement overLED-base augability to rank passages by relevance to questions. Table4 mented with InfoNCE loss 2[9], but is outperformed by shows the recall of evidence passages within various LED-large. This agrees with findings from Dasigi et al. percentages of the top ranked passages, averaged over [24] that LED-large generally outperformsLED-base for all questions inQASPER, for the retrievers in both zero- retrieval but not QA. Thus, we consider onlyLED-base shot and finetuned settings. As questions are posed for in our subsequent experiments with downstream QA. a specific document, our retrievers consider a variable number of passages per question because documents 5.4. Question Answering vary in length. Since top- penalizes longer documents when is small, we measure recall using top- % for Here, we focus on the efect of using retrieval mecha ∈ {1, 5, 10, 20} . nisms on QASPER’s plain text passages for answer pre
In the zero-shot setting,BM25 outperformsDPR by an diction. We report Answer- 1 scores for extractive, abaverage of +15.58 gain in recall on the test data. These stractive, and Boolean answer types. We also finetune results support findings from Sciavolino et al. [15], who DPR and ELECTRA_CE models on QASPER’s train split reported that DPR trained on Natural Questions 2[8] and compare againstLED variations. underperformedBM25 when faced with the new ques- Table 5 shows UnifiedQA’s Answer- 1 using the tion patterns and entities found in their EntityQuestionshighest-ranked passage from each retriever. We observe dataset. Thus, DPR requires finetuning and is less gener- that UnifiedQA (first 5 rows) generally yields higher alizable than BM25, which has no trainable parameters. Answer- 1 scores across answer types, datasets, and re
ELECTRA_CE, with average recall gains of +7.2 and trievers thanLED baselines (last 2 rows). The exception is +13.78 over BM25 and DPR, respectively, is the clear win- extractive answers from the test set whereBM25 reports ner. We hypothesize thatELECTRA_CE’s success is due a lower Answer- 1 than LED-base. Similarly, a zero-shot to the explicit interaction between every token of the DPR retriever performs worse than bothLED models. question and passage through its cross-attention mecha- Among zero-shot retrievers, ELECTRA_CE yields the nism, ofering a more expressive similarity function than best performance on the overall test set with a +7.92 and +6.54 Answer- 1 increase over LED with and without 60 1 rF55 e w sn50 A 45 40
Retriever ELECTRA_CE-FT DPR-FT ELECTRA_CE BM25 DPR our experiments include: 1. DiT demonstrates superior text extraction perfor
mance to pdfminer.six and TesseractOCR. 2. Zero-shot ELECTRA_CE ofers the best retrieval performance for all top %- (where ∈ {1, 5, 10, 20}). 3. DRC adapts to new domains through weaklysupervised training on evidence passages leading to substantially improved answer prediction over LED baselines.
tomating manual knowledge extraction from scientific This work was performed under the auspices of the U.S.
PDF documents. We showed thatDRC greatly improves Department of Energy by Lawrence Livermore National
over existing baselines, which act on clean texts and Laboratory under Contract DE-AC52-07NA27344.
sidestep the challenge of PDF-to-text extraction. Through
extensive experiments, we evaluate our pipeline
components in both zero-shot and finetuned settings. In practice, References
datasets as comprehensive asQASPER are few and may [1] M. Fadaee, O. Gureenkova, F. Rejon Barrera,
not be feasible for niche domains. In such cases, a fully C. Schnober, W. Weerkamp, J. Zavrel, A new
neuzero-shot pipeline is mandatory for document QA, and ral search and insights platform for navigating and
DRC can be weakly supervised to adapt to specific do- organizing AI research, in: Proceedings of the First
mains. Our DRC sets a new benchmark forQASPER and Workshop on Scholarly Document Processing,
Asserves as a proof of concept for an end-to-end document sociation for Computational Linguistics, Online,
QA system, from PDF to answer. Key takeaways from
tion Processing Systems, volume 34, Curran puting Machinery, New York, NY, USA, 2021, p.
Associates, Inc., 2021, pp. 25968–25981. URL: 2356–2362. URL: https://doi.org/10.1145/3404835.
https://
ArXiv abs/1901.04085 (2019). M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova,
[20] X. Zhang, A. Yates, J. Lin, Comparing score ag- L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le,
gregation approaches for document retrieval with S. Petrov, Natural questions: a benchmark for
quespretrained transformers, in: Advances in Informa- tion answering research, Transactions of the
Assotion Retrieval: 43rd European Conference on IR Re- ciation of Computational Linguistics (2019).
search, ECIR 2021, Virtual Event, March 28 – April [29] A. Caciularu, I. Dagan, J. Goldberger, A. Cohan,
1, 2021, Proceedings, Part II, Springer-Verlag, Berlin, Utilizing evidence spans via sequence-level
conHeidelberg, 2021, p. 150–163. URL: https://doi. trastive learning for long-context question
answerorg/10.1007/978-3-030-72240-1_11. doi:10.1007/ ing, arXiv preprint arXiv:2112.08777 (2021).
978-3-030-72240-1_11. [30] J. Johnson, M. Douze, H. Jégou, Billion-scale
simi[21] K. Clark, M. Luong, Q. V. Le, C. D. Manning, ELEC- larity search with gpus, IEEE Transactions on Big
TRA: pre-training text encoders as discriminators Data 7 (2019) 535–547.
rather than generators, in: 8th International
Conference on Learning Representations, ICLR 2020,
Addis Ababa, Ethiopia, April 26-30,
X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, Ms marco: A human generated machine reading comprehension dataset, arXiv preprint arXiv:1611.09268 (2016). [23] D. Khashabi, Y. Kordi, H. Hajishirzi, Unifiedqav2: Stronger generalization via broader cross-format training, arXiv preprint arXiv:2202.12359 (2022). https://huggingface.
co/allenai/unifiedqa-v2-t5-large-136320.0 [24] P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith,
M. Gardner, A dataset of information-seeking questions and answers anchored in research papers, in:
NAACL, 2021. [25] I. Beltagy, M. E. Peters, A. Cohan, Longformer:
The long-document transformer, arXiv preprint arXiv:2004.05150 (2020). [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
URL: https://proceedings.neurips.cc/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pd.f [27] J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep,
R. Nogueira, Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations, in: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, Association for Com