Information retrieval using transformer architectures, especially pretrained models like BERT, has seen great improvements. However, due to the quadratic complexity of the self-attention mechanism, for long documents, directly using such models is unsatisfactory. Truncating long documents is a widely adopted approach. Other researchers also propose to separate a long document into passages, each of which can be treated by a standard BERT model. The other solution is modifying the self-attention mechanism to make it sparser. However, these approaches either lose information or have high computational complexity and memory requirement. We propose a slightly diferent approach that firstly pre-ranks passages within a long document according to the query, after which the filtered top-ranking passages are combined for later ranking to obtain the document relevance score. Experiments on IR collections demonstrate the SOTA level efectiveness of the proposed approach.
Document information retrieval (IR) is used in many applications in our daily lives, including
as web search. Benefiting from deep neural networks, Neural Information Retrieval (Neural
IR) has led to the development of numerous interesting IR models. The transformer model [
To tackle this issue in IR, three techniques have been presented. The first kind is truncation
[
Relevance Scoren (another document) Query-document Relevance Score1 Label1
query
However, all three techniques have drawbacks such as information loss, high computational
costs, memory requirements or requiring modifying CUDA kernels [
From a diferent perspective, we propose a new framework [
This extended abstract describes this framework and the remainder of the paper is organized as follows: Section 2 will describe the proposed framework with classical IR approach for block ranking. While section 3 will describe two semantic block matching approaches. Then Section 4 will show the experimental results.
To begin, we introduce the proposed framework, which utilizes classical IR functions such
as BM25 for intra block ranking. It is illustrated in Figure 1. The query-block scoring part,
as can be seen, is used to identify relevant blocks across the whole document, which may be
regarded as a pre-ranking strategy. A neural IR network that generates relevance scores for a
learning-to-rank (LTR) loss is represented by the Deep Neural IR Network. In this study, we
focus here on two state-of-the-art neural IR models, namely Vanilla BERT [
The IDF is based on documents rather than blocks, as using blocks instead of documents might lead to bias, as important words in a document are likely to appear in many blocks. KeyB(vBERT) We call the model KeyB(vBERT) when the deep neural IR network is a BERT model. The most relevant blocks are concatenated together in their order of appearance in query bbbbbbbblllllllloooooooocccccccckkkkkkkk86543271 (BdEoSbRcclTlooeSrCcvienekalgelpclBaetclTvoiotceypk)ls qqSCSSCbuuloEEELLeecrSrSkPPPyy1 block ssc(aoemrviBnaegElmmRooTddeells) in diferent time slices block9 block7 blockn blockn One Long Document SEP
CLS EmbeddingL(ienveaalrmLoadyeel)r bBeEfRorTe, edvoecrlyevbelolctokkisensscoinrepdu,t to
only for block scoring, no gradient.
(traBinEmRoTdel) ECmLSbeddinLg(tirnaeinarmLoadyeel)r (QaQuSneocurtoeyhrr-eyder-o1ddcoocSc)corenLLToRss LLaabbeelln1 the document and with the query to form the input of BERT. The number of selected blocks depends on the capacity of BERT (512 tokens).
KeyB(PARADE) PARADE [
KeyB(vBERT) We propose here a strategy that aims at exploiting BERT to compute the
relevance score of a block. The overall architecture of the model proposed is depicted in Figure 2,
in which the same BERT model and linear layer are used at diferent time slices, first to compute
a query-block representation, from which ([CLS] embedding from BERT) the relevance score of
the block is derived, and then to compute the query-document representation ([CLS] embedding)
based on the top ranked blocks, finally to obtain the score of the document. This second part is
identical to the KeyB(vBERT) model, the only diference lying in the way the blocks are selected.
For the first part, both the BERT model and the linear layer are just utilized for scoring and are
not trained (hence the phrase "eval model" used in Figure 2) which reduces the complexity.
Extend for late interaction approach (ICLI) Previous approaches are interaction based
methods for long documents which is computation expensive. We propose to seek a solution for
late interaction [
We address this problem here through a BERT-based dense intra-ranking and contextualized late interaction (ICLI) with multi-task learning and the architecture is shown in Figure 3. Firstly a long document is also segmented into passages. Then each contextualized token in the passages intra passage ranking FFN2 query CLS
FFN1 BERT … … pre-stored select first passage and other top ranking passages for late interaction score fine-grained aggregation interaction W
FFN2
doc score FFN1
BERT
passage1 CLS …
pa…ssageN … CLS …
from a long document
can be obtained by the BERT model. The tokens are passed to a one-layer feedforward neural
network 1 to obtain the compressed embeddings (dimension size 128). The query tokens
and passage tokens are interacted as in ColBERT. A long document may contains many passages
and plenty of potentially not relevant passages. Similar as above methods, we propose to select
relevant passages before late interaction. In fact this can be done by BM25 and we call this
ICLI-BM25. To deal with the potential issue of exact matching, we also want to include semantic
matching in this approach. Inspired by the eficient dense retrieval, we want to take advantage
of the [CLS] embeddings and use dot-product. To do so, the [CLS] embedding is also passed to
a one-layer feedforward neural network 2 to obtain the embedding (dimension size 128)
for dense passage ranking. The first passage is always selected so that the [CLS] dot-product
of the first passage can be compared with the document level label to obtain the loss ℒ1 and
train the model to generate good [CLS] embeddings. The document score is the aggregation
of passage scores through a weighted sum, then another loss is obtained ℒ2 for training good
document embeddings. We use multi-task learning to train the model. As the two losses may
have diferent scales, we combine them to obtain the final loss in a parameter learning way:
ℒ = 2112 ℒ1 + 2122 ℒ2 + log(1 + 12) + log(1 + 22), where 1 and 2 are two learning parameters
for multi-task learning [
We report here the results on the document reranking task of TREC 2019 Deep Learning Track
[
We take other baseline results from QDS-Transformer [
Results Experimental results are displayed in Table 1. The overall average results of KeyB(vBERT) models, particularly KeyB(vBERT), which employs the BERT itself to choose blocks, exceed the baseline models by a large margin and achieve SOTA level efectiveness, with a score of 0.707 for NDCG@10. Similar to KeyB(vBERT) models, experimental results show the proposed KeyB(PARADE5)25 is also efective. In terms of NDCG@10, the proposed approach with five passages outperforms the original PARADE with 16 passages. In terms of the suggested late interaction based approach ICLI, the results reveal that the proposed approaches outperform baseline models when extended for long document retrieval, with the exception of PARADE in terms of MAP. ICLI surpasses the original ColBERT approach, with ICLI-dot achieving 0.705 in terms of NDCG@10, which is 8.46 percent greater than the original ColBERT method.
Comparing with sparse attention based methods, it is shown that the proposed select blocks models obtains comparable or better results. They all outperform QDS-Transformer in terms of NDCG@10. KeyB(vBERT) and KeyB(PARADE5)25 obtain better results in terms of MAP, while others are slightly lower. It’s worth mentioning that, unlike QDS-Transformer, our methods don’t necessitate altering CUDA kernels.
In conclusion, the proposed KeyB models are interaction based models while ICLI models are eficient late-interaction models for long document retrieval. These results show that the proposed pre-ranking framework for IR is efective, and that using learning or semantic matching for block selection has more potential. In the future, we will also seek such a solution for PARADE model.
This work has been partially supported by MIAI@Grenoble Alpes (ANR-19-P3IA-0003) and the Chinese Scholarship Council (CSC) grant No.201906960018. relevance matching, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 1049–1058. [15] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [16] I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, 2020. arXiv:2004.05150.