Recent advances in Information Retrieval (IR) have utilized high-dimensional embedding spaces to enhance the retrieval of relevant documents. The Manifold Clustering Hypothesis suggests that, although document embeddings are high-dimensional, the documents relevant to a specific query lie on a lower-dimensional manifold that depends on the query. This idea has motivated new retrieval methods, but current approaches still find it hard to clearly separate relevant signals from irrelevant noise. To address this issue, we present a new method called Eclipse, which uses information from both relevant and non-relevant documents. Our method calculates a centroid from the non-relevant documents and uses it as a reference to detect and estimate noisy dimensions in the relevant ones, leading to better retrieval results. Extensive experiments on three in-domain and one out-of-domain benchmarks demonstrate an average improvement of up to 21.03% (resp. 22.88%) in mAP(AP) and 12.04% (resp. 14.18%) in nDCG@10 w.r.t. the DIME-based baseline (resp. the baseline using all dimensions). Our results pave the way for more robust, pseudo-irrelevance-based retrieval systems in future IR research. We make the code available on Github1.
Dense retrieval models [
In this section, we begin by outlining the classical Relevance Feedback model introduced by Rocchio [26], followed by a comprehensive overview of the Dimension Importance Estimation paradigm.
Rocchio. Rocchio’s algorithm is a foundational method in information retrieval, refining query vectors by pulling them toward relevant documents and pushing them away from irrelevant ones. As modern IR systems rely on high-dimensional embeddings, moving beyond traditional vector space models requires exploring how to identify an optimal subset of query dimensions, rather than solely optimizing entire query vectors.
Dimension Importance Estimation (DIME). Faggioli et al. suggest that queries and documents
exist in a lower-dimensional, query-dependent subspace of their high-dimensional latent space R. By
projecting embeddings onto this subspace, a dense IR system can retain only the most informative
dimensions for distinguishing relevance. DIMEs assign importance scores to dimensions using a
querydependent function. This score allows the system to rank the dimensions, retaining those with higher
scores and discarding the less important ones. The selected dimensions thus form a low-dimensional,
query-dependent subspace of R. Two methods for estimating the importance of dimensions are PRF
DIME and LLM DIME. The PRF DIME method utilizes pseudo-relevance feedback by assuming that the
top- documents retrieved by a similarity measure, such as BM25 [25], are likely relevant to the query
[26, 30]. These documents are combined into a centroid vector p used to captures the alignment to the
query q, helping to rank and select the most relevant dimensions. LLM DIME, on the other hand, uses
a synthetic document a, generated by an LLM [
In this section we introduce Eclipse, a novel framework designed to improve dense vector retrieval by including non-relevant documents in the decision-making of dimension importance estimation.
Formally, for a given query q ∈ R, which is embedded in a latent space using a bi-encoder, we follow the same procedure as in DIME to retrieve a set of documents from the corpus. These documents are ranked using similarity measures such as cosine similarity or inner product. This set of documents, denoted as = {d1, d2, . . . , d}, contains pseudo-relevant documents, whose content captures mainly relevant information and typically found at the top positions, and potentially pseudo-irrelevant documents at the bottom positions, whose content captures mainly irrelevant information. Now, fixing a parameter 0 < − < , we can define pseudo-irrelevant feedback by aggregating the embeddings of the bottom − documents in into an irrelevant representative embedding p as: p = − 1 − − 1 ∑︁ d− .
=0
We define Eclipse as a weighted diference between a pseudo-relevant representative embedding p* and the irrelevant representative embedding p as: * () = (q · p* ) − (q · p). (1)
In Eq. (1), the embedding p* depends on the original DIME used to compute the relevant signal. This formulation allows for the extension of any framework of DIME. Using pseudo-relevant feedback we can instantiate the vector p by aggregating the top 0 < + < − − document embeddings from 1 ∑︀+ as: p = + =1 d.
We can also instantiate an LLM-based approach using the following pipeline: (1) Zero-shot prompting an LLM using the query ; (2) Use an encoder to embed the generated text into a latent vector representation a ∈ R; (3) Set p = a. Efectiveness metrics of our methods Eclipse ( , ) and baselines on diferent query sets and biencoders. In bold, the best performance observed for each triple IR system, test collection, and evaluation measure. Superscripts a and b indicate that the result is statistically significantly (p < 0.05) better than Baseline or standard DIMEs, respectively.
AP
AP
Retained 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 ANCE TAS-B ANCE TAS-B ....222254446738 ....222255679501 .2...22277657422aab ..2..22276654711aab .241 ....333379867507 ....433307992630a .4...33419800070ab .4...33309898242ab .375 ..2..22255546433aab ..2..22266657434aab .2...22266659216aab ..22..2266556197aabb .239 ....554411776669a ....554422896792a .5...45429199691aab ..5..55402191072aab .480 ....222265631775b ..22..2278645213aabb ..22..2287659550aabb ..22..2277658906aabb .238 ....333386889655 ..4..44301088023aab .4...43420801887ab .4...43321993054aab .374 ..2..11128792979abb ..2..22230112649aab .2...22231123684aab ..2..22232122282aab .197 ....444442130883b ....444445646662aa ....444456641798aa ..4..44475648578aa .428 DL ’20 RB ’04
∈ R control the balance between the relevant and irrelevant document signals. Rather than using a convex combination, we apply independent weighting to each term. This method provides greater flexibility and demonstrates superior performance in our experiments.
In our experiments, we compare our proposed Eclipse against the state-of-the-art DIMEs for dense IR
systems. We experiment with three dense retrieval models: ANCE [29], Contriever [16], and TAS-B
[
Datasets. We evaluate our methodology on three widely used benchmark collections for in-domain
evaluation: TREC Deep Learning 2019 (DL ’19) [
Hyperparameters. We define four primary hyperparameters that influence diferent aspects of the model’s decision-making process: +, − , , and . The parameter + ∈ {1, . . . , 10} (resp. − ∈ {1, . . . , 14}) determines the number of relevant (resp. irrelevant) documents, used to build our pseudorelevance embeddings. The hyperparameter controls the strength of the relevant representative embedding, while modulates the denoising efect of the irrelevant representative embedding. Both are positive values increasing linearly from 0.1 up to 1. For combinations where = we test the base case of = = 1.
Baselines. We compare our method to standard DIMEs, PRF DIME and LLM DIME. We use GPT4
[
In our experiments, we investigate the following research questions: RQ1: Can non-relevant documents be leveraged using irrelevant feedback to improve state-of-the-art DIME approaches? RQ2: Are metrics of the retrieval pipeline impacted diferently by nonrelevant results when used for dimension importance estimation? Results for RQ1: Table 1 compare both versions of Eclipse with standard DIMEs (PRF and LLM) on the TREC DL ’19, DL ’20, DH, and RB ’04 datasets, using the ANCE, Contriever, and TAS-B models. We report the performance using the best configuration for all the DIMEs (standards and Eclipse ) in the table. The most interesting results is over ANCE, where Eclipsereduce the percentage of retained dimensions needed to surpass the baseline when using all the dimensions to just 40-60%, demonstrating that explicitly modeling both positive and negative feedback in the DIME framework yields a robust improvement. The gains are especially notable, with improvements of 21.03% in AP and 12.04% in nDCG@10 relative to DIMEs, and even higher margins over the standard baseline: 22.88% (AP) and 14.18% (nDCG@10).
Eclipse exhibits superior performance in the traditional evaluation protocol, improving performance up to 21.03% (resp. 22.88%) in AP and 12.04% (resp. 14.18%) in nDCG@10 w.r.t. the DIME-based baseline (resp. the baseline using all dimensions). In particular, both PRF Eclipse and LLM Eclipse show statistically significant improvement with respect to their DIME counterparts and Baseline.
Results for RQ2: To understand how the presence of nonrelevant documents in the dimension importance estimation pipeline afects diferent aspects of the retrieval pipeline, we analyzed the recall performance of LLM Eclipse compared the standard LLM DIME. Table 2 demonstrates that LLM Eclipse achieves consistent recall improvements over LLM DIME across multiple datasets and bi-encoders, with the most notable gains observed for low and medium relevance documents. This efect is especially pronounced in the DL collections, where recall increases of up to 16.91% are observed for marginally relevant documents. As a result, this explain why LLM Eclipse yields a larger boost in AP, which is sensitive to recall across all relevance levels. In contrast, improvements in nDCG@10 are more modest, reflecting the smaller gains for highly relevant documents that dominate the top-ranked results.
We present Eclipse, a novel method designed to enhance dense retrieval by exploiting pseudo-irrelevant feedback. This approach ofers improved separation between relevant and non-relevant dimensions within document embeddings. Unlike conventional DIME methods that rely solely on relevance signals, Eclipse introduces a contrastive perspective by utilizing irrelevant documents.
Eclipse achieves an average improvement of up to 21.03% (and 22.88% for AP) and 12.04% (and 14.18% for nDCG@10) compared to the DIME-based baseline (and the baseline using all dimensions).
By emphasizing relevant embedding dimensions, Eclipse promotes moderately relevant documents within the ranking, leading to marked gains in AP. Future research should focus on predicting a unique percentage of retained dimensions for each queries. Another unexplored section is the use of irrelevant documents generated by LLMs as a substitute for human-generated documents.
[16] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. [17] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November 2020. Association for Computational Linguistics. [18] Omar Khattab and Matei Zaharia. Colbert: Eficient and efective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 39–48, New York, NY, USA, 2020. Association for Computing Machinery. [19] Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. Approximate nearest neighbor search on high dimensional data — experiments, analyses, and improvement.
IEEE Transactions on Knowledge and Data Engineering, 32(8):1475–1488, 2020. [20] Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9:329–345, 04 2021. [21] Xueguang Ma, Minghan Li, Kai Sun, Ji Xin, and Jimmy Lin. Simple and efective unsupervised redundancy elimination to compress dense vectors for passage retrieval. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2854–2859, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. [22] Iain Mackie, Jefrey Dalton, and Andrew Yates. How deep is your learning: the dl-hard annotated deep learning dataset. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2335–2341, New York, NY, USA, 2021.
Association for Computing Machinery. [23] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, 2018. [24] N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. [25] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond.
Found. Trends Inf. Retr., 3(4):333–389, April 2009. [26] J.J. Rocchio. Relevance Feedback in Information Retrieval. Prentice Hall, Englewood Clifs, New
Jersey, 1971. [27] Gabriele Tolomei, Cesare Campagnano, Fabrizio Silvestri, and Giovanni Trappolini. Prompt-to-os (p2os): revolutionizing operating systems and human-computer interaction with integrated ai generative models. In 2023 IEEE 5th International Conference on Cognitive Machine Intelligence (CogMI), pages 128–134. IEEE, 2023. [28] Ellen M. Voorhees. Overview of the trec 2004 robust track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), Gaithersburg, MD, 2004. NIST Special Publication 500-261, National Institute of Standards and Technology (NIST). [29] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021. [30] Jinxi Xu and W. Bruce Croft. Improving the efectiveness of information retrieval with local context analysis. ACM Trans. Inf. Syst., 18(1):79–112, January 2000.