Dense Information Retrieval approaches are considered state-of-the-art and are based on projecting the queries and documents in a latent space, where each dimension encodes a latent characteristic of the text. In this paper, we enunciate the Manifold Clustering (MC) Hypothesis: projecting queries and documents onto a subspace of the original representation space can improve retrieval efectiveness. Based on the MC hypothesis, we define the Dimension IMportance Estimators (DIME). DIMEs operate on the query representation to estimate the expected importance of each dimension. Such DIMEs can be used to truncate the representation only to the most important dimensions. We describe two DIMEs, one based on the response generated by a Large Language Model (LLM), and one that relies on the user's active feedback. Our experiments show that the LLM-based DIME enables performance improvements of up to +11.5% (moving from 0.675 to 0.752 nDCG@10) compared to the baseline methods using all dimensions. Even more impressively, the DIME based on the active feedback allows us to outperform the baseline by up to +0.224 nDCG@10 points (+58.6%, moving from 0.384 to 0.608).
Information Retrieval (IR) systems have benefited from the emergence of pretrained Large
Language Models (LLMs), leading to the development of new systems with improved retrieval
efectiveness over the previous state-of-the-art IR systems [
We conjecture that it is possible to find a subspace of the original latent space that best represents the query and the associated relevant documents. Thus, we formulate the following Manifold Clustering hypothesis (MC hypothesis) for dense IR systems:
lie in a query-dependent lower-dimensional manifold of the representation space.
If our MC hypothesis holds, there is a query-dependent, low-dimensional manifold in the latent space where retrieval is more efective since the query and its relevant documents are closer than in the original latent space. In other terms, we assume it is possible to devise a subset of the dimensions of the latent space, optimal to represent the query and the documents and discard the other ones. This assumption corresponds to restricting ourselves to seek for linear subspaces of the original latent space, one for each query.
In this work, we describe two methods to estimate which dimensions to retain and which ones to discard, and we call them Dimension IMportance Estimators (DIMEs). Thorough experimentation of the proposed DIMEs with state-of-the-art dense IR systems on various TREC collections show impressive performance improvements: up to +0.126 (+52.8%, moving from 0.238 to 0.364) in AP and +0.224 (+58.6%, moving from 0.384 to 0.608) in nDCG@10. 1.1. The Dimension Importance Estimation Framework According to the MC hypothesis, by reducing the number of dimensions considered when computing the similarity between dense query and documents’ representations, we can improve the retrieval result. Importantly, this does not mean that we train a representation model with fewer dimensions, but we take a complete representation and remove some of the dimensions from it at query time. The task is now to determine which dimensions to be removed. To this end, we employ some heuristics, which we refer to as Dimension IMportance Estimators (DIMEs). A DIME is a function that takes in input the representation q ∈ R of a query and – possibly – some additional information and outputs a real number for each dimension of q describing its importance. Given a generic DIME , we compute the projection of the query on the top dimensions. In practical terms, this corresponds to setting to 0 the − dimensions that are not among the top ones according to the DIME scores. Finally, we use the novel representation of the query to rank the documents, by leaving unaltered the original representations of documents. This operationalization allows for DIMEs seamless integration in already deployed retrieval pipelines: there is no need for re-indexing the collection, but it is suficient to operate on the query representations only.
LLM DIME. LLMs are the current state of the art for generating documents. Therefore, given a query , we harness their power to generate an artificial document that can be used to determine which dimensions of q are the most important. In more detail, we employ a state-of-the-art LLM to generate an answer in response to the query. We are not interested in investigating if the answer returned is correct, as it will not be presented to the user but used only for computing the DIME. To avoid introducing any form of bias, we do not perform any prompt engineering: we directly input the verbatim query to the LLM, without any form of preprocessing, granting the highest possible reproducibility. Once the text in response to the query has been generated by the LLM, we compute its representation a in the latent space. Then, the DIME based on LLM feedback is defined as () = q · a. The dimension importance is given by the product of the -th dimension of the representations of the query and the LLM-generated answer.
Active-Feedback DIME. This DIME constructs upon the LLM DIME, by replacing the document generated by the LLM, with an actual, human-assessed, relevant document. This importance estimator cannot be a suitable option in an ofline scenario, as it requires knowing, for each query, at least one relevant document. Nevertheless, it can be particularly efective when it comes to online situations. Let thus us assume to have access to a relevant document in response to a query and let s be its representation in the latent space. The DIME based on Active-Feedback is defined as () = q · s. In other terms, the weight of each dimension is the product of the -th dimension of the relevant document representation and the -th dimension of the query representation.
While this DIME has a specific area of application, i.e., real-time retrieval, it is also efective in showing the power of DIMEs in identifying the optimal dimensions. In turn, it represents a sort of middle solution between the superior performance of the oracle DIME and the performance of the other, more practical DIMEs.
In our experimental analysis1, we examine three dense retrieval models: ANCE [
AP Retained 0.2 0.4 0.6 0.8 1
Table 1 shows the performance achieved if we retain a varying fraction of the representation dimensions based on the two DIMEs described before.
Concerning the DIME based on an LLM ( ), we notice that, on ANCE, the improvement ranges from +0.005 for AP on DL ‘20, to +0.020 for nDCG@10 on DL ‘19– while the improvement is present, it is not statistically significant. On the contrary, for both Contriever and TAS-B, we can observe an impressive improvement over the baseline. Indeed, the improvement for Contriever is between +0.023 (+9.55%) (Average Precision (AP) for RB ‘04) up to +0.077 (+11.5%) in the case of nDCG@10 for DL ‘19. For TAS-B on the other hand the improvement is between 0.021 (+8.96%) in the case of AP for DL HD, to 0.053 (+11.2%) for DL ‘19. The analysis highlights the large impact of using DIMEs for zero-shot application of IR models: when it comes to the RB ‘04 collection, in almost all scenarios there is a significant improvement over the baseline for both Contriever and TAS-B.
We now consider the scenario in which the user provides us some feedback, using the DIME . To simulate such feedback, for each query, we randomly pick a document with maximum relevance among those annotated for the query. First of all, it is interesting to notice that in all scenarios there is an improvement over the baseline. In particular, in the case of Contriever and TAS-B, the improvement is significant (and very large), regardless of the collection or evaluation measure considered. The maximum improvement is observed on the DL HD, where Contriever and TAS-B reach an impressive improvement in nDCG@10 of +0.220 (+57.7%) and +0.225 (+58.6%), respectively. ANCE, on the other hand, remains the most challenging model, with improvements that are not significant, although they are quite large in some cases (e.g., +0.056 of nDCG@10 with DL HD).
This paper introduces the MC hypothesis for the latent space learned by dense IR neural models: “high-dimensional representations of queries and documents relevant to them often lie in a query-dependent lower-dimensional manifold of the representation space”. According to this hypothesis, for a given query there is a subspace of the learned representation space where the representations of relevant documents tend to cluster closer around the query representation. To address the task of finding such a space, we define the problem of Dimension Importance Estimation and a novel class of models, the DIMEs. Given a dense IR model and a query, a DIME identifies the most important dimensions to induce the optimal document ranking. We propose a DIME that exploits a pseudo-relevant document generated by a LLM which allows us to gain +11.5% in the best scenario, moving from 0.675 to 0.752 of nDCG@10. We also propose an active-feedback DIME that, by using a single relevant document is capable of largely improving the retrieval performance of dense IR models. The improvement is as big as +52.8% (moving from 0.238 to 0.364 of AP) and +58.6% (moving from 0.384 to 0.608 of nDCG@10).
Among future developments, we plan to tackle the automatic selection of the optimal number of dimensions to be retained. Additionally, we plan to explore DIME based on other signals, such as previous utterances in the conversational search scenario or query reformulations. Finally, we plan to develop DIMEs based on linear combinations of the dimensions.