In our research, we aim to assess the efectiveness of Large Language Models (LLMs) in performing query expansion in the context of Information Retrieval (IR). Some recent solutions proposed and studied in the literature to perform this task have proven efective considering specific LLMs, datasets, or prompt engineering techniques. In this paper, we intend to deepen this analysis with a more comprehensive and up-to-date view of their efectiveness, by comparing the results obtained from such solutions in the context of Zero-Shot (ZS) and Chain-of-Thought (CoT) learning, so as to be agnostic with respect to Few-Shot (FS) learning that requires additional training data from the dataset considered for evaluations, and using a variety of LLMs also of the latest generation. Results obtained across various LLMs generally demonstrate the superiority of utilizing recent LLM-based solutions for query expansion when employed in a prompt engineering scenario based on Zero-Shot learning. This showcases the intrinsic efectiveness of such recent LLMs even characterized by a modest number of parameters.
In an Information Retrieval System (IRS), the proper formulation of a query has a substantial impact
on the efectiveness of the system in retrieving relevant search results. However, it is well known in
the literature how uncertainty and vagueness are challenges encountered when formulating a query
due to imprecise or ambiguous user input [
To tackle these hurdles efectively, it is often necessary to reformulate the query, ensuring it aligns
more closely with the user’s information needs. This often involves employing a specialized strategy
such as query expansion, which entails integrating additional related terms to encompass potential
interpretations of the user’s intent. Many approaches have been proposed over the years to address
this tasks [
Nowadays, in particular, with the rapid progress of the so-called generative AI, Large Language Models
(LLMs) have been efectively applied to the query expansion problem in IR through the application of
prompt engineering techniques [
CEUR
ceur-ws.org
query expansion. Such a model was tested in the original paper in a Few-Shot (FS) learning setting [
Such solutions have proven their efectiveness with respect to specific and distinct LLMs, datasets, or prompt engineering techniques; however, it is our opinion that they need to be further tested comparatively against the use of diferent LLMs (both with a large and small number of parameters, both proprietary and open-source) and against prompt learning techniques that are agnostic to additional labeled data with respect to the datasets used for evaluations but rely only on the potential of the pre-trained models. Hence, in our study, we examine the capabilities of several recent LLMs ranging from 7 to 20 billion parameters, in both Q2D and Q2E operations, with respect to multiple IR subtask considering diferent datasets (and contexts), in both a ZS and a CoT setting.
The findings align closely with existing literature, afirming the eficacy of the Q2D approach for query expansion tasks using LLMs guided by prompt engineering techniques. A notable observation, especially when compared to prior comprehensive studies, is that contemporary LLM models consistently outperform their predecessors in Zero-Shot learning scenarios. This underscores the inherent efectiveness of these recent LLMs, even when operating with a relatively limited number of parameters.
The remainder of the article is structured as follows: Section 2 delves into previous literature pertaining to the task under consideration; Section 3 delineates the models and prompt engineering techniques employed with LLMs for executing the task; Section 4 expounds upon and deliberates the outcomes of the comparative evaluation; and finally, Section 6 encapsulates the study’s findings and identifies avenues for future research.
Query expansion in Information Retrieval (IR) is a technique used to improve the efectiveness of
search queries by automatically supplementing them with additional relevant terms [
At its core, query expansion enhances IRSs by broadening query terms into additional terms that
convey the same concept or information need, thereby increasing the probability of a lexical match
with documents in the corpus. Early research on query expansion predominantly centered on
PseudoRelevance Feedback (PRF) [
It is in this scenario that the capabilities associated with pre-trained generative AI models open up intriguing possibilities for query expansion. Although the literature is still quite limited in this regard, some work is demonstrating the efectiveness of LLMs for the task of expanding queries using only prompt engineering techniques [10]. In this regard, there are actually three main techniques that are used nowadays [11], previously mentioned in the Introduction, which are worth briefly detailing in this section. Zero-Shot learning refers to the ability of a model to perform a task even when it has not been explicitly trained on examples of that task. In other words, the model is able to generalize from its training data to perform new tasks it has not seen before. Few-Shot learning extends the idea of ZS learning by allowing the model to be further fine-tuned on a very small number of examples (a “few shots”) for a given task. Finally, Chain-of-Thought learning emphasizes the ability of a language model to maintain context and coherence over longer passages of text. It is about the model’s capability to follow and understand a chain of related ideas or thoughts within a conversation or text.
Not directly related to the query expansion task, but equally useful to discuss in this section as a
starting point, is the approach presented in [12]. Here the authors propose Hypothetical Document
Embeddings (HyDE) for dense retrieval, where ZS learning instructs an LLM (i.e., GPT-3,
text-davinci003) to generate a hypothetical document from a query , by employing the following prompt: “Write a
paragraph that answers the question”. The document , encoded into an embedding vector, may capture
relevance patterns but also contain non-relevant information or hallucinations. Hence, this vector is
employed to identify a neighborhood in the corpus embedding space, where similar real documents are
retrieved based on vector similarity. This second step ground the generated document to the actual
corpus, with the encoder’s dense bottleneck filtering out the incorrect or non-relevant information. A
notable approach that has some similarities with the previous one but this time developed specifically to
perform query expansion is Query-to-Document (Q2D) [
Further elaborated in [
As we have seen, the literature works presented above each have some drawback stemming either from the use of a single LLM or from the utilization of training data in the prompt engineering phase, which furthermore has also shown not to yield better results. Given that our objective is to conduct a comparative evaluation across multiple LLMs, and we do not consider any data referencing the datasets we will use for evaluations (thus neither FS learning nor PRF), in this approach, we explore the query expansion problem in both a ZS and a CoT scenario.
Similar to what was done in [
+ = concat ({ × }, prompt ) where + is the expanded query, is the number of times the original query is repeated,1 and prompt returns the expansion terms for , which may consist of the pseudo-document in the case of using the Q2D model or a set of expansion keywords in the case of using the Q2E model, both of which were previously illustrated in Section 2. The symbol refers to the specific configuration used to generate pseudo-document or keywords, with respect to the prompt engineering technique considered. Thus, for example, = / indicates the use of the model driven by an LLM using learning for the generation of the pseudo-document for query expansion. The list of configurations used is shown in the next section.
In the context of the Zero-Shot learning scenario, the prompts utilized for generating the pseudodocument () and the expansion keywords () are as follows: () Write a passage that answers the following query: [query] () Write a list of keywords for the following query: [query]
In the context of the Chain-of-Thought learning scenario, the prompt utilized for generating the pseudo-document () is as follows: ()
These prompts were executed by each of the LLMs detailed subsequently in the next section,
concerning the various configurations outlined below:
• Q2D/ZS: the Zero-Shot version of the Q2D model [
Experimental comparative evaluation is conducted in this study with regard to the utilization of various LLMs, encompassing both proprietary and recent open-source models (Section 4.1). Evaluation is conducted on a variety of datasets, encompassing a range of tasks and domains (Section 4.2), utilizing standard metrics for assessing Information Retrieval Systems (IRSs) efectiveness (Section 4.3), and benchmarked against established baselines (Section 4.4). Subsequently, detailed results of this comparative evaluation are presented (Section 4.5). All experiments have been carried out within a sparse retrieval setting, using BM25 [14, 15] as implemented by pyTerrier [16] with its default parameters A series of recent LLMs has been considered, which spans both open-source and proprietary models, ranging from 7B to 20B parameters. Specifically, we include: • GPT-4 [17]: it is the well-known and proprietary large multimodal model (accepting image and text inputs, emitting text outputs) developed by OpenAI, updated with text up to June 13th 2023.
In this paper, we use its GPT-4-0613 version;3
1Since LLM output may be verbose, this is therefore a necessary ploy to preserve the importance of the terms of the original
query in the expanded query. In this work we considered = 5 , as already done in previous works [
Additionally, we include a comparison with the top-performing system from [
The datasets used pertain to assess the efectiveness of LLMs for the considered task with respect to distinct subtasks of Information Retrieval. In particular, the considered datasets are MS MARCO [26], developed for Passage Retrieval, and a subset of those contained in BEIR [27, 28], a benchmark dataset for Zero-Shot evaluation of IR models across diferent domain/task combinations, encompassing Medical IR, Entity Retrieval, Fact Checking, etc.
1. MS MARCO [26]:10 the MicroSoft MAchine Reading COmprehension dataset is a collection of
datasets focused on deep learning in search. In this article, the employed dataset is that referred
to Passage Retrieval (PR). Based on the passages and questions available in the Question Answering
(QA) dataset,11 a PR task is formulated. With a pool of 8.8 million passages, the aim is to rank them
according to their relevance. Relevance labels are derived from passages marked as containing
the answer in the QA dataset;
2. NFCorpus [29]:12 it is an extensive English retrieval dataset tailored for Biomedical Information
Retrieval. It encompasses 3,244 natural language queries, sourced from the NutritionFacts.org
Website. Alongside these queries are 169,756 automatically extracted relevance judgments,
pertaining to 9,964 medical documents. These documents, characterized by their
terminologyrich language, primarily originate from PubMed;
4https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
5https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
6https://huggingface.co/Qwen/Qwen1.5-7B-Chat
7https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
8https://huggingface.co/google/gemma-1.1-7b-it
9Comparison with FLAN is made only in terms of Recall (details on the use of this measure are provided in Section 4.3), the
only measure used in [
Some of the most commonly used evaluation metrics in Information Retrieval (IR) are employed to assess the efectiveness of the system in retrieving relevant results. Specifically, the following metrics are considered: • Recall at 1K (Recall@1K): it computes the recall, or the proportion of relevant documents retrieved, at various cutof points (in our case, at 1K, as in the previous literature works); 13https://sites.google.com/view/fiqa/home 14https://touche.Webis.de/data.html 15https://allenai.org/data/scifact 16https://github.com/allenai/scidocs 17http://tiny.cc/dbpedia-entity 18https://ir.nist.gov/trec-covid/ 19https://ai.google.com/research/NaturalQuestions 20http://climatefever.ai/ • Mean Reciprocal Rank at 10 (MRR@10): it measures the average reciprocal rank of the top 10 retrieved relevant documents; • Normalized Discounted Cumulative Gain at 10 (nDCG@10): it normalizes the DCG score by the ideal DCG score at 10, providing a measure of ranking quality considering both relevance and position of retrieved documents.
Classical PRF-based query expansion methods have been considered as baselines. In particular, as illustrated in [40]: • Bose-Einstein weighting (1) (BE1): this refers to a weighting scheme inspired by the Bose-Einstein (BE) statistics [41], often used in IR to assign weights to terms in a query-document context. It typically involves incorporating term frequency and document length normalization to improve retrieval efectiveness; • Bose-Einstein weighting (2) (BE2): similar to BE1, Bose-Einstein weighting (2) is another variant of the weighting scheme inspired by Bose-Einstein statistics. It may involve diferent formulations or adjustments tailored to specific retrieval tasks or datasets; • Kullback-Leibler weighting (KL): this weighting scheme utilizes the Kullback-Leibler (KL) divergence [42], a measure of dissimilarity between two probability distributions. In the context of IR, KL weighting is often employed to compute the similarity between the language model of the query and that of the documents in the collection, facilitating more efective retrieval by considering the relevance of documents based on their language models.
In Tables 1, 2, and 3, the results of the comparative evaluation are presented, in terms of Recall@1k, MRR@10, and nDCG@10 respectively.
Based on previous studies, particularly [
Based on our experimental evaluation, it becomes evident that across nearly all datasets and a comprehensive range of LLMs – regardless of their parameter count, source availability (open-source or proprietary), and sophistication – contemporary pre-trained models exhibit remarkable efectiveness for our task, especially when applied in a Zero-Shot scenario. This underscores the robustness and adaptability of modern LLMs, highlighting their eficacy across diverse settings and configurations.
The datasets used in this work are publicly accessible, as indicated in Section 4.2. The data generated by the seven LLMs considered in the context of this research across the three prompt settings – namely Query-to-Document in Zero-Shot (Q2D/ZS), Query-to-Entity in Zero-Shot (Q2E/ZS), and Query-toDocument with Chain-of-Thought (Q2D/CoT) – are made publicly available at the following address: https://github.com/ikr3-lab/QueryExpansionLLMs.
Our research delved into the efectiveness of Large Language Models (LLMs) for query expansion in Information Retrieval (IR) contexts, particularly focusing on the application of prompt engineering techniques. We conducted a comprehensive analysis, comparing the performance of recent LLM-based 2 e l b a T a p m o C : 3 e l b a T solutions across various learning scenarios, including Zero-Shot (ZS) and Chain-of-Thought (CoT) learning, while aiming to remain agnostic to Few-Shot (FS) learning that necessitates additional training samples (though limited in number). Our study builds upon prior research, which showcased the potential of LLMs, notably the Query-to-Document (Q2D) model, in improving query expansion tasks. By extending these investigations to encompass a broader spectrum of LLMs, datasets, and learning scenarios, we ofer a more nuanced understanding of their eficacy. Our findings afirm the superiority of recent LLM-based solutions, particularly in Zero-Shot learning scenarios, underscoring their robustness and adaptability across diverse contexts.
Looking ahead, several avenues for future research emerge. A deeper exploration of the peculiarities of individual domains and the typical form of queries within those domains could be undertaken. Additionally, investigating the interplay between diferent prompt designs and LLM architectures could uncover synergies for further enhancing performance. Furthermore, examining the generalization capabilities of LLMs across various languages could extend the applicability of these models in real-world IR applications.
We acknowledge the support of: the PNRR ICSC National Research Centre for High Performance Computing, Big Data and Quantum Computing (CN00000013), under the NRRP MUR program funded by the NextGenerationEU; CINECA under the ISCRA initiative, for the availability of high-performance computing resources;21 CSC – IT Center for Science, Finland,22 for the availability of high-performance computing resources; the Italian MUR under the PRIN 2022 Project KURAMi: “Knowledge-based, explainable User empowerment in Releasing private data and Assessing Misinformation in online environments” (20225WTRFN);23 the University of Milano-Bicocca under the ATEQC 2024 Project PriQuaDeS: “Next-generation Privacy- and Quality-preserving Decentralized Social Web Applications”. 21https://www.hpc.cineca.it/hpc-access/access-cineca-resources/iscra-projects/iscra-general-informations/ 22https://csc.fi/en/ 23https://kurami.disco.unimib.it/ [9] Z. Zheng, K. Hui, B. He, X. Han, L. Sun, A. Yates, Contextualized query expansion via unsupervised chunk selection for text retrieval, Information Processing & Management 58 (2021) 102672. URL: https://www.sciencedirect.com/science/article/pii/S0306457321001576. doi:https://doi.org/10. 1016/j.ipm.2021.102672. [10] B. Chen, Z. Zhang, N. Langrené, S. Zhu, Unleashing the potential of prompt engineering in large language models: a comprehensive review, 2023. arXiv:2310.14735. [11] L. Reynolds, K. McDonell, Prompt programming for large language models: Beyond the few-shot paradigm, in: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–7. [12] L. Gao, X. Ma, J. Lin, J. Callan, Precise zero-shot dense retrieval without relevance labels, 2022.
arXiv:2212.10496. [13] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al., Scaling instruction-finetuned language models, arXiv preprint arXiv:2210.11416 (2022). [14] S. E. Robertson, K. S. Jones, Relevance weighting of search terms, Journal of the American Society for Information science 27 (1976) 129–146. [15] S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al., Okapi at trec-3,
Nist Special Publication Sp 109 (1995) 109. [16] C. Macdonald, N. Tonellotto, Declarative experimentation ininformation retrieval using pyterrier, in: Proceedings of ICTIR 2020, 2020. [17] J. Achiam, et al., GPT-4 technical report, arXiv preprint arXiv:2303.08774 (2023). [18] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023). [19] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, S. Sanghai, Gqa: Training generalized multi-query transformer models from multi-head checkpoints, arXiv preprint arXiv:2305.13245 (2023). [20] I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150 (2020). [21] J. Bai, et al., QWEN technical report, arXiv preprint arXiv:2309.16609 (2023). [22] AI@Meta, Llama 3 model card (2024). URL: https://github.com/meta-llama/llama3/blob/main/
MODEL_CARD.md. [23] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S.
Kale, J. Love, et al., Gemma: Open models based on gemini research and technology, arXiv preprint arXiv:2403.08295 (2024). [24] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M.
Dai, A. Hauth, et al., Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023). [25] Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al., Ul2: Unifying language learning paradigms, in: The Eleventh International Conference on Learning Representations, 2022. [26] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, L. Deng, MS MARCO: A human-generated machine reading comprehension dataset (2016). [27] E. Kamalloo, N. Thakur, C. Lassance, X. Ma, J.-H. Yang, J. Lin, Resources for brewing beir:
Reproducible reference models and an oficial leaderboard, arXiv preprint arXiv:2306.07471 (2023). [28] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, I. Gurevych, Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, arXiv preprint arXiv:2104.08663 (2021). [29] V. Boteva, D. Gholipour, A. Sokolov, S. Riezler, A full-text learning to rank dataset for medical information retrieval, in: Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20–23, 2016. Proceedings 38, Springer, 2016, pp. 716–722. [30] A. Bondarenko, M. Fröbe, M. Beloucif, L. Gienapp, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, et al., Overview of touché 2020: argument retrieval, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11, Springer, 2020, pp. 384–395. [31] A. Bondarenko, P. Braslavski, M. Völske, R. Aly, M. Fröbe, A. Panchenko, C. Biemann, B. Stein, M. Hagen, Comparative web search questions, in: Proceedings of the 13th International Conference on Web Search and Data Mining, 2020, pp. 52–60. [32] J. Callan, The lemur project and its clueweb12 dataset, in: Invited talk at the SIGIR 2012 Workshop on Open-Source Information Retrieval, 2012. [33] D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, H. Hajishirzi, Fact or fiction:
Verifying scientific claims, arXiv preprint arXiv:2004.14974 (2020). [34] A. Cohan, S. Feldman, I. Beltagy, D. Downey, D. S. Weld, Specter: Document-level representation learning using citation-informed transformers, arXiv preprint arXiv:2004.07180 (2020). [35] K. Balog, R. Neumayer, A test collection for entity search in dbpedia, in: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, 2013, pp. 737–740. [36] F. Hasibi, F. Nikolaev, C. Xiong, K. Balog, S. E. Bratsberg, A. Kotov, J. Callan, Dbpedia-entity v2: a test collection for entity search, in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017, pp. 1265–1268. [37] E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soborof, L. L. Wang, Trec-covid: constructing a pandemic information retrieval test collection, in: ACM SIGIR Forum, volume 54, ACM New York, NY, USA, 2021, pp. 1–12. [38] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al., Natural questions: a benchmark for question answering research, Transactions of the Association for Computational Linguistics 7 (2019) 453–466. [39] T. Diggelmann, J. Boyd-Graber, J. Bulian, M. Ciaramita, M. Leippold, Climate-fever: A dataset for verification of real-world climate claims, arXiv preprint arXiv:2012.00614 (2020). [40] G. Amati, C. J. Van Rijsbergen, Probabilistic models of information retrieval based on measuring the divergence from randomness, ACM Transactions on Information Systems (TOIS) 20 (2002) 357–389. [41] V. Yukalov, Theory of cold atoms: Bose–einstein statistics, Laser Physics 26 (2016) 062001. [42] T. Van Erven, P. Harremos, Rényi divergence and kullback-leibler divergence, IEEE Transactions on Information Theory 60 (2014) 3797–3820.