Suggesting related questions on search result pages is a popular feature of search engines which helps users satisfy their information needs. In this paper, we study the problem of related question generation in e-commerce search. We study the efectiveness of various approaches for the task that use a Large Language Model (LLM) with diferent inputs. In particular, by experimenting with the TREC Product Search dataset, we show that leveraging information from the top products in the result list can improve the performance of an approach that only uses the query. Our analysis also reveals that the most promising approach for the task is to use the diferent types of product information to generate questions and let the LLM select the best ones among them.
Presenting related questions to users in the Search Results Page (SERP) is a popular feature of
search engines like Google, also known as People Also Asked (PAA) [
We envision that in the near future LLM technology will lead to integrated information discovery within e-commerce, and such questions can serve as entry points for this integration, as shown in Figure 1.
While presenting shopping questions has many benefits, such as eficient information access and
making it easier for uninitiated shoppers to begin their journey, it has not been adopted at scale thus
far. The reasons for this are twofold. First, unlike Web search engines, e-commerce systems do not
have access to large amounts of log data from which to mine questions [
This paper takes a first step in this direction by studying the task of generating related questions in
e-commerce search. Using LLMs for the task has the potential of alleviating the problems that have
previously impeded the adoption of the related question functionality in e-commerce search engines.
In particular, the vast amount of world knowledge in LLMs facilitates the generation of high quality
questions, thus removing the dependency on logs [
The input to our system includes a query and a result list of products from a retrieval system. Using this input to represent a shopping mission, we aim to generate a set of questions that would interest many users. We study diferent generation approaches using diferent inputs, which include the search query as well as information from the top products in the result list (like product titles and reviews).
We evaluated the diferent question generation approaches with the public TREC Product Search
dataset using automatic LLM-based annotations [
The problem of generating related questions for a search query has already been studied in the context
of Web search [
We study the problem of generating related questions to a search query in e-commerce search. The input is a query and a ranked list of top products = {1, 2, ..., } returned by the e-commerce retrieval engine. Each product is represented by diferent types of textual information. In this study, we focus on the title, description, and reviews of the product; we leave the study of other fields like product attributes and bullet points for future work. Given and , our goal is to generate a set of related questions = {1, 2, ..., }. Since our focus is on the setting of e-commerce search engines, the questions should be generated such that they would help users to make a purchase decision. Another requirement is that we want the questions to have the right level of specificity such that they are interesting to as many users as possible; generating personalized questions is an alternative strategy to address this, but is not in the focus of this paper. Also note that we do not aim to generate a ranked list of questions. This is because the ranking should be done based on user engagement, ideally via an online algorithm. The set of questions will be then presented to users where they can click on them to get a quick and useful response. This paper focuses solely on question generation; online question ranking and answer generation are out of our scope and left for future work.
You are an AI assistant designed to generate questions that many users would be interested in knowing their answer after issuing a search on an e-commerce website. You will receive in the input a user query [and the top search results]. Your goal is to generate questions related to that query. [You are encouraged to use the information from the search results to come up with the most interesting questions. If any of the search results are not relevant, you can ignore them.] Generate exactly ten (10) related questions the users may be interested in asking.
Related questions requirements: (1) The questions must be relevant to the query. (2) The questions must be about broad/general topics related to the query. (3) The queries should not be specific for a single item in the result list but related to the query in general. (4) The questions must help the users in acquiring knowledge that can help them make a purchase decision. (5) If you don’t understand the query or if it is inappropriate, just provide general questions that are relevant to any product.
You are an AI assistant designed to predict how many users would find a question useful for making a purchase decision. In an online e-commerce website, users are searching for products using a search query and are presented with a list of questions that they tap on to get an answer. You are given in the input a search query, and two product-related questions. Your task is to judge which question is more interesting and useful to users.
A good question should: (1) Ask additional information that is not asked by the user query. (2) Ask about the general knowledge regarding products instead of asking for product recommendations. (3) Help users understand a specific aspect regarding related products.
Along with your selection, provide a short explanation for your reasoning. If two questions are equally good, use a special score of 0.
The focus of this study is on how to efectively leverage an LLM to generate related questions for e-commerce search engines. To this end, we investigate the performance of several zero-shot prompts that use diferent types of input from the SERP; the system instructions can be found in the top part of Figure 2. We decided to focus on zero-shot methods since we assume the setting where no information is available for us regarding what kind of questions users might ask and we would like to provide the LLM with flexibility to generate the best questions it can. We experiment with the following approaches. Query-based Generation The most straightforward approach that we study uses only the query as an input to the LLM. The advantage of only using the query is that the model has the greatest flexibility in generating questions since it is provided with the most minimal input. Furthermore, the quality of the generated questions is expected to be less sensitive to noise in the result list. Using only the query is also likely to have some disadvantages. In particular, the model may not have enough knowledge to understand challenging domain-specific queries without some context from the result list. For example, the model may have dificulty in knowing some brand names or specific model numbers of products. Furthermore, since we rely mostly on the LLM’s world knowledge, there is the risk of asking questions that are not interesting or useful with respect to the actual products presented in the result list. Retrieval-based Generation To mitigate the risks of relying solely on the search query, we experiment with prompts that include information about the top products in the result list. We examine the efectiveness of using diferent types of product information to represent it in the prompt. The first two approaches use either the Title or the Description of the product. We also experiment by representing a product with a customer Review which is randomly selected from the set of reviews. We note that selecting a random review may not result in an optimal performance but is suficient for us to obtain initial insights for the efectiveness of using reviews for question generation; we leave the study on how select the best set of reviews for future work.
Combining Multiple Product Fields Finally, to understand whether the diferent approaches are complementary, we study the performance of combining them together. The All Fields approach uses the query and all types of product information in a single prompt. An alternative approach that we study is Fusion which pools all questions from the diferent fields and prompts the LLM to select the most promising ones out of them.
5.1.1. Dataset
We use the publicly available 2023 TREC Product Search dataset [
For calculating the evaluation metrics, we compare between every pair of questions that are at the same position, resulting in 16K comparisons in total. Based on the LLM annotations, we report the win-rate of an approach that uses information from the result list compared to only using the query. The win-rate is defined as the portion of winning cases out of all of the cases where the questions are not equally good. An alternative approach would be to perform point-wise evaluation. However, our preliminary investigation of the point-wise strategy showed that all approaches generate good quality questions in at least 85% of the cases with no major diferences. We hence focus on the pair-wise evaluations to better highlight the diferences between approaches. Finally, we also conduct a pair-wise human annotation experiment with three annotators on a set of 100 questions pairs. 1We also experimented with 5, 15, and 20 products which resulted in similar performance.
5.2.1. Main Results The results in Table 2 demonstrate the strong performance of the approaches that leverage product information from the result list. In particular, we can see that for the vast majority of cases, the win-rate is greater than 0.5, indicating improvements when using almost any type of information and LLM. Comparing the performance of broad and specific queries, we can see that in most of the cases, the win-rate is similar. Still, we can see that in some cases, questions for specific queries perform slightly better than for broad ones. One possible explanation for this can be that for specific queries, since they usually refer to a single or a very small number of products, leveraging information from the result list makes it easier to identify useful product aspects, where for broad queries it may be suficient to just rely on the knowledge of the LLM.
The results also show that GPT performs better than Claude, especially for Description and All Fields. Our manual examination of the results indicated that the GPT model has better capability to filter out noise in the relatively lengthy context of those two approaches. Focusing on the results from GPT, we can see that the best approach is to use all types of product information to generate questions. Examining the individual product fields, we can see that using the description is the most promising approach, but it requires a good capability of the model to distill the useful information from it. Finally, we can see that using just the title is the most robust to model changes.
To further validate the results from the automatic evaluation, we performed human annotations. Specifically, we focused on the All Fields approach and randomly sampled 100 question pairs to compare it with the Query approach. We asked three annotators to determine the winner between every pair of questions and all annotators were assigned the same questions. The results showed that for 83% of the questions, there was a majority agreement on a winner. For this portion of questions, the win-rate was measured as 54%, attesting to the strength of the All Fields approach. Finally we measured the overall inter-annotator agreement using Fleiss Kappa [27] and achieved the value of 0.24 which indicates slight agreement. The relatively low value of agreement may be due to the subjectivity of the task which may require a larger number of annotators; we leave the extension of such human study for future work. 5.2.2. Product Category Analysis We analyze the question generation performance for queries from diferent categories in Table 3. Overall, we can see that most result list-based approaches provide consistent improvements across diferent product categories with most win-rates being higher than 0.5. It is also interesting to see that some categories are uniformly good/bad performing across diferent approaches and models. For example, while queries in the Electronics category have performance that is at least 7% higher than all queries, for Health & Household there is performance degradation of around 10%. One possible reason for the diference across categories is that some of them may have queries that are easy for the LLM to just leverage its internal knowledge to generate useful questions while for others, there can be domain specific details that the model is lacking understanding of. This can happen for certain electronic products for niche areas, for instance. We note that using the product category can be a useful tool that is easy to implement in a production system to control/block the generation for certain use cases. 5.2.3. Query Dificulty Analysis In Table 4, we report the performance of diferent groups of queries with diferent levels of retrieval performance (as measured by @10). The results indicate mostly positive correlation between the retrieval efectiveness and the question generation performance. This finding aligns with the reasoning that relying on irrelevant products has the potential of generating irrelevant questions to the query. The results also show the sensitivity to the retrieval accuracy which is to a lesser extent in the case of GPT compared to Claude. This finding aligns with our previous observation regarding GPT’s better ability to handle noise. Finally, we can see that even though there is some sensitivity with respect to the retrieval accuracy, all approaches, for most levels of accuracy, outperform the query approach which indicates stable performance when using product information from the result list.
Claude GPT [0.0, 0.1) [0.1, 0.5) [0.5, 1] [0.0, 0.1) [0.1, 0.5) [0.5, 1] 0.53 0.54 0.56 0.55 0.56 0.54 0.49 0.51 0.53 0.55 0.53 0.55 0.48 0.48 0.49 0.58 0.59 0.59 0.49 0.51 0.52 0.59 0.61 0.62 0.50 0.52 0.55 0.56 0.55 0.57 5.2.4. Question Position Analysis We analyze the win-rate of questions with respect to their position in the output of the LLM in Figure 3. The graphs reveal a very interesting trend of question performance. According to the results, a very high performance is observed for the first question which then drops and followed by a steady performance increase that peaks at the last question. A possible explanation for this can be the better likelihood of the model to come up with the most interesting question at the first position as no other questions were generated previously. The performance increase afterwards, on the other hand, may be attributed to the presence of prior tokens that better condition the generation of the subsequent new questions. Finally, we can see that regardless of the observed trend, the performance of questions in diferent positions is mostly consistently better than when only using the query. 5.2.5. Case Study Several examples of questions generated by either using only the query or by using a single field of the product are presented in Table 5. The first observation from the table is the relatively high diversity of questions across all approaches which can be attributed to each one relying on diferent types of Title Reviews Description 9 10 0.60 data sources. Furthermore, most questions seem relevant and useful which makes it reasonable to use approaches that leverage several sources of information like All Fields or Fusion. It is also interesting to see that all approaches generate questions in a decreasing order of broadness. For example, for all approaches the first question asks about the features or benefits of a specific product. Comparing the Query approaches to the others, we can see that the result list-based approaches are generally longer and more specific by mentioning non-trivial aspects of products that are likely extracted from the result list. For example, mentioning the “stabilization technology” of a GoPro. Finally, we can see that the diferent result list based approaches can difer in style and topic which resonates with their respective data source. For instance, using reviews, we can see questions that are likely to be addressed by reviews like quality and potential problems. Title questions, on the other hand, mention broader technical product aspects that can be found in titles such “weatherproof”.
This paper studied the problem of generating related questions to a search query in e-commerce search engines, in line with the increasing integration of Generative AI experiences in e-commerce [28]. Our empirical analysis demonstrated the benefit of using information from products in the result list as input to the LLM as compared to prompting it only with the search query. Furthermore, we showed that leveraging various types of product information is the most promising approach. For future work, we plan to conduct an online study to more accurately measure the usefulness of questions to real users.
This work studies the efectiveness of generative AI approaches for the task of generating related questions for e-commerce search. Generative AI tools were not employed in any way to assist with the writing of this paper. for Computational Linguistics, Miami, Florida, US, 2024, pp. 563–572. URL: https://aclanthology. org/2024.emnlp-industry.42/. doi:10.18653/v1/2024.emnlp-industry.42. [15] S. Gao, Z. Ren, Y. Zhao, D. Zhao, D. Yin, R. Yan, Product-aware answer generation in e-commerce question-answering, in: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019, pp. 429–437. [16] Y. Wang, K. Song, L. Bing, X. Liu, Harvest shopping advice: Neural question generation from multiple information sources in e-commerce, Neurocomputing 433 (2021) 252–262. [17] O. Rozen, D. Carmel, A. Mejer, V. Mirkis, Y. Ziser, Answering product-questions by utilizing questions from other contextually similar products, arXiv preprint arXiv:2105.08956 (2021). [18] X. Li, Z. Chen, J. I. Choi, N. Vedula, B. Fetahu, O. Rokhlenko, S. Malmasi, Wizard of shopping: Target-oriented e-commerce dialogue generation with decision tree branching, arXiv preprint arXiv:2502.00969 (2025). [19] H. Roitman, U. Singer, Y. Eshel, A. Nus, E. Kiperwasser, Learning to diversify for product question generation, arXiv preprint arXiv:2207.02534 (2022). [20] Z. Zhang, K. Zhu, Diverse and specific clarification question generation with keywords, in:
Proceedings of the web conference 2021, 2021, pp. 3501–3511. [21] N. Vedula, O. Rokhlenko, S. Malmasi, Question suggestion for conversational shopping assistants using product metadata, arXiv preprint arXiv:2405.01738 (2024). [22] Q. Mei, D. Zhou, K. Church, Query suggestion using hitting time, in: Proceedings of the 17th
ACM conference on Information and knowledge management, 2008, pp. 469–478. [23] Y.-C. Lien, R. Zhang, F. M. Harper, V. Murdock, C.-J. Lee, Leveraging customer reviews for ecommerce query generation, in: European Conference on Information Retrieval, Springer, 2022, pp. 190–198. [24] J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep, R. Nogueira, Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations, in: Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), 2021, pp. 2356–2362. [25] S.-C. Lin, J.-H. Yang, J. Lin, In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval, in: Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), 2021, pp. 163–173. [26] Y. Liu, H. Zhou, Z. Guo, E. Shareghi, I. Vulic, A. Korhonen, N. Collier, Aligning with human judgement: The role of pairwise preference in large language model evaluators, arXiv preprint arXiv:2403.16950 (2024). [27] J. L. Fleiss, Measuring nominal scale agreement among many raters., Psychological bulletin 76 (1971) 378. [28] K. Dhole, N. Vedula, S. Kuzi, G. Castellucci, E. Agichtein, S. Malmasi, Generative product recommendations for implicit superlative queries, in: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop), Association for Computational Linguistics, Albuquerque, USA, 2025, pp. 77–91. URL: https://aclanthology.org/2025.naacl-srw.8/. doi:10.18653/v1/2025.naacl-srw.8.