Query-document relevance prediction is a critical problem in Information Retrieval systems. This problem has increasingly been tackled using (pretrained) transformer-based models which are finetuned using large collections of labeled data. However, in specialized domains such as e-commerce and healthcare, the viability of this approach is limited by the dearth of large in-domain data. To address this paucity, recent methods leverage these powerful models to generate high-quality task and domain-specific synthetic data. Prior work has largely explored synthetic data generation or query generation (QGen) for QuestionAnswering (QA) and binary (yes/no) relevance prediction, where for instance, the QGen models are given a document, and trained to generate a query relevant to that document. However in many problems, we have a more fine-grained notion of relevance than a simple yes/no label. Thus, in this work, we conduct a detailed study into how QGen approaches can be leveraged for nuanced relevance prediction. We demonstrate that - contrary to claims from prior works - current QGen approaches fall short of the more conventional cross-domain transfer-learning approaches. Via empirical studies spanning three public e-commerce benchmarks, we identify new shortcomings of existing QGen approaches - including their inability to distinguish between different grades of relevance. To address this, we introduce label-conditioned QGen models which incorporates knowledge about the different relevance. While our experiments demonstrate that these modifications help improve performance of QGen techniques, we also find that QGen approaches struggle to capture the full nuance of the relevance label space and as a result the generated queries are not faithful to the desired relevance label.
The task of modeling how relevant a document is to a query is among the most central problems
in Information Retrieval, and a key component of many IR systems. The e-commerce domain
is no exception, with improved relevance models leading to higher consumer engagement and
user satisfaction [
The advent of Large Language Models (LLMs) such as GPT [
Generate Queries
Labeled Data (ESCI / MS-MARCO) Vanilla QGen
hiking pack LabelCond QGen hiking pack teton backpack zero-shot application apply QGen
Task
Finetune Test Product Corpora (WANDS / HomeDepot)
Test Data
(WANDS / HomeDepot)
LLaMa [
Recently, an alternative paradigm has emerged to overcome the lack of query logs –
synthetically generated query logs i.e., Query Generation (QGen). Recent works have successfully
demonstrated the use of such techniques across different verticals and IR problems, including
Question Answering [
Thus our first contribution is providing the first detailed empirical understanding of QGen
approaches in the e-commerce domain. Using data from three different e-commerce benchmarks,
1The one notable exception is the recently released ESCI dataset [
Our second contribution involves experiments and analyses that demonstrate that (unlike claims reported in prior works) QGen approaches are outperformed by the more conventional (cross-domain) transfer learning style approaches. Via detailed analyses, we identify a set of key reasons (that we have not seen discussed – or perhaps identified – in prior works) explaining why QGen approaches fall short. For example, we observe that the best existing QGen baseline produces at least one problematic (from the lens of faithfulness / correctness) query for 80+% of products.
Per our study, a key reason responsible for the shortcomings of existing QGen techniques, is
their simplification of the label space. More specifically, QGen techniques simplify the problem
of query-document (product) relevance into a simple binary one i.e., relevant or not. In fact, most
existing approaches only use the relevant query-document pairs, by training the model to produce
the associated (relevant) query given the document. This yes/no binarization is unfortunately
a gross over-simplification of the complex relationship between queries and documents. For
example, TREC relevance judgments are often rated on a 4-point Likert scale. Thus ignoring this
nuance seems sub-optimal – as evidenced in our results. Additionally, as noted in Reddy et al.
[
Powerful transformed-based models like GPT [
Finetune-Based Typically, such a QGen model [
In this work, we explore the application of these existing QGen approaches to a much harder relevance prediction task, where it has multiple relevance classes which are nuanced as opposed to only binary relevance prediction (e.g in MS-MARCO). In the next section, we describe our adaptations to the above QGen approaches which conditions on all labels.
As mentioned above, the task of relevance prediction for e-commerce entails – given a userissued query and a product, predict the degree of relevance (e.g. highly relevant, partially relevant, irrelevant) between them. Consider an example from Table 1, where we can see the ifne-grained difference in queries across different relevance labels for the same ESCI product. Simply binarizing this task or only considering queries from one relevance label, as done in the above strategies, could risk losing the nuance. Therefore, we extend the above described vanilla QGen techniques to our nuanced relevance prediction task by conditioning the query generation on the relevance label. Below we describe our adaptations: • Finetune-Based-LabelCond: we use the entire training portion of the available data and not just the relevant query-document portion, to train the QGen model. Specifically, each annotated query-document-label triple is transformed such that the label is prepended to the document and the model is trained to the output the query , as shown in Table 1. • Prompt-Based-LabelCond: we follow PROMPTAGATOR and instead of using all 8 examples of just the relevant label, we use 2 labels per each relevance label, where again, the label is prepended to the respective example as shown in Table 1.
As before, the QGen model is applied to the product corpora of the target domain, which generates query-product examples for all labels, on which a downstream task model is then trained. In the next section, we describe in detail this entire process.
We conduct experiments for the zero-shot setting, where we assume that we do not have any
training data for our dataset of interest. We use two e-commerce datasets as our target, namely,
WANDS and HomeDepot, both described below. These datasets were selected to fulfill the
following desiderata – a) they provide a significantly-sized test sets in the e-commerce domain
and has real-world impact, and b) they have fine-grained nuance in the relevance judgements. To
understand the effectiveness of QGen over the more conventional transfer learning approaches,
we compare the cross-domain transfer learning approach (which is non-QGen) with two QGen
approaches (vanilla vs label-conditioned). For the zero-shot cross-domain transfer learning, we
train a downstream relevance prediction model on existing datasets, namely, ESCI and
MSMARCO, where MS-MARCO is more general-purpose while ESCI is e-commerce focussed,
albeit much smaller in size. The QGen models are similarly trained on ESCI and MS-MARCO,
and applied to the two target datasets to create training data for the downstream task (Figure 1).
We now present the datasets in more detail.
4.1. Data
MS-MARCO Bajaj et al. [
ESCI Reddy et al. [
Table 2 shows some examples for all these datasets and in Table 3 we describe the statistics for each dataset. 3https://www.wayfair.com/ 4There is no designated train/test split provided so for reproducibility we use the entire data as our test set. 5https://www.kaggle.com/c/home-depot-product-search-relevance/ 6http://www.homedepot.com. 7The original dataset had 74,068 examples but 279 of those had parsing issues.
Label
E S C I
We use the pretrained mT5-XXL [
Train QGen We finetune a QGen model using the training portion of MS-MARCO and ESCI. We use the same train/dev/test splits as provided with the respective datasets and transform the input/output as shown in Table 1. We finetune the QGen model for 100k additional steps with a constant learning rate of 1e-4, Adafactor optimzer, batch size 128, input sequence length 256, target length 32.8 The best checkpoint for subsequent steps was selected using the performance of BLEU on the validation set.
Apply QGen Next, we apply the above trained models to generate query-product pairs on WANDS and HomeDepot. For the label-conditioned models, the input text is a concatenation of the desired label and the product, whereas for its vanilla QGen counterparts the input text is simply the product information. Since *-LabelCond QGen models have the ability to generate 8100k steps amount to approximately 8 epochs which we thought were sufcfiient given the computational and time requirements. queries for different relevance labels, unlike their vanilla counterparts which only can generate queries for one relevance label, we generate queries for all relevance labels for a given product. Similar to the training setup, we use input sequence length of 256 with target length to be 32. As an additional filtration step, we remove duplicate queries i.e. if the same query is generated for different labels of the same product, we only retain that query-product-label triple which has the highest model probability.9
We automatically evaluate the generated synthetic data for its utility to the downstream task. To do that we evaluate the models trained on above-generated data on the respective test sets. We split the resulting filtered QGen data into a train and validation set with a 90:10 ratio such that there is no product overlap across the two sets. We experiment with two styles of downstream models, classification and ranking. classification We use a pretrained mt5-XXL based encoder-only model to perform multiclass classification, and report NDCG. 10 For ESCI-based QGen models, this would become a four-class classification task. We finetune the mt5-encoder for additional 25000 steps with a constant learning rate of 1e-4.11 We use a batch size of 64 with input sequence length of 608. The reason we chose NDCG, a ranking metric, instead of accuracy is because there is a label mismatch between ESCI and WANDS, which helps avoid an oversimplification by deterministic mapping across the two label sets. This helps evaluate whether the model is correctly ranking exactly-relevant over partially-relevant over irrelevant. In order to compute NDCG, we need a relevance score output for each query-document pair. So, from a downstream model based on the four-way ESCI classification model, we output the prediction probability (|) where is the concatenation of input query, product title, product description and is the output label (E/S/C/I). We then compute a final score by taking an expectation of the prediction probabilities by multiplying it with the label weight: =
∑︁ ={,,,}
( |) * = { = 3.0, = 2.0, = 1.0, = 0.0}
An astute reader may wonder why we go through the trouble of training a multi-class model as
opposed to using a ranking model. The reason is that we want the ability to generate new queries
for different relevance labels. This is important for search engines where queries/products that
have had more user-clicks are often indexed and served with priority (to avoid latency). However,
rare or new products often are not covered as they do not have any query associated with them, so
having the ability to generate queries across relevance labels for such products becomes crucial
to increase coverage. However, for completeness, we do report results from a neural re-ranker
and find them to underperforming than the classification model (details in section 5).
9see Table 6 for number of duplicate queries.
10We had also tried an encoder-decoder model for the classification task but found encoder-only model to outperform
slightly.
11We tried two other learning rates of 1e-3 and 5e-5 but found them to be under-performing.
ranking In the neural re-ranker setup, we use the RankT5 model [
Below, we briefly summarize all the model variants we experiment with.
First, we describe the baselines which do not use QGen:
• Random where for the target datasets the documents for a given query are randomly
ranked.
• Zero-shot (ESCI) where we train a downstream model for multi-class classification on all
of the ESCI training data and apply it directly to WANDS and Homedepot test data.
• Zero-shot (MS-MARCO) where we train a ranking model using RankT5 [
Next, we describe the baselines which use existing QGen approaches: • Prompt-Based (ESCI) where we randomly sample 8 query-product pairs from ESCI having Exact (E) relevance label and similar to PROMPTAGATOR prompt FLAN-137B to generate one relevant query for a new WANDS/Homedepot product. For the downstream application, we follow the ranking setup described in subsection 4.3. • FineTune-Based (MS-MARCO) where we finetune the QGen model on only those querypassage pairs from MS-MARCO that have Relevant label. For every new target product, we generate one relevant query and use the retriever to retrieve 35 documents as negative examples following the ranking setup.
Finally, we describe our adaptations of the above QGen approaches:
• Finetune-Based-LabelCond (ESCI) where we finetune the QGen model on all ESCI
examples, and for every new target product generate queries for all four relevance labels.
12Due to hardware restrictions we could not train the mt5-XXL model variant with their code setup.
13We used the mt5-BASE model finetuned with the unsupervised objective proposed by Izacard et al. [
For the subsequent downstream model, we initialize it with the multi-class classification model trained on all ESCI data (which we had used in our zero-shot setting), and further ifnetune it on the synthetic data, following the classification setup. • FineTune-Based-LabelCond (MS-MARCO) where we finetune the QGen model on MSMARCO examples which and for every new product generate two queries for each of the two relevance labels. We use the ranking setup to train the downstream model and initialize it with the MS-MARCO-finetuned-ranking model (used in the zero-shot setting). • Prompt-Based-LabelCond (ESCI) where we prompt FLAN-137B with 8 ESCI examples comprising of 2 examples per each relevance label. For every new target product, we generate queries for all four relevance labels, and follow ranking setup to train the downstream model.
In this section, we present the results of two major QGen families (finetune-based vs promptbased) comparing them with cross-domain transfer learning approach. Since WANDS is a more recent and challenging dataset in comparison to HomeDepot14, we focus on WANDS for our discussion. We report results for WANDS in Table 4 and for HomeDepot in Table 5. Here are our main findings:
Zero-shot Transfer Learning wins over any QGen! Overall, we find that zero-shot transfer learning outperform all QGen approaches, both vanilla and label-conditioned. This is unlike what existing works such as INPARS and PROMPTAGATOR where QGen approaches give the best downstream performance. This could be attributed to the difficulty of the downstream task, which in this case is a nuanced relevance prediction task, while the existing works focus on binary relevance which is much simpler.
Label-conditioned QGen wins over vanilla QGen! Within the QGen approaches, we find that our adaptation of conditioning on all relevance labels outperforms the vanilla versions which do not. From the results of FINETUNE-BASED and FINETUNE-BASED-LABELCOND trained on MS-MARCO, we find that exposing the QGen models to all labels (in the case of MS-MARCO they are binary) performs better by +3.3 NDCG@10 points. Therefore, we finetune with all labels on a related dataset (ESCI) for WANDS and find that it outperforms even the MS-MARCO-based QGen models. For prompt-based QGen models, we find that its label-conditioned counterpart underperforms its vaniila variant. However, the prompt-based vanilla variant is far behind (-8.3 NDCG@10 points) the finetune-based vanilla variant to begin with.
In-domain training is important! We find that for both transfer learning and QGen
approaches, transferring from a related domain is important in downstream performance. For
instance, within the transfer learning models, the model trained on ESCI (zero-shot (ESCI))
gives the best downstream performance, even outperforming the model trained on MS-MARCO
(zero-shot (MS-MARCO)), which is trained on nearly 10 times larger training data than ESCI.
This again emphasizes that having a related dataset to transfer from is essential for
downstream performance, similar to Gururangan et al. [
Below we discuss the probable reasons for the shortcomings of QGen approaches. We inspect three QGen models which have been trained with all labels, namely, PROMPT-BASEDLABELCOND (ESCI), FINETUNE-BASED-LABELCOND (ESCI) and FINETUNE-BASEDLABELCOND (MS-MARCO), for the number of duplicate queries generated by the model. Specifically, a duplicate query here refers to the QGen model producing the same query across different relevance labels for the same product. In Table 6 we report the results for WANDS. As you recall, for each of the WANDS 42,994 products, the QGen models trained on ESCI, generated 171,976 queries, one for each of the four relevance labels. For QGen models using MS-MARCO, we generate 85,988 queries, one of each of the two relevance classes. In Table 6 we find that the FINETUNE-BASED-LABELCOND (ESCI) QGen model produces duplicate queries for 81% of the products, which suggests that simply prepending label information in the input context is insufficient for the model to learn how to generate discriminative queries. We would also like to highlight the fact that this is happening despite exposing the QGen model to the entire ESCI training data which is 1.6 million examples, of which only 5 of the 1.1 million products had duplicate queries. In Table 7 we report the distribution of generated queries across different labels, after applying the filtration step (described in subsection 4.2) where we remove the duplicate queries. Clearly, noise in the synthetic queries causes errors in the subsequent downstream models. Interestingly, despite PROMPT-BASED-LABELCOND (ESCI) and FINETUNE-BASED-LABELCOND (MS MARCO) models having more number of valid queries, they still underperform FINETUNE-BASED-LABELCOND (ESCI).
The reason why FINETUNE-BASED-LABELCOND (MS-MARCO) is underperforming its
ESCI counterpart could be attributed to a) the difference in domain and, b) the style of queries. For
instance, queries from MS-MARCO-trained-QGen models are more formal what-style questions,
while queries from ESCI-trained-QGen models are more informal and similar in style to the
gold queries. Although, PROMPT-BASED-LABELCOND (ESCI) has far fewer duplicate queries
it severely underperforms probably because of poor overall quality. In Table 8 we present the
generated queries from different QGen model for a product. We also provide the user-issued or
gold query from the WANDS test set for the same product.15 For PROMPT-BASED-LABELCOND
(ESCI) we see that the query for the highest relevance label i.e. ‘E’ focuses on the entity bed
frame with free storage plans, while from the product description we know that it is mainly
about a bed frame which is made from acacia wood and additionally has storage. Nowhere
does the product talk about storage plans. In fact, the query for the next relevance label ‘S’
is more relevant than the one for ‘E’. Clearly, exposing the models to only 8 examples, as
proposed by PROMPTAGATOR [
This highlights that even though QGen techniques offer a promising solution for adapting
models to new domains, they need further investigation and analyses to make them more effective
across different tasks.
16Dai et al. [
ESCI data 17Given that, for WANDS in Table 4 the PROMPT-BASED-LABELCOND (ESCI) is almost 21 points behind its ifnetuned counterparts, we did not apply this additional step which would require additional model training.
Synthetic Question Generation has come a long way from relying on simple but rigid heuristics
[
QGen for QA Pre-transformer era had seq-to-seq models trained with attention to read an
input sentence and generate a question with respect to an answer which is contained in that
sentence e.g. for factiod QA [
Label-Conditioned QGen Some previous work have looked at label conditioning in QGen
models for classificication tasks. Kumar et al. [33], Yang et al. [34] find that pre-pending
class-labels to input text is quite effective in class-conditional test generation and thereby data
augmentation. They show the effectiveness of this approach for classification tasks (e.g. SST-2
with binary relevance, SNIPS with 7 intents, and TREC with six-classes, SNLI, commonsense
reasoning) across different pretrained LMs including auto-encoder LM (BERT Devlin et al. [35]),
auto-regressive LM (GPT-2 Radford et al. [
From the above results, it is apparent that QGen approaches, although offering a promising direction especially for zero-shot settings, need considerable work to outperform transfer learning. Clearly, simply adding label information in the input context does not provide a sufficient signal for the model to generate discriminative queries. We need to explicitly enforce this signal throughout the QGen training process. In this work, we only generate one query, but using beam search we could generate multiple queries for a given product-label combination, resulting in a diverse collection. Another challenge in working with QGen approaches is that the typical strategy for evaluating the synthetic data is to evaluate it on a downstream task, requiring two additional steps after training a QGen model: applying the QGen model for generating queries and then training a downstream task model, to understand the effect of synthetic data. So if a researcher wanted to experiment with multiple QGen models, they would have to run three times the number of experiments to understand which QGen model is the best one, which is a waste of resources and time. This means that we need to come up with an intrinsic evaluation metric that correlates well the downstream task performance. Our next steps are focused on addressing these issues.
We would like to thank the anonymous reviewers for the valuable feedback and suggestions. We would also like to thank Honglei Zhuang and Rolf Jagerman for helping us run and adapt RankT5 model for our experiments. ontology, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 6424–6439. URL: https://aclanthology.org/2021. acl-long.502. doi:10.18653/v1/2021.acl-long.502. [33] V. Kumar, A. Choudhary, E. Cho, Data augmentation using pre-trained transformer models, in: Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, 2020, pp. 18–26. URL: https://aclanthology.org/2020.lifelongnlp-1.3. [34] Y. Yang, C. Malaviya, J. Fernandez, S. Swayamdipta, R. L. Bras, J.-P. Wang, C. Bhagavatula, Y. Choi, D. Downey, Generative data augmentation for commonsense reasoning, arXiv preprint arXiv:2004.11546 (2020). [35] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [36] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7871–7880. URL: https:// aclanthology.org/2020.acl-main.703. doi:10.18653/v1/2020.acl-main.703.