Addressing the “vocabulary mismatch” issue in information retrieval is a central challenge for e-commerce search engines, because product pages often miss important keywords that customers search for. Doc2Query [1] is a popular document-expansion technique that predicts search queries for a document and includes the predicted queries with the document for retrieval. However, this approach can be ineficient for e-commerce search, because the predicted query tokens are often already present in the document. In this paper, we propose Doc2Token, a technique that predicts relevant tokens (instead of queries) that are missing from the document and includes these tokens in the document for retrieval. For the task of predicting missing tokens, we introduce a new metric, “novel ROUGE score”. Doc2Token is demonstrated to be superior to Doc2Query in terms of novel ROUGE score and diversity of predictions. Doc2Token also exhibits eficiency gains by reducing both training and inference times. We deployed the feature to production and observed significant revenue gain in an online A/B test, and launched the feature to full trafic on Walmart.com.
CEUR ceur-ws.org
The vocabulary gap problem in e-commerce search is a central challenge, as it arises from discrepancies between the vocabulary used by customers and sellers when describing products. Customer queries are often short and ambiguous, while product descriptions tend to be more detailed and explicit. For instance, a customer might search for “small building set”, intending to ifnd a set that ofers simpler building experiences for young children. However, in the product catalog, those products are often characterized by piece count and target age group, which do not align directly with this search query.
Diferent approaches have been proposed to address the vocabulary mismatch issue. In the
context of lexical retrieval, query expansion [
In this paper, we propose a new document expansion technique, called Doc2Token, which generalizes Doc2Query for application in e-commerce search, as depicted in Figure 1. Our task is to, given a product, generate relevant keywords that are absent from the product’s indexed metadata to ensure that the product is retrieved when customers search using these keywords. We call these “novel tokens”. We observed that Doc2Query’s predicted queries often contain tokens already in the product metadata instead of novel tokens, which makes it ineficient for our task. In contrast, Doc2Token is designed to predict novel tokens (instead of queries). The approach is to prepare a dataset of pairs of product and novel token, then train a seq2seq model on that dataset. By design, Doc2Token eficiently generates tokens with a high probability of being novel, rather than producing long and redundant sequences like in Doc2Query. Using the product shown in Figure 1 as an example, Doc2Query predicts “6 year old boy toy”, “3 in 1 creator”, “sea animal toy”, and “building toy for boy”. On the other hand, Doc2Token provides a more diversified set of tokens, including “small”, “kit”, and “tank”. To incorporate the Doc2Token predictions into the search system, we added them to the product’s metadata and indexed for retrieval matching and ranking.
To assess performance on the novel-token-prediction task, we introduce a new evaluation
metric called “novel ROUGE score”, denoted by “nROUGE”, to measure the ROUGE score [
Our contributions are summarized as follows: ing novel tokens.
ciency compared to Doc2Query. • We propose a novel technique, Doc2Token, for document expansion in e-commerce search, encompassing both the training setup and the modification of the loss function. • We introduce a new metric, “novel ROUGE score”, to evaluate the performance of predict• We demonstrate that Doc2Token achieves improvements in both efectiveness and efi
We define the task of novel token prediction in this section. For a product , let and represent the product text and the set of tokens extracted from , respectively. The goal is to predict novel tokens, i.e., tokens that are absent from . To achieve this, we first collect a list of relevant queries (section 3.1) for the product based on historical search logs. Next, for each product, we assemble a set of unique tokens, disregarding their sequence, through the subsequent process. We concatenate all queries, divide the concatenated sequence into individual tokens, count their frequencies, and exclude tokens already present in product . As a result, for product , we have a target token set ⋃ queries for product , and ∉ (
, ), where represents the th unique token from target
denotes the frequency of token . Instead of using all tokens as
one training target, we divide them into training instances. We train a seq2seq generative
input and outputs a sequence of tokens ̂ in an autoregressive manner. More formally,
language model, T5 [
̂ = Decoder(Encoder( )).
To account for the token frequency, we modify the T5 loss as follows. The loss for a product weighted( , ̂ ) = ( ) ∗ 5 ( , ̂ ), where is the smoothing factor, set to 0.5 in our implementation. This value is chosen to balance the contribution of token frequency to the overall loss. In practice, we use the cross-entropy (1) (2)
For model inference, we employ beam search [
We sampled the product-query pairs from user engagement data on Walmart.com with at least a certain number of add-to-cart (ATCs) over a two-year period. Then we did the following preprocessing steps. The product information used throughout the experiment includes product title, product type, brand, color, gender and description.
RF RF + FMF + PTF (Doc2Query)
Relevance filter (RF). The engaged products are not always relevant to the search query, because
users’ decisions are influenced by factors other than relevance, such as price, visual appeal,
ranking, etc. Additionally, the minimal match criteria [
Price token filter (PTF). Customer queries sometimes include price and deal intent (e.g., “under $500” or “on sale”). Such phrases are not very useful for our task, since price and deals can lfuctuate rapidly, so it does not make sense to include them as training labels. We utilized regular expressions to identify and eliminate these phrases from the query.
Overlapping token filter (OTF). This step excludes all query tokens that are present in the product information. (This is a stronger extension of the full match filter.) After this step, only novel tokens remain.
Detailed data statistics can be found in Table 1. In later sections, we show results for both Doc2Query and Doc2Token. For the Doc2Query dataset, we applied the first three filters, resulting in 14.9M product-query pairs. For the Doc2Token dataset, we built upon the Doc2Query dataset by further dividing the queries into tokens and applying the overlapping token filter, resulting in 10.3M product-token pairs. For each dataset, we partitioned it by product into training, validation, and test sets in an 8:1:1 ratio to ensure no product overlap between the sets.
To evaluate model performance, we utilize the standard ROUGE score, a widely-used evaluation
metric for summarization tasks [
= 1 ∑
ℎ ∑
∈ ∑ ∈ ( ̂ ( ℎ ℎ
(, ̂ ) (, ̂ ) primarily relies on unigrams. Therefore, we measure the ROUGE scores in terms of unigrams. In our context, we formulate the ROUGE score as follows. where , ̂ represent the reference text and predicted text for product , respectively. denotes the number of tokens, and token in the predicted text. is the number of products.
denotes the number of co-occurrences of a reference
However, a higher ROUGE score does not necessarily indicate better performance at predicting novel tokens. We observed that text predictions with high ROUGE scores often exhibit substantial token overlap with the product information. To assess the performance for novel tokens, we introduce a new metric, “novel ROUGE score” (nROUGE), where the reference text consists solely of novel tokens. Formally, the nROUGE score is defined as follows.
= 1 ∑
= 1 ∑ ∑
∈ ∑ ∈ ∗ ∗ ( ( ̂ ℎ ℎ ∗ )
(, ̂ ) (, ̂ but not in the product information. where ∗ represents the novel reference text for product , i.e., the tokens in the reference text
For the model training, we fine-tuned the public T5-base model using eight A-100 Nvidia GPUs, learning rate of 1e-4, batch size of 64, maximum input sequence of 256, and maximum output sequence of 32. For model inference, we employed a top-k beam search strategy with a beam size of 10. The input of the model is a text of product information consisting of the product title and some attributes, such as brand, color, etc.
In this section, we compare our proposed Doc2Token model to the baseline Doc2Query model from both efectiveness and eficiency perspectives. To assess efectiveness, we measure performance based on the ROUGE and nROUGE scores. In terms of eficiency, we report the resources used for model training and inference. We evaluated on four models, including both Doc2Query and Doc2Token models with and without full-match filter in data preprocessing step. )
, ) , )
, ) (3) (4) (5) (6)
w/o FMF
w/ FMF Doc2Token w/ (FMF + OTF) 3.4.1. Ofline evaluation results In Table 2, we present the results for each model based on the top 10 predictions with various beam score cutofs. The predictions were chosen if their beam scores exceeded the respective cutof value. These cutof values were tuned, for each model, to achieve the optimal nROUGE F1 score. Additionally, to assess models’ eficiency in predicting novel tokens, we concatenated the predictions and calculated the total number of predicted tokens and the number of predicted novel tokens. For the Doc2Query models, we present the result without any beam score cutof, as well as the result that achieves the highest nROUGE F1 score. For the Doc2Token models, we reported three results: one without any cutof, one with the optimal nROUGE F1 score, and a result generating a similar number of novel tokens as the optimal Doc2Query model.
To evaluate the efectiveness of the full-match filter in data preprocessing, we trained the models without incorporating that step. We observed no apparent impact on the ROUGE F1 score. However, there was a substantial improvement in the nROUGE F1 score for both Doc2Query (from 0.438 to 0.481) and Doc2Token (from 0.469 to 0.500) at the optimal cutof values. This is expected, as with the full-match filter, our training data primarily relies on labels containing novel tokens.
In our comparison between the Doc2Query and Doc2Token models, we observed that the Doc2Query models tend to achieve higher ROUGE F1 scores than the Doc2Token models. However, Doc2Token excels in achieving superior nROUGE F1 scores. Comparing the models with a similar number of predicted novel tokens, for example, the Doc2Query model with a cutof of 0.51 and Doc2Token model with a cutof of 0.29, the Doc2Token model outperforms the Doc2Query model in both nROUGE precision and nROUGE recall, yielding a higher nROUGE F1 score. With optimal cutofs, the Doc2Token model shows the superior performance compared to the Doc2Query model by achieving a 3.95% higher nROUGE F1 score from 0.496 to 0.500. This improvement is statistically significant, with a 95% confidence interval for the Doc2Token F1 score of (0.498, 0.501) obtained through bootstrap resampling. Moreover, the Doc2Token model is more eficient in generating novel tokens, achieving nearly 100% of predicted tokens being novel. In contrast, Doc2Query produces only 20% novel tokens, indicating a higher degree
w/o FMF w/o (FMF + OTF) all all of redundancy. This is expected, as the Doc2Token model is designed to predict more diverse novel tokens. 3.4.2. Model eficiency Table 3 presents the results for training and inference times. The training time is primarily afected by the size of the training data. Without the full-match filter, splitting queries into tokens explodes the data size, resulting in a longer training time for Doc2Token compared to Doc2Query. However, with the full-match filter, the situation changes: the Doc2Token strategy significantly reduces the dataset size, leading to shorter training times than Doc2Query. For inference time, we sampled 100,000 products from the test dataset and conducted model inference on the top 10 results with a batch size of 16 using a single K80 GPU machine. The inference time for Doc2Token is faster than that for Doc2Query, as the output of Doc2Token is generally shorter. The results are in agreement with the eficiency discussions from Table 2.
Doc2Token “float”, “kid”, “floaty”, “floater”, “salvavida”, “swimmy”, “baby”, “floatation”, “children”, “life”
germany cd”
Table 4 showcases two example products along with its corresponding Doc2Query and
Doc2Token predictions. The Doc2Query model produces the top 5 queries, while the Doc2Token
model generates the top 10 tokens. The novel tokens, after the process of stemming [
We evaluated the performance of the Doc2Token feature from both relevance and engagement perspectives. For the relevance evaluation, we enlisted the human annotators to assess the top 10 ranked products from impacted queries. These assessments were based on a three-point scale (exact match, substitute, irrelevant), considering factors such as product title, image, and product page at Walmart.com. We then computed the NDCG@10 based on this 3-point scale, showing a 0.49% lift (p-value=0.066). For engagement assessment, we conducted a two-week A/B test for the feature on live trafic. The test revealed a statistically significant 0.28% lift in revenue (p-value = 0.013). While the NDCG@10 improvement is statistically marginal, the statistically significant revenue increase demonstrates the efectiveness of the Doc2Token feature. By introducing relevant products in the retrieval process, the Doc2Token feature is able to enhance the end-to-end search results, assisting customers in finding what they are searching for.
In this study, we present Doc2Token, a novel document expansion technique for e-commerce search engines. We introduce the novel ROUGE score, a new metric crafted to evaluate the eficacy of document expansion endeavors. Our analysis has demonstrated that Doc2Token surpasses Doc2Query in terms of eficiency and efectiveness in addressing the vocabulary mismatch challenge. The Doc2Token feature has been deployed and evaluated online, resulting in a significant improvement in both relevance and revenue.