Accurate explicit and implicit product identification in search queries is critical for enhancing user experiences, especially at a company like Adobe which has over 50 products and covers queries across hundreds of tools within those products. Whether users come to learn about and purchase new products, to launch or download products they have already purchased, or to get help on products, accurate product identification is key to surfacing relevant search results and product cards. In this work, we present a novel approach to training a product classifier from user behavioral data. Our semantic model led to: >25% relative improvement in CTR (click through rate) across the deployed surfaces; a >50% decrease in null rate; a 2x increase in the app cards surfaced, which helps drive product visibility.
CEUR ceur-ws.org
Adobe boasts over 50 products for a variety of creative use cases (e.g. editing photos, videos, and audio, creating illustrations, animations, and vector graphics). When users come to Adobe.com or to Creative Cloud (CC, a subset of Adobe products focused on creativity), it is critical to route them to the right product for their use case. Users issues queries on these surfaces to learn about and purchase new products, to launch or download products they have already purchased, or to get help on products. In all of these cases, accurate product identification is key to surfacing relevant results.
Adobe.com and CC have product-focused search experiences that are augmented by contextual app card suggestions in autocomplete and at the top of search results. App cards provide users an easy way to discover, learn more about, or simply launch the Adobe product that matches their query intent. These app cards are the most clicked items on app-agnostic surfaces like CC and Adobe.com and are critical in driving new-user acquisition and product discovery, as well as providing existing users with help for their queries. Example app card triggering in autocomplete for implicit product intent is shown in Figure 1.
Initially, this matching was done via regular expression rules and simple named entity recognition. While this approach gave product experts the ability to curate the experience for end users, there were multiple problems with this approach. • Scalability: Due to the rule-based nature of the matching, this approach was hard to scale to torso and tail queries. Minor variations in phrasing would result in app cards not triggering and queries with implicit product intent (e.g. edit video should trigger app cards for Premiere Pro and Rush) rarely triggered app cards. This led to a high null rate (>50%). Furthermore, it was particularly hard to scale across the diferent languages supported by Adobe products. • Non-uniformity: There was a lack of cohesion between the behavior of app cards shown in autocomplete and search results, which are maintained by diferent teams. This led to a poor user experience.
Our system was able to solve both issues found in the previous approaches. We present a low-latency query-to-product semantic matching system that provides contextual app card suggestions for the search and autocomplete services.
Product disambiguation has often been modeled as a (named) entity recognition (NER) approach
in industry. Most approaches to query product disambiguation have been to model tokens
within queries and extract relevant entities based on the set of supported products [
In addition to NER techniques, other works have focused on a semantic search approach of
mapping product embeddings and query embeddings in the same semantic space [
Our work utilizes a mixture of components from previous approaches. We pretrain a language
model (LM) on our internal document set to learn the intricacies of Adobe products [
We support 46 Adobe products in our training dataset. The datasets are in English but come from multiple locales since English queries are used in combination with non-English in most locales. Also, Adobe products have the same oficial name in all locales and languages. In order to learn a good representation of the products and to tackle a diverse set of queries, we utilize four datasets, from user clicks to expert-maintained spreadsheets. These are described below.
Adobe HelpX Behavioral Dataset Adobe HelpX1 gets millions of unique visitors every year looking for tutorials and learn content (Adobe HelpX articles, Adobe help videos) related to Adobe products. We utilize the click logs from user queries → HelpX article clicked to generate our query → product dataset. From each help article, we extract the related product from the metadata. We use logs from January 2021 – August 2022 for our training set. This dataset is noisy but provides a large and diverse dataset that is critical for learning a good representation. A sample row from the dataset is shown in Table 1.
We utilize a relevance field that is derived by using the log of the click ratio of the querydocument pair. This is important because for a given query, there may be multiple clicked documents and we wish to pay more attention to query-document pairs with more clicks. Since we can show multiple app cards to the user (Figure 1), the applications use the relevance scores to determine which app cards to show and their order. We take the log of the max click ratio to allow less frequently clicked documents to be part of the learning process.
= ( (( ( ⟶ ) ⟶ )) ) where represents the specific query; represents the particular document clicked; represents the number of clicks for the pair; represents the set of documents clicked for , ie = { … }.
HelpX Document Dataset Adobe HelpX documents are curated by Adobe content creators to provide information about Adobe products, from tutorials to product announcements. We utilize this high-quality resource by considering the document title and description as unique query-product training pairs. Since this dataset is curated and high quality, it is given a higher weight during training (relevance = 1).
Product NER Explicit Dataset From user queries in Creative Cloud, we utilize a rule-based product NER to extract query-product pairs containing explicit product names. This dataset allows us to train on a wide set of explicit, high precision intent queries.
Adobe Express Dataset Adobe Express is one of the newer Adobe products and hence has very few user clicks in our behavioral datasets.2 To bolster additional training data for this product and to learn a good representation for its queries, we utilize top Express in-product queries as part of our dataset.
Finally, we merged the four datasets. Each unique query may have 1 or more products assigned to it. The overall dataset counts are shown in Table 2. (1) Dataset Adobe HelpX Behavioral Dataset Adobe HelpX Document Dataset Adobe Express Dataset Product NER Explicit Dataset
We divide the task of learning a good representation of user queries into two parts. The first part is pretraining a language backbone to utilize for downstream finetuning. The second is training the classifier head on top of the language backbone. 2We did not add other product-specific datasets since there was already suficient HelpX data for them and because in-product help-related queries are routed to HelpX.
We found open-source language models (LMs) like BERT [
To counter this, we pretrained a LM based on Microsoft DeBERTa v3 [
Once we had our domain-specific LM backbone, we trained a classifier to predict Adobe products given a user query. We utilized the training datasets described in section 3 for this classifier head. We experimented with freezing the LM backbone (no weights are updated in the LM) and found the best combination to be to freeze the backbone for the initial few epochs and then train the full system for a few additional epochs.
We utilized a classic 2-hidden-layer Multilayer Perceptron network, with a 0.5 dropout rate and a learning rate of 1e−5 and trained the classifier in a multi-label approach, i.e. each product was given a probability score between 0–1 given a query. The multi-label approach is necessary because a large number of implicit product queries are associated with multiple products and even explicit product queries can be associated with multiple products (e.g. photoshop is primarily associated with three Photoshop products (web, mobile, and desktop) as well as with Photoshop Express).
We use the Weighted Binary Cross Entropy loss function for our training and leverage the relevancy weights (see equation 1) to pay more attention to more important examples during training.
We reserve 10% of our initial dataset (section 3) for evaluation. We compute per-product and per-source metrics. Since the dataset comprises past user queries, it reflects the final product use cases (see section 1 and figure 1). However, it is focused on explicit product mentions since the production app cards are primarily triggered for explicit mentions. Even with explicit product intent, a given query may have multiple products associated with it based in the past user click behavior. Each of these is considered in the quantitative evaluation. As shown in Table 4, precision and recall are well balanced and result in an F1 score of .949. Detailed per-dataset analysis is shown in Table 5. Results on clean and easier datasets like the HelpX document dataset and the explicit NER dataset outperform those on behavioral data. This is because the former two datasets often have the product in the query itself, thus making it easier to predict.
We also see that the model is robust and can identify products in queries with spelling errors. This includes both small errors (1 edit distance) and large errors (2-3 edit distance). A few examples are shown in Table 6.
Through quantitative evaluation on the test set (previous subsection), we determined that the model did well when the product was mentioned in the query. We then focused on queries with implicit information about the product, e.g. keyframe caddy, fashion poster, etc. We utilized a set of 2700 production CC queries for evaluation. These queries were previously unseen by the model. We leveraged Adobe-internal product experts (e.g. product managers for CC) to judge relevancy of the predicted product to the user query. For each query, the model predicts the most likely output(s). Then the product experts mark the suggested output as correct/relevant or incorrect/irrelevant. In the cases where multiple products were predicted, the evaluators were asked to mark the predications as correct/relevant only when all the products predicted were useful. That is, the entire product intent prediction from the model had to be correct, not just a subset of the predictions. Table 7 shows the accuracy results for the 2700 queries in the qualitative evaluation. 3Spark is the original name for Adobe Express.
We AB-tested the new product intent model for showing app cards in autocomplete and at the top of search results.4 The new model was tested for all locales against the production model on the CC app and CC web site. Although the LM is trained for English queries, the large number of English queries in non-English locales and the fact that Adobe product names are identical in all languages means that the model triggers app cards in all locales.
As hypothesized, app cards surfaced significantly more with the new model due to triggering on implicit product intent queries and on misspelled queries. Previously, a few fixed queries and key terms triggered app cards, but there was no semantic model to provide broader coverage. The AB test showed a 2-fold increase in surfacing and a >50% decrease in queries with no app cards. We did not expect app cards for all queries since some queries do not have app intent; so, there should always be some queries without app cards surfacing.
Both autocomplete and search result app cards saw an increase in click-through rate for all surfaces and an increase in the unique users who engaged with app cards. Overall, there was a >25% relative improvement in CTR (click-through rate) across the deployed surfaces.5 These increases reflect the fact that the increased surfacing, e.g. for queries with implicit product intent, was high precision and provided information users needed.
Accurate product identification is critical for enhancing user experiences, especially at a company like Adobe which has over 50 products. Users on Adobe gateway surfaces such as Adobe.com and Creative Cloud are looking to learn about, license, download, launch, and get help with Adobe products. Given the broad selection of products, new customers often do not know which product they want and so ask implicit product queries around capabilities (e.g. photo editing), while returning users tend to issue explicit product queries. In this work, we present a novel approach to training a product classifier from user behavioral data. Our semantic model led to: • >25% relative improvement in CTR (click-through rate) • a >50% decrease in null rate • a 2x increase in the app cards surfaced, which helps drive product visibility.
As future work, first, we are training a multi-lingual version of the model to better support
nonEnglish queries with implicit product intent (e.g. images gratuites (French: ‘free pictures’) which
is associated with the Adobe Stock image marketplace). Second, we are experimenting with
better long prompt understanding for product disambiguation. This is particularly important
for RAG based systems [