In this paper, we address the problem of evaluating whether results served by an e-commerce search engine for a query are good or not. is is a critical question in evaluating any e-commerce search engine. While this question is traditionally answered using simple metrics like query click-through rate (CTR), we observe that in ecommerce search, such metrics can be misleading. Upon inspection, we nd cases where CTR is high but the results are poor and vice versa. Similar cases exist for other metrics like time to click which are oen also used for evaluating search engines. We aim to learn the quality of the results served by the search engine based on users' interactions with the results. Although this problem has been studied in the web search context, this is the rst study for e-commerce search, to the best of our knowledge. Despite certain commonalities with evaluating web search engines, there are several major dierences such as underlying reasons for search failure, and availability of rich user interaction data with products (e.g. adding a product to the cart). We study largescale user interaction logs from Flipkart's1 search engine, analyze behavioral paerns and build models to classify queries based on user behavior signals. We demonstrate the feasibility and ecacy of such models in accurately predicting query performance. Our classier is able to achieve an average AUC of 0.75 on a held-out test set.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
SIGIR 2018 eCom, Ann Arbor, Michigan, USA © 2018 Copyright held by the owner/author(s). 978-x-xxxx-xxxx-x/YY/MM. . . $15.00 DOI: 10.1145/nnnnnnn.nnnnnnn
Search engines are a fundamental component of most modern
Internet applications, and evaluating their performance on a query
is not only needed for evaluating their overall performance, but is
also critical in the iterative process of improving the algorithms
that power them. is is important since bad performance of a
search engine leads to customer arition as described in White and
Dumais [
More recent work [
e most popular user activity signal in the aforementioned work
is clicks and it is used to dene the Click-rough Rate (CTR) metric.
e CTR of a query is oen used as a proxy for the performance of
the search engine on that query, and this approximation is based on
the assumption that clicks on search results are a reliable indicator
of performance. However, Hassan et al. [
We validate this observation for e-commerce search by studying the distributions of the ordinal ratings of search results given by human experts to queries having a wide range of CTR values randomly sampled from Flipkart search query-logs. We discretized the CTR values into 5 buckets with the bucket boundaries at the 20th, 40th, 60th, and 80th percentiles of the CTR values of our sampled queries. e distributions of search result ratings across these percentile-based CTR buckets is shown in Figure 2. e details of how queries are sampled from our query-logs and how the associated search results are rated by human experts are given in Section 3.
From Figure 2 it is evident that while the fraction of queries
whose results are rated as poor decreases as we go from the
lowest CTR bucket to the highest CTR bucket, a signicant fraction
of queries whose results are rated as bad still exists even in the
highest CTR bucket. Figure 1 shows an example of a search engine
results page (SERP) that appears in Flipkart’s mobile app for the
query “sling bags women lavie”. e query has good results even
though it belongs to the 0-20% CTR bucket from Figure 2. is
highlights the need for a richer set of user activity signals beyond
click behavior. Guo et al. [
Our classier is able to achieve an average AUC of 0:75 on a held-out test set. On certain product categories like Mobile Phones, we achieve an average AUC of 0:88 on the held-out test set.
Summarily, the primary contributions of our work are: (1) We identify a rich set of user activity signals that help predict whether the results for any search query would be rated as bad or good by human experts. (2) We demonstrate that it is possible to use user activity signals to automate the evaluation of search engine performance for e-commerce applications. (3) We analyze the performance of our classier and derive insights into the eectiveness of automated systems for evaluating search engine performance that are of particular interest to e-commerce applications. 2 2.1
Evaluating search engine performance has been well-studied in
the domain of web search. Topical relevance based metrics like
nDCG [
Several methods were proposed to automatically measure
various characteristics of the documents retrieved for a query, which
can then be used for measuring overall system performance. Clarity
score [
User click behavior has been used as an alternative to expert
judgments for automatically tuning retrieval algorithms (predicting
document relevance) as well as estimating IR evaluation metrics
[
ere has been considerable work in the area of analyzing user
satisfaction at a session level rather than at an individual query
level. Fox et al. [
Most studies have been geared towards web search where user
search goals are dierent from those in product/e-commerce search.
However, there has been some work recently in the context of
product search. Singh et al. [
At Flipkart, regular search quality analysis is done for a random sample of queries (stratied on query volume segment) from search logs by a team of quality experts. ey are requested to rate queries on a ve-point scale (PBAGE: Poor-1, Bad-2, Average-3, Good-4, Excellent-5) based on result relevance. To ensure the consistency of labeling across experts, inter-rater agreement is continuously monitored. In this work, we make use of the expert editorial judgments for the month of January 2018.
We selected 18,613 queries from this randomized set of expertlabeled queries which occurred more than 100 times in a week in order to ensure reasonable user activity data. is set of queries corresponded to 127M query impressions, 149M clicks and 14M other interactions (e.g. lters application, sort application) from
Table 1 lists the metrics along with their descriptions that we extracted for every query instance. We characterize the user behavior metrics as Activity time, Positional and Activity aggregates. We characterize the non-user metrics as ery text characteristics and Meta aspects.
Activity time features capture the time taken by the user for various activities. timeToFirstClick is the time taken by the user to click a product aer the results are displayes. timeToFirstCart is similar (a) Time to rst click (b) Time to rst cart (c) ery duration (d) First Click Position to timeToFirstClick except it captures time taken to add a product to the cart. queryDuration is the total time spent in interacting with the query results including all interactions with product pages, cart etc. Figures 3a-3c show the distribution of Activity time features with respect to query performance. We observe interestingly that time taken for rst click increases with the query performance. is is counter-intuitive in that when the query performance is good, it still takes users longer to click. is is however potentially explained with Figure 4a, which shows the distribution of the number of clicks against query performance. We observe that when the query performance is low the total number of clicks is lower and it increases with query performance. Intuitively, the users usually don’t click any products when the query performance is poor but when they click products for a poorly performing query they do it faster. Similar paern is observed for the add-to-cart behavior, in Figures 3b and 4c.
Positional features correspond to the position of result interaction. posFirstClick captures which position the user clicked rst. A lower position value indicates that the results were shown near the top of the page. We observe that the average position of the rst result click increases with improving query performance. is is correlated with the previous observation where time to rst click of poorly performing queries is lower and correspondingly the user is clicking the results in lower positions (faster). e total number of clicks is low when query performance is low. Similar to Activity Time features, users usually don’t click products when the query performance is poor but then they click products for a poorly performing query they do it at lower positions.
Activity aggregates features capture the aggregated summary of user’s actions for a query. We observe that all the activity aggregates are positively correlated with the query performance – i.e. increasing user activity indicates beer query performance. Number of product clicks (numClicks: Figure 4a), product swipes (numSwipes: Figure 4b), cart additions (numCarts: Figure 4c), lters applied (numFilters: Figure 4d), sort applied (numSorts: Figure 4e), product impressions per query (numImpressions: Figure 4f), query successful click through rate (clickSuccess: Figure 4g), query successful cart conversion rate (cartSuccess: Figure 4h) are all positively correlated with query performance.
ery text characteristics features capture the textual properties
of the query. chareryLen and worderyLen are length of query
in characters and words respectively. LMScore is the perplexity
score of the query based on a language model[
Meta aspect features include additional information about the query. queryCat indicates the e-commerce product category. ese are broad lines of business, namely Mobile Phones, Books, Electronics, Lifestyle, and Home and Furniture. Each query is assumed to belong to one of these categories. e intuition for using this feature is that the query performance and user behavior might be dependent on the specic categories. queryType indicates the type of query which is classied amongst three kinds, namely “Product”, “FacetCategory’ and “Category.” eries in which the exact product that the user is looking for is mentioned are called “Product” queries (e.g. iPhone X ). eries which refer to a broad group of products are called “Category” queries (e.g. shoes). “FacetCategory” queries typically contain one or more aributes followed by a category (e.g. red Nike shoes). For both queryCat and queryType, we make use of modules which are able to assign appropriate values for a given query (details of these modules is outside the scope of this paper). queryCount is the total number of times the query was issued by users in the past week. isAutoSuggestUsed indicates whether the user selected the query from the suggested queries (auto-suggest). e intuition is that the queries suggested by the search engine typically perform beer than query issued by user. isGoodNetwork indicates whether the user has a good Internet connection (dened as WiFi or LTE) while issuing the query. is is important, as the user experience and behavior might be altered if he/she doesn’t have a good Internet connection leading to bad experience independent of the search engine’s performance. numProductsFound indicates the total number of products found in the search index for the query. e intuition here is that the number of products found in conjunction with the type of the query may indicate if the search engine is not able to retrieve relevant results. 5 5.1
In this work, we formulate the problem of query performance
prediction as a binary classication task, as is done in [
We aggregate the metrics, described in previous section, across all the instances of the query in the week to obtain aggregate user behavior corresponding to the query. For metrics which may not have values for all query instances (e.g. timeToFirstClick), we only include instances for which values are present, in the aggregate calculation. ese aggregate metrics are used as features for the classication model. We experiment with various descriptive statistics for the features, namely, average, median, standard deviation, inter-quartile range5. We bin each numeric feature into 10 percentile buckets and convert them to one-hot encoded features. We also dened certain interaction features such as clickSuccess queryCount.
We split the labeled data into 80% training and 20% test set. During training, we performed feature selection using recursive feature elimination along with model hyper-parameter tuning. e hyper-parameter tuning is done using ve-fold cross validation with class-stratication and optimized for area under the ROC curve (AUC). 5.2
We analyze the results of our model along the following aspects: performance of learnt classier, feature importance, performance across e-commerce categories, performance across query types and performance across query volume. We use AUC to evaluate the prediction performance.
5.2.1 Performance of classifier. We train a binary random forest model based on the methodology described earlier in section 5.1. Figure 5 shows the AUC curve and Figure 6 shows the PR curve. 5In all the gures above, we show qualitative analysis of the features with only the “averaged” metric which suciently indicates the paerns. e overall test AUC obtained is 0.75. We observe that the classier is able to achieve a reasonably good performance, thus establishing that it is feasible to predict query performance based on user interaction signals.
One application of this predictive model is to enable automated interventions for unsatisfactory queries i.e. when the classier is condent that the results are poor, we can enable certain interventions like triggering an interactive intent solicitation module. Towards that end, we need a reasonably high precision operating point. Based on discussion with business/product team, the operating point that can be used is 85% precision where we will be able to achieve 20% recall with the current model. Rohan Kumar, Mohit Kumar, Neil Shah6, and Christos Faloutsos for “Product” queries and larger number of products for “Category” queries. Interestingly numFilters and numSorts which indicate further renement of results are also indicative of query performance, where based on Figures 4d and 4e we observe positive correlation with query performance. One surprising observation is that none of the Activity Time features are amongst the top 10 features; even though they are indicative, they are less indicative than other structured features like lters and sorts applied.
5.2.3 Performance across categories. Table 2 shows the performance of the model across the e-commerce categories (described in Section 4). We observe that the model is able to predict the query performance in “Mobile” categories considerably beer than all other categories. We conjecture this is due to model’s performance across query types (detailed below in section 5.2.4). e “Mobile” category has 7x more “Product” queries compared to the “Lifestyle” category. Additionally, “Mobile” category has 3x less “Facet Category” queries. e model is able to perform much beer for ‘Mobile’ category due to the underlying query type distribution which is biased towards “Product” queries. is is fairly important from a business perspective as the “Mobile” category contributes to a signicant portion of overall sales.
5.2.4 Performance across query types. Table 3 shows the results across query types. ere are three query types, namely, “Product,” “Facet Category” and “Category” as discussed in Section 4.
We observe that performance of “Product” queries, where the user’s intent and language is very specic, is signicantly beer than other query types. We conjecture that indicators like numProductsFound and numClicks are particularly indicative of the query performance for “Product” queries.
5.2.5 Performance across query volume segments. eries are categorized into three segments based on weekly volume: Head, TorsoHigh and TorsoLow. Table 4 shows that classier performance improves as the volume increases. e average queryCount for queries belonging to the Head segment is about 34x that of queries belonging to TorsoBoom segment. Despite the huge dierence in amount of data available per query, the classier is able to predict performance for queries in all three segments reasonably well.
Measuring search engine performance is essential to building and improving retrieval algorithms. ery performance evaluation allows for a ne-grained measure of search performance. CTR can be a noisy metric in that high CTR queries may still have poor performance, and vice versa. A more sophisticated analysis of search behavior is needed to distinguish poor and well performing queries. In this work, we successfully demonstrate that query performance can be predicted based on user’s interaction with the result set. is is the rst study to our knowledge that has collectively dened these signals in the context of query performance prediction for e-commerce search. Specically, we propose and use several user interaction signals that help characterize query performance and enabled us to achieve good classication performance using these signals. Notably, our model achieved an overall AUC of 0.75 in the binary SAT/DSAT prediction task. We have analyzed the results across various factors like category of the query, query type and query volume. Key takeaways from the performance analysis are (a) We achieve signicantly higher AUC of 0.90 on certain categories like “Mobile” making the result very promising from business impact perspective, (b) Classier performance varies across query types (“Product”, “Facet Category” and “Category”) and is best for “Product” queries, and (c) Classier performance improves with engagement volume, and is beer for Head queries than TorsoBoom queries.
Future Work e study can be extended to have a ner prediction target of issue type like spell error, vocabulary gap, selection gap etc. which would make the classier prediction more easily actionable by giving ner details on the query. Even richer signals of user activities can be used for prediction. For example, the notion of good dwell time (healthy engagement such as reading or voting on reviews) and bad dwell time (unhealthy engagement such as changing seller) might be used. Reducing the number of observations required (currently set to 100) for robustly predicting query performance would be another avenue of future work. is would allow the classier to scale an even larger number of queries which do not have many instances in a xed time period. 7
We thank Mr. Priyank Patel and Mr. Subhadeep Maji for their helpful comments.