Many tasks in e-commerce involve providing users with relevant and personalized information that reflects their particular interests and afnfiities. Although tasks such as search and recommendation are commonplace in most large-scale e-commerce platforms nowadays, there is a growing need in providing new features and capabilities, most of which would benefit from personalized user-centric strategies. However, designing custom solutions takes a considerable amount of time and effort, so the availability of generic behavioral representations that can be used in a variety of tasks, is of great practical importance In this work, we propose a generic feature extraction model that effectively reduces the time and costs of building personalized solutions by leveraging a diverse and general set of tasks suitable to ecommerce. During pre-training, the model predicts attribute embeddings of the next and purchased items in a navigation session. At run-time, instead of discarding the output prediction heads, we use them to build a novel item representation that incorporates user preferences observed in the navigation session. Equipped with these representations, we perform experiments on two different datasets, demonstrating competing performance and a high degree of complementary with other approaches.
Being able to generate personalized responses is key for e-commerce, as it improves the overall
user experience by presenting results that align with their interests and affinities. Applications of
personalization range from generating customized product recommendations [
Besides the modeling capabilities of these designs, another important factor is scale, especially in high-traffic scenarios like large auction platforms/marketplaces, and streaming service providers. In these cases, the large number of items that change over time (items being constantly added or removed) or the real-time nature of the interactions (low latency response), make the actual design face constraints that go beyond downstream performance. Furthermore, given the variety of problems that involve deciding what to present to a specific user in a given context, there is a practical need for developing generic approaches that reduce the effort required by designing custom solutions.
In this work, we tackle these problems and propose a generic learning framework that leverages user navigation data to learn a generic model that allows us to compute session-based representations that perform well on various tasks. We learn the model by leveraging supervision signals well suited for e-commerce applications. Importantly, generating such signals does not require any additional effort since they can be inferred automatically from historic navigation logs.
Our contributions can be summarized in the following: • We propose a multi-task self-supervised pre-training approach from user activity logs. Our approach is designed to be independent of the size of the product catalog. • A feature extraction mechanism that leverages the trained sequential model and enables its use across multiple applications. This mechanism can be easily integrated into existing solutions and incorporate problem-specific features in a straightforward manner. • We show that the model and the new representation can be computed incrementally with a minimum effort, making them suitable for online personalization settings. • We report experiments on two different e-commerce datasets and show consistent improvements w.r.t a variety of models. We also provide a disaggregate analysis of the session data that provide new insights into the particularities of the problem.
In this section, we review relevant work on personalization and representation learning in the
context of e-commerce. We refer the interested reader to [
In [
Instead of fine-tuning the model for each downstream task, [
[
[
[
A particularity of most of the models described above is that they are trained on some variation
of the "next-item prediction" task. This task is usually formulated as a classification problem over
the items in the catalog. In an e-commerce setting, this number can be too large from a practical
standpoint (number of parameters in the last layer of the model). Moreover, with new items
being added and removed constantly, the effectiveness and traceability of such a system over time
becomes an issue. An alternative formulation, proposed by [
Lastly, none of the described models provide a mechanism to incorporate problem-specific
features when using them for new downstream tasks. That means that all downstream tasks
must rely solely on the user session and not on other features that may be relevant to the specific
problem [
Let be the set of items1 available to the user. An item ∈ can be described by a collection of different attributes () = {()} =1, e.g. its price, title, product description, etc. Let be the set of events that capture the different ways a user can interact with an element in , e.g. viewing it, buying it, marking it as a favorite, etc. A user session can be modelled as a sequence = {(, )}=1, with (, ) ∈ × of item-event pairs. We use (, ) or () instinctively to denote the event acting on item . 1While a product represents a manufactured object or commodity, an item depicts something being sold at a given price and conditions.
In our work, we aim at learning a representation that is efficient (can be computed incrementally) and generic (exhibits good off-the-shelf performance on a variety of tasks), and which encodes information about the behavior of users exploring based on their preferences.
We focus on two types of events that are particularly relevant in our setting: viewing an item and/or buying it. In what follows, we present a model and learning formulation that account for these goals.
Let us assume the availability of a large set of user session data = {}=1. As mentioned above, we restrict the event set to = {, }. Let us further assume an item ∈ can be encoded based on any given attribute by a mapping : → R, ∈ {1, . . . , }, i.e. embeddings for the publication title, price and/or product description. These representations are item-centered and capture generic knowledge about the type of data (modality) being represented. They do not account for anything related to user behavior during navigation. We propose to train a model on top of these representations in order to capture the dynamics induced by the user during a navigation/shopping session.
We choose a recurrent network (GRU) [
To train our model we define a series of tasks tailored to the type of events we mentioned above. Concretely, given a large collection of user session data, we train a model to predict if an item is being viewed and/or purchased. At a given time step, the query item can be the current, the next, or the last one along the sequence. If the last element in the sequence is being purchased, we say the whole session corresponds to a purchase event. Identifying these types of sessions early is important from a business perspective. Sessions and events types are shown schematically in Fig. 1.
Regarding view events, instead of trying to predict the id of the following item in a navigation session, we seek to predict each of its attributes. The rationale behind this approach over the more usual of casting the problem as a classification task is as follows. First, many items in e-commerce correspond to the same product being sold by different users, each of which has a different identifier. Casting the problem as a classification over the set of item ids not only makes the number of free parameters grow but also makes training such a model difficult due to the long-tailed distribution of item interactions, i.e. most items in the catalog would not have enough samples to properly train the classification layer. Our formulation, on the other hand, is independent of the catalog size and it can be easily adapted to accommodate new attributes and tasks.
The set of pre-training tasks and associated losses our model is trained on will be discussed in the sections below. The tasks correspond to predicting whether the current item is purchased or not, whether the session ends in a purchase or not, the attributes of the next item viewed by the user, and, if applicable, the attributes of the item being purchased.
In what follows, we use ℎ = (()) to denote the hidden state of the recurrent network at the -th step of the sequence.
Given an item we would like to predict if the associated event is a purchase or not. This task is posed as a binary classification problem with the loss:
|S| ℒ = ∑︁ ∑︁ ℓ(1(ℎ), )
S∈ =1 where 1 : R → is a classifier and ℓ(· , · ) the cross entropy loss.
This is similar to the above but instead of predicting if the current item is being purchased, we try to anticipate if the session will end on a purchase based on the navigation history up to the current step.
|S| ℒ = ∑︁ ∑︁ ℓ(2(ℎ), |S|).
S∈ =1 Here, || denotes the event associated with the last item in S and 2 a classifier as before. (2)
Given an item , we aim at predicting the attributes (their embeddings) of the next item in the sequence. The loss term associated with attribute can be written as:
||− 1 ℒ = ∑︁ ∑︁ ℓ( (ℎ), (+1)).
S∈ =1 where (+1) is the -th attribute of the ( + 1)-th item.
The overall loss for this task, ℒ , is the combination of the following partial terms: (1) (3) • Item title, ℒ: regression to the average word embedding of the next item publication title.
We use the cosine loss, i.e. one minus the cosine similarity between the attribute embedding
and its prediction, as the regression loss for this task.
• Product ID, ℒ: same as before, but regressing towards an embedding of the product ID. Note
that different items can be mapped to the same ID, e.g. same product offered by different sellers.
By regressing towards (an embedding of) the product ID, we make the model independent of
the number of products in the catalog [
We also explore a variation of this term based on the classification of the "price bucket" in which the product falls. In this case, the size and number of buckets can be set from statistics computed over the training set or guided by the application, e.g. pre-defined price ranges.
This task is the same as the above but instead of predicting the attributes of the next item in the sequence, we try to predict those of the one being purchased. In this case, we consider only sequences that end in a purchase. Eq. (3) can be modified as follows:
S∈
||− 1 ℒ′ = ∑︁ 1||= ∑︁ ℓ(′ (ℎ), (|S|)).
=1 (4) where 1 is an indicator variable that takes the value one if y true and zero otherwise. As before, the overall loss for this task, ℒ , will be the sum of the ℒ s corresponding to the attributes ′ of the purchased item.
There are many ways in which we can aggregate Eqs. (1)–(4) into a single loss to be optimized.
Aggregation methods vary both in their technical details as well in the intuitions they rely on. For
linear aggregation, losses can be weighted by considering each task uncertainty, learning speed,
or per-task performance [
In light of these results, we opt for the following strategy. We combine losses linearly. We first
balance each term by multiplying it by a constant value so that all loss terms have roughly the
same magnitude. During training, we follow [
Training a model on the tasks described above allows us to capture important yet generic user behavior patterns from data. We can think of these representations as user-generic as they originate from learning from a large collection of real user interactions within an e-commerce context. Some applications, however, would benefit from building a more personalized characterization of the user, based on their particular interests.Examples are search personalization, where we have to reorder search results in relation to the recent navigation behavior, or push notification selection, where we have to select a set of candidates, dates, and times to send a push notification to a user.
To build such representations, first note that most prediction heads in our model correspond to regression heads, i.e. their output account for the model’s best guess for the attribute embeddings of the next/purchased item in a navigation sequence. We leverage such predictions and build a representation that contrasts the predicted embeddings with those of the attributes of each item in a (short) list of candidates. Formally, let , = (ℎ) denote the embedding predicted by the model for attribute at the -th step. Let denote the -th product in a list of candidates (e.g. items returned by an initial search query) and consider the following set of descriptive statistics: =m1,a..x., (,, ,)
1 ∑︁ (,, ,) =1 (,, ,) (5) (6) (7) where , = () is the representation for attribute of product and (, ) denotes a similarity measure between and , e.g. cosine similarity for vector embeddings or 1 minus a distance for scalar attributes (e.g. price).
Eq. (5)–(7) compute simple descriptive statistics that seek to capture generic patterns from the items the user has recently interacted with, in relation to a candidate item. Eq. (5) computes the maximum similarity of the current model prediction with all the items in the navigation history. Eq. (6) aims at capturing ’s average relevance to the current session. Finally, (7) computes the similarity between and the prediction for the last item visited by the user.
In this context, the computation of (,, ,) can be thought of as a simple attention mechanism with item-to-item similarity at its core, and can retrieve properties about the relationship between a given candidate and all past interactions. This formulation can be extended with more problem-dependant statistics as long as the final vector has a fixed size.
All these coefficients can be computed efficiently with a minimum of caching, i.e. by maintaining a running max and running average of the similarities. Using the regression heads presented in Sec. 4 we compute a total of 18 coefficients: 9 correspond to heads predicting the next event, and 9 to session purchases. In both cases, we extract 3 features for each of the 3 attributes of the item. This is a compact representation that can be computed for any new item given a user session.
We term this representation session conditioned item descriptor (SCID). Figure 2 illustrates the processes to compute the SCID coefficients for a given attribute and a session of length . In the experiments, we also consider replacing the GRU with a simple moving average model, while pre-training using the same heads and tasks configurations. The computation of this SCID variant remains the same.
In this section, we present our experimental setup, show results on different datasets, and discuss our findings.
We run experiments on two different datasets: Coveo Data Challenge (CDC) [
CDC is a publicly available session-level dataset reflecting 10M interactions over 57K different products in an e-commerce website. It accounts for more than 4M user sessions. Interactions in this dataset include add to cart, remove from cart, view, purchase, and search events. The associated challenge considered two different sets of tasks: session-based recommendation and cart abandonment. We focus on the former, i.e. given the first elements of a session, the task is to predict future interactions. We consider the following problems: next item prediction and search personalization. For the first, we set the target as the last item in each sample sequence. For the second, we use the available annotations and ask the model to predict the item clicked by the user from the list shown after a search query. In both cases, we use mean reciprocal rank # of unique items # of sessions # of interactions # of purchases 25/50/75 pct. of session length
(MRR) as the performance metric. Preliminary results showed that other ranking metrics such as NDCG, precision, and recall at k were highly correlated with MRR, thus we decided to report only MRR for simplicity.
MeLi-Sessions is an in-house dataset that we use to evaluate different scenarios relevant to our application domain. It consists of 1M user sessions over more than 4M items. Interactions include view and purchase events. Samples in the dataset were collected between April 2021 and July 2021. Sessions in this dataset are longer than CDC, with a median of 12 interactions (vs. 3 for CDC). For evaluation, we consider the following tasks: search personalization, session-based recommendation, and product ads. The recommendation task is similar to search personalization with the difference that the trigger is an item instead of a search query. The list of candidates, in this case, is obtained from co-occurrence statistics w.r.t the item we use as trigger. As before, we use the MRR as our metric. The product ads task corresponds to estimating the conversion rate of a sponsored item. In this case, we use the area under the ROC curve as our metric.
Table 1 shows summary statistics computed on the training set of both CDC and MeLi-Sessions. From the table, we see that although the number of session samples in MeLi-Sessions is four times smaller than CDC, sessions are longer and the number of unique items is considerably larger. Also, an important aspect of MeLi-Sessions is that it accounts for e-commerce tasks in Spanish.
We split the datasets chronologically into two disjoint subsets. For CDC, we use sessions occurring before May 20th for pre-training (∼ 80% of the total) and those after this date for evaluation, unless specifically stated otherwise. We follow similar criteria for the MeLi-Sessions dataset. We use data from April to July for pre-training and the subsequent month for evaluation. In both cases, there is no overlap between train and test subsets, and the test sets are posterior to the training sets in each case.
We pre-train a GRU network (single layer, 256-dimensional hidden state) using the losses
described in Sec. 4. Prediction heads consist of two-layer MLPs with ReLU non-linearities for
CDC and single-layer MLPs for MeLi-Sessions. (Pre-)training is run using a learning rate of
0.001 and the Adam optimizer. We use an early stopping heuristic with a patience parameter of 4.
As mentioned before, we follow [
In the following, we present and discuss the results for the different downstream tasks for both datasets.
In this section, we show experimental results and discussions on the CDC dataset.
7.1.1. Item and Attribute Encoders
We build a representation using the embeddings provided with the dataset. We concatenate the
description and category embeddings, adding the normalized price range as an additional scalar
feature. Description embeddings correspond to the ones provided with the dataset for textual
meta-data. We compute category embeddings as the average description embedding for the items
in each category.
7.1.2. Model Pre-Training
We adapt the attribute prediction tasks in Sec. 4.3-4.4 to those available in the dataset. Price
estimation is approached as a 10-way classification problem using the provided annotations.
Instead of title embeddings, we use embeddings of the item description provided by the authors
of the dataset. Finally, instead of category IDs, we regress towards an embedding of the product
category computed as the average of the description embeddings for the items of each category.
We pre-train our model as described above and use the trained model to extract the features that
feed a downstream predictor. We do not perform any task-specific fine-tuning, i.e. we freeze the
model weights and use it as a simple feature extractor.
7.1.3. Next Item Prediction
For this task, we train a gradient boosting model as implemented in LightGBM [
When reporting results using SCID features (first block in the table), we show two different scores. The first corresponds to the case of SCID features computed using GRU embeddings, and the second to SCID features computed based on moving average embeddings.
Results are shown in Table 2. From the table, we see that combining either MA or GRU with the BL features gives a slight but consistent improvement. If we consider the combination of BL with either of the SCID features (based on MA or GRU models), we observe an additional improvement mostly coming from sessions in which the target item was already seen. The improvement brought by adding SCID is higher for the GRU-based model. If we compare the effect of adding the GRU-based SCID to the BL baseline, the relative improvement for the "Seen" case is 9.6% while for the "Not Seen" case is only 1.2%. This can be attributed to the fact that when the target item was seen during navigation, its similarity with the items in the navigation session is a strong indicator of user interest, i.e. the max similarity statistic is a maximum (for a cosine similarity) and the average similarity increases with the number of times the user visited the target item during navigation. From the table, we also observe little improvement for the combination of more than 2 descriptors.
As a reference, we show results obtained by the top-performing teams during the competition.
Teams DeepBlueAI and NVIDIA Merlin [
All the list of candidates for each query. We include this default CDC baseline for reference. We also include a custom baseline model built from the concatenation of the description and query embeddings along with the price of the query item. As before, we show disaggregate results for sessions in which the item clicked by the user was previously seen or not.
From the table, we see that GRU performs on par with MA, showing an improvement for sessions where the target item has already been seen. All feature combinations perform better than the baseline model alone. The performance of MA and GRU models gets surpassed by a large margin by either of the corresponding SCID. In this case, the model that uses a GRU as the base sequential model shows the best performance overall. It brings relative improvements of 63.9% and 8% for the "Seen" and "Not Seen" cases, respectively, over the GRU-only counterpart. In this case, in contrast to the next item prediction task, we see an increase in performance for sessions that do not include the target item. When combined, either MA+SCID or GRU+SCID seems to saturate over their SCID variants alone, showing the expressiveness of the SCID on this task. Also, besides the large gain brought by the SCID, we observe that CDC default is a competitive baseline compared to both our sequential variants (MA and GRU).
In this section, we show experimental results and discussions on the MeLi-Sessions dataset.
7.2.1. Item and Attribute Encoders
We use a concatenation of FastText [
✓ ✓ ✓ were trained internally on historical data collected previous to the construction of the dataset. Once we have a pre-trained model on session data, we use it as a monolithic feature extractor as before, i.e. without performing any task-specific fine-tuning. 7.2.2. Search Personalization and Session-based Recommendation We approach both of these problems by training a binary classifier to predict the item clicked by the user from the list of candidates shown after a search query (Search) or retrieved using co-occurrence statistics with the last item the user interacted with (Recommendation). We use the classification score as a relevance measure to rank the items in this list. This model is trained on top of different feature combinations, and results are shown in the "Search" and "Recommendation" groups of Table 4. We consider a set of problem-specific features (BL) used internally for each task (concatenation of the embeddings for the search query / last item, and the item being ranked in the candidate list) and MA, GRU, and SCID features as before. We also consider different combinations by concatenation and show disaggregate results for the search and recommendation tasks.
When comparing the performances of individual models, we see the problem-specific strategy without personalization performs on par (search) or better (recommendation) than the sequential models alone. Although there seems to be no difference between MA and GRU when considered alone, the SCID descriptors computed from them observe a clear improvement, with the GRUbased model performing better than the MA-based one. For the SCID features, if we consider the overall metrics ("All"), we observe a marked improvement for both descriptor variants in search and recommendation. The performance gain observed on these tasks is due mainly to a large gain in sessions for which the target has already been seen. For sessions in which the target was not previously seen, there is a noticeable decrease in performance, possibly due to the intrinsic difcfiulty of the task. The good complementarity between SCID and the other representations allows us to recover and boost this loss by a margin. We observe that, when combining the BL features with either MA or GRU, there is no gain in the overall performance. It is only after combining these models with the SCID that we observe a large increase on all sides, especially for the GRU-based SCID. These improvements are more noticeable for sessions in which the target item has already been seen. For instance, in the search personalization task, while there is a 21% relative improvement after adding the SCID features to the BL+MA combination, the improvement is 30% for sessions in which the target has been seen and 1.6% for those in which has not. For session-based recommendation, the overall/seen/not seen improvements are 14.6%/16.1%/4.5%, respectively. For the combination of BL+GRU, the improvements after adding the SCID features are even more noticeable: 28.8%/45.2%/4.4% for search and 21.2%/23.8%/4.1% for recommendations. 7.2.3. Product Ads This task is similar to session-based recommendation, but where the candidates originate from a list of advertised products. The goal here is to predict if an item within this list was finally bought or not. Table 5 shows the results using the ROC-AUC metric. As before, we consider a baseline model using problem-specific features describing the placement and past performance of the ad, and two models that combine session embeddings with the SCID features from Sec. 5. We observe similar trends as before but with lower relative improvements overall. In this case, we do not show disaggregated results since we observed very similar behavior in both cases. Interestingly, although there is a switch in the best-performing SCID model when considered alone, the GRU-based SCID seems to exhibit better complementarity compared to the one based on a moving average, showing a 2.7% relative improvement compared to BL alone or combined with either of the sequential models.
Figure 3 show qualitative examples of the effect of our model on search personalization for the query "bts kpop" (left) and "camperas de abrigo dama" (lady coat jackets, right) in the MeLiSessions dataset. The top row shows the last 5 items the user interacted with, with the most recently viewed on the right. The middle and bottom rows show the top-5 results as ranked (ordered in decreasing order from left to right) by the baseline model and our BL+GRU+SCID, respectively. The item clicked by the user is highlighted in green. In both cases, we can observe that the results provided by our model are more consistent with the user navigation ✓ ✓ ✓ ✓
SCID ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
All history, regarding both product type and price range. In the first example, results shown by the baseline, although relevant to the search query, ignore the user intent as reflected in hers/his recent navigation history. In the second, the price range of the top-ranked results by the baseline is one order of magnitude higher than what the user actually searched for.
In this paper, we proposed a multi-task feature learning and extraction model in the context of e-commerce. Our approach is based on the pre-training of a sequential model using a set of auxiliary tasks relevant to our application domain, for which supervision is easily obtained from user navigation logs.
Instead of training a set of prediction heads that are subsequently discarded, we leverage them and for each new item, we compute a discriminative descriptor that shows consistent performance improvements w.r.t other alternatives on a variety of tasks. The presented approach allows its application on a broad set of e-commerce problems using standard modeling techniques, thus lowering the difficulty of deploying session-based solutions.
Our analysis considers two different scenarios that account for two different user navigation/shopping behaviors, i.e. sessions for which the target item was already seen by the user and sessions for which it has not. We observe that the second is the more challenging case, as it calls for a greater degree of diversity of the model responses, as there is no proxy (item revisits) that helps the model to succeed on the final predictions. However, the former option is relatively easier, which raises the question of whether it may introduce a bias in the training procedure towards simpler solutions.
The findings and insights gained from this work can serve as a foundation for further
advancements in the following directions: a) parametrizing SCID statistics, results have demonstrated
the importance of SCID in session-based recommendation systems. Moving forward, future
research can focus on parametrizing SCID, enabling its learning instead of relying on heuristics;
b) comparing with efficiency-optimized transformer architectures , in this study, we examined
our GRU model’s performance in session-based recommendation. Future research can explore
different optimized transformer architectures like Block-Recurrent transformer [
We thank the authors of the CDC dataset for providing us with the data and guidelines for the experiments of Sec. 7.1. preprint arXiv:2203.07852 (2022). [30] S. Wang, B. Z. Li, M. Khabsa, H. Fang, H. Ma, Linformer: Self-attention with linear complexity, arXiv preprint arXiv:2006.04768 (2020). [31] Y. Tay, D. Bahri, L. Yang, D. Metzler, D.-C. Juan, Sparse sinkhorn attention, in: International Conference on Machine Learning, PMLR, 2020, pp. 9438–9447.