In large-scale ranking systems, cascading architectures have been widely adopted to achieve a balance between eficiency and efectiveness. The pre-ranking module selects candidates for the subsequent ranking module, while maintaining eficiency and accuracy under online latency constraints. In this paper, we propose a novel neural network architecture called RankTower, which is designed to eficiently capture user-item interactions while following the user-item decoupling paradigm to ensure online inference eficiency. The proposed approach employs a hybrid training objective that learns from samples obtained from the full stage of the cascade ranking system, optimizing diferent objectives for varying sample spaces. This strategy enhances the pre-ranking model's ranking capability and alignment with the existing cascade ranking system. Experimental results conducted on public datasets demonstrate that RankTower significantly outperforms state-of-the-art pre-ranking models.
Corpus Recal Pre-Ranking Ranking Re-Ranking Item 1 Item 2 Item 3
In industrial information services, such as recommender systems, search engines, and advertisement systems, the cascading architecture ranking system has been widely used to achieve a balance between eficiency and efectiveness. A typical cascade ranking system, as illustrated in Figure 1, consists of multiple sequential stages, including recall, preranking, ranking, and re-ranking stages. Pre-ranking is commonly regarded as a lightweight ranking module characterized by a simpler network architecture and a reduced set of features. Compared to ranking models, pre-ranking models are required to score a larger number of candidate items for each user and demonstrate higher inference efifciency. Given the emphasis on eficiency, pre-ranking typically employs a straightforward vector-product-based model.
We propose a novel pre-ranking framework called RankTower to address these challenges. The primary contributions are as follows: • We introduce the RankTower architecture, which comprises three key components: Multi-Head Gated imum Similarity Layer. This architecture eficiently captures user-item interactions while following the user-item decoupling paradigm to ensure online inference eficiency. • We employ a full-stage sampling strategy by drawing the training samples from diferent stages of Woodstock’22: Symposium on the irreproducible science, June 07–11, 2022, ∗Corresponding author. (L. Li) 0000-0002-1213-4343 (Y. Yan); 0009-0006-9933-2436 (L. Li) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License
CEUR
ceur-ws.org
User Multi-Embedding
Maximum Similarity Layer Gated Cross-Attention
Network Multi-Head Gated
Network
Multi-Head Gated
Network Embedding Layer
Embedding Layer User Tower
Item Tower
Online Serving
Vector Database
Item Multi-Embedding the cascade ranking system. Tightly coupled with this sampling approach, we strategically integrate a hybrid loss function that combines distillation and learning-to-rank losses. This synergistic approach facilitates comprehensive learning of the ordering dynamics underlying user interactions while aligning with the inherent patterns of the cascade ranking system. • Experiments on public datasets demonstrate that RankTower significantly outperforms state-of-theart pre-ranking models in terms of prediction accuracy and inference eficiency. 2.
The RankTower architecture, as shown in Figure 2, introduces three main modules: Multi-Head Gated Network for computing diversified user and item representations, Gated Cross-Attention Network for modeling bi-directional useritem interactions, and Maximum Similarity Layer for eficiently capturing user-item interactions to compute the final prediction.
RankTower follows the user-item decoupling paradigm, enabling eficient online serving by pre-computing and caching user and item embeddings. During online serving, only the gated cross-attention layers require forward propagation, while other operations remain parameter-free, optimizing computational eficiency.
The dataset for building the pre-ranking model consists of instances ( , , , ) , where and are the user feature and item feature respectively, and are the user and item input embeddings obtained by concatenating respective feature embedding vectors, ∈ {0, 1} indicates the user-item binary feedback label, is the logged ranking model prediction that for knowledge distillation. and ̂ denote the pre-ranking model’s logit and prediction.
ℎ of the user multi-embedding is: The Multi-Head Gated Network is an enhanced MLP augmented with a gating mechanism for extracting diverse user and item representations. The MLP output is multiplied by an instance-aware gating vector, modeled by a two-layer MLP. The input embedding does not receive gradients from the gating network during training for stability. For example, given an user input embedding , the ℎ-th sub-space (1)
(2) ℎ = ( )ℎ ∘ (
( ))ℎ ∈ ℝ× ,
ℎ = 1, ⋯ , ℎ of the item multi-embedding is:
where ∘ denotes the Hadamard product, denotes the activation function of the gating network: Sigmoid() , denotes the MLP layer for modeling the user input embedding,
denotes the gating MLP for facilitating selective attention, is the batch size and is the embedding size of each sub-space.
Similarly, for item input embedding , the ℎ-th sub-space ℎ = ( )ℎ ∘ (
( ))ℎ ∈ ℝ× ,
ℎ = 1, ⋯ ,
In the ofline processing stage, we will periodically batch ℎ into the vector database for online serving usage. inference and store all the user/item’s embeddings ℎ and
The Gated Cross-Attention Network employs the crossattention mechanism to efectively model the interaction between user embedding and item embedding. It utilizes the Gated Attention Unit as the main building block, along with residual connections and layer normalization for training stability.
The Bi-Directional Gated Cross-Attention Network interchangeably utilizes user and item embedding as queries and keys-values for bi-directional attention. Specifically, with the user multi-embedding = Concat(e1u, ..., ) and item multi-embedding = Concat(ei1, ..., ), the crossattention compute the user attended embedding ℰ and item attended embedding ℰ as follows: Add & LayerNorm
Add & LayerNorm Gated Attention
Unit Q
KV
Gated Attention
Unit Q
KV
User
The Maximum Similarity Layer computes the final probability prediction based on the user and item attended embeddings. Specifically, each user sub-space computes the ... ...
...
...
Pre-Ranking Model Impression Samples
Candidate Samples ...
...
Random Samples User
The pre-ranking models trained exclusively on impression samples, same as ranking models, sufer from sample selection bias. The pre-ranking model, which operates on the outputs of recall models, aims to identify the most relevant candidates set for the ranking model. Consequently, aligning the item distribution between the training and serving phases is essential to mitigate this sample selection bias and improve model efectiveness.
As illustrated in Figure 4, we implemented full-stage sampling to draw training data from impression samples, candidate samples, and random samples to mitigate sample selection bias. Moreover, we strategically applied various distillation and learning-to-rank losses to diferent sample scopes to efectively learn the ordering of user behaviors and the sequencing of the sample stages.
The RankTower model is trained using user-level listwise samples containing multiple positive items and multiple objectives. The training samples for each user are sourced from various stages of the cascade ranking system, as shown in Figure 1. Detailed definitions and relationships among these components are provided below:
The items output by the ranking model and viewed by the user consist of both positive and negative samples. Positive samples are items that have received various types of feedback.
The item candidates in the ranking or pre-ranking stages that are not viewed by the user are categorized based on their progression through the cascade ranking pipeline. Ranking candidates, which have advanced to the ranking stage, are generally considered as hard negative samples due to their higher relevance and quality compared to the pre-ranking candidates. Pre-ranking candidates are regarded as relatively easy negative samples because they were filtered out before reaching the ranking stage.
Items that are randomly sampled from the item corpus to serve as negative samples. These random samples are considered the easiest negative samples but are included to further enhance the generalization capability of the preranking model. The incorporation of random samples ensures that the model remains efective and adaptable when encountering previously unseen items during the serving phase, thereby improving its robustness and ability to handle diverse item distributions.
Our framework incorporates two types of labels: hard labels and soft labels. Hard labels represent various types of positive user feedback on impression samples, while soft labels are predictions made by the ranking models, used knowledge distillation. Both categories of labels require an aggregation function to consolidate the diferent user behaviors into a single scalar value for the pre-ranking model’s learning.
The aggregation of hard labels is highly dependent on the specific business problem, requiring that labels be aggregated according to their orders of importance.
For instance, in online advertising, eCPM can be utilized based on the pricing model of the platform. In an e-commerce context, one might establish a relative preference order based on the depth of user feedback, such as Purchase > Add to Cart > Click. For scenarios like feed ranking or video recommendations, user feedback signals can be aggregated using a weighted sum approach. Additionally, we incorporate a general impression label applicable across business scenarios, for learning the pattern of the cascade ranking system. The label assigned a value of 1 for impression samples and 0 otherwise.
The user feedback labels help the pre-ranking model in learning the revenue or engagement level associated with diferent user behaviors. The exposure label facilitates the pre-ranking model’s ability to learn and replicate the ranking patterns in the downstream cascade ranking system.
For soft labels, we use the ranking objective function as aggregation function. This approach ensures that the soft labels are seamlessly integrated into the training process, maintaining the consistency between the pre-ranking model and the ranking model.
The pre-ranking model focuses on achieving both consistency and ranking accuracy through the following techniques: • Knowledge Distillation: The ranking model’s predictions are used as soft labels to transfer knowledge from the ranking model (teacher) to the pre-ranking model (student). • Ranking Capability:
Fine-grained and coarsegrained ranking losses are applied to improve ranking performance and retrieval capability, respectively. • Diverse Training Samples: The model is trained on samples from diferent stages and varying easy/hard sample levels to achieve robust generalization and optimize hierarchical objectives.
Our synergistic framework is designed to learn both the hierarchy of user behaviors and the pattern of the cascade ranking system. For instance, in the context of online advertising, the model is expected to understand the following order of importance: converted items > clicked items > exposed items > candidate items and randomly sampled items.
As the main goal for the pre-ranking model is to output a high-quality item set for the ranking model, hence we used a listwise loss for distilling the knowledge from the ranking model as follows: ℒDistillation(, ) = − ∑ log ∈
exp( ) ∑∈ exp( ) (9) where is the prediction of the ranking model (soft label), is the logit of the pre-ranking model, ℐ is the impression samples set. Note the distillation process from the ranking model to the pre-ranking model is conducted exclusively on impression samples. As the ranking model is trained solely on these impression samples, its ability to generalize to candidate samples and random samples is inherently limited.
The fine-grained ranking loss is applied to both impression and candidate samples, which directly correspond to the sample scope used in serving. We employ the SoftSort , a diferentiable sorting loss, to learn user behavior and the patterns of the cascade ranking system. This loss function aims to precisely rank items according to the varying degrees of positive feedback they receive and efectively diferentiate positives from impression samples and negatives from candidate samples. dimensional real vectors ∈ ℝ :
Consider the SoftSort operator defined by metric function d = | ⋅ | and temperature parameter for sorting SoftSort () = softmax ( −d( ()
1 , 1 ) ) (10)
The output of SoftSort operator is a permutation matrix
of dimension . The softmax operator is applied row-wise,
thereby relaxing the permutation matrices into a set of
unimodal row-stochastic matrices. In simple words: the -th
row of the SoftSort operator is the
softmax of the negative
distances to the -th largest element [
ℐ∪ model. We use the tr to compute the element sum of the matrix SoftSort ( ) ∘ log (SoftSort () ). is the logit of the pre-ranking
(11) is the
The coarse-grained ranking loss, applied to all samples
(impression, candidate, and random), aims to separate positive
and negative samples while supporting ranking among
positives by distinguishing varying degrees of relevance. We
propose the Adaptive Margin Rankmax (AM-Rankmax) loss,
an extension of the Rankmax loss [
Consider the Rankmax loss for ranking problems with binary labels only: ∶ >0 =1 ℒ (, ) = ∑ log ∑ ( − + 1)+ (12)
The Rankmax loss is reminiscent of pairwise losses. To extend the Rankmax loss to more general ranking problems involving multi-level positive labels, we introduce the adaptive margin with following enhancements: • The loss is applied only when < , which is more suitable for multi-level positive label scenario. • The margin adjusts based on whether is positive or negative, to further enhancing the diferentiality between positive and negative samples. • The margin scales with the label distances between samples, reflecting varying degree of positive samples.
The adaptive margin function is: (, ) = ⋅
I( = 0) + ( , ) (13) where is a constant for adding additional margin between negative and positive items, I is the indicator function. The metric function can take various forms, for example ( , ) = 1 or ( , ) = | − | . The adaptive margin Rankmax loss is then given by: AM−Rankmax(, ) = ∑ log ∑ ( − + (, ) )+ ∶ >0 ∶ < (14) where y = ( )∈ ℐ∪ ∪ ℛ is the hard label from all the samples and z = ( )∈ ℐ∪ ∪ ℛ is the logit of the preranking model.
The AM-Rankmax loss function can efectively adapt to scenarios with multiple positive labels of varying levels. This enhancement allows the model to handle diferent degrees of positive feedback, thereby improving its ability to generalize and accurately rank items in complex settings.
We design a hybrid ranking loss that integrates both distillation and ranking objectives. The hybrid ranking loss is the weighted sum of three losses: ℒHybrid(, ) = 1ℒDistillation(, ) + 2ℒSorting(, ) + 3ℒAM−Rankmax(, ) (15) where 1, 2 and 3 are weights for each sub-objective. Balancing distillation and ranking losses is crucial for the pre-ranking model to inherit the ranking model’s capabilities while generalizing to broader sample spaces. Weighting ifne-grained and coarse-grained ranking losses ensures a balance between precise ranking and overall retrieval robustness.
We conduct experiments on three large-scale public datasets from online advertising, e-commerce, and short video recommendation domains to evaluate the efectiveness of RankTower. The experiments provide a comprehensive description of the evaluation metrics, and comparisons with stateof-the-art pre-ranking models. We aim to answer the following questions through our experiments: • Q1: How does our proposed RankTower perform for pre-ranking task? Is it efective and eficient under extremely high-dimensional and sparse data settings? • Q2: How do diferent settings on dataset sampling and training losses influence the performance of RankTower?
We evaluate our model using real-world datasets: Alimama1, Taobao2, and KuaiRand3. For each dataset, we keep users with at least 100 impressions and 20 instances of positive feedback. The data is split into 70% for training, 10% for validation, and 20% for testing. As all labels in the datasets are binary, we aggregate them by summing the labels to form the hard label.
We consider Recall@K and NDCG@K for evaluating the performance of the models, and we set to 100 for all experiment metrics.
Recall@K is the fraction of relevant retrieved within the top recommendations. It’s mainly used for measuring ranking system’s capability on retrieving relevant items.
NDCG@K measures the quality of the ranking by considering both the relevance and the position of items within the top recommendations. Items with higher relevance ranked at higher position contribute more to the metric.
We compare RankTower with the following pre-ranking
models: LR [
The overall performance of diferent model architectures is listed in Table 1. We have the following observations for model efectiveness: • LR exhibits the lowest performance compared to the other neural network-based models. • Two-Tower brings the most significant relative improvement in performance, highlighting the importance of learning deep feature interactions. • COLD achieves strong performance among the competing models, indicating the significance of learning useritem feature interactions. 1https://tianchi.aliyun.com/dataset/408 2https://tianchi.aliyun.com/dataset/649 3https://kuairand.com/ • ARF outperform other models without utilizing listwise ranking losses, highlighting the importance of using listwise ranking losses. • RankTower achieves the best prediction performance, attributed to its efective modeling of bi-directional useritem feature interactions and the design of full-stage sampling and hybrid loss functions.
To gain deeper insights into the proposed model, we conduct experiments on the KuaiRand dataset and compare model performance on diferent settings, including: 1) the efect of full-stage data sampling; 2) the efect of listwise ranking losses; and 3) the efect of distillation from the ranking model.
We conduct an ablation study to evaluate the impact of each sampling component on the model’s performance. As shown in Table 2, the full-stage sampling strategy achieves the best overall performance. Training the pre-ranking model solely with impression samples hinders its ability to generalize to unexposed items, negatively afecting retrieval performance. We also observe that candidate samples are more important than random samples, as they significantly enhance the model’s ability to discriminate between relevant and non-relevant items.
To better understand the properties of the proposed hybrid loss, we compare it with several widely used ranking losses in the industry. The experiment results, as shown in Table 3, indicate that the hybrid loss consistently outperforms other alternatives, surpassing both its individual components: the Sorting loss and the AM-Rankmax loss. Moreover, our proposed AM-Rankmax demonstrates superior performance compared to the original Rankmax loss and the Softmax loss. We conduct an ablation study on the distillation component and further compare Softmax loss with other alternatives.
The Table 4 demonstrate the eficacy of transferring knowledge distillation. Among various loss function experimented for distillation, the Softmax loss outperforms the other alternative losses. The Softmax loss, being a listwise ranking loss, proved more adept at distilling the ranking model’s capabilities compared to the weighted logloss, which essentially is a pointwise approach and exhibited suboptimal performance in learning the relative ranking distribution. In contrast, the pairwise logloss, focusing solely on pairwise ordering of ranking model’s predictions without considering the relative proximity of predictions, exhibited overfitting to the ranking model’s outputs.
This paper introduces the RankTower model, designed to enhance the performance of the two-tower model by efectively capturing bi-directional latent interactions between user and item. To ensure consistency with existing casecade ranking system, a hybrid loss function and full-stage sampling approach are integrated into the model’s optimization framework. Comprehensive experiments demonstrate that RankTower significantly outperforms state-of-the-art preranking models. In future work, we aim to study how to efectively and jointly optimize the cascade ranking system in an end-to-end fashion.