In large-scale ranking systems, cascading architectures have been widely adopted to achieve a balance between efficiency and effectiveness. The pre-ranking module selects candidates for the subsequent ranking module, while maintaining efficiency and accuracy under online latency constraints. In this paper, we propose a novel neural network architecture called RankTower, which is designed to efficiently capture user-item interactions while following the user-item decoupling paradigm to ensure online inference efficiency. The proposed approach employs a hybrid training objective that learns from samples obtained from the full stage of the cascade ranking system, optimizing different objectives for varying sample spaces. This strategy enhances the pre-ranking model's ranking capability and alignment with the existing cascade ranking system. Experimental results conducted on public datasets demonstrate that RankTower significantly outperforms state-of-the-art pre-ranking models.
In industrial information services, such as recommender systems, search engines, and advertisement systems, the cascading architecture ranking system has been widely used to achieve a balance between efficiency and effectiveness. A typical cascade ranking system, as illustrated in Figure 1, consists of multiple sequential stages, including recall, preranking, ranking, and re-ranking stages. Pre-ranking is commonly regarded as a lightweight ranking module characterized by a simpler network architecture and a reduced set of features. Compared to ranking models, pre-ranking models are required to score a larger number of candidate items for each user and demonstrate higher inference efficiency. Given the emphasis on efficiency, pre-ranking typically employs a straightforward vector-product-based model.
We propose a novel pre-ranking framework called Rank-Tower to address these challenges. The primary contributions are as follows:
β’ We introduce the RankTower architecture, which comprises three key components: Multi-Head Gated Network, Gated Cross-Attention Network, and Maximum Similarity Layer. This architecture efficiently captures user-item interactions while following the user-item decoupling paradigm to ensure online inference efficiency.
β’ We employ a full-stage sampling strategy by drawing the training samples from different stages of The Architecture of RankTower the cascade ranking system. Tightly coupled with this sampling approach, we strategically integrate a hybrid loss function that combines distillation and learning-to-rank losses. This synergistic approach facilitates comprehensive learning of the ordering dynamics underlying user interactions while aligning with the inherent patterns of the cascade ranking system.
RankTower significantly outperforms state-of-theart pre-ranking models in terms of prediction accuracy and inference efficiency.
The RankTower architecture, as shown in Figure 2, introduces three main modules: Multi-Head Gated Network for computing diversified user and item representations, Gated Cross-Attention Network for modeling bi-directional useritem interactions, and Maximum Similarity Layer for efficiently capturing user-item interactions to compute the final prediction.
RankTower follows the user-item decoupling paradigm, enabling efficient online serving by pre-computing and caching user and item embeddings. During online serving, only the gated cross-attention layers require forward propagation, while other operations remain parameter-free, optimizing computational efficiency.
The dataset for building the pre-ranking model consists of instances (π₯ π’ , π₯ π , π¦, π), where π₯ π’ and π₯ π are the user feature and item feature respectively, π π and π πΌ are the user and item input embeddings obtained by concatenating respective feature embedding vectors, π¦ β {0, 1} indicates the user-item binary feedback label, π is the logged ranking model prediction that for knowledge distillation. π§ and Ε· denote the pre-ranking model's logit and prediction.
The Multi-Head Gated Network is an enhanced MLP augmented with a gating mechanism for extracting diverse user and item representations. The MLP output is multiplied by an instance-aware gating vector, modeled by a two-layer MLP. The input embedding does not receive gradients from the gating network during training for stability. For example, given an user input embedding π π , the β-th sub-space π β π’ of the user multi-embedding is:
where β denotes the Hadamard product, π denotes the activation function of the gating network: Sigmoid(π₯), ππΏπ π’ denotes the MLP layer for modeling the user input embedding, πππΏπ π’ denotes the gating MLP for facilitating selective attention, π΅ is the batch size and π is the embedding size of each sub-space.
Similarly, for item input embedding π πΌ , the β-th sub-space π β π of the item multi-embedding is:
In the offline processing stage, we will periodically batch inference and store all the user/item's embeddings π β π’ and π β π into the vector database for online serving usage.
The Gated Cross-Attention Network employs the crossattention mechanism to effectively model the interaction between user embedding and item embedding. It utilizes the Gated Attention Unit as the main building block, along with residual connections and layer normalization for training stability.
The Bi-Directional Gated Cross-Attention Network interchangeably utilizes user and item embedding as queries and keys-values for bi-directional attention. Specifically, with the user multi-embedding πΈ π’ = Concat(e
The cross-attention mechanism with two parallel branches is designed to simultaneously attend to both user preferences and item characteristics. This bidirectional approach allows the model to capture user-item interactions more accurately. The overall structure of the Gated Cross-Attention Network is illustrated in Figure 3.
The Gated Attention Unit introduces a gating mechanism to facilitate selective attention for better learning the dependency between user embedding and item embedding. Specifically, the Gated Attention Unit effectively enables an attentive gating mechanism as follows:
where π π , π πΎ , π π are the query, key, and value input, π is the non-linear activation function for projection layer, π is the sigmoid function for computing gating value. With the learned projection π, πΎ, π, and the gating value π, we compute the attention weights, followed by gating and a post-attention projection.
where π΄ β β π» π’ Γπ» π contains user to item attention weights. This example assumes that we use user embedding as the query, and item embedding as key and value.
The Maximum Similarity Layer computes the final probability prediction based on the user and item attended embeddings. Specifically, each user sub-space computes the
where π and π are the sub-space indexes of user-attended embedding and item-attended embedding, respectively, and π is the learnable temperature scalar for re-scaling the cosine similarity. Note that the Maximum Similarity Layer does not have any parameters which is suitable for online serving.
The pre-ranking models trained exclusively on impression samples, same as ranking models, suffer from sample selection bias. The pre-ranking model, which operates on the outputs of recall models, aims to identify the most relevant candidates set for the ranking model. Consequently, aligning the item distribution between the training and serving phases is essential to mitigate this sample selection bias and improve model effectiveness.
As illustrated in Figure 4, we implemented full-stage sampling to draw training data from impression samples, candidate samples, and random samples to mitigate sample selection bias. Moreover, we strategically applied various distillation and learning-to-rank losses to different sample scopes to effectively learn the ordering of user behaviors and the sequencing of the sample stages.
The RankTower model is trained using user-level listwise samples containing multiple positive items and multiple objectives. The training samples for each user are sourced from various stages of the cascade ranking system, as shown in Figure 1. Detailed definitions and relationships among these components are provided below:
The items output by the ranking model and viewed by the user consist of both positive and negative samples. Positive samples are items that have received various types of positive user feedback, while negative samples are items that have been exposed to the user without receiving user feedback.
The item candidates in the ranking or pre-ranking stages that are not viewed by the user are categorized based on their progression through the cascade ranking pipeline. Ranking candidates, which have advanced to the ranking stage, are generally considered as hard negative samples due to their higher relevance and quality compared to the pre-ranking candidates. Pre-ranking candidates are regarded as relatively easy negative samples because they were filtered out before reaching the ranking stage.
Items that are randomly sampled from the item corpus to serve as negative samples. These random samples are considered the easiest negative samples but are included to further enhance the generalization capability of the preranking model. The incorporation of random samples ensures that the model remains effective and adaptable when encountering previously unseen items during the serving phase, thereby improving its robustness and ability to handle diverse item distributions.
Our framework incorporates two types of labels: hard labels and soft labels. Hard labels represent various types of positive user feedback on impression samples, while soft labels are predictions made by the ranking models, used knowledge distillation. Both categories of labels require an aggregation function to consolidate the different user behaviors into a single scalar value for the pre-ranking model's learning.
The aggregation of hard labels is highly dependent on the specific business problem, requiring that labels be aggregated according to their orders of importance.
For instance, in online advertising, eCPM can be utilized based on the pricing model of the platform. In an e-commerce context, one might establish a relative preference order based on the depth of user feedback, such as Purchase > Add to Cart > Click. For scenarios like feed ranking or video recommendations, user feedback signals can be aggregated using a weighted sum approach. Additionally, we incorporate a general impression label applicable across business scenarios, for learning the pattern of the cascade ranking system. The label assigned a value of 1 for impression samples and 0 otherwise.
The user feedback labels help the pre-ranking model in learning the revenue or engagement level associated with different user behaviors. The exposure label facilitates the pre-ranking model's ability to learn and replicate the ranking patterns in the downstream cascade ranking system.
For soft labels, we use the ranking objective function as aggregation function. This approach ensures that the soft labels are seamlessly integrated into the training process, maintaining the consistency between the pre-ranking model and the ranking model.
The pre-ranking model focuses on achieving both consistency and ranking accuracy through the following techniques:
β’ Knowledge Distillation: The ranking model's predictions are used as soft labels to transfer knowledge from the ranking model (teacher) to the pre-ranking model (student).
β’ Ranking Capability: Fine-grained and coarsegrained ranking losses are applied to improve ranking performance and retrieval capability, respectively.
β’ Diverse Training Samples: The model is trained on samples from different stages and varying easy/hard sample levels to achieve robust generalization and optimize hierarchical objectives.
Our synergistic framework is designed to learn both the hierarchy of user behaviors and the pattern of the cascade ranking system. For instance, in the context of online advertising, the model is expected to understand the following order of importance: converted items > clicked items > exposed items > candidate items and randomly sampled items.
As the main goal for the pre-ranking model is to output a high-quality item set for the ranking model, hence we used a listwise loss for distilling the knowledge from the ranking model as follows:
where π is the prediction of the ranking model (soft label), π§ is the logit of the pre-ranking model, π β is the impression samples set. Note the distillation process from the ranking model to the pre-ranking model is conducted exclusively on impression samples. As the ranking model is trained solely on these impression samples, its ability to generalize to candidate samples and random samples is inherently limited.
The fine-grained ranking loss is applied to both impression and candidate samples, which directly correspond to the sample scope used in serving. We employ the SoftSort, a differentiable sorting loss, to learn user behavior and the patterns of the cascade ranking system. This loss function aims to precisely rank items according to the varying degrees of positive feedback they receive and effectively differentiate positives from impression samples and negatives from candidate samples.
Consider the SoftSort operator defined by metric function d = | β | π and temperature parameter π for sorting πdimensional real vectors π β β π :
The output of SoftSort operator is a permutation matrix of dimension π. The softmax operator is applied row-wise, thereby relaxing the permutation matrices into a set of unimodal row-stochastic matrices. In simple words: the π-th row of the SoftSort operator is the softmax of the negative distances to the π-th largest element [1]. We then employ the softmax cross entropy between the permutation matrices of label π¦ and the permutation matrices of logit π§. The SoftSort loss function is hereby defined as: β Sorting (π§, π¦) = βtr(J π (SoftSort π π (π¦) β log SoftSort π π (π§))) (11) where J π is a π Γ π matrix of ones, y = (π¦ π ) πβπ β βͺπ π is the hard label and z = (π§ π ) πβπ β βͺπ π is the logit of the pre-ranking model. We use the tr to compute the element sum of the matrix SoftSort π π (π¦) β log (SoftSort π π (π§)).
The coarse-grained ranking loss, applied to all samples (impression, candidate, and random), aims to separate positive and negative samples while supporting ranking among positives by distinguishing varying degrees of relevance. We propose the Adaptive Margin Rankmax (AM-Rankmax) loss, an extension of the Rankmax loss [2] that introduces an adaptive margin based on the pair's nature and label distance, thereby extending the Rankmax loss to address ranking problems with ordered or continuous positive labels. Consider the Rankmax loss for ranking problems with binary labels only:
The Rankmax loss is reminiscent of pairwise losses. To extend the Rankmax loss to more general ranking problems involving multi-level positive labels, we introduce the adaptive margin with following enhancements:
β’ The loss is applied only when π¦ π < π¦ π , which is more suitable for multi-level positive label scenario.
β’ The margin adjusts based on whether π¦ π is positive or negative, to further enhancing the differentiality between positive and negative samples.
β’ The margin scales with the label distances between samples, reflecting varying degree of positive samples.
The adaptive margin function is:
where πΌ is a constant for adding additional margin between negative and positive items, I is the indicator function. The metric function πΏ can take various forms, for example πΏ(π¦ π , π¦ π ) = 1 or πΏ(π¦ π , π¦ π ) = π½|π¦ π β π¦ π | π . The adaptive margin Rankmax loss is then given by:
where y = (π¦ π ) πβπ β βͺπ π βͺπ β is the hard label from all the samples and z = (π§ π ) πβπ β βͺπ π βͺπ β is the logit of the preranking model.
The AM-Rankmax loss function can effectively adapt to scenarios with multiple positive labels of varying levels. This enhancement allows the model to handle different degrees of positive feedback, thereby improving its ability to generalize and accurately rank items in complex settings.
We design a hybrid ranking loss that integrates both distillation and ranking objectives. The hybrid ranking loss is the weighted sum of three losses:
where π 1 , π 2 and π 3 are weights for each sub-objective. Balancing distillation and ranking losses is crucial for the pre-ranking model to inherit the ranking model's capabilities while generalizing to broader sample spaces. Weighting fine-grained and coarse-grained ranking losses ensures a balance between precise ranking and overall retrieval robustness.
We conduct experiments on three large-scale public datasets from online advertising, e-commerce, and short video recommendation domains to evaluate the effectiveness of Rank-Tower. The experiments provide a comprehensive description of the evaluation metrics, and comparisons with stateof-the-art pre-ranking models. We aim to answer the following questions through our experiments:
β’ Q1: How does our proposed RankTower perform for pre-ranking task? Is it effective and efficient under extremely high-dimensional and sparse data settings?
β’ Q2: How do different settings on dataset sampling and training losses influence the performance of RankTower?
We evaluate our model using real-world datasets: Alimama 1 , Taobao2 , and KuaiRand3 . For each dataset, we keep users with at least 100 impressions and 20 instances of positive feedback. The data is split into 70% for training, 10% for validation, and 20% for testing. As all labels in the datasets are binary, we aggregate them by summing the labels to form the hard label.
We consider Recall@K and NDCG@K for evaluating the performance of the models, and we set π to 100 for all experiment metrics.
Recall@K is the fraction of relevant retrieved within the top πΎ recommendations. It's mainly used for measuring ranking system's capability on retrieving relevant items.
NDCG@K measures the quality of the ranking by considering both the relevance and the position of items within the top πΎ recommendations. Items with higher relevance ranked at higher position contribute more to the metric.
We compare RankTower with the following pre-ranking models: LR [3], Two-Tower [4], DAT [5], COLD [6], Int-Tower [7] and ARF [8]. Recall@K NDCG@K Recall@K NDCG@K Recall@K NDCG@K LR 0. The overall performance of different model architectures is listed in Table 1. We have the following observations for model effectiveness:
β’ LR exhibits the lowest performance compared to the other neural network-based models.
β’ Two-Tower brings the most significant relative improvement in performance, highlighting the importance of learning deep feature interactions.
β’ COLD achieves strong performance among the competing models, indicating the significance of learning useritem feature interactions.
β’ ARF outperform other models without utilizing listwise ranking losses, highlighting the importance of using listwise ranking losses.
β’ RankTower achieves the best prediction performance, attributed to its effective modeling of bi-directional useritem feature interactions and the design of full-stage sampling and hybrid loss functions.
To gain deeper insights into the proposed model, we conduct experiments on the KuaiRand dataset and compare model performance on different settings, including: 1) the effect of full-stage data sampling; 2) the effect of listwise ranking losses; and 3) the effect of distillation from the ranking model.
We conduct an ablation study to evaluate the impact of each sampling component on the model's performance. As shown in Table 2, the full-stage sampling strategy achieves the best overall performance. Training the pre-ranking model solely with impression samples hinders its ability to generalize to unexposed items, negatively affecting retrieval performance. We also observe that candidate samples are more important than random samples, as they significantly enhance the model's ability to discriminate between relevant and non-relevant items.
To better understand the properties of the proposed hybrid loss, we compare it with several widely used ranking losses in the industry. The experiment results, as shown in Table 3, indicate that the hybrid loss consistently outperforms other alternatives, surpassing both its individual components: the Sorting loss and the AM-Rankmax loss. Moreover, our proposed AM-Rankmax demonstrates superior performance compared to the original Rankmax loss and the Softmax loss.
We conduct an ablation study on the distillation component and further compare Softmax loss with other alternatives. The Table 4 demonstrate the efficacy of transferring knowledge distillation. Among various loss function experimented for distillation, the Softmax loss outperforms the other alternative losses. The Softmax loss, being a listwise ranking loss, proved more adept at distilling the ranking model's capabilities compared to the weighted logloss, which essentially is a pointwise approach and exhibited suboptimal performance in learning the relative ranking distribution. In contrast, the pairwise logloss, focusing solely on pairwise ordering of ranking model's predictions without considering the relative proximity of predictions, exhibited overfitting to the ranking model's outputs.
This paper introduces the RankTower model, designed to enhance the performance of the two-tower model by effectively capturing bi-directional latent interactions between user and item. To ensure consistency with existing casecade ranking system, a hybrid loss function and full-stage sampling approach are integrated into the model's optimization framework. Comprehensive experiments demonstrate that RankTower significantly outperforms state-of-the-art preranking models. In future work, we aim to study how to effectively and jointly optimize the cascade ranking system in an end-to-end fashion.