RankTower: A Synergistic Framework for Enhancing Two-Tower Pre-Ranking Model Yachen Yan1,∗ , Liubo Li1 1 Credit Karma, 760 Market Street, San Francisco, California, USA, 94012 Abstract In large-scale ranking systems, cascading architectures have been widely adopted to achieve a balance between efficiency and effec- tiveness. The pre-ranking module selects candidates for the subsequent ranking module, while maintaining efficiency and accuracy under online latency constraints. In this paper, we propose a novel neural network architecture called RankTower, which is designed to efficiently capture user-item interactions while following the user-item decoupling paradigm to ensure online inference efficiency. The proposed approach employs a hybrid training objective that learns from samples obtained from the full stage of the cascade ranking system, optimizing different objectives for varying sample spaces. This strategy enhances the pre-ranking model’s ranking capability and alignment with the existing cascade ranking system. Experimental results conducted on public datasets demonstrate that RankTower significantly outperforms state-of-the-art pre-ranking models. Keywords Recommender Systems, Pre-Ranking, Learning to Rank, Differentiable Sorting Item 1 Item Recall Pre-Ranking Ranking Re-Ranking Item 2 Corpus Online Serving Item 3 Maximum Similarity Layer Vector Database Figure 1: The Architecture of Cascade Ranking System Gated Cross-Attention Network 1. Introduction User Multi-Embedding Item Multi-Embedding In industrial information services, such as recommender Multi-Head Gated Multi-Head Gated Network Network systems, search engines, and advertisement systems, the cascading architecture ranking system has been widely used Embedding Layer Embedding Layer to achieve a balance between efficiency and effectiveness. A typical cascade ranking system, as illustrated in Figure 1, consists of multiple sequential stages, including recall, pre- User Tower Item Tower ranking, ranking, and re-ranking stages. Pre-ranking is commonly regarded as a lightweight ranking module char- Figure 2: The Architecture of RankTower acterized by a simpler network architecture and a reduced set of features. Compared to ranking models, pre-ranking models are required to score a larger number of candidate items for each user and demonstrate higher inference ef- the cascade ranking system. Tightly coupled with ficiency. Given the emphasis on efficiency, pre-ranking this sampling approach, we strategically integrate a typically employs a straightforward vector-product-based hybrid loss function that combines distillation and model. learning-to-rank losses. This synergistic approach We propose a novel pre-ranking framework called Rank- facilitates comprehensive learning of the ordering Tower to address these challenges. The primary contribu- dynamics underlying user interactions while align- tions are as follows: ing with the inherent patterns of the cascade ranking system. • We introduce the RankTower architecture, which comprises three key components: Multi-Head Gated • Experiments on public datasets demonstrate that Network, Gated Cross-Attention Network, and Max- RankTower significantly outperforms state-of-the- imum Similarity Layer. This architecture efficiently art pre-ranking models in terms of prediction accu- captures user-item interactions while following the racy and inference efficiency. user-item decoupling paradigm to ensure online in- ference efficiency. 2. Model Architecture • We employ a full-stage sampling strategy by draw- ing the training samples from different stages of The RankTower architecture, as shown in Figure 2, intro- duces three main modules: Multi-Head Gated Network for Woodstock’22: Symposium on the irreproducible science, June 07–11, 2022, Woodstock, NY computing diversified user and item representations, Gated ∗ Corresponding author. Cross-Attention Network for modeling bi-directional user- Envelope-Open yachen.yan@creditkarma.com (Y. Yan); liubo.li@creditkarma.com item interactions, and Maximum Similarity Layer for effi- (L. Li) ciently capturing user-item interactions to compute the final Orcid 0000-0002-1213-4343 (Y. Yan); 0009-0006-9933-2436 (L. Li) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License prediction. Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings RankTower follows the user-item decoupling paradigm, enabling efficient online serving by pre-computing and caching user and item embeddings. During online serv- ing, only the gated cross-attention layers require forward Add & LayerNorm Add & LayerNorm propagation, while other operations remain parameter-free, optimizing computational efficiency. Gated Attention Gated Attention Unit Unit Q KV Q KV 2.1. Preliminary The dataset for building the pre-ranking model consists of instances (𝑥𝑢 , 𝑥𝑖 , 𝑦, 𝑝), where 𝑥𝑢 and 𝑥𝑖 are the user fea- ture and item feature respectively, 𝑋𝑈 and 𝑋𝐼 are the user and item input embeddings obtained by concatenating re- User Item spective feature embedding vectors, 𝑦 ∈ {0, 1} indicates the Multi-Embedding Multi-Embedding user-item binary feedback label, 𝑝 is the logged ranking model prediction that for knowledge distillation. 𝑧 and 𝑦̂ Figure 3: The Architecture of Gated Cross-Attention Network denote the pre-ranking model’s logit and prediction. 2.2. Multi-Head Gated Network The Multi-Head Gated Network is an enhanced MLP aug- mented with a gating mechanism for extracting diverse user ℰ𝑢 = LN(𝐸𝑢 + GAU(𝑄 = 𝐸𝑢 , 𝐾 = 𝐸𝑖 , 𝑉 = 𝐸𝑖 )) ∈ ℝ𝐵×𝐻𝑢 ×𝑘 and item representations. The MLP output is multiplied by (3) an instance-aware gating vector, modeled by a two-layer MLP. The input embedding does not receive gradients from ℰ𝑖 = LN(𝐸𝑖 + GAU(𝑄 = 𝐸𝑖 , 𝐾 = 𝐸𝑢 , 𝑉 = 𝐸𝑢 )) ∈ ℝ𝐵×𝐻𝑖 ×𝑘 the gating network during training for stability. For exam- (4) ple, given an user input embedding 𝑋𝑈 , the ℎ-th sub-space The cross-attention mechanism with two parallel 𝑒𝑢ℎ of the user multi-embedding is: branches is designed to simultaneously attend to both user 𝑒𝑢ℎ = 𝑀𝐿𝑃𝑢 (𝑋𝑈 )ℎ ∘ 𝜎(𝑔𝑀𝐿𝑃𝑢 (𝑋𝑈 ))ℎ preferences and item characteristics. This bidirectional ap- (1) proach allows the model to capture user-item interactions ∈ ℝ𝐵×𝑘 , ℎ = 1, ⋯ , 𝐻𝑢 more accurately. The overall structure of the Gated Cross- where ∘ denotes the Hadamard product, 𝜎 denotes the ac- Attention Network is illustrated in Figure 3. tivation function of the gating network: Sigmoid(𝑥), 𝑀𝐿𝑃𝑢 denotes the MLP layer for modeling the user input embed- 2.3.2. Gated Attention Unit ding, 𝑔𝑀𝐿𝑃𝑢 denotes the gating MLP for facilitating selec- The Gated Attention Unit introduces a gating mechanism tive attention, 𝐵 is the batch size and 𝑘 is the embedding to facilitate selective attention for better learning the de- size of each sub-space. pendency between user embedding and item embedding. Similarly, for item input embedding 𝑋𝐼 , the ℎ-th sub-space Specifically, the Gated Attention Unit effectively enables an 𝑒𝑖ℎ of the item multi-embedding is: attentive gating mechanism as follows: 𝑒𝑖ℎ = 𝑀𝐿𝑃𝑖 (𝑋𝐼 )ℎ ∘ 𝜎(𝑔𝑀𝐿𝑃𝑖 (𝑋𝐼 ))ℎ 𝑄 = 𝜙(𝑋𝑄 𝑊𝑄 ), 𝐾 = 𝜙(𝑋𝐾 𝑊𝐾 ) (2) ∈ ℝ𝐵×𝑘 , ℎ = 1, ⋯ , 𝐻𝑖 (5) 𝑉 = 𝜙(𝑋𝑉 𝑊𝑉 ), 𝑈 = 𝜎(𝑋𝑄 𝑊𝑈 ) In the offline processing stage, we will periodically batch where 𝑋𝑄 , 𝑋𝐾 , 𝑋𝑉 are the query, key, and value input, 𝜙 inference and store all the user/item’s embeddings 𝑒𝑢ℎ and is the non-linear activation function for projection layer, 𝜎 𝑒𝑖ℎ into the vector database for online serving usage. is the sigmoid function for computing gating value. With the learned projection 𝑄, 𝐾, 𝑉, and the gating value 𝑈, we 2.3. Gated Cross-Attention Network compute the attention weights, followed by gating and a The Gated Cross-Attention Network employs the cross- post-attention projection. attention mechanism to effectively model the interaction between user embedding and item embedding. It utilizes the 𝑂 = (𝑈 ⊙ 𝐴𝑉 )𝑊𝑜 (6) Gated Attention Unit as the main building block, along with residual connections and layer normalization for training 𝑄𝐾 𝑇 𝐴 = softmax( ) (7) stability. √𝑑𝑘 2.3.1. Cross Attention Mechanism where 𝐴 ∈ ℝ𝐻𝑢 ×𝐻𝑖 contains user to item attention weights. This example assumes that we use user embedding as the The Bi-Directional Gated Cross-Attention Network inter- query, and item embedding as key and value. changeably utilizes user and item embedding as queries and keys-values for bi-directional attention. Specifically, 𝐻 with the user multi-embedding 𝐸𝑢 = Concat(e1u , ..., 𝑒𝑢 𝑢 ) and 2.4. Maximum Similarity Layer 𝐻 item multi-embedding 𝐸𝑖 = Concat(e1i , ..., 𝑒𝑖 𝑖 ), the cross- The Maximum Similarity Layer computes the final proba- attention compute the user attended embedding ℰ𝑢 and bility prediction based on the user and item attended em- item attended embedding ℰ𝑖 as follows: beddings. Specifically, each user sub-space computes the Fine-Grained Coarse-Grained Distillation Loss Ranking Loss Ranking Loss ... ... ... Pre-Ranking Model User Item 1 ... Item N Item N+1 ... Item N+M Item N+M+1 ... Item N+M+L Impression Candidate Random Samples Samples Samples Figure 4: The Synergistic Framework for Learning User Behavior Ordering and Full-Stage Sample Ordering maximum cosine similarity with all item sub-spaces, and positive user feedback, while negative samples are items the scalar outputs are summed across user sub-spaces: that have been exposed to the user without receiving user feedback. 𝐻𝑢 𝑝 𝑞 𝑠 = ( ∑ Max 𝐶𝑂𝑆𝐼 𝑁 𝐸(ℰ𝑢 , ℰ𝑖 ))/𝜏 (8) 𝑝=1 𝑞∈{1,⋯,𝐻𝑖 } 3.1.2. Candidate Samples where 𝑝 and 𝑞 are the sub-space indexes of user-attended The item candidates in the ranking or pre-ranking stages embedding and item-attended embedding, respectively, and that are not viewed by the user are categorized based on their 𝜏 is the learnable temperature scalar for re-scaling the cosine progression through the cascade ranking pipeline. Ranking similarity. Note that the Maximum Similarity Layer does not candidates, which have advanced to the ranking stage, are have any parameters which is suitable for online serving. generally considered as hard negative samples due to their higher relevance and quality compared to the pre-ranking candidates. Pre-ranking candidates are regarded as rela- 3. Pre-Ranking Model Optimization tively easy negative samples because they were filtered out before reaching the ranking stage. The pre-ranking models trained exclusively on impression samples, same as ranking models, suffer from sample selec- tion bias. The pre-ranking model, which operates on the 3.1.3. Random Samples outputs of recall models, aims to identify the most relevant Items that are randomly sampled from the item corpus to candidates set for the ranking model. Consequently, align- serve as negative samples. These random samples are con- ing the item distribution between the training and serving sidered the easiest negative samples but are included to phases is essential to mitigate this sample selection bias and further enhance the generalization capability of the pre- improve model effectiveness. ranking model. The incorporation of random samples en- As illustrated in Figure 4, we implemented full-stage sam- sures that the model remains effective and adaptable when pling to draw training data from impression samples, can- encountering previously unseen items during the serving didate samples, and random samples to mitigate sample phase, thereby improving its robustness and ability to han- selection bias. Moreover, we strategically applied various dle diverse item distributions. distillation and learning-to-rank losses to different sample scopes to effectively learn the ordering of user behaviors and the sequencing of the sample stages. 3.2. Label Aggregation Our framework incorporates two types of labels: hard la- 3.1. Full-Stage Sampling bels and soft labels. Hard labels represent various types of positive user feedback on impression samples, while soft The RankTower model is trained using user-level listwise labels are predictions made by the ranking models, used samples containing multiple positive items and multiple knowledge distillation. Both categories of labels require an objectives. The training samples for each user are sourced aggregation function to consolidate the different user behav- from various stages of the cascade ranking system, as shown iors into a single scalar value for the pre-ranking model’s in Figure 1. Detailed definitions and relationships among learning. these components are provided below: 3.2.1. Hard Labels 3.1.1. Impression Samples The aggregation of hard labels is highly dependent on the The items output by the ranking model and viewed by the specific business problem, requiring that labels be aggre- user consist of both positive and negative samples. Posi- gated according to their orders of importance. tive samples are items that have received various types of For instance, in online advertising, eCPM can be uti- model to the pre-ranking model is conducted exclusively lized based on the pricing model of the platform. In an on impression samples. As the ranking model is trained e-commerce context, one might establish a relative prefer- solely on these impression samples, its ability to generalize ence order based on the depth of user feedback, such as to candidate samples and random samples is inherently Purchase > Add to Cart > Click. For scenarios like feed rank- limited. ing or video recommendations, user feedback signals can be aggregated using a weighted sum approach. Addition- 3.3.2. Fine-Grained Ranking Loss ally, we incorporate a general impression label applicable across business scenarios, for learning the pattern of the The fine-grained ranking loss is applied to both impression cascade ranking system. The label assigned a value of 1 for and candidate samples, which directly correspond to the impression samples and 0 otherwise. sample scope used in serving. We employ the SoftSort, a The user feedback labels help the pre-ranking model in differentiable sorting loss, to learn user behavior and the pat- learning the revenue or engagement level associated with terns of the cascade ranking system. This loss function aims different user behaviors. The exposure label facilitates the to precisely rank items according to the varying degrees pre-ranking model’s ability to learn and replicate the rank- of positive feedback they receive and effectively differenti- ing patterns in the downstream cascade ranking system. ate positives from impression samples and negatives from candidate samples. Consider the SoftSort operator defined by metric func- 3.2.2. Soft Labels tion d = | ⋅ |𝑝 and temperature parameter 𝜏 for sorting 𝑛- For soft labels, we use the ranking objective function as dimensional real vectors 𝑠 ∈ ℝ𝑛 : aggregation function. This approach ensures that the soft labels are seamlessly integrated into the training process, maintaining the consistency between the pre-ranking model −d(𝑠𝑜𝑟𝑡(𝑠)1𝑇 , 1𝑠 𝑇 ) SoftSort𝑑𝜏 (𝑠) = softmax( ) (10) and the ranking model. 𝜏 The output of SoftSort operator is a permutation matrix 3.3. Hybrid Loss Functions of dimension 𝑛. The softmax operator is applied row-wise, thereby relaxing the permutation matrices into a set of uni- The pre-ranking model focuses on achieving both consis- modal row-stochastic matrices. In simple words: the 𝑟-th tency and ranking accuracy through the following tech- row of the SoftSort operator is the softmax of the negative niques: distances to the 𝑟-th largest element [1]. We then employ the • Knowledge Distillation: The ranking model’s predic- softmax cross entropy between the permutation matrices of tions are used as soft labels to transfer knowledge label 𝑦 and the permutation matrices of logit 𝑧. The SoftSort from the ranking model (teacher) to the pre-ranking loss function is hereby defined as: model (student). • Ranking Capability: Fine-grained and coarse- ℒSorting (𝑧, 𝑦) = −tr(J𝑛 (SoftSort𝑑𝜏 (𝑦) ∘ log SoftSort𝑑𝜏 (𝑧))) grained ranking losses are applied to improve rank- (11) ing performance and retrieval capability, respec- where J𝑛 is a 𝑛 × 𝑛 matrix of ones, y = (𝑦𝑖 )𝑖∈𝒟ℐ ∪𝒟𝒞 is the tively. hard label and z = (𝑧𝑖 )𝑖∈𝒟ℐ ∪𝒟𝒞 is the logit of the pre-ranking model. We use the tr to compute the element sum of the • Diverse Training Samples: The model is trained on matrix SoftSort𝑑𝜏 (𝑦) ∘ log (SoftSort𝑑𝜏 (𝑧)). samples from different stages and varying easy/hard sample levels to achieve robust generalization and 3.3.3. Coarse-Grained Ranking Loss optimize hierarchical objectives. The coarse-grained ranking loss, applied to all samples (im- Our synergistic framework is designed to learn both the pression, candidate, and random), aims to separate positive hierarchy of user behaviors and the pattern of the cascade and negative samples while supporting ranking among pos- ranking system. For instance, in the context of online adver- itives by distinguishing varying degrees of relevance. We tising, the model is expected to understand the following propose the Adaptive Margin Rankmax (AM-Rankmax) loss, order of importance: converted items > clicked items > ex- an extension of the Rankmax loss [2] that introduces an posed items > candidate items and randomly sampled items. adaptive margin based on the pair’s nature and label dis- tance, thereby extending the Rankmax loss to address rank- 3.3.1. Distillation Loss ing problems with ordered or continuous positive labels. Consider the Rankmax loss for ranking problems with As the main goal for the pre-ranking model is to output a binary labels only: high-quality item set for the ranking model, hence we used a listwise loss for distilling the knowledge from the ranking model as follows: 𝑛 ℒ𝑅𝑎𝑛𝑘𝑚𝑎𝑥 (𝑧, 𝑦) = ∑ log ∑ (𝑧𝑖 − 𝑧𝑗 + 1)+ (12) 𝑗∶𝑦𝑗 >0 𝑖=1 exp(𝑧𝑖 ) ℒDistillation (𝑧, 𝑝) = − ∑ 𝑝𝑖 log (9) The Rankmax loss is reminiscent of pairwise losses. To 𝑖∈𝒟𝐼 ∑𝑗∈𝒟𝐼 exp(𝑧𝑗 ) extend the Rankmax loss to more general ranking prob- where 𝑝 is the prediction of the ranking model (soft label), lems involving multi-level positive labels, we introduce the 𝑧 is the logit of the pre-ranking model, 𝒟ℐ is the impression adaptive margin with following enhancements: samples set. Note the distillation process from the ranking • The loss is applied only when 𝑦𝑖 < 𝑦𝑗 , which is more • Q2: How do different settings on dataset sampling suitable for multi-level positive label scenario. and training losses influence the performance of RankTower? • The margin adjusts based on whether 𝑦𝑖 is positive or negative, to further enhancing the differentiality between positive and negative samples. 4.1. Experiment Setup 4.1.1. Datasets • The margin scales with the label distances between samples, reflecting varying degree of positive sam- We evaluate our model using real-world datasets: Ali- ples. mama1 , Taobao2 , and KuaiRand3 . For each dataset, we keep users with at least 100 impressions and 20 instances The adaptive margin function is: of positive feedback. The data is split into 70% for training, 10% for validation, and 20% for testing. As all labels in the 𝑚(𝑖, 𝑗) = 𝛼 ⋅ I(𝑦𝑖 = 0) + 𝛿(𝑦𝑖 , 𝑦𝑗 ) (13) datasets are binary, we aggregate them by summing the where 𝛼 is a constant for adding additional margin be- labels to form the hard label. tween negative and positive items, I is the indicator function. The metric function 𝛿 can take various forms, for example 4.1.2. Evaluation Metrics 𝛿(𝑦𝑖 , 𝑦𝑗 ) = 1 or 𝛿(𝑦𝑖 , 𝑦𝑗 ) = 𝛽|𝑦𝑖 − 𝑦𝑗 |𝑝 . The adaptive margin We consider Recall@K and NDCG@K for evaluating the Rankmax loss is then given by: performance of the models, and we set 𝑘 to 100 for all ex- periment metrics. 𝐿AM−Rankmax (𝑧, 𝑦) = ∑ log ∑ (𝑧𝑖 − 𝑧𝑗 + 𝑚(𝑖, 𝑗))+ Recall@K is the fraction of relevant retrieved within 𝑗∶𝑦𝑗 >0 𝑖∶𝑦𝑖 <𝑦𝑗 the top 𝐾 recommendations. It’s mainly used for measuring (14) ranking system’s capability on retrieving relevant items. where y = (𝑦𝑖 )𝑖∈𝒟ℐ ∪𝒟𝒞 ∪𝒟ℛ is the hard label from all the NDCG@K measures the quality of the ranking by con- samples and z = (𝑧𝑖 )𝑖∈𝒟ℐ ∪𝒟𝒞 ∪𝒟ℛ is the logit of the pre- sidering both the relevance and the position of items within ranking model. the top 𝐾 recommendations. Items with higher relevance The AM-Rankmax loss function can effectively adapt to ranked at higher position contribute more to the metric. scenarios with multiple positive labels of varying levels. This enhancement allows the model to handle different de- 4.1.3. Competing Models grees of positive feedback, thereby improving its ability to generalize and accurately rank items in complex settings. We compare RankTower with the following pre-ranking models: LR [3], Two-Tower [4], DAT [5], COLD [6], Int- Tower [7] and ARF[8]. 3.3.4. The Hybrid Ranking Loss We design a hybrid ranking loss that integrates both distil- 4.2. Model Performance Comparison (Q1) lation and ranking objectives. The hybrid ranking loss is the weighted sum of three losses: Table 1 ℒHybrid (𝑧, 𝑦) = 𝜆1 ℒDistillation (𝑧, 𝑝) Performance Comparison of Different Algorithms on Alimama , + 𝜆2 ℒSorting (𝑧, 𝑦) (15) Taobao and KuaiRand Dataset. Alimama Taobao KuaiRand + 𝜆3 ℒAM−Rankmax (𝑧, 𝑦) Model Recall@K NDCG@K Recall@K NDCG@K Recall@K NDCG@K LR 0.4802 0.3237 0.4792 0.2685 0.6713 0.5027 Two-Tower 0.5123 0.3428 0.5019 0.2921 0.6902 0.5258 where 𝜆1 , 𝜆2 and 𝜆3 are weights for each sub-objective. DAT 0.5161 0.3472 0.5089 0.3013 0.6955 0.5312 Balancing distillation and ranking losses is crucial for the COLD IntTower 0.5210 0.5215 0.3518 0.3519 0.5123 0.5101 0.3070 0.3051 0.7011 0.6960 0.5349 0.5309 pre-ranking model to inherit the ranking model’s capabili- ARF 0.5318 0.3655 0.5215 0.3117 0.7096 0.5497 RankTower 0.5462 0.3794 0.5301 0.3223 0.7182 0.5551 ties while generalizing to broader sample spaces. Weighting fine-grained and coarse-grained ranking losses ensures a The overall performance of different model architectures balance between precise ranking and overall retrieval ro- is listed in Table 1. We have the following observations for bustness. model effectiveness: 4. Experiments • LR exhibits the lowest performance compared to the other neural network-based models. We conduct experiments on three large-scale public datasets • Two-Tower brings the most significant relative improve- from online advertising, e-commerce, and short video rec- ment in performance, highlighting the importance of ommendation domains to evaluate the effectiveness of Rank- learning deep feature interactions. Tower. The experiments provide a comprehensive descrip- tion of the evaluation metrics, and comparisons with state- • COLD achieves strong performance among the compet- of-the-art pre-ranking models. We aim to answer the fol- ing models, indicating the significance of learning user- lowing questions through our experiments: item feature interactions. • Q1: How does our proposed RankTower perform for pre-ranking task? Is it effective and efficient under extremely high-dimensional and sparse data 1 https://tianchi.aliyun.com/dataset/408 2 settings? https://tianchi.aliyun.com/dataset/649 3 https://kuairand.com/ • ARF outperform other models without utilizing listwise 4.3.3. Effect of Distillation from Ranking Model ranking losses, highlighting the importance of using list- We conduct an ablation study on the distillation component wise ranking losses. and further compare Softmax loss with other alternatives. • RankTower achieves the best prediction performance, The Table 4 demonstrate the efficacy of transferring attributed to its effective modeling of bi-directional user- knowledge distillation. Among various loss function ex- item feature interactions and the design of full-stage sam- perimented for distillation, the Softmax loss outperforms pling and hybrid loss functions. the other alternative losses. The Softmax loss, being a list- wise ranking loss, proved more adept at distilling the rank- ing model’s capabilities compared to the weighted logloss, 4.3. Model Study (Q2) which essentially is a pointwise approach and exhibited sub- To gain deeper insights into the proposed model, we conduct optimal performance in learning the relative ranking distri- experiments on the KuaiRand dataset and compare model bution. In contrast, the pairwise logloss, focusing solely on performance on different settings, including: 1) the effect pairwise ordering of ranking model’s predictions without of full-stage data sampling; 2) the effect of listwise ranking considering the relative proximity of predictions, exhibited losses; and 3) the effect of distillation from the ranking overfitting to the ranking model’s outputs. model. Table 4 4.3.1. Effect of Full-Stage Sampling Experiment Results for Different Distillation Losses. We conduct an ablation study to evaluate the impact of Recall@K NDCG@K each sampling component on the model’s performance. As Distillation (Softmax) 0.7182 0.5551 shown in Table 2, the full-stage sampling strategy achieves Distillation (Weighted Logloss) 0.7130 0.5519 Distillation (Pairwise Logloss) 0.7071 0.5432 the best overall performance. Training the pre-ranking No Distillation 0.7108 0.5495 model solely with impression samples hinders its ability to generalize to unexposed items, negatively affecting retrieval performance. We also observe that candidate samples are more important than random samples, as they significantly 5. Conclusion enhance the model’s ability to discriminate between relevant and non-relevant items. This paper introduces the RankTower model, designed to enhance the performance of the two-tower model by effec- Table 2 tively capturing bi-directional latent interactions between Experiment Results for Different Sampling Strategies. user and item. To ensure consistency with existing casecade Recall@K NDCG@K ranking system, a hybrid loss function and full-stage sam- Full-Stage Sampling 0.7182 0.5551 pling approach are integrated into the model’s optimization w/o random samples 0.7125 0.5437 framework. Comprehensive experiments demonstrate that w/o candidate samples 0.7040 0.5401 RankTower significantly outperforms state-of-the-art pre- w/o candidate & random samples 0.6981 0.5323 ranking models. In future work, we aim to study how to effectively and jointly optimize the cascade ranking system in an end-to-end fashion. 4.3.2. Effect of Listwise Ranking Losses To better understand the properties of the proposed hybrid References loss, we compare it with several widely used ranking losses in the industry. The experiment results, as shown in Table 3, [1] S. Prillo, J. Eisenschlos, Softsort: A continuous relax- indicate that the hybrid loss consistently outperforms other ation for the argsort operator, in: International Confer- alternatives, surpassing both its individual components: the ence on Machine Learning, PMLR, 2020, pp. 7793–7802. Sorting loss and the AM-Rankmax loss. Moreover, our pro- [2] W. Kong, W. Krichene, N. Mayoraz, S. Rendle, L. Zhang, posed AM-Rankmax demonstrates superior performance Rankmax: An adaptive projection alternative to the compared to the original Rankmax loss and the Softmax softmax function, Advances in Neural Information loss. Processing Systems 33 (2020) 633–643. [3] H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, Table 3 Experiment Results for Different Ranking Losses. et al., Ad click prediction: a view from the trenches, in: Proceedings of the 19th ACM SIGKDD international Recall@K NDCG@K conference on Knowledge discovery and data mining, Hybrid Loss 0.7182 0.5551 2013, pp. 1222–1230. Sorting 0.7128 0.5516 [4] P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, L. Heck, AM-Rankmax 0.7132 0.5507 Rankmax 0.7105 0.5492 Learning deep structured semantic models for web Softmax 0.7109 0.5498 search using clickthrough data, in: Proceedings of ApproxNDCG 0.7006 0.5436 the 22nd ACM international conference on Information RankNet 0.7072 0.5452 & Knowledge Management, 2013, pp. 2333–2338. [5] Y. Yu, W. Wang, Z. Feng, D. Xue, A dual augmented two- tower model for online large-scale recommendation (2021). [6] Z. Wang, L. Zhao, B. Jiang, G. Zhou, X. Zhu, K. Gai, Cold: Towards the next generation of pre-ranking system, arXiv preprint arXiv:2007.16122 (2020). [7] X. Li, B. Chen, H. Guo, J. Li, C. Zhu, X. Long, S. Li, Y. Wang, W. Guo, L. Mao, et al., Inttower: the next gen- eration of two-tower model for pre-ranking system, in: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 3292–3301. [8] Y. Wang, Z. Wang, J. Yang, S. Wen, D. Kong, H. Li, K. Gai, Adaptive neural ranking framework: Toward maximized business goal for cascade ranking systems, arXiv preprint arXiv:2310.10462 (2023).