Dividing ads ranking system into retrieval, early, and final stages is a common practice in large scale ads recommendation to balance the eficiency and accuracy. The early stage ranking often uses eficient models to generate candidates out of a set of retrieved ads. The candidates are then fed into a more computationally intensive but accurate final stage ranking system to produce the final ads recommendation. As the early and final stage ranking use diferent features and model architectures because of system constraints, a serious ranking consistency issue arises where the early stage has a low ads recall, i.e., top ads in the final stage are ranked low in the early stage. In order to pass better ads from the early to the final stage ranking, we propose a multi-task learning framework for early stage ranking to capture multiple final stage ranking components (i.e. ads clicks and ads quality events) and their task relations. With our multi-task learning framework, we can not only achieve serving cost saving from the model consolidation, but also improve the ads recall and ranking consistency. In the online A/B testing, our framework achieves significantly higher click-through rate (CTR), conversion rate (CVR), total value and better ads-quality (e.g. reduced ads cross-out rate) in a large scale industrial ads ranking system.
The goal of the ads ranking system is to select the optimal ads to display to users. Due to latency
constraints, it is impractical to predict ranking score for each ad out of large-scale candidates.
Therefore, a multi-stage ranking process is widely adopted, which uses progressively more
complex models to narrow down the number of ads [
While retrieval is often rule-based, both early stage and final stage ranking use ranking score predicted by machine learning models.
After we obtain the final stage ads ranking score, the system will run an ads auction to decide
the winning set of ads to show to the user. To ensure that the winning ad maximizes value for
both user and businesses, we use total value [
In general, it is determined by ads quality models, which predict scores of multiple quality events (e.g. crossing out ads, hiding ads). Our framework mainly focuses on learning estimated action rates (i.e. ads CTR) and ads quality.
Despite the multi-stage ads ranking system being a common practice, it has the fundamental
problem of multi-stage inconsistency: the early stage ranking system fails to pass good ads to
the final stage ranking system [
In order to address those issues, we propose a multi-task learning framework for early stage ranking to learn the relevant information of ads total value in final stage ranking. Due to the latency constraint, we cannot learn all components of ads total value in one light-weighted early stage ranking model. Instead, we focus on joint learning of ads CTR and ads quality events. There are three major benefits for our framework: • Ranking consistency improvement In order to solve total value definition inconsistency issue on ads quality, we present a new objective for early stage ads quality, called consolidated quality score (CQS). Instead of replicating every final stage ads quality event model in early stage, the CQS consolidates all final stage ads quality objectives together to be a single objective. We derive the CQS task label from the final stage total quality scores. In addition, we add a distillation task from the final stage CTR model. Both of tasks significantly improve the ads recall for early stage ranking. • Resource saving by model consolidation In ads auction, the CTR model’s prediction is essential to estimate the action rates for various post-click conversions, while ads quality models are necessary for determining the quality score of each ad. Consequently, the primary serving costs stem from the ads CTR model and quality models, due to their large serving trafic. With the multi-task learning for CTR and ads quality events, we can reduce serving costs with shared model architectures and features. • Mitigation of selection bias We leverage the data augmentation to mitigate the selection bias of early stage model. We logged more final stage non-impression data in the training data as the augmented data. When computing the CTR loss, instead of treat them as negative samples, we use the final stage CTR prediction as the pseudo-label. For CQS task, the augmented data also has its label as each non-impression ads in final stage still participate in ads auction.
In order to better understand the impact of jointly learning CTR and ads quality, we also build a ofline recall simulation framework. In the current multi-stage ranking system, the ifnal stage has more accurate prediction for higher precision, whereas the early stage need to optimize for recall. The results show that our framework can improve simulated soft recall for early stage ranking. In the online experiment, we also observe the reduction of total value divergence between early stage and final stage, which implies better ranking consistency. We also conduct ablation study for each key component in our multi-task learning framework. In the online A/B testing, our framework achieves better ads quality, CTR and CVR, compared with the separate serving baseline.
Early stage ads ranking, also known as the pre-ranking stage, has great potential to improve
overall ranking performance as they decide candidates for final stage ads ranking. Most of
the prior work discussed how to improve the efectiveness while maintain eficiency for early
stage ads ranking [
Multi-task learning is widely used in recommendation system [
In this section, we discuss the key components for our framework: model architecture, model training, and evaluation metrics.
Instead training separate early stage CTR and quality models, we propose a multi-task learning
framework to train a single model on those objectives, as shown in Figure 2. We utilize DLRM[
Learning all quality events in a single model can be challenging. First, the data collection process of diferent quality events varies significantly, which makes it dificult to log all quality events in one data pipeline. For instance, there could be quality events derived from survey-based assessments, whose logging infra that is diferent from the one used for logging the CTR training data. Moreover, it is challenging for a single model to fulfill the model capacity constraint for eficient inference, while still predicting multiple tasks for quality events. To address these issues, we propose Consolidated Quality Score (CQS) to consolidate all quality events in early stage ranking. We define the CQS in Equation 2, as the input of the mapping function to compute the
, which denotes the final ads quality score of an ad. The indicates the model prediction of the quality event . The is the associated multiplier, so as to control the quality event’s power in the ads auction. The CQS can be easily logged into training data during the ads auction.
(1) (2) (3)
With the final stage CQS as the label, we not only unblock the quality data logging, but also solve the total value definition inconsistency issue in early stage ranking. Also, the early stage CQS can adapt to final stage quality event changes automatically in a flexible manner and maintain stable multi-stage status. We utilize mean square error as the loss function: where the
is the final stage consolidated quality score, and is the early stage ranking predicted value. is the number of samples.
In addition to the CTR task and CQS task, we also add one more task for teacher distillation. This task is not used for serving. There are two benefits for using final stage pCTR as the teacher model to distill early stage CTR model. First, distilling knowledge from a teacher model to a student model is a common approach to improve student model’s performance without = ∑ =1 = () ∗
=
1 =1 ∑( − ) , 2 additional capacity cost [16]. The final stage CTR model is much more complex compared to early stage CTR model, rendering it a reasonable choice to be a teacher model. Second, using the final stage CTR model as the distillation teacher can improve the ranking consistency since the early stage learns the final stage prediction information directly. Although this task can improve ranking consistency, we cannot use this task to replace the original CTR task during serving, because the model cannot learn good calibration without ground-truth click label. The distillation logistic regression Loss is employed in our teacher task.
ℎ = −[ ∗ log( ) + (1 − ) ∗ log(1 − )], (4) where is the final stage CTR prediction (between 0 to 1) and is the CTR task head prediction in our multi-task learning framework. The loss function measures the dissimilarity between the early stage CTR prediction and final stage CTR prediction, which helps improve consistency.
The serving trafic of CTR model is the subset of that of quality models. For instance, for post-impression conversion types, they do not need the CTR action to complete the conversion, but they still need quality score to rank. Therefore, compared with original CTR pipeline, we add remaining serving trafic for CQS task in the consolidated pipeline.
During model training, the CTR task will only be trained on its serving trafic to avoid unused feedback loop.
In order to mitigate selection bias, we enrich the data with non-impression ads. We randomly subsample the early stage non-impression ads as the augmented data. We treat the final stage CTR prediction as the pseudo-label for those non-impression data, in order to further improve the ranking consistency. During the online training, we have developed data augmentation framework to logging specific model’s prediction in the non-impression data.
During ofline experiments, we find adding CQS task leads to negative transfer for CTR task. This is expected since the correlation between CTR and quality score ads is low and ads quality is designed for relevance and integrity. Considering CTR is an important optimized ad event, we tune the weight of the CQS task in the loss to reduce the negative impact of the CTR task. In the final settings of our framework, we adjust the loss weight of CQS to be 1.5, which has the neutral impact on NE of the CTR task. In addition, adding the CTR teacher task can boost the CTR performance significantly. We tune the task weight of CTR teacher to be 2, in order to achieve best performance for CTR.
There are several common ofline evaluation metrics for ads ranking models, such as
AreaUnder-ROC (AUC) [
Calculating the accurate recall is impractical considering the large-scale ads candidates in early stage. In order to have a better measurement on recall, we leverage the ofline simulated recall for multi-objective early stage ranking system. We replay a small trafic with full ad requests in a simulator, a separate ranking flow but copies all components from production flow. Since the simulator will not serve any production trafic, we can relax the timeouts between stages, to ensure that all ads from retrieval stages are ranked. As is shown in Figure 3, after obtaining ads candidates from the retrieval stage, we will pass all ads in a ads request to the simulator and log top ads in the replay log. Those top ads will be marked as positive samples and rest of ads in the same ads request will mark as negative samples. We guarantee the production flow and replay flow has the same amount of final ads candidates to reproduce the production flow. After we have the replay log, we can utilize recall metrics to measure model’s ofline performance, with top candidates in replay logs as the golden set. There are two types of recall metrics: • Hard recall counts of intersection between top ads picked by the model and golden set divided by at the ad request level. This is the widely-accepted definition of recall.
The recent work [
We choose soft recall as the major ofline metric for ranking consistency because it is more reasonable for measuring the value of early stage ads. Also, we observe the variance of soft recall among diferent ad requests is much smaller than hard recall.
In this section, we conduct both ofline experiments and online A/B testing to justify the benefit of our framework. In order to understand each technique better, we first built a simple dedicated CQS model to verify the benefit of consolidating all early stage quality models. Then we further iterate on the production CTR model with our proposed multi-task learning framework. For ofline metrics, we compare the recall metric for overall multi-task predictions. Compared with other ofline metrics, we find the recall metric is more efective to reflect early stage ranking model’s online performance, such as impression based total value, CTR, CVR and total value divergence (TVD). The impression based total value is a metric to measure the potential business value of ads after user impression, as we run ads auction depends on the total value of ads. The TVD is computed by the following equation on final stage ads candidates, as an online metric for ranking consistency: = ∑|
∑| −
| | For ads quality metrics, we select two quality metrics: • Ads cross-out (Xout) happens when a user clicks ”×” and selects ”I don’t want to see this” at the top-right of an ad. We use the ads cross-out rate to measure this quality event, where the lower ads cross-out rate implies better ads quality. • Ads Survey for Quality (ASQ) is a survey-assessment based metrics for ads quality related signals. It estimates the user rating for ads, where higher is better.
To address the total value definition inconsistency issue between the early and final stages, we study a simple CQS model to consolidate all early stage ads quality models. The ofline soft recall shows significant improvement compared with using separate early stage ads quality models. For online metrics, we observe better quality of ads with lower ads cross-out rate and higher ASQ score. In addition, the total value divergence between early and final stage significantly decreases with the increased impression based total value. Although the CQS model does not afect any CTR or CVR model, the CTR and CVR also increase, which implies the business power of ads quality models. The better ads quality can bring long term value for ads ranking performance, with better ads experience for users. Another benefit for CQS is to save serving CPU cost significantly as we consolidate multiple simple early stage quality models together.
Given the baseline CQS model, we further iterate the CTR model on multi-task framework we proposed. Compared with the production CTR model, we refresh the features add top 50 important CQS features from the CQS model feature importance rank. With more CQS top features, our multi-task learning framework can have neutral MSE performance compared with (5) Recall (+) Xout rate (-) ASQ (+) TVD (-) CTR (+) CVR (+) Total Value (+) total CPU (-) +3.2% -1.8% +0.02 -7.9% +1.7% +2.0% +1.0% -0.7% the baseline CQS model. In table2, the multi-learning framework achieves better soft recall than production CTR and baseline CQS models. The online experiment also shows better ads quality and CTR, as well as higher CVR and impression based total value. The total value divergence is further reduced as we add final stage teacher distillation task in our framework. Since we add more features and two more tasks to the original CTR model, the total CPU is slightly smaller than that of separate CTR and CQS models.
Recall (+) Xout rate (-) ASQ (+) TVD (-) CTR (+) CVR (+) Total Value (+) total CPU (-) +12.2% -3.5% +0.005 -5.7% +0.4% +0.8% +3.0% -0.06%
We also set up several comparable models for the ablation study in Table 3, in order to exclude the impact of diferent feature sets compared with production models. We build four baseline models: 1) Dedicated CTR model by removing CQS tasks from our multi-task learning framework. 2) Dedicated CQS model with both CTR and teacher tasks removed. 3) Our multi-task learning framework without teacher task 4) Our multi-task learning framework trained on impression ads only. According to Figure 3, building dedicated CTR and dedicated CQS models can achieve NE or MSE gain over our framework, which implies negative transfer [19] issue in multi-task learning. Without teacher task, the CTR task performance regresses a lot, while the MSE becomes better. The teacher task is essential to help close the performance gap between final stage ranking and early stage ranking.
During the online experiments, the version with dedicated CTR and CQS models shows significant increase on Xout rate and drop for ASQ, although the dedicated CQS model has better MSE performance than our proposed framework. For ads CTR, although the dedicated CTR model can improve the ads CTR with better ofline NE performance, the CVR and impression based total value is slightly worse. The higher CTR but lower CVR implies that the ad is very eye-catching, but the user clicking on the ad may not the right demographic for which the ad targets. The poor ads quality can be the explanation of the lower CVR, as the ads quality reflect the user experience on ads. Such results manifest that the single ofline metric for a individual ranking model may not be reliable to reflect online performance. The soft recall metric can mitigate this issue which takes multi-objectives into consideration. The larger total value divergence also reflects ranking consistency issue, where the total value between early stage and final stage has large distribution gap. The multi-task learning between CTR and CQS can force the model to learn the coexistence of estimated action rate and ads quality, and improve the total value divergence.
Without the teacher distillation task, the MSE loss for CQS becomes better but the online quality metrics turn out to be worse than our multi-task framework. The online CTR reduces after removing the teacher task, and the total value divergence becomes worse. The impression based total value has regression, which is expected with worse CTR and ads quality.
The augmented data shows great potential in improving early stage ranking performance. After filtering out the augmented data, the ofline simulated recall is worst among all baselines. The model also sufers from impression based total value regression, CVR drop and CTR drop, as well as worse ads quality. The augmented data plays a critical role to improve ads recall with selection bias mitigated.
Based on these results, we can draw the following conclusions: • Each component in our multi-task learning framework is essential to improve the performance of early stage ads ranking model. The CQS task solves the issue of total value definition inconsistency between early and final stage. The teacher task helps close the performance gap of CTR and improve the ranking consistency. The augmented data mitigates the selection bias with better ads recall. • Ads recall and ranking consistency are important for early stage ads ranking. If we only focus on the individual objective of each ads ranking model and optimize for precision, the online overall performance may not improve due to the poor ranking consistency and low ads recall.
We propose a novel multi-task learning framework to improve early stage ads ranking performance. This framework can be generalized to other user cases since the CQS can be applied to any ads ranking system with the ads quality component. We also design the ofline recall evaluation metric for the multi-task learning framework in early stage ranking, which has been verified to reflect the model online performance in an industrial ads ranking system.
For future work, we plan to improve the stability of the CQS task. As MSE loss is prone to outliers, we will conduct more experiments for robust regression loss. Also, more techniques [20, 19] can be explored to avoid negative transfer between the CQS and CTR tasks. In addition, we manually tune the weights for diferent tasks in the current framework. This can be improved with learnable loss weight techniques [21, 22], which can adjust weight automatically for multiple tasks. recommendation systems, CoRR abs/1906.00091 (2019). URL: https://arxiv.org/abs/1906. 00091. [16] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531 (2015). [17] G. Zhou, X. Zhu, C. Song, Y. Fan, H. Zhu, X. Ma, Y. Yan, J. Jin, H. Li, K. Gai, Deep interest network for click-through rate prediction, in: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 1059–1068. [18] X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y. Shi, A. Atallah, R. Herbrich, S. Bowers, et al., Practical lessons from predicting clicks on ads at facebook, in: Proceedings of the eighth international workshop on data mining for online advertising, 2014, pp. 1–9. [19] W. Zhang, L. Deng, L. Zhang, D. Wu, A survey on negative transfer, IEEE/CAA Journal of
Automatica Sinica (2022). [20] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, C. Finn, Gradient surgery for multi-task learning, Advances in Neural Information Processing Systems 33 (2020) 5824–5836. [21] A. Kendall, Y. Gal, R. Cipolla, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7482–7491. [22] O. Sener, V. Koltun, Multi-task learning as multi-objective optimization, Advances in neural information processing systems 31 (2018).