With the rising prominence of short video platforms, the challenge of efective content recommendation has become more pressing, especially given the diverse range of video content and the sparsity of implicit user feedback signals. This research delves into the formulation of "successful video watches", particularly for short video platforms. We introduce various functional formulations to model video watch behavior and use these to define ranking objectives for training recommender models. Our findings reveal that, in contrast to the naive percentage-based thresholds, our proposed formulation - grounded on duration and watch percentile - aligns better with user retention and boosts engagement metrics. Moreover, while the standard approach tends to bias content recommendations towards extremes in video length, our methodology ensures a more balanced content recommendation, greatly impacting user experience on streaming platforms. This study underscores the potential nuances and implications of training recommender systems for video content.
seconds of a 400 second video. Understanding the factors that define a successful video play, keeping in mind the In recent years, video platforms have witnessed an un- video duration, is vital for developing efective recomprecedented surge in popularity, transforming the way mendation strategies, improving user engagement, and users consume and engage with digital content. These optimizing content delivery. platforms ofer a vast and diverse range of videos, encom- In this paper, we present an in-depth investigation passing varying lengths, genres, and categories, which into diferent formulations of successful video watches, pose unique challenges for efective content recommen- and propose various functional formulations that help dation. To successfully train content recommender sys- us model video watch behavior on short video platforms. tems, platform designers have relied on implicit signals Subsequently, we use these formulations to define the in the form of user feedback data. However, in the con- ranking objectives and train candidate generation and text of short video platforms, leveraging implicit signals, ranker models. Specifically, we train Field Aware Facsuch as likes, shares, or downloads, remains a challenge torization Machine model based on these objectives, and due to the inherent sparsity of such signals. investigate how diferent formulations of successful video
Video streaming time, on the other hand, is a widely watches impact various user engagement and business available signal and is often leveraged to train and eval- metrics. uate recommender systems. Given the heterogeneity of Compared with naive formulation of percentage based the video content, with a large number of short and long threshold, the proposed formulation based on duration videos uploaded hourly on such platforms, naively choos- and watch percentile is better correlated with user reing a label derived from video streaming time inadver- tention on the platform, and also results in better user tently causes bias towards certain type of video content. engagement metrics when used as an objective for the For example, a label based on successfully watching (say) FFM model. We also investigate how the platform level 50% video will result in a larger proportion of shorter content distribution changes when these formulations video watches being tagged as successful watches, as are used to train the recommender system, and highlight compared to longer videos. Indeed, shorter videos tend that naive formulation of successful video watch biases to have a larger watch perentage; e.g. users often would the surfaced content towards the extremes, either on very watch 10 seconds of a 20 second video, than watch 200 short or very long videos; whereas the proposed formulation strives a better balance in terms of the video content Workshop on Learning and Evaluating Recommendations with Impres- surfaced, , significantly influencing the recommendations sions (LERI) @ RecSys 2023, September 18-22 2023, Singapore surfaced to users. We contend that our findings have im$ srijanskt@gmail.com (S. Saket); saibabavelugoti@sharechat.co plications on how recommender systems are trained and (S. B. R. Velugoti); erishabh@gmail.com (R. Mehrotra) evaluated on video streaming platforms. https://www.linkedin.com/in/srijansaket/ (S. Saket); https://rishabhmehrotra.com/ (R. Mehrotra)
0009-0006-9460-1203 (S. Saket); 0000-0002-0836-4605 (R. Mehrotra)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) (a) Histogram plot of videos with duration bins on the x-axis
(b) Median duration of videos across categories
Recommender systems modulate what billions of people
are exposed to on a daily basis. Over the past decade a lot
of research has gone into specifying how these systems
are optimized for user engagement signals such as clicks,
streams, likes, or a weighted combination of such sets
[
Recent work has started to explore the impact the
objective choice would have on the platform. Zahra et al.
[
In the domain of short video recommendations,
duration bias has been an under explored topic, especially
for early stage content [
duration in Fig.2. It’s evident that the definition of success is influenced by the video’s duration: when using a percentage-based definition (p50)[Fig.2a], shorter videos are considerably favored over longer ones. Conversely, with a fixed threshold-based definition (WT20)[Fig.2b], longer videos are preferred over shorter ones. We anticipate that an ideal label should not exhibit such bias towards video duration, given that such a bias would significantly alter the overall content consumption pattern on the platform.
This also highlights the need for more nuanced formulations of video watch signal, which we consider in the next section. hibit sparsity in their occurrence, making them unreliable for capturing user preferences. In contrast, video watches is relatively less sparse compared to other signals, providing a more abundant source of information for modeling [Table:1]. However, the challenge lies in appropriately formulating the video watch signal. Indeed, a biased label choice might lead to the unintended promotion of shorter videos at the expense of longer-format videos, potentially cannibalizing the latter’s visibility and engagement. Striking the right balance in label choice becomes essential for developing accurate and fair recommendation models in short video platforms.
We begin by describing the product context and production data used in Section 3.1 and present preliminary analysis that highlight the biases that might exist with naive formulations of video watch signal (Section 3.2). In Section 3.3 we propose a number of alternate formulations of the successful video watch signal; which we use to train the recommendation model described in Section 3.4.
We consider production trafic from one the largest short video applications serving 200 million users, and randomly sampled user interaction data over the course of one week across 41,316,850 users, containing 14,549,333 video posts in the Hindi language, capturing both implicit signals such as Video Play - indicating a successful completion of a recommended video beyond a specified 3.3. Proposed Formulations of Video threshold, Skip, click, like, share, and favorite. We lever- Watch Signal age Field-aware Factorization Machines (FFM) to extract As highlighted in Figure 2, naive formulation of video 32-dimensional embeddings from user-item interactions watch signals often causes duration bias in the video confor each signal. To ensure real-time learning, we adopted tent surfaced as recommendations. To mitigate this, we a dynamic approach that continuously updated the em- introduce additional formulations of video watch signal beddings with every new interaction data point. The that can serve as labels for training candidate generation learning process utilized 7 days of logged data, while of- or ranking models. These formulations can be categolfine evaluation was conducted on 1 day of unseen data. rized into two main categories: binary and continuous signals, each ofering distinct approaches for defining 3.2. Prevalence of Duration bias them. For continuous signals, we further explore diferAs baseline formulation of video watch signal, we pro- ent definitions based on watch percentages, watch time, pose two simple approaches: and percentile watches.
Table 2 provides details of the proposed formulations.
Fixed Threshold (WT20) defines a binary label for The "WT20" signal employs a fixed time threshold of 20 successful video watch based on whether the user seconds, serving as a quick engagement measure. In conwatched the video for more than 20 seconds. trast, the "L1PD" signal is based on the logarithm of the video duration and dynamically adjusts the threshold Fixed Percentage (p50) defines a binary label for suc- based on the ratio of watch time to video duration, adaptcessful video watch based on whether the user ing to varying content lengths. The "p50" signal sets the watched the video for more than 50% of the video threshold at 50% of video duration, wherein we assume duration. if a user streams atleast half the video, it is a successful video watch.
The "SVP" signal introduces a nuanced binary metric of successful watch, stratifying videos into duration-based
WT20 L1PD SVP P50
{︃1 if watch_time > 20s WT20 =
0 else {︃1 if watch_time > log(1 + duration) L1PD =
0 else ⎧1 if watch time > (− ) & duration ∈ [5, 19) ⎪⎪ ⎪⎪⎪1 if watch time > (− ) & duration ∈ [20, 34) ⎪⎪ ⎪⎨1 if watch time > (− ) & duration ∈ [35, 49) SVP = ⎪1 if watch time > (− ) & duration ∈ [50, 63) ⎪⎪ ⎪⎪⎪1 if watch time > (− ) & duration > 64 ⎩⎪⎪0 otherwise p50 = {︃1 if watch time > 0.5 × video duration
0 else Continuous
RootLogTimeWatch (RLTW) RootLogPercentileWatch (RLPW) RootLogTimePercentileWatch (RLTPW) √︀log(1 + watch_time) √︀log(1 + watch_percentile) √︀log(1 + watch_time * watch_percentile) bins. Each bin’s success definition is established from We present details of one such model in the next section. historical user watch behavior, allowing for targeted evaluation of engagement within specific temporal segments. 3.4. Training recommenders based on Referring to Table.2, within a particular bin, a value of 1 Video Play Signal is assigned if the video watch time surpasses the speciifed threshold; otherwise, a value of 0 is assigned. The To evaluate the eficacy of the proposed formulations of threshold is decided based upon the min, max and mean the video watch signal, we train a Field-Aware Factorizaof watch time within the duration bucket, from user his- tion Machines (FFM) model takes userId, videoId and latorical watch time data. bel as the training data input and learns a 32-dimensional
For the group of continuous signals, we leverage log- vector based representation that captures interactions arithmic and square root functions, given their ability between user and video features. FFM extends the trato gracefully handle a range of video duration and scale ditional Factorization Machines by introducing field indown the scores from a wide array of duration range. formation, which is crucial in recommendation systems The "RLTW " metric is centered on raw watch time, and where attributes can belong to distinct categories or dois quantified by taking the square root of the log of watch mains. The FFM formula can be expressed as follows: time.
Additionally, "RLPW " incorporates percentile of video wofatthchei,rlirneksipnegctvivideeboinplsa, yesntaobltihnegpceorncteenxtti-leawdiasrteriabsusteiossn- ˆ() = 0 + ∑︁ + ∑− ︁1 ∑︁ ⟨, ⟩ · ment. We generate uniform 1-second intervals for videos =1 =1 =+1 of varying lengths. Then we calculate the percentile dis- where represents the input features, 0 is the bias tribution of watch time within each interval. To get the term, are linear weights, are the feature embedvalue of the label, first, associate the video play event dings, and ⟨, ⟩ denotes the inner product between with its corresponding duration interval. Subsequently, the embeddings of features and . within that interval, correlate the observed watch time Once trained, the FFM model provides us with user and with the respective percentile value. Finally, we propose video embeddings which we store in vector databases "RLTPW ", which leverages the interplay of watch time and leverage approximate neareast neighbor search apand watch time percentile, yielding a composite signal proaches to fetch a recommendation of top-k closest reflecting both engagement magnitude and relative posi- videos to a given user embedding. These top-k fetched tioning. videos consitute the recommendations shown to the user.
Together, these suggested indicators ofer a wide range Specifically around training, there are two modes of of choices, ofering binary and continuous objectives that learning these embeddings via FFM models: batch and can be used to define labels to train recommender models. real-time. The batch setup updates the embeddings at a specified frequency by collecting data over a period of Table 3 few hours whereas the real-time method dynamically up- Label/ROC_AUC_SCORE dates them based on every interaction data point. In this work, we designed a real-time embedding update system Label Like wherein FFM continually refines embeddings, enabling RLTW 0.5778 the model to adapt to evolving user preferences and item RLPW 0.5745 characteristics, enhancing the accuracy and relevance of RLTPW 0.5803 generated recommendations. We omit the specific im- L1PD 0.5853 plementation details of the realtime embedding system SWVPPER 00..55740837 since it is outside the scope of the current work, but we posit that the findings presented in this paper should generalize beyond FFM models.
We next present a detailed evaluation on how the pro- 4.1.4. Remarks on signals: posed formulations of the video watch signal performs on a number of evaluation criterion. We begin by looking at the correlation of these signals with user retention on the platform (Section 4.1) and present an analysis on the overlap of information between these signals and traditional like/share/favorite signals (Section 4.2). Further, we use these labels to train a FFM based recommender models and evaluate its performance on recall based user engagement metrics (Section 4.3). Finally, we demonstrate the impact the design choice of this signal could have the overall platform level content distribution (Section 4.4).
Among percentile, watch time, and duration, percentile appears to be the most influential, followed by watch time and then duration. This hierarchy is attributed to the fact that duration-based measurements lack broad applicability across various data subsets due to their threshold nature. 4.1.5. Comparison between D1, D3, and D7 retention:
form threshold and percentage-based methods. The ra- In order to determine the efectiveness of the proposed lationale behind this is yet to be determined. Additionally, bels as potential training signals for ranker models aimed
4.1.1. How did we chose the labels: An optimal label for training any machine learning model is one that demonstrates a correlation with user retention. Hence, gauging this correlation is of paramount significance. UserIDs are linked to video watch events, leading to the formation of labels, and subsequently, their correlation with retention is measured. 4.1.2. Key takeaway: The outcomes of this analysis highlight that the proposed employment of quantile-based labels with square root transformations exhibits a stronger correlation with user retention when compared to the previous SVP signal and threshold-based approaches. This correlation is even higher than that achieved by the L1PW approach. 4.1.3. Remarks on functions:
correlation value (0.12 vs. 0.09), indicating a higher correlation with long-term retention compared to short-term retention. Notably, no significant variance is observed across trends.
In conclusion, we anticipate that training a machine learning model with this objective should enhance user retention, though it’s important to note that this is a view of correlation rather than causation.
Our objective is to assess whether the suggested labels convey similar information or exhibit varying degrees of overlap. Upon examination, it becomes evident that these labels difer and convey distinct information, as indicated by the heatmap[Fig.4]. Consequently, this dissimilarity implies that the recommendations derived from them should also difer. Once we establish the absence of label redundancy, we proceed to examine how these labels influence user satisfaction and the distribution of content in later sections. at enhancing user satisfaction, a comprehensive evalua- embeddings, suggestions are formulated in accordance tion framework was established. The core objective was with suggested markers. Subsequently, the recall is calcuto ascertain the suitability of the labels in capturing user lated concerning the ’like’ label as well as various other engagement and subsequently training a Field-Aware indicators of user engagement on unseen data. Factorization Machine (FFM) model for improved ranking. 4.3.3. Results and Insights
Curve (ROC-AUC) was employed as the principal evaluation metric. In this context, the predicted scores were generated using the embeddings learned from the specific labels, and the actual labels were represented by the ’like’ label in the corresponding column. The other assessment criterion selected is recall. Using the learnt 4.3.1. Methodology The evaluation results ofer valuable insights into the efectiveness of the proposed labels in capturing user To assess the viability of the labels, a seven-day dataset engagement and guiding the ranker model. Notably, the was utilized for training the FFM model. Subsequently, AUC scores varied across the diferent labels, shedding the trained model was employed to generate embeddings light on their relative performance. for the candidate items, which were then used to rank The key observations from the evaluation is that the the items. The evalution was done on one day of unseen AUC score was highest for the "L1PD" label, indicating user data. The process was anchored in the analysis of its efectiveness in capturing and predicting user eninteraction signals, which are pivotal indicators of user gagement. The AUC scores followed a distinct order engagement. where "L1PD" was followed by "percentile based signals," "watch_time based" and "SVP" labels, respectively. The 4.3.2. Evaluation Metrics comparison highlights the varying degrees of efectiveness in capturing user engagement among the evaluated labels. From Table:4, it’s evident that RLTPW demonstrates the highest recall concerning user engagements, along with exhibiting elevated values in relation to the newly proposed metrics. This alignment is consistent with the correlation pattern we previously identified.
through the proposed label, aligning with our initial assumptions. This observation reinforces our anticipation that the proposed label would exhibit less bias towards content duration.
Our approach involved using labels to steer the model’s behavior, subsequently influencing the content displayed to users. As a result, this choice of labels directly afects the content distribution across the platform. However, these interactions can lead to unintended and interest- 5. Discussion & Conclusion ing outcomes. Furthermore, this interplay generates a feedback loop, amplifying potential negative efects. In We posit that defining and understanding "successful the following sections, we demonstrate the alteration video plays" in the context of video platforms that surin content distribution across two aspects: duration of face a diverse range of videos is a complex endeavor. Our video and categories. investigations underscore the pitfalls of relying on naive label definitions, and higlights that such approaches suf4.4.1. Across duration fer from duration bias and distort the content distribution on the platform in unintended ways, thereby proFrom our analysis, we derive several insights shown in moting either excessive short or excessive long videos. [Fig.5]. When employing a threshold-based label, shorter We highlight that the proposed signals based on watch videos are favored, while the same preference holds for time, and watch percentile are more aligned with user the percentage-based label. We observed a tendency to- retention, and when these labels were incorporated as wards longer videos when considering logarithmic watch recommender objectives, we observed a positive impact times, although additional verification is necessary. In on various engagements metrics, attesting to their potenlight of these findings, our proposed approach aims to tial in enhancing the recommendation quality and user strike a balanced combination, taking into account mul- satisfaction. tiple factors for optimal content distribution. 4.4.2. Across categories We validate our findings from the ofline experiments, which demonstrate that various suggested labels present diverse content types to users. In [Fig.6], we depict the relative variation in content exposure across categories between the potentially optimal label (RLTPW) and a reference label (p50). Positive values indicate a higher recommendation frequency by the reference label (p50) compared to the optimal label (RLTPW), and vice versa for negative values. It’s noteworthy that categories such as Politics and News exhibit increased recommendations
As we move forward, several areas warrant further exploration. First, a pressing question remains about what the optimal metrics for gauging video success should be. While we have made headway in establishing some promising objectives, a comprehensive evaluation of how these formulations fare as evaluation metrics remains to be explored. Second, while our study introduced various functional formulations, we imagine future work to explore learnt formulations of video successful watch signals. Third, to truly gauge the applicability and effectiveness of our findings, they need to be validated in real-world, online scenarios wherein the recommender models trained on these signals are deployed online and evaluated. Lastly, an underexplored dimension of this research pertains to content creators. The ways in which these formulations and definitions afect the creators, both in terms of their motivation and the content they produce, is a crucial aspect to understand.