Many online platforms struggle to control the massive increase in clickbait content. Publishers and professional content creators strive to get users attention in a highly competitive online environment, thus trying to write catchy, superlative, and misleading titles. In our setting, we run a semi-open on-line content platform with hundreds of professional publishers and thousands of content creators. We examined several diferent approaches, how to systematically penalize a clickbait item in industrial recommender system. To compare the eficiency of selected approaches, we conducted large-scale online AB experiments, reported results indicating KPI shifts, and shared several insights from application of the approach on our platform.
Our content platform serves millions of active users each day, delivering personalized recommendations
from a wide range of content creators. Users receive a customized mix of on-line media, including news,
entertainment articles, videos, and podcasts. Clickbait (CB) often employs sensationalist headlines
or descriptions that exaggerate the content’s value or importance to lure users into clicking. These
headlines often create a "curiosity gap" by providing just enough information to spur interest, but not
enough to satisfy it, forcing users to click to learn more [
In this work, we investigate several approaches to integrate clickbait mitigation into an industrial recommendation system, ranging from heuristic methods to reinforcement learning, validated through extensive online A/B testing.
The initial step in mitigating clickbait involves using a model that assesses the degree of clickbaitiness in article titles. In our setting, we employ an internal BERT-like model trained on a large, carefully 0.8 r o t fca 0.6 n o i t a c i iltp 0.4 l u m 0.2 0 0 annotated dataset. The model classifies each article title and outputs a continuous score between 0 and 1, where 0 means that there is no clickbait and 1 signals a clickbait title. This model is static and used in all of our experiments.
We explore several methods to integrate clickbait mitigation into the recommendation system. First, we introduce a model-agnostic approach that requires no changes to the model architecture or the training process. Next, we modify the model itself by incorporating clickbait signals, both as an input feature and as an auxiliary learning objective, allowing the model to account for undesirable behavior through its loss function. Finally, we apply an actor-critic framework from reinforcement learning to guide the model’s output distribution to more desirable recommendations. Each of these approaches is described in detail below.
One of the most straightforward, model-agnostic solutions to applying any kind of promotion or penalization logic based on some item attribute is to modify the score of an item that is used for ranking. In our case, we decided to reduce the item score by multiplying by a factor derived from the clickbaitiness of the item. To compute this factor, we use a custom, adjustable sigmoid function: (, ℎ, , , ) = (1 − ) + 1 + ︁( 1− )︁ 1− 2 (−ℎ−ℎ) where c is the clickbaitiness of given item, h and l are the locations of the higher and lower control points, d is the relative drop of the value at the control points, and w is the weight, controlling the lowest value of the curve and therefore the overall efect of clickbait penalization. The c parameter is tied to a given item, and all the other parameters are treated as hyperparameters of the solution. The function, including a visualization of the hyperparameters and their values used in the experiment, can be seen in Figure 1.
(1)
The presence of clickbait in recommendation can be considered as a bias. We decided to experiment with
an existing model debiasing technique that employs a shallow tower architecture[
Even more direct solution how to penalize the model for recommending clickbait was to handle the
clickbaitiness at loss level. While we tried reducing the weight of training data rows for clickbait
items, the most promising approach turned out to be the utilization of the predicted CTR and item’s
clickbaitiness into an additional auxiliary task loss [
= log( 1 − ) * (2*¯ − 1) (2)
This allows the model to directly learn which items to promote over others based on their clickbaitiness. The efect of this approach can be adjusted by modifying the loss weight.
While auxiliary task solution does properly punish high-clickbait articles, it still needs interactions to
learn such punishment from. We therefore looked for solutions that are more in the realm of model
alignment rather than only standard supervised learning from implicit interactions. Inspired by RLHF
[
We sample from a catalogue of available articles and perform an actor-critic cycle with KL divergence bounds over frozen model with the previously mentioned clickbait loss to align the model. (3) (4) (5) (6) critic = (¯ ; − )
= (frozen; actor) actor = * − * ¯
The impact of this method can be regulated by adjusting the loss weight as well as by modifying the way the items are sampled from catalogue.
The main pipeline of our recommender system follows the 4-stage framework as described by Higley et
al. [
In order to assess the performance of our recommender system, we perform randomized A/B tests on live trafic. For each of the tested variants, we include A/A variant. The size of each variant as well as test duration are chosen empirically w.r.t. statistical significance. The unit of randomization in our AB testing system is the user.
To evaluate the efectiveness of the tested methods in reducing the amount of clickbait items in recommendations, and to see its impact on the KPIs, we report these user-level metrics: • Number of impressions per clickbait category - to show the shifts in the amount of recommended CB, we categorize items into three categories according to their clickbait score corresponding to non-CB, light CB and heavy CB. • Click-through rate - the number of clicks divided by number of item impressions. • Number of clicks • Spent time - the sum of amount of time user spent consuming clicked items. • Bounce rate - the ratio of clicks resulting in immediate leave after the content of article is displayed.
Unlike other mentioned metrics, reducing bounce rate means improvement.
All metrics are computed per user on personalized portion of trafic and their reported changes are in relative scale.
Clickbait mitigation can lead to various outcomes. In the short-term, it most likely leads to a reduced number of clicks in the system. Clickbait titles have generally higher click-through-rate than other article titles. The same may not be true for other metrics that try to measure user satisfaction, such as spent time, the overall number of user interactions, etc. Our findings suggest that some clickbait titles are connected with a high number of bounces, i.e. users leaving the article in a short time and not reading a significant part of it. However, there are also clickbait titles that keep the users interested in the articles. Therefore, it is not straightforward to predict the outcomes of the mitigation on these metrics.
Examining the experimental results reveals that each tested approach exhibited distinguishably diferent behavior. All of the methods successfully reduced the exposure of CB, some of them more aggressively than others and with varying impacts on KPIs. Without any doubt, the heuristic postprocess method is the overall favorite, substantially reducing the exposure of heavy CB similarly to the most aggressive Auxiliary task method, but with much lower, manageable impact on KPIs. Another positive aspect is that the method is easy to set up and fine-tune to desired trade-of, model-agnostic and therefore potentially applicable to other than personalized recommendation scenarios.
The debiasing via shallow tower variant has also shown interesting results. It is not as powerful in suppressing CB as other methods, however the efect is still not negligible, especially considering the improvements in majority of KPIs. The slight drop in number of clicks is compensated by an increase in spent time, meaning that users spent more time consuming the items rather than browsing the recommendations, which is considered a positive change. One potential limitation of this method is the inability to control the extent of clickbait suppression.
The auxiliary task produced highly contradictory results. On the one hand, it reduced CB impressions more efectively than the other approaches. On the other hand, it led to a significant decline in clickrelated metrics. Interestingly, despite this drop, users spent more time consuming the content overall, and the bounce rate decreased significantly more than with the other methods. These mixed outcomes are intriguing and give us flexibility in choosing which KPIs to prioritize.
Reinforcement learning resulted in a strong reduction in clickbait, but this came at the cost of an overall decline in both click and time-based metrics. Its outcomes were somewhat similar to those of the auxiliary task, with the added downside that even time spent on site decreased slightly. This is the most challenging setup, and so far, it hasn’t delivered the level of performance we were hoping for.
Although we were open to use machine learning approaches to mitigate the presence of clickbait in recommendations, the heuristic post-process method demonstrated the best trade-of - it efectively reduced the exposition of heavy clickbait and positively impacted bounce rate with a rather minor cost in terms of the performance in clicks and CTR. A natural continuation of this research will be to explore a combination of multiple approaches, such as shallow tower and post-process methods, to further enhance the results. This combination may leverage the strengths of both heuristic and ML solutions to improve clickbait mitigation and user satisfaction.
We thank Vít Líbal and Daniel Procházka for their extensive feedback on this paper. We thank Josef Florian and Petr Zelenka for their assistance with conducting the experiments. We thank Minh Duc Pham for sigmoid function reduction.
During the preparation of this work, the authors used ChatGPT, Microsoft Copilot and Perplexity AI in order to: Paraphrase and reword, Improve writing style. Further, the authors used SciSpace in order to: Citation management. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.