Personalised user experiences have improved engagement in many industry applications. When it comes to news recommendations, and especially for a public service broadcaster like the BBC, recommendation systems need to be in line with the editorial policy and the business values of the organisation. In this paper we describe how we develop recommendation systems for news articles at the BBC. We present three models and describe how they compare with baseline approaches such as random and popularity. We also discuss the metrics we use, the unique challenges we face and the considerations needed to ensure the recommendations we generate uphold the trust and quality standards of the BBC.
Computing methodologies
Machine learning approaches.
The BBC is one of the world’s leading public service
broadcasters. Its services—television, radio, digital—reach more
than 80% of UK’s adult population every week [
According to the mission of the BBC, the organisation
must “act in the public interest, serving all audiences through
the provision of impartial, high-quality and distinctive output
and services which inform, educate, and entertain” [
The structure of this paper is as follows. Section 2 defines the problem addressed in the current work, and Section 3 discusses prior related work. Section 4 describes the methodology including the data, models, and evaluation approaches. Finally, results are presented and discussed in Sections 5 and 6. 1Please note that ‘News’ capitalised refers to the UK channel, whereas lowercase regards to the type of content. 2https://www.bbc.com/mundo
Our goal is to build recommendation systems for news
articles. Recommendations in the news domain have been
characterised distinctly in the literature [
Non-signed in users. The majority of users on any BBC news platform are not signed in. This means that we have limited information about the user and the items they have previously interacted with. We typically work with sessionbased information, i.e. user-item interactions that occurred within 30 minutes from each other. This means that our recommendation models need to achieve high accuracy for cold-start user scenarios or predict the user’s taste after as little as one item interaction.
Many cold-start items. The publication cycle on any news platform is rapid and unrelenting. BBC News is no diferent. Fresh items are regularly uploaded and any recommendation system we implement should be able to serve an item within minutes of publication. Additionally, articles may become obsolete or gain sudden relevance following an event—consider for example the case of breaking news. Recommendation approaches must thus be able to take these characteristics into account, not being based solely on a user’s history, but considering the content and context of the articles they read.
Architecture constraints. Because of the popularity of BBC news, multiple stakeholders (internal and external) rely on and set the requirements for the news platform. Any changes to the system architecture that could afect other stakeholders need to be thoroughly investigated and justiefid. Our recommendation models often have to adapt to the existing architecture which means that our system architecture choices are somewhat constrained.
Mistakes are not tolerated. BBC news, and the Mundo platform in particular, are consumed by millions of users. For the majority of these users, this is the only BBC platform they visit. News is also a very sensitive domain as is not just entertainment but is also the way in which people inform and educate themselves. Mistakes in data-driven recommendations could lead to misinformation or compromise our quality standards, something which will largely impact our audience. The bar for the performance of the system is set very high to limit the risk of unexpected behaviour.
Fairness and impartiality. The BBC has built its trust
after many years of thoughtful manual curation and expert
editorial guidance. It commits to delivering content in a fair,
impartial and honest way and data-driven recommendations
should live up to, and advance, these standards.
Algorithmic fairness and impartiality in recommendation systems
are increasingly discussed in the literature [
The above challenges drive the decisions we make around which models and evaluation strategies to implement. For example, we place significant focus upon ofline evaluation to avoid unexpected behaviour; we use a variety of metrics to track the quality of recommendations; we consider recency-based systems an essential baseline for news recommendations; and we adopt content-based approaches to tackle the cold-start scenarios. More details about our choices and how they relate to these challenges are provided in Sections 3 and 4. 3
Recommendation systems in the news domain have been
investigated for more than a decade [
Beyond the news domain, recommendation systems have
been investigated in a variety of industrial applications.
Approaches vary between traditional content-based and
collaborative filtering while, more recently, the advent of deep
neural networks has facilitated the development of hybrid
strategies [
Considering the system architecture, some neural
networkbased approaches for recommending textual content are
endto-end (for example [
The BBC collects detailed user interaction data for its digital
services, providing information about users and the
circumstances of their visits to BBC websites. For the purpose
of this analysis, we used 15-days worth of data from BBC
Mundo, spanning from the 6th to 20th April 2019. We define
a sequence, or visit, as any succession of user interactions (i.e.
page views) within 30 minutes from each other. Page views
were aggregated into sequences according to this definition.
In this dataset, the average number of user interactions we
collected per day was in the order of millions. As shown in
Figure 1, most recorded sessions included only a single article
read (i.e., a sequence of length 1) which is a common
observation in news delivery platforms [
Like all statistical learning models, to robustly evaluate recommender system performance, the data is required to be appropriately split. In traditional machine learning problems where the raw data takes the form of input-output pairs, this split is relatively straightforward. Assuming there is enough data, a common split might be 80%, 10%, 10% into training, validation and test sets respectively. For recommender systems, the temporal nature of the data makes the situation a little diferent. While we still need to perform a train/validation/test split, referred to from now on as the test split, we also need to perform an additional split, henceforth be referred to as the query split. The query split describes the process of transforming a temporal sequence of consumption logs into a single or group of feature-target pairs suitable for ingestion into algorithmic learning models.
For the test split, our initial thought was to discard the temporal dimension and sample user sessions according to pre-determined train/test/validation fractions. While the simplicity of this approach is attractive, we decided that to maximise the similarity between our ofline testing framework and our online production environment was more important. The temporal approach we implemented is displayed in Figure 3 where we choose a thirteen-day period for training, the next day for validation and the following day for test. As we have the capacity to train and serve fresh consumer-facing models every day, we aim for this ofline approach to reflect our production environment suficiently for inferences in the former to provide valuable information about the latter.
For the query split, we take a user session from a given period defined earlier in the current section and divide it into the maximum number of trigrams while preserving temporal order. Then, for each trigram, the first two elements (articles vectors) represent the user profile while the third and final element is the groundtruth item used as a target for our models. The length of the user profile was chosen based upon two factors: (1) our client-side serving infrastructure is currently limited to providing the current and previous article; and (2) exploratory analysis indicated that minimal gains were made from increasing the number of items that make up the user profile. 4.2
All recommendation models we implemented were constrained by the need to have compatibility with our current system architecture. This consists of three main components. The ifrst is responsible for generating article embeddings. The second takes user data and article embeddings as input and produces a user embedding. Finally, the outputs of the first and the second modules are combined by the third component, which ranks the recommended articles for a user, based on a nearest neighbour search in the latent article space (Figure 4).
The content representation module generates article
embeddings. The article embeddings were derived using a Latent
Dirichlet Allocation (LDA) model as found performant in
related research [
The user representation module generates user embeddings. The user embeddings are derived from the article embeddings and previous user interactions. Our experiments focused primarily on developing models to derive user embeddings. We explored neural network approaches that combine both content and user data as well as models based only on user interactions (i.e. Cosine-based collaborative filtering model, Section 4.2.2).
The output of the user representation module is
subsequently processed by the recommendation generation module.
This component takes as input a user embedding and
performs an approximate nearest neighbour search in the article
latent space, returning as output the articles with the
smallest distance to the user embedding. The distance is
computed using the angular metric from the Python package
ANNOY [
We evaluated three diferent models to derive the user embeddings: a) a weighted average of item embeddings (Section 4.2.1), b) a cosine-based collaborative filtering method (Section 4.2.2), and c) a rank-optimised neural network (Section 4.2.3). The sections below describe each approach in detail.
Content data Content representation module
Article embeddings (LDA)
User data User representation module
User embeddings Recommandation generation
module Nearest neighbours
Recommendations 4.2.1 Weighted average of item embeddings. The first user representation model we tested derived the user embeddings from the weighted average of item embeddings, for all items consumed by a given user within a session. The most recently consumed item was weighted by a factor while the rest of the items in the user’s session were weighted by 1 − . 4.2.2 Cosine-based collaborative filtering. The second approach was a combination of simple user-item collaborative filtering and a session-based approach. Since users do not need to log in to view the articles, we had no explicit user profile and instead treated each session as a user. To generate the sparse user-item matrix, we took the article IDs for all user sessions within a given time window. The inputs to the model at prediction time were the IDs of the articles viewed in the current user session, and the output was the highest scored items based on these interactions. Our metric for scoring the articles to recommend was the cosine distance of the current user session and all other user sessions. Serving environment
Recommended articles KNN
Training environment
Binary cross- entropy loss Sigmoid Dot product Prediction model
User embedding
Article embedding Multi-layer perceptron
Multi-layer perceptron
OR Client (current + previous article vector)
User profile (current + previous article vector)
Groundtruth article vector
Negative
sampled
article
vector
4.2.3 Rank-optimised neural network. Motivated by the
awareness that a simple linear combination of a user’s current and
previous article representations led to modest performance
gains over using solely the current article, we sought to
explore non-linear combinations of these vectors. Artificial
neural networks are ideally suited to fitting such non-linear
functions, while we were encouraged by the results reported
by others that have successfully used deep architectures to
solve information retrieval problems, e.g. [
The challenge we faced was to design a neural network architecture which learned a latent representation of a user profile (current and previous article) to minimise the distance between itself and the latent representation of the most appropriately recommended article (in this case, the subsequently consumed article). One way of reflecting this problem is a pointwise architecture that behaves in a way similar to a regression problem. The model illustrated in Figure 5 takes a user profile (two concatenated 75-length vectors) and an article as input (a 75-length vector), passes each through a five-layer perceptron (with 1024, 512, 256, 128 and 75 hidden units, each with rectified linear activation functions). The model then minimises the binary cross-entropy between the target and the inner product of the final layer of the two perceptrons. Batch normalisation placed before the activation functions of the initial layers was found to significantly boost performance while also halving convergence time, facilitating greater experimentation. Training runs including dropout layers produced no improvement in accuracy so were not included in the final model. Negative articles were randomly over-sampled from the population of positive articles, whereby each training user profile has one positive article and five negative articles. Once this model had been trained, two further models were derived from it for use in the prediction environment. The first, the user model, took only the user profile as input and returned the final layer of the connected five-layer perceptron. The second, the article model, took only a single article as input and returned the iffth layer of its own five-layer perceptron. The article model was then used to transform all of the raw LDA embeddings into the article model embedding space before being fed into our vector-based nearest neighbour index. 4.3
The aim of our work is not only to increase user
engagement with BBC products, but also to inform, educate, and
entertain—according to the mission of our organisation. We
build recommendation systems taking into account these
values and develop evaluation strategies that reflect our mission.
This section focuses on ofline evaluation metrics and the
baselines we use in our experiments. Online evaluation is also
a big part of our work but goes beyond the scope of this
paper which focuses on preliminary results.
4.3.1 Metrics. When developing recommendation models
ofline, we currently monitor and optimise performance with
reference to a suite of six quantitative metrics. For all metrics
(with the exception of inter-list diversity) a value can be
computed for each groundtruth/recommendations list pair.
The overall metric is computed as the mean value over all
groundtruth/recommendations list pairs within the test
period. For each metric, in addition to calculating the overall
value, we also estimate the item-normalised value by rfist
taking the mean metric value for every unique groundtruth item.
This value provides an insight into the performance of an
algorithm independently of the test set bias towards popular
groundtruth items. All metrics were calculated upon
recommendation lists of length = 100. We use a relatively large
motivated by the finding that deeper cut-ofs in ofline
experiments provide greater robustness and discriminative
power [
Normalised Discounted Cumulative Gain (NDCG). It measures the gain of a document based on its ranked position in the top 100 list, with lower ranks discounted by a logarithmic factor, and normalises the result by the maximum gain of an ideal top 100 list.
Hitrate. A recall-based metric whereby a recommended list of items is assigned 1 if it contains the groundtruth item, and 0 otherwise.
Intra-list diversity. It estimates the average distance between every pair of items in a recommendations list. For the experiments reported here, distance between two articles is measured as the ANNOY angular distance (described formally in Section 4.2) between two article embeddings.
Inter-list diversity. It measures how diverse the recommended items across multiple lists are. It compares two lists of recommendations and computes the ratio of unique items in these lists over the total number of recommended items between these lists.
Popularity-based surprisal. It measures how novel or
surprising the items in a list are. It is formally defined as the
log of the inverse popularity of an item (i.e. the probability
of observing an item in the recommendations) [
Recency. : Measures how recent the recommended items are. It calculates the time diference between the recommendation request and the age of the recommended items using a Gaussian decay function. The mean is set to 1 and the standard deviation is chosen such that articles of 7 days old or more receive a score less than 0.5.
The ideal recommendation engine would optimise all these metrics providing recommendations that are relevant to the user, but that are also diverse, recent, and avoid the popularity bias. In practice this is usually a trade-of as an algorithm that provides more accurate results is, conversely, less likely to produce diverse ones (and vice versa). In line with our values and objectives, we sometimes choose algorithms that favour diverse and recent content at the cost of a certain degree of accuracy. 4.3.2 Baselines. We compare our user models to four baseline approaches and require that each new user model outperforms the existing ones. We consider the following recommenders as baselines: ∙ Random recommender : Produces random recommendations. ∙ Recency-based recommender : Ranks item by recency and returns the top most recent items. ∙ Popularity-based recommender : Ranks items by popularity and returns the top most popular items. ∙ Content similarity recommender : Finds the nearest neighbours of an item (e.g., the last item consumed by a user) using the ANNOY angular distance between item embeddings.
Our ofline experiments report results on the four baselines defined in above and the three models defined in Section 4.2. We use the NDCG metric to comment on the accuracy of the systems and the remaining metrics defined in Section 4.3.1 to comment on qualitative aspects of the recommendations. 5
The NDCG scores for each recommender system are shown in Figure 6. The scores from all metrics are summarised in Table 1.
Accuracy scores recorded for the baselines models were in line with expectations. Compared to a random selection of items (the random model), all other baselines show clear performance improvements for both overall and item-normalised NDCG. The popularity and recency recommenders returned higher values than the content-based similarity (CS) model for NDCG overall; however, if the most popular items are factored out by looking at the item-normalised score, the opposite is true. The recency recommender scored particularly high NDCG overall which confirms our expectation that users in a news platform prefer to consume fresh content.
Of the implemented models, the cosine-based collaborative filtering (CF) model (Section 4.2.2) outperformed all baselines and other models by a significant margin, this being the case both for overall and item-normalised NDCG and hitrate. However, this significant advantage in accuracy comes at a cost to inter-list diversity and surprisal, where both other models returned higher scores. However, this efect was not observed with the intra-list diversity metric, indicating that individual CF lists contained more diverse content while the lists of the other models were more distinct.
The weighted average (WA) model (described in Section 4.2.1, with optimised at 0.7) achieved accuracy scores surpassing all the baselines in item-normalised NDCG, although as expected, this was not the case for NDCG overall. This suggests that the model consistently projects into relevant regions of the embedding space, and that the nearest neighbours are not just most popular candidates. Despite returning marginally higher NDCG scores, the WA results are salient mainly for how similar they are across the board, to the CS baseline that lacks information from the previous article.
The rank-optimised neural network (NN) model (Section 4.2.3) returned accuracy scores that were a clear step up from both other LDA-based models (CF and WA). This was the case for both variants of NDCG and particularly so for hitrate, indicating that the NN model was optimised more for recall than precision and could possibly benefit from further reranking procedures. The NN model also distinguished itself from CF and WA models in the diversity and surprisal metrics. Results suggest the NN model produces more distinct lists (indicated by higher inter-list diversity) but that those lists are more topically homogenous (indicated by lower intra-list diversity and surprisal metrics). 6
The first cycle of research in our journey to find the best news recommender for BBC Mundo is complete. In Section 2 we have outlined the characteristics of the problem we address: a majority of non-signed in users; a large number of cold-start items; architectural constraints; and high quality demands, not only in terms of accuracy, but also in what concerns fairness and impartiality of recommendations.
One of the lessons we learned is that—unsurprisingly—
balancing the diferent aspects of our problem is hard. One
model may satisfy one of our requirements, whilst failing
to fulfil another. A pure collaborative filtering approach is
currently our best option to maximise ofline scoring
accuracy, but that comes at the cost of reducing diversity (and
a degree of recency, dependent upon how regularly we
retrain). Moreover, the performance of the CF model was not
entirely unexpected, as it has been shown [
Fundamentally, we believe there is scope to optimise the NN approach further so that it will perform more competitively with CF. To achieve this end we have multiple strategies. These fall into three categories: model architecture, data, and training improvements.
We know that learning to rank in a pointwise framework
is not optimal. Both pairwise and listwise approaches should,
in theory, achieve better results (see [
A key reason for this may be the under-representation
of negative examples in our training set. Adopting a higher
proportion of negative training examples may address this,
but also using more informed negative sampling techniques
may be required (such as weighted approximate-rank
pairwise loss [
Changes to our training process may also lead to
significant gains. In addition to increasing compute resources for
the exploration of the hyperparameter space, reducing the
training/testing window from the order of days/weeks to the
order of hours may provide greater scope for
experimentation (as has been reported elsewhere [
A further change that may prove fruitful is to expand the richness of the input to the user profile model. This may include expanding the size of the user journeys in the training set beyond 3 (a constraint which, incidentally, did not apply to the CF model at training), while also introducing contextual information about the user.
Finally, another direction to be explored in the future
regards content representation. In experiments not reported in
the current work, raw article text has been encoded through
an LDA model. However, our system architecture afords
enough flexibility to replace the current content model with
alternative article embeddings and test diferent approaches.
In particular, we are interested in taking sub-word
information into consideration [
Our results demonstrate the dificulty of acquiring all the desired characteristics of an ideal news recommender. Ultimately, we expect ensemble approaches may represent the best solution. Here we may take the cold-start benefits of the content-based neural approach and combine it with the less diverse but more accurate list of items generated by a collaborative filtering model. 7
In this paper we evaluated three approaches to provide news recommendations for the BBC Mundo service. The systems we have built are compatible with BBC serving infrastructure, a use case which includes millions of daily users and new content in the order of several thousand articles per week. In spite of our experiment being only the initial step of a journey that promises to be much longer, our models outperformed random, popularity-based, recency-based and content-similarity baselines. It is worth noticing though, that these results do not reflect current online performance. More work is needed to ensure these models, when deployed, meet the quality and editorial standards of the BBC. Future challenges do not concern only achieving higher accuracy, but also conforming to the principles of algorithmic fairness and impartiality. We encourage the community to collaborate in helping us create the way forward towards fair and engaging recommendations and applications with responsible machine learning.