This paper presents the recommendation algorithms used by the Insight UCD team participating in the CLEF-NewsREEL 2014 online news recommendation challenge.
Recommender systems have become an essential part of our day-to-day lives, when it comes to dealing with the overwhelming amount of information available, especially online. Recommender systems improve user experience and increase revenue in the context of online retail stores (Amazon, eBay), online news providers (Google News, BBC) and many more. In this work we present a wide range of online recommender algorithms and compare their performance in the scope of the CLEF-NewsREEL 2014 online challenge.
In CLEF-NewsREEL 2014, participating teams connect to the Plista Open
Recommendation Platform [
The rest of this article is organised as follows: in Section 2 we present some
related research in news recommendation; in Section 3 we provide a more detailed
description of the Plista ORP framework[
Further complications for collaborative ltering arise from the dynamic
nature of the users and the news articles themselves [
Content based approaches can run into problems where some measures of
similarity identify news articles which are in fact about di erent topics.
Extracting the constantly changing distribution of topics in the news presents a
challenge [
Plista provides the ORP platform for making live recommendations to a
number of their client sites. Plista communicates with the teams via an HTTP API,
by sending (JSON) messages. These messages are triggered when a user reads
an article or clicks on a recommendation (event noti cation), and whenever an
article is created or updated by the publisher (item update). Requests for article
recommendations (recommendation request ) are sent as message to the teams,
to which a response must be given within a certain timeframe (100ms). The
resulting dataset is unique in many respects, providing detailed user pro le
information for some users where available and cross-domain data from 11 di erent
news providers. The dataset is fully described in [
With each message type (request, event, update) the Plista ORP framework provides additional metadata regarding the current user and article. Although the Plista ORP API documentation lists almost 60 elds of information, in practice we found many were unclear, or not useful or detailed enough to use. Fields like Filter allowosr typify the ones we simply didn't understand (they had no description either), and popular demographic signals like age, gender and income, expressed as probabilities (male vs female, age and income brackets) turned out to be too vague to be of use in reality. In the end, we settled on 8 metadata elds for use in our recommenders: geo user The geolocation of the user time weekday The day of the week category The subcategory under which an article is published within a given domain, e.g. sport, world, local etc. time hour The hour of the day item source The unique item (article) identi er publisher The publisher (domain) of an article. keyword Keywords with their corresponding counts occurring in the article user cookie The user id, which is not necessarily unique
Note that the category, publisher and keyword elds only provide a numerical id rather than a textual representation. For category and publisher it was possible to exploit URLs in order to determine this information, but for keywords there was no way to uncover what the words actually were, or how they were chosen. Nonetheless, we found that despite this the keywords still provided a useful signal. In Section 5 we provide a deeper analysis of the data provided and used. 3.1
In this section we present the challenges of producing online news recommendations. In the section that follows, we detail the system architecture that we have implemented to cope with these challenges.
Traditionally, recommender systems are evaluated o ine, with plenty of time
to build complex models of user browsing behaviours and preferences. In
realtime online scenarios like CLEF-NewsREEL however, such leisure is not a orded;
not only do teams have to respond within a very tight timeframe (100ms), they
also have to deal with factors like the lack of rich user browsing history as users
are not obliged to register and therefore do not always have persistent pro les
or identi ers. Moreover, a user can access the news sites from di erent devices,
and many users can do so from the same shared device, further complicating
the ability to reliably track their browsing, as pointed out in [
Our system architecture, shown in Fig 1, is implemented in Python and is designed to accomplish our goals of scalability, speed and extensibility. The rst
Fig. 1: System Architecture point of interaction with Plista is NGINX, a load balancing HTTP server, which lters and redirects the di erent types of Plista messages to individual python processes (using the Tornado framework, one for each di erent types of message). All Tornado servers write to a permanent archive database, for which we use MongoDB. In order to make content-based recommendations we use an ElasticSearch instance, which keeps only data that is relevant to recommendable news articles, (articles often expire after a few days and get agged as unavailable for recommendation).
Rather than trying to guarantee accurate recommendations in under 100ms, we precompute the recommendations and store them in a Redis instance. For each of our recommendation algorithms, we have a long running process which continually reads the latest events and articles from the database to build its recommendations. With this o ine approach, there is a danger that we might send back a recommendation for an article which has expired during the time it took to compute the recommendations. To deal with this possibility, we refresh our recommendations in the background as frequently as possible so that our recommendations have a minimal lag. A typical refresh time for the most demanding recommenders (usually the content-based ones) is less than 5 minutes. We have constructed the system with a goal of compromising between accuracy or freshness and scalability. Our architecture is capable of sending recommendations back to Plista with a typical response time of about 3ms. We can easily scale it to handle many di erent algorithms for testing and evaluation purposes. 4
In this section we describe our recommender algorithms. For each we describe the algorithm by how it selects the candidate set i.e. the larger set of items that will be considered for recommendation, and its ranking strategy, i.e. how it ranks the candidate items for recommendation, before returning the top-N to the user. We refer to the target user as the user for whom the recommendations are required, and the current article as the news article the user is currently reading. We have implemented sixteen recommenders in total: six popularity-based recommender, seven content-based recommenders, a feedback-based recommender, a recencybased recommender and an ensemble recommender.
Before returning the top-N recommendations to the user, we apply three basic lters to the candidate set: Exclude items from other domains Recommendations must come from the same domain as the current article; we do not consider cross-domain recommendations although this is something we would like to investigate in the future, once we have an understanding of how single-domain recommenders work in this space.
Exclude already read articles We do not make recommendations of articles we know the target user has already read, however we do allow previously recommended articles to be re-recommended if the user did not click on the recommendation the rst time. As always, it's hard to interpret non-clicks in recommender systems; a more mature model might limit the number of times the same recommendation is ultimately shown to a user.
Exclude non-recommendable items These are items that are agged as nonrecommendable (usually older articles) by Plista.
Each of the six popularity recommenders rank their candidate set by item popularity, i.e. the number of users who have read an article. However, they di er in the way they compile their candidate sets. We have developed a basic as well as more advanced recommenders that use additional features. They can be described as follows: Popularity - Basic The candidate set is all items in the dataset. Popularity - Geolocation The candidate set is all items that have been read by users in the same geographical location as the target user.
Popularity - Item Categories Every item is associated with zero or more news categories (business, sports, politics, etc.). The candidate set is all items whose categories intersect with the target article's categories. Popularity - Weekday The candidate set is all items that have been seen at least once in the same week day as the target article.
Popularity - Hour The candidate set is all items that have been seen at least once in the same hour as the target article.
Popularity - Freshness The candidate set is all articles that have been published or updated in the last 24 hours.
Similar to the Lucene-based recommender described in [
Each of the content-based recommenders we deploy use the conventional
TFIDF vector space model[
articles must also be fresh i.e. published or updated in the last 24 hours. Content - Title and Summary + New As above but candidate set articles must be brand new i.e. published in the last hour, rather than fresh. Content - Keywords The candidate set is all articles, represented by their keywords provided by Plista (recall that Plista provides a set of keywords and their frequencies within a document, although how the keywords are selected, or what they actually are is not disclosed).
Content - German Entities The candidate set is all articles, represented by their entities; using AlchemyAPI we extract entities, in German, from the full text of the articles.
Content - English Entities The candidate set is all articles, represented by their entities. This time, we use Google Translate to rst translate the articles into English and then use OpenCalais to extract entities from the full text of the articles.
ommender, we represent articles using both their entities and the context surrounding them in the article. In OpenCalais context represents the relevant text before and after an entity.
Positive Implicit Feedback The candidate set is all articles that have been successfully recommended in the past (by any team's algorithm) to some user, i.e. clicked by the user. The more popular an article is as a recommendation, i.e. the more clicks it has, the higher the algorithm ranks it. Most Recent The candidate set is all articles in the dataset. Candidate articles are then ranked in reverse chronological order of when they were published or updated.
Ensemble Recommendations are produced based on a combination of all the popularity- and content-based recommenders above. The candidate set is the union of candidate sets from each recommender. Candidate articles are then ranked using the sum of the rankings for the top n articles from each recommender. If, for any recommender, an article does not occur in the top n recommendations, it receives the maximum ranking of n, plus a penalty of 1, as in equation 1. We set n = 100 in these experiments.
rank(a) =
X r2recommenders (rankr(a) if a 2 recommendations(r) n + 1 otherwise
(1)
In this section we give a brief overview of the data we receive from Plista, in order to better understand the performance of our recommender algorithms. We will nish with a discussion on possible evaluation methods. 5.1
To illustrate the nature and dynamic quality of the data and how it informs our choice of recommender algorithms, we look closely at a one week sample, between Wed, 25 May '14 and Tue, 30 Jun '14. During this time period we observe 11 websites. They range from domain-speci c news e.g. sport, (sport1.de) to more general news (tagesspiegel.de) to online home and gardening stores (wohnen-undgarten.de). There were approximately 8 million visits from 1.75 million unique users to 400 thousand unique articles published. The mean number of observed articles per user is approximately 4, and the data sparsity is 99:99%.
Over 90% of the tra c generated cumulatively from the websites is event noti cations (impressions and clicks). This tra c is unevenly distributed across the domains, with sport1.de contributing almost 40% and together with ksta.de and channelpartner.de over 80% of all tra c. We nd that most of the users that visit tagesspiegel.de also visit most of the rest of the websites, indicating the broad appeal of this domain. Surprisingly enough, there is a noticeable overlap between users of sport1.de (sports website) and wohnen-und-garten.de (online home and gardening store). On the other hand, users who frequent motor-talk.de rarely visit any of other websites. The uneven patterns of cross-domain tra c are certainly interesting, but for simplicity we have restricted our algorithms to only make recommendations in the same top-level domain for this work. 5.2
We show the average and maximum CTR results as reported by Plista in Fig. 2, and we compare the CTR with our algorithms in Table 1. We also show the precision results from an o ine evaluation we additionally performed, as we discuss later in this section. With the objective of testing many algorithms the nal averaged CTR < 1% is not particularly competitive. We can also see that there is a noticable di erence between the average and maximum CTR. This could be due to a number of reasons including network congestion, server overload and change in algorithm performance based on features such as the domain, time of the day, user geolocation, etc. In future work we plan to get a better understanding of the speci cs of the problem and bring the average CTR closer to the maximum. In contrast, the scalability of our architecture is apparent in the high number of requests that we were able to respond to.
We show the distribution of article popularity in Fig. 3(a). Although, most articles have very low popularity among users, there are plenty with higher popularities suggesting that popularity-based recommenders should perform comparatively well and that is what we nd - three of the top ve of our recommenders are popularity-based.
The complex dynamics of the news cycle are apparent in the relative performances of our popularity recommendations based on di erent timeframes. We nd better performance recommending articles that are popular over the course of a day compared to the course of an hour, indicating particular daily cycles in users' news consumption. It will be interesting to examine further the optimal timeframe over which recommendations should be made, and how this varies across each domain.
Our popularity recommendations based on geographic location of the user (see Fig. 3(b)), also performed well, and this highlights the importance of accounting for local and spatial contexts in news recommendations. Similar performance is found with category-based recommendations. Our intuitive understanding is that users tend to read news from the same category, and to further test this idea we looked at a transition matrix for user clicks from category to category in a single domain Fig. 4 (we chose the news domain ksta.de). We clearly see just a few dominant categories and clicks between these categories account for the vast majority of items that are read. However, it is clear that there are important patterns of activity between certain less popular categories and future recommender algorithms will attempt to exploit these.
The strong correlations in the click-transitions of a domain's categories suggests that content based recommenders are likely to have some success. We nd high CTR for our content-based algorithms Content - Title and Summary, although the best performing variant is one which returns fresh recommendations. While users seem to express some degree of preference for articles that are similar, the behaviour is very di erent across domains and also changes over time.
Content-based recommenders have shown to be quite successful in the past. Our Content - Title and Summary recommenders perform among the best of our algorithms. This shows that users enjoy articles that are related to the target article. The best performing variant is one which returns fresh recommendations.
Alchemy and OpenCalais are entity extraction services which tag text and use linked data sources to identify entities in text such as people, companies, etc., and relationships between entities. While each service returns largely similar results, there are di erences which can be signi cant (noticeable gaps in knowledge about non-English entities). The resulting recommendations achieve comparatively good scores, although it should be noted that for technical reasons we did not run all of our entity recommenders for the entire duration of the competition. Based on the maximum CTR they achieved (2.44%) we expect to look more closely at tuning these algorithms for further use.
The fact that the Content - German Entities outperforms Content - English Entities suggests that something is lost during the translation of articles into another language, even with entities which one might expect would be more reliably translated than full text. We also nd that including contextual text that OpenCalais provides reduces recommendation accuracy. This is likely because this text rstly is not cleaned and contains stopwords, and secondly is likely to be more useful for identifying sentiment associated with entities, rather than more accurately classifying what the article is about.
The Plista competition provides a single evaluation metric - CTR. There are
many possible approaches to evaluation [
Article popularity (a) Distribution of the popularity of articles. The most popular articles (the ones to the right on the x-axis) come from sport1.de and are about football.
2.5M rs2.0M e su1.5M fo1.0M .o0.5M N 0.0M 47 50 84 15 6 8 9 6 2 0 1 9 2 7 2 4 0 75 66 95 8 8 6 8 488 488 488 588 588 688 788 741 596 190 788 788 788 8 8 8 18 18 14 18 1 1 1 1 1 1 1 2 1 4 1 1 1 18 18 18
20 most common geolocations (b) Histogram of number of users seen for the top 20 most common geolocations. We can see that there is a considerable amount of users concentrated at the big German cities, for example almost 2.5 million user could be located at Berlin and over 1.25 million at Munich. CTR provided by Plista, and provides us with many interesting suggestions for further work to improve our algorithmic techniques and evaluations. 6
We have presented our results for the CLEF-NewsREEL 2014 Challenge. We have described our scalable architecture for delivering recommendations and detailed various algorithmic approaches that we have taken. We have highlighted some of the issues with the dataset which impact on the type of algorithms we can implement. For future work we intend to examine more closely the user features which are most relevant for collaborative ltering approaches. The dynamic click behaviour associated with browsing patterns is a rich area to exploit, and we also intend to improve on our content-based algorithms with better entity detection and similarity measures.
The Insight Centre for Data Analytics is supported by Science Foundation Ireland under Grant Number SFI/12/RC/2289