The huge amount of news articles published every hour makes it hard for users to nd the relevant news matching the user's expectations. The main challenges when developing a recommender for the news domain are the continuous changes in the set of items, the contextdependent relevance of items, as well as the requirements with respect to scalability and response time. In this work, we present a scalable and distributable implementation of a real-time news recommender system based on the Akka framework. Our approach focuses on optimizing the recommendation precision. It is able to adapt to the continuous changes of the set of relevant news articles as well as it considers the di erent user preferences dependent from the hour the day. Our implementation ensures that tight response time constraints are ful lled and the system can be easily extended to streams of much larger volume. We implement three di erent recommendation algorithms namely, Most Popular Items, Most Recent Items, and Most Recent Items of the Most Popular Categories. A time-dependent delegation strategy is used for assigning requests to a recommender algorithm. We evaluate the developed recommender system in the CLEF-NewsREEL challenge 2016. The evaluation shows that the recommender performs very successfully; the developed recommender has won the online evaluation in several timeframes.
The amount of available information in the World Wide Web is more and more
increasing. This richness of information overwhelms users if not handled
sophisticatedly. A ltered view on the huge amount of available content helps users to
nd interesting items. Recommender systems are developed to support users in
ltering out the most relevant items matching the individual user's preferences.
Recommender algorithms are used in many modern e-commerce applications.
In this paper we focus on recommending news articles. With the spreading of
handheld devices, such as smartphones and tablets, and the ubiquitous
availability of internet connectivity, online news portal are becoming an important
channel for real-time information. The main challenges for recommender systems
in the news domain are the continuous changes in the set of potentially relevant
items as well as the limited accuracy of user tracking (since users do not have
to log in). User preferences often highly depend on the context and the speci c
domain. Furthermore, recommender systems in online settings must ensure tight
response time constraints and be able to handle heavy load peaks [
The CLEF NewsREEL challenge [
Task 1: The Living Lab Scenario In the Living Lab task participating teams
must provide recommendations for di erent news portals in real time. The teams
receive data describing freshly published articles as well as information about
the interactions between users and items. Four types of messages are used for
modeling the di erent types of information. The messages are transferred via
HTTP connections and encoded in the JSON format. Recommendation Requests
expect recommendable items as an answer and have to be replied within 100 ms.
Impressions and Item Updates inform about the activity on the publisher's web
sites. Error Messages indicate technical problems. Teams can register their
algorithms and then compare the performance via the Click-Through-Rate (CTR)
on a leader-board. The web portal allows participant registering new algorithms.
In addition, the portal visualizes the evaluation results. The portal is called the
Open Recommendation Platform (ORP) [
Task 2: The Simulated Stream Scenario For the o ine evaluation scenario
NewsREEL provides a large dataset consisting of a recorded data stream. The dataset
contains all interaction data for two months [
The remaining paper is structured as follows. In the next Section 2, we describe the scenario and the challenges in detail. Subsequently, we explain our approach and the details of the implemented algorithms in Section 3. The evaluation results are discussed in Section 4. Finally, Section 5 provides a conclusion and an outlook to future work. 2
Recommending news in real time is a challenging task due to the continuous
changes in the item set, the fuzzy user identi cation, the context-depended user
preferences as well as the requirements with respect to scalability and response
time. In 2015 several di erent recommender algorithms have been tested in the
NewsREEL challenge [
In order to consider context-dependent user preferences, we implement three di erent recommender algorithms. In addition, we learn a time-dependent delegation strategy selecting the most promising algorithm based on context parameters. In the next paragraphs we explain the di erent recommender algorithms in detail. 3.1
The Most Popular Recommender ranks news articles by the number of impressions. In order to take into account the continuous changes in the stream of items and user-item interactions, our algorithm uses a window-based approach. For each publisher, a sliding window is de ned consisting of a xed number of impressions. If a user clicks on an article, an impression is received and stored in the window. When the maximum number of impressions for the sliding window is reached, the oldest entry is deleted from the window. To provide recommendations a ranking of the news articles of the current window is computed. For each article, the number of impressions within the window is counted. The articles having received the largest number of impressions in the sliding window are recommended to users. 3.2
The Most Recent Recommender ranks news articles based on the most recent user-item interaction criterion. Similar to the Most Popular Recommender, a sliding window is used, separately optimized for each publisher. The implemented algorithm recommends the news items most recently added to the sliding window (that stores the news articles for the speci c publisher). 3.3
In order to provide recommendation focused on speci c user interests, we consider the categorization of the news items provided in the NewsREEL challenge. We re ne the Most Popular Recommender by computing the popularity separately for each category and each publisher. Only articles from the most popular category are recommended. In contrast to the most popular recommender, this approach provides recommendations focused on the category for that the user already has showed an interest. The disadvantage is, that a smaller number of data is aggregated when computing the item ranking since only the data assigned for the requested category is considered. This might reduce the stability of the provided recommendations due to the smaller number of items considered for each category. 3.4
We use a delegation strategy of assigning incoming requests to recommender algorithms. The delegation strategy is trained on the click events received in the most recent 15 minutes. The intersections of the clicked recommendations and the rankings from the three algorithms are compared. The largest intersection wins the competition and the winner algorithm is chosen to answer the next recommendation requests. 3.5
Based on our positive experiences in NewsREEL 2015 we decided to implement
the recommender algorithms in the Akka framework. The Akka framework
provides a distributed real-time engine implementing the actor model [
We evaluated our developed recommender component in the CLEF NewsREEL challenge.
The ORP website (http://orp.plista.com) allows researchers to register implemented algorithms. In addition, the websites lists and visualizes several key gures describing the performance of the recommender algorithms. The portal does not only shows the CTR and the number of recommendation requests for our algorithms; it also lists the CTR of the other teams actively participating in the online evaluation. The teams are ranked based on the Click Through Rate (CTR) describing the proportion of clicks to the number of answered requests. Two baseline algorithms have been used. The rst algorithm, named Berlin, uses a most popular strategy for recommending items. It considers the most recently requested 50 distinct items. The second baseline algorithm (maintained by the team named baseline) uses a most recent strategy implemented based on a ring bu er. We compare the performance of our recommender and the active recommender teams in the online challenge for two di erent timeframes.
Load Balancer s t s e u q e R Analyzed Recommenders Algorithms Two recommender implementations have been analyzed. The rst recommender, called xyz uses the Most Popular Items algorithm only. The second recommender, xyz-2.0 uses the delegation strategy. Recommendation requests are answered by the Most Popular Items, the Most Recent User-item interactions, or Most Popular Category algorithm. Analyzed Timeframes In the following paragraphs we analyze two timeframes in detail. The rst period includes 4 days in March (March 5th until March 8th). The second timeframe includes one week in April 2016 (April 10th until April 16th).
The CTR of the algorithm xyz and the baseline recommenders are visualized in Table 1). The results show that our recommender outperforms the baseline recommenders. The average variance of the xyz-recommender is slightly lower compared to the baseline recommenders. A-A Testing: Within the rst time frame, the xyz-recommender is registered four times in the ORP-interface. All instances map to the same recommender instance. Therefore, it is possible to compare the reached CTR. As the same instances are mapped, no di erences in the CTR are expected. We are comparing the CTR for one publisher, namely sport1.de, who has the largest number of requests in this period (99.43 %) in Table 2. The variance is shown for the four days and is generally on a low level. Nevertheless, it varies between the days. On the rst and last day it is notably higher compared to the other days. r t c _ s w e n n i l r e b e s i w i w c c b a
N
B Algorithm e n i l e s a b
M M t f l e d u t _ c f l d g i d a i r
P S F
Delegation Strategy Evaluation We analyze the delegation strategy used by the xyz-2.0 recommender. Table 4 shows the ratio of recommendations answered by the three implemented recommenders. We analyze at the same timeframe as depicted in Figure 3. The Most Popular recommender answered the majority of requests (> 95%). Only a small percentage of requests has been delegated to the Most Recent and the Most Popular Category recommender (< 5%). 5
In this paper we presented our recommender components developed based on the Akka framework. We evaluated two di erent versions of the recommender approach in the online scenario. In the analyzed timeframes our recommender outperformed the competing teams. This shows that the recommender approach has a big potential. Unfortunately, we could not participate in the complete evaluation period; so the analyzed timeframes are relatively short. We will keep on participating in the online evaluation to verify the signi cance of the observed CTR performance.
Recommender System Performance In the rst analyzed evaluation timeframe, our recommender xyz outperformed the CTR of the baseline recommenders; but there were other teams in the challenge reaching a better CTR than our recommender xyz.
In the second analyzed evaluation timeframe, we compared the performance of the algorithms xyz and xyz-2.0 with the other participating teams. Our algorithms reached the best CTR in this timeframe.
A-A Testing: For estimating the variance of the results, we performed an AA testing. Our A-A tests showed only minor di erences between the di erent instances of our algorithms. This indicates the variance of the CTR is low. Hence, there is a small random component in the data. This veri es the signi cance of the reached CTR in the analyzed timeframes.
The Delegation Strategy The algorithm xyz-2.0 uses a delegation strategy to answer recommendation requests either by a Most Popular, a Most Recent useritem Interaction or a Most Recent Category recommender. Our evaluation shows that the majority of requests have been delegated to the most popular algorithm. This is an explanation why the CTR of the algorithms xyz and xyz-2.0 are very similar. However, the use of the delegation strategy leads to a CTR improvement. Future Work In this paper we showed that a combination of di erent Most Popular algorithms reaches a high CTR. As future work, we plan to put a stronger focus on the delegation strategy and on optimizing the window size considered in the delegation component. In addition, we are working on considering additional aspects that can be used for measuring the popularity of news articles, such as number of clicks and total time spend on the news items.
The research leading to these results was performed in the CrowdRec project, which has received funding from the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement No. 610594.