Interest Beat (inbeat.eu) is a service for recommendation of content. InBeat was designed with emphasis on versatility, scalability and extensibility. The core contains the General Analytics INterceptor module, which collects and aggregates user interactions, the Preference Learning module and the Recommender System module. In this paper, we describe InBeat general architecture, putting emphasis on its highperformance architecture that was used in the CLEF-NEWSREEL: News Recommendation Evaluation Lab.
The large amount of content to choose from causes the Information Overload problem for visitors of news websites. Based on the analysis of past usage patterns, recommender systems can make a personalized list of preselected content, alleviating the users of the e ort entailed in the process of choosing the content they should consume next and limiting the number of choices they need to make.
There are various existing recommendation systems and algorithms, both
in academia and industry.While recommender systems were used on large
ecommerce websites for more than a decade [
In this paper, we introduce the InBeat architecture and algorithms in the context of its participation in the CLEF-NEWSREEL: News Recommendation Evaluation Lab (further only Challenge), which focused on recommending news InBeat API InBeat Logic
Load Balancer
API Instance
API Instance
API
Instance
Business Logic Cache
Storage
Processing
Logic articles in real-time.In section 2 presents the system architecture and technical solution. Section 3 presents InBeat as a news recommender system in the Challenge's on-line task, including the details on performance. Section 4 presents our attempt at the o -line task. Finally, Section 5 provides a list of other InBeat use cases along with some re ections on the organization of the Challenge. 2
InBeat's was designed for scalability and low utilization of resources per one client. A simpli ed overview of the technical solution is given in Figure 1. The main entry point is load balancer nginx 4, which distributes the workloads across multiple instances of API applications. Nginx is focused on high concurrency, performance and low memory usage. It uses asynchronous event-driven approach and is able to provide high performance under high loads.
Instances are created with Node.js5, which is a platform for building fast and scalable applications. Node.js uses an event-driven, non-blocking I/O model that 4 http://nginx.org/ 5 http://nodejs.org/ makes it lightweight and e cient, suitable for data-intensive real-time applications that run across distributed devices. As storage we selected MongoDB 6, which met our demands on scalability, performance and high availability. Its key advantage is schema-less design, which allows to use custom set of attributes in an individual use case without the need to update the schema. MongoDB also supports map/reduce procedures that are used for aggregation of data and for providing di erent views on data by InBeat. We use MongoDB mainly as a storage of historic data (e.g. latest N thousands interactions per news portal or descriptions of items) and aggregated statistics (e.g. number of interactions per item etc.). The Cache is implemented with Redis7, a highly scalable in-memory, key-value store. It provides essential performance improvement under high workloads, reducing the latency of business logic including storage operations. 3
The CLEF-NEWSREEL: News Recommendation Evaluation Lab8 is focused on recommending news articles in real-time. The emphasis of the Challenge is on scalability and response time limitation. Recommendations had to be provided in real-time (within 100 ms), and the winning criterion was set to the total number of successful recommendations, rather than the prediction accuracy (clickthrough rate). There are practical problems with real time processing of recommendations that are not incurred when there is \unlimited time" to provide the recommendation. It is necessary to balance the architecture and technologies with the complexity of the involved algorithms. The InBeat platform deals with real time processing of data using special features of NoSQL databases and map/reduce principles. 3.1
Task De nition This section describes simpli ed de nition of the news recommender task. Inputs: The main inputs are the users' interactions and news item descriptions. { interaction(type; userId; itemId; context) where type = fimpressionjclickg and context describes the features of the user (e.g. browser version, geolocation, etc.) and special features related to items and their presentation (e.g. keywords, position). { item(itemId; domain; description) where domain is the identi er of items from the same group (e.g. news portal) and desciption provides more detailed information about items (e.g. title, text, time of last update).
Outputs: Set of recommended items for the speci c user who is reading the item within a given context.
{ (userId; itemId; context) ! fitemx; itemy; :::g 6 http://www.mongodb.org/ 7 http://redis.io/ 8 http://www.clef-newsreel.org/ 3.2 In this section we describe a set of algorithms we used in the Challenge. Top Interacted This algorithm is based on the daily popularity of news items. To avoid excessive e ect of high short-time popularity of one item the interactions are aggregated on a daily basis. This approach deals with an evolution of popularity over time and decrease an in uence of peaks appeared at the speci c days. We implemented the algorithm using simple incremental updates in a MongoDB collection represented as a triple (Date; ItemId; count). The result is s list of items sorted by the number of interactions.
Most Recent Since we are in the highly dynamic news domain, the recency of an article plays an important. Our baseline recency-based algorithm uses a simple heuristic based on the newest news item within the same group as the group of the item the user is reading at the time of the request. The results is ordered list of items sorted by creation time.
Rule Based Since this algorithm is more complex than the previous one, we decided to describe it in greater detail.
For each interaction(type; userId; itemId; context) stored in our database, we prepared one entry in the training dataset as described in Table 1. Interactions are described only by the contextual features that are provided by the platform (e.g. Location, Browser, ...) and by an identi er of item the user interacted with.
The training dataset was used to learn association rules. The contextual features could appear only in the rule body (antecedent) and the identi er of the item only on the right side of rule (consequent). We used association mining algorithm apriori implemented in R - arules 9. Example of a rule: isp = \281" ^ os = \431229" ! item = \1124541"
Additional mining setup is as follows. We used latest N thousands interactions as training dataset from our database. We experimentally set N to ve thousands. The apriori algorithm is experimentally constrained with minimal support of ve interactions, and minimum con dence of 0.2. 9 http://cran.r-project.org/web/packages/arules/
All discovered rules are imported into our simple rule engine. The engine nds all rules that match the contextual features of a recommendation request. The consequent of each matching rule represents a recommended item. The output is a list of unique item identi ers from the right side of the matching rules.
The association rule-based algorithm was added at the end of Challenge and thus did not participate in all evaluation weeks. 3.3
Performance In this section, we present the performance of the InBeat recommender in the Challenge. The metrics used in the Challenge to select the winning recommender systems was the cumulative number of clicks (number of successful recommendations) over the three di erent evaluation periods. The additional metrics provided by the organizers include number of impressions and click-through rate.
Sum of the number of impressions with the number of clicks can be interpreted as the performance of the systems { the ability to process large number of interaction on the server.
Figure 2 shows the network tra c on our server infrastructure within the last week of the Challenge. During this period, InBeat handled thousands of recommendation requests. The peaks in the graph correspond to the higher number of interaction in daytime. Note that the gap between Monday and Tuesday is caused by the maintenance break of our infrastructure. Figure 3 depicts the CPU load on the server. The server load was kept mostly under ten percent even in peak periods. InBeat was run on a single virtual machine assigned four Core i7@3.20GHz cores and 8GB of RAM.
Figure 4 presents the results for the last evaluation period. The table is sorted by the cumulative number of clicks. InBeat team is on the third position. The table provides only results that are aggregated per team participating in the Challenge. There are no speci c results for each recommendation algorithm. In click-through rate, the second metric, InBeat is on the fourth position.
Since the CTR reported in Figure 4 is the average for all algorithms, also report the numbers for the individual InBeat algorithms:
Fig. 3. CPU Load on InBeat server. { Top Interacted has 1:4% CTR, { Most Recent has 0:8% CTR, { Rule Based has 1:5% CTR.
The most successful algorithm is Rule Based, which we explain by the fact that it takes into account both popularity and contextual features. Most Recent is in uenced only by temporal aspects and Top Interacted takes into account only the popularity. 4
In addition to the on-line challenge CLEF-NEWSREEL o ered an o -line task.
The dataset consisted of 84 million records of recommendation data across
multiple portals [
Task de nition From the o -line dataset, we used a subset of 26,875 records.10 The data were preprocessed to the form shown at Table 1 and randomly split to a training 10 From the portal with the smallest number of records. dataset (90%) and test dataset (10%). The task was to predict the label (item viewed), given the description of the context. In our approach, we have not exploited the temporal relations between entries in the training dataset. Each row was considered as a separate training instance. 4.2
Algorithms We have used several standard approaches as a baseline, and compared it with the proprietary InBeat rule learning solution.
Baseline: Decision trees, SVM. We used the implementations of common machine learning algorithms available in RapidMiner 5: Decision trees (CHAID and the RapidMiner's "Decision Tree\), and SVM (LibSVM).
Rule learning { basic. R-arules implementation of the apriori algorithm, in the fastest setup described in Section 3. The resulting rule set is sorted according to con dence, support, and rule length (shorter is ranked better/higher). The top ranked rule matching a test object is used for classi cation.
Rule learning { with pruning. R-arules implementation of the apriori
algorithm. After learning, the rules are pruned using our database coverage pruning
implementation [
Rule learning { pruning with extension. Same as the previous setup, but the rules in the pruned rule set are additionally post-processed using rule extension (working version of our experimental algorithm). The rules are matched with test data using the same approach as in the basic setup.
Rule learning { pruning with extension and mixture classi cation. Same as the previous setup, but all rules matching the test object contribute to the nal classi cation using the rule mixture approach (a working version of our experimental algorithm). 4.3
Performance The results are depicted on Table 2. The baseline algorithms were run with default parameters in the RapidMiner environment.11 Rule learning was run with 0.02 minimum con dence and 0.001 minimum relative support threshold.
The worst results were provided by CHAID and Decision Tree algorithms, the best performing result was provided by SVM (with RBF kernel). 11 For CHAID and DecTree we tried several di erent parameter con gurations, but with no or negligible improvement.
SVM CHAID DecTree RL-Basic RL-PR RL-PR-Ext RL-PR-Ext-Mix Accuracy 0.15 0.02 0.02 0.02 0.06 0.08 0.11
The results clearly indicate that while our experimental rule-based system surpassed other symbolic learning algorithms incl. CHAID and Decision Tree, it was not able to beat the SVM with RBF kernel baseline. 5
In this paper we introduced the InBeat experimental recommender system. There is an on-going work on extending the set of recommendation algorithms available in InBeat and their customization to the constraints imposed by the Challenge platform, which remains open for further experiments.
It should be noted that InBeat or its modules are or were deployed also
in other use cases. The rst version of its component was conceived as a web
analytics solution compatible with Google Analytics [
Acknowledgement. This work was supported by the Czech Technical University grant SGS14/104/OHK3/1T/18, by the University of Economics, Prague by grant IGA 26/2011 and by the EC project FP7-287911 LinkedTV.