In real-world scenarios, recommenders face non-functional requirements of technical nature and must handle dynamic data in the form of sequential streams. Evaluation of recommender systems must take these issues into account in order to be maximally informative. In this paper, we present Idomaar-a framework that enables the e cient multi-dimensional benchmarking of recommender algorithms. Idomaar goes beyond current academic research practices by creating a realistic evaluation environment and computing both e ectiveness and technical metrics for stream-based as well as setbased evaluation. A scenario focussing on “research to prototyping to productization” cycle at a company illustrates Idomaar's potential. We show that Idomaar simplifies testing with varying configurations and supports flexible integration of di erent data.
Increasingly, we witness a shift of recommender system research
toward large-scale systems developed for industry settings. The
trend was already well described in Amatriain’s 2012 tutorial on
building large-scale real-world recommender systems at ACM
RecSys 2012 [
Benchmarking the performance of recommender systems by these
aspects is challenging. Mark Levy pointed this out during his
keynote at the ACM RecSys 2013 workshop on Reproducibility and
Replication in Recommender Systems [
We introduce Idomaar to address this challenge.1 It enables
researchers to evaluate di erent algorithm with respect to multiple
criteria. The framework uses large-scale static data sets to simulate
live data streams, bringing o ine evaluation closer to online A/B
testing. By comparing the performance of recommender algorithms
operating in a live system (e.g., as studied in the living lab
NewsREEL [
The reference framework Idomaar is a tool to evaluate recommendation services in real-world settings. As opposed to typical recommender system evaluation which assumes static information, real-world applications process data in form of a stream of information. In fact, users, items, and interaction amid both collections continue generating events fed to the recommender system. For instance, new users register or existing users cancel their subscription; new items emerge; users consume items. Such information must be ingested and processed as soon as possible (e.g., by updating the recommendation models) in order to be available. All these messages are asynchronously handled. However, the system also has to synchronously serve incoming recommendation requests within strict time constraints. Practically, the whole flow of incoming messages is managed by means of queues.
Idomaar mimics the work flow of such real-world scenario by using state-of-the-art technologies (e.g., Apache Flume and Apache Kafka) to manage data streaming. The architecture has been split into four main modules, as depicted in Fig. 1: the data container, the evaluator, the orchestrator, and the computing environment.
The data container stores the data (entities and relations, in
accordance to the format defined in [
The recommender system runs within a virtual machine, referred to as computing environment, whose environment is created with Vagrant (https://www.vagrantup.com/) and where all required libraries are automatically provisioned with Puppet (https://puppetlabs.com/). The recommender system subscribes to the required Apache Kafka channel and receives the asynchronous messages (i.e., users, items, interactions, etc.). Recommendation requests are synchronously sent via a HTTP interface (or, alternatively, a 0MQ interface).
The evaluator compares recommendations generated by the computing environment with the ground truth. In addition to standard user metrics (RMSE, recall, precision), Idomaar evaluates business metrics (e.g., scalability, response time, throughput), so to provide a 360-degree evaluation of the recommendation infrastructure.
Finally, the orchestrator coordinates all processes, including launching and provisioning the computing environment, instructing the evaluator to split data into training, test, and ground truth, feeding the recommender system with the incoming messages in accordance to their timestamp, collecting the generated recommendations, and computing the quality metrics.
Moving from an o ine toward an online scenario (where data stream is not simulated from historical information, but the real flow of data) means either replacing the Apache Flume source with another one (e.g., that reads from log data) or ingesting the data directly into the Apache Kafka queue.
Various frameworks have been proposed to facilitate evaluating
recommender systems. Ekstrand et al. [
Idomaar was used in the “research to prototyping to operating”
cycle of a recommender system service provider to validate its
usefulness. The focus of the validation was on multidimensional
evaluation that simultaneously takes e ectiveness and technical
constraints into account. The only limitation identified was the
monitoring of performance metrics like CPU and memory usage
(cf. [
With respect to the cycle itself, we found that having a standard in terms of data formats and APIs increases the reusability of code in all phases and helps data scientists to produce code that can be transformed into e ective prototypes. Our “research to prototyping to operating” cycle shows:
Idomaar allowed easy testing using di erent datasets with different algorithms that share the same input types and subjects (e.g., implicit or explicit events, sessions or users).
The Idomaar format is flexible enough to change subjects, events types, or to integrate contextual information, both on events and on recommendation requests.
Idomaar can be considered as a suited tool for recommender system research: reuse of code speeds up prototyping and standardization of datasets helps merging di erent data sources.
In the future, Idomaar will go beyond classical recommender systems domains (e.g., movies or products) and consider additional types such as actions or navigation trees. Supporting generic objects and additional evaluation functions promise to establish Idomaar as standard research tool for recommender system. Such a standard could provide valuable support for the current trend of researchers participating in community-wide recommender system challenges. Idomaar has already been applied in such a challenge. 5.
In this paper, we present the Idomaar framework, which enables the e cient, reproducible evaluation of recommender algorithms in real-world stream-based scenarios. Idomaar simplifies the multidimensional evaluation taking into account precision-related metrics as well as technical aspects.
Acknowledgment: The research leading to these results was performed in the CrowdRec project, which has received funding from the EU 7th Framework Programme FP7/2007-2013 under grant agreement No. 610594.