Recommender systems add value to vast content resources by matching users with items of interest. In recent years, immense progress has been made in recommendation techniques. The evaluation of these has however not been matched and is threatening to impede the further development of recommender systems. In this paper we propose an approach that addresses this impasse by formulating a novel evaluation concept adopting aspects from recommender systems research and industry. Our model can express the quality of a recommender algorithm from three perspectives, the end consumer (user), the service provider and the vendor (business and technique for both). We review current benchmarking activities and point out their shortcomings, which are addressed by our model. We also explain how our 3D benchmarking framework would apply to a speci c use case.
Recommender systems identify items suitable for speci c users in large content collections. Despite recent commercial and research e orts, a systematic evaluation model that addresses and considers all aspects and participants of the recommender system is still missing. In this paper we propose a 3D Recommender System Benchmarking Model that covers all dimensions that impact the e ectiveness of recommender systems in real-world settings. The concept builds on a study of benchmarking settings from research and industry and provides a common comparison of recommender systems, independent of setting, data and purpose. Our benchmarking concept captures three evaluation aspects which are shared in all recommender systems, independent of whether they are research systems or industrial products. As three main evaluation dimensions we identify user requirements, business requirements and technological constraints, each represented by a set of qualities which ensure the general applicability of these procedures. For each particular recommendation problem, the instantiation and relevance of these requirements should be speci ed.
The motivation behind this framework is the growing importance of recommender systems. Users cannot be assumed to have the necessary overview to specify their information needs in vast content collections. However, with a variety of data and the recommendation task, the comparison of algorithms, approaches and general concepts becomes infeasible due to the inherent di erences in requirements, design choices, etc. This calls for a comprehensive benchmarking framework that sets data- and task-speci c requirements driven by particular real-world applications.
The bene ts of benchmarking. Benchmarks formulate
standardized tasks making it possible to compare the
performance of algorithms. They have been highly successful in the
areas of information retrieval, e.g. Text Retrieval Conference
(TREC) [
Existing recommendation benchmarks. Today's benchmarks are limited by their simpli ed views of users and of data. The problem setting of the Net ix Prize1, groundbreaking at its time, was focused on a single functional requirement : the qualitative assessment of recommendation was simpli ed to the root mean squared error of predicted ratings. Its simpli ed view treated users as needing no further output from the recommender system than a rating on individual items. The data set was equally restricted to user ratings, additional information available in a real-world recommender system environment were not considered. Furthermore, the Prize did not take non-functional requirements into account, which arise from business goals and technical parameters of the recommendation service, though aspects as scalability, reactivity, robustness and adaptability are key for the productive operation of recommender systems.
The series of context-aware movie recommendation (CAMRa) challenges explored the usefulness of contextual data in recommendations. The 2010 challenge2 provided special features on the movie mood, movie location, and intended audience (Moviepilot track), as well as social relationship between users and user activities on a movie-related social site (Filmtipset track). The time of the recommendation was also considered as context (Week track). Although the challenges expanded the data sources used, the evaluation translated real-world user needs into the classi cation accuracy metrics to evaluate the system in the contest, and non-functional requirements of the solutions were not investigated.
The limitations of the Net ix Prize and CAMRa series are characteristics of currently existing benchmarks and data sets. The concept presented in this paper approaches this
User Requirements challenge by placing central focus on real-world user needs; large, heterogeneous, multi-source data sets and evaluating both functional (quality-related) and non-functional (technical and business goals-related) requirements.
In order to extend the state of the art of evaluation, we propose a concept for evaluation metrics that incorporates the needs from all perspectives in the recommendation spectrum. The concept de nes a set of benchmarking techniques that select the correct combination of (i ) data sets, (ii ) evaluation methods and (iii ) metrics according to a three dimensional requirement space: business models, user requirements and technical constraints, see Fig. 1.
Business models allow a company to generate revenue.
Di erent models lead to di erent requirements in terms of
the expected value from a recommender system. For instance,
in a pay-per-view video-on-demand business model, the goal
of the recommender system is to increase sales to allow the
company to maximize revenues. However, in subscriber-based
video-on-demand business models, the driving forces may be
to get users to return to the service in the future (a typical
showcase where recommender systems help [
User requirements re ect users' perspectives.
Recommenders are assets for user satisfaction and persuasion, i.e.,
they try to in uence a user's attitude or behavior [
Technical constraints. Recommender systems in reallife must take into account a number of technical requirements and constraints. These can be classi ed as data and system constraints, scalability and robustness requirements. Data constraints relate to the service architecture, e.g. satellite TV lacks a return channel for feedback, hindering the use of collaborative ltering algorithms. System constraints derive from hardware and/or software limitations, e.g. in a mobile TV scenario, the processing power in the hand-held device is limited; excluding resource-heavy algorithms on the client side. Scalability requirements derive from the need of instant recommendations to all users on all items. These requirements are particularly strict in linear TV, where viewers are used to quick responsiveness. Robustness requirements are needed to create good services, able to work in case of data or component failure in distributed systems. 3. 3.1
Existing evaluation methods for recommender systems can
be classi ed into system-oriented evaluation, user-oriented
evaluation or a combination of both [
In system-oriented evaluation (o -line) users are not involved in the evaluation, instead, a data set is partitioned into training and test sets. Using the training set, data points in the test set are predicted. In user-oriented evaluation (on-line) feedback from users interacting with the system is collected by explicit questions or implicit observing.
Competitions and challenges built around recommender systems are mostly organized to nd the most accurate models. As described in Table 1, recommender systems are mostly evaluated o -line and often, the business value of the technologies is not examined. Even though the accuracy may in uence user satisfaction and revenue increase indirectly, there exists no way to evaluate the dimensions of user requirements and business models. In most of the cases, the o -line evaluation scheme is chosen. Algorithms are often evaluated by error, ranking or classi cation accuracy measures. Many challenges (e.g. Net ix Prize) use explicit ratings to pro le users, other recommender scenarios (e.g. item-2-item recommendation) are not addressed. Technical constraints are uncommon in contests, the exception being the RecLab Prize3. If a certain method performs well on a data set, the integrability in a real-world system is still not addressed. This de ciency is partially solved by online testing methods (as seen in CAMRa): recommender systems were tested in a real environment, but an objective metric to show the real applicability of the tested system is missing. In the RecLab Prize, the evaluated metric is revenue increase generated by the system. The organizers also speci ed non-functional requirements to be eligible for the semi- nal (top 10 teams), but user requirements are not considered. These approaches all contain metrics and methods moving towards our 3D model, but none of them provide a comprehensive model. 3.2
On-line evaluation is the only technique able to measure
the true user satisfaction; conducting such evaluations is
however time consuming, and cannot be generally applied,
rather only to limited scenarios [
Classi cation metrics measure how well a system is able
to classify items correctly, e.g. precision and recall.
Predictive metrics measure to what extent a system can predict
ratings of users. As rated items have an order, predictive
accuracy metrics can be used to measure the item ranking
ability. Coverage metrics measure the percentage of items
for which the system can make recommendations [
A recommender system can recommend accurate items,
have good coverage and diversity and still not satisfy a user,
if they are trivial [
Challenge Task(s) Metric Net ix Prize minimize rating prediction error RMSE KDD-Cup'07 1: predict who rated what RMSE 2: predict number of ratings RecLab Prize Increase revenue revenue lift KDD-Cup'11 KDD-Cup'12 CAMRa'10 CAMRa'11 CAMRa'12 minimize rating prediction error RMSE split popular/unpopular items ErrorRate prediction followed users MAP@3 o -line click trough rate prediction MAE, AUC context-aware; 1: temporal, 2: MAP, P@N, o -line emotional, 3: social AUC & online group recommendation ErrorRate o -line rater identi cation nd users for speci c items impact on-line Mode o -line o -line
User indirect: error measure not addressed online & not addressed o -line o -line indirect: error measure not addressed nd interesting or irrelevant items exploring interesting users not addressed not addressed & sources ad targeting (CTR) contextual information in- not addressed not addressed uences preference group & target recommen- indirect: satisfaction not addressed dation split interesting and irrel- increase audience not addressed evant content
Business Technical
not addressed not addressed
detect trends & pop- not addressed
ular items
revenue lift
response/learning
time, scalability
not addressed
satisfaction are especially di cult to measure o -line.
Diversity is important for the usefulness of a recommendation
and therefore there is a need to de ne an intra-list similarity
metric [
Real-world recommender systems should satisfy (1 ) functional requirements that relate to qualitative assessment of recommendations and (2 ) non-functional requirements specied by the technological parameters and business goals of the service. Functional and non-functional requirements should be evaluated together: without the ability to provide accurate recommendations, no recommender system can be valuable. As poor quality has adverse e ects on customers, it will not serve the business goal. Similarly, if the recommender does not scale with a service, not being able to provide recommendation in real time, neither users nor service provider bene t from it. Thus, a trade-o between these requirements is needed for an impartial and comprehensive evaluation of real-world recommenders. Scalable recommenders provide good quality recommendations independently of the data size, growth and dynamic. They are able to (1 ) process huge volumes of data during initialization using computation resources linearly scalable with data size; and (2 ) serve large amounts of parallel recommendation requests in real time without signi cant degradation in service quality. In our model, scalability is found on the technical requirement axis.
Reactivity ensures good recommendations in real-time where the time threshold depends on the use case, typically in the range of 10{1000 ms. Adaptability is important to react for changes in user preferences, content availability and contextual parameters. In our 3D model, reactivity and adaptability belong to the user requirement axis.
Robustness is needed to handle partial, missing or corrupted data both in the system initialization and operational phases. Robustness belongs to the business axis of our model.
Generally speaking, none of the requirements are
mutually exclusive, instead, optimization should be based on a
combination of them { adapted for the setting in which the
recommender system will be deployed [
This example of a Video-on-Demand (VoD) service from the IPTV industry serves as a potential scenario for our model. Business goals include increased VoD sales and customer retention, but may have additional aspects (promoting content). The technical constraints are partly speci ed by the middleware and the hardware/software con guration of the service provider, these all in uence the response time of the service which is crucial. Via the service interface, the user gets recommendations based on the context, which might be translated into di erent recommendation tasks. From a user perspective, easy content exploration and context dependent recommendation may be the most important aspects.
We proposed a 3D Recommender System Benchmarking model that extends the state-of-the-art and addresses both functional and non-functional real-word application-driven aspects of recommender systems. Following the proposed concept, the benchmarking activities within the community will encompass the full range of other recommender system use cases and algorithmic approaches. The comprehensive evaluation methodology will boost the development of more e ective recommender systems, and make it possible to focus research resources productively and for industry technology providers to increase the uptake of recommender technology.