Recommender systems have become an important personalization technique on the web and are widely used especially in e-commerce applications. However, operators of web shops and other platforms are challenged by the large variety of available algorithms and the multitude of their possible parameterizations. Since the quality of the recommendations that are given can have a significant business impact, the selection of a recommender system should be made based on well-founded evaluation data. The literature on recommender system evaluation o↵ ers a large variety of evaluation metrics but provides little guidance on how to choose among them. This paper focuses on the often neglected aspect of clearly defining the goal of an evaluation and how this goal relates to the selection of an appropriate metric. We discuss several well-known accuracy metrics and analyze how these reflect di↵ erent evaluation goals. Furthermore we present some less well-known metrics as well as a variation of the area under the curve measure that are particularly suitable for the evaluation of recommender systems in e-commerce applications. H.4 [Information Systems Applications]: Miscellaneous; D.2.8 [Software Engineering]: Metrics-complexity measures, performance measures General Terms
Theory recommender systems, e-commerce, evaluation, metrics, measure, area under the curve, auc, informedness, markedness, matthews correlation, precision, recall, roc
There is a large variety of algorithms for recommender
systems that were published in research and have been
implemented by the industry. Almost every author claims that
a particular algorithm or implementation is superior to
another in a certain respect, which makes it di cult to choose
among them. Currently a comprehensive and objective
comparison of existing recommender systems is hard to find and
the available results are sometimes contradictory for di↵
erent data sets or metrics. How e cient and successful a
specific recommender system is also depends on the specific
purpose of a recommender system and the characteristics of
the domain it is applied to. It is very unlikely that there is a
single best solution for any domain and context. If we want
to increase the e↵ ectiveness of recommendations we have to
determine the best fit for a given scenario through thorough
evaluation of available algorithms and parameterizations.
This is particularly important for the usage of recommender
systems in e-commerce applications where the choice of
algorithms can have a significant business impact. In a web
shop a better recommender system can have a direct e↵ ect
on the company’s revenue since the recommendations can
significantly influence the users’ buying decisions [
In this paper we will focus on the specific evaluation demands of recommender systems in e-commerce applications. We discuss the importance of defining a su ciently detailed goal and analyze which aspects of the recommender’s user interface and the used preference data influence a reasonable choice of metrics. We give an overview of applicable accuracy metrics, explain how to choose among the large variety and highlight some metrics that are particularly well-suited to the evaluation of recommender systems. In order to discuss accuracy metrics in detail a discussion of non-accuracy measures has to be omitted due to space constraints. 2.
Over time numerous quantitative metrics and qualitative techniques for o✏ ine and online evaluations of recommender systems have been published in research. We will start by giving a short overview of some of the most important publications that are relevant to this paper.
Herlocker et al. provide a comprehensive overview of the
existing methods for evaluating collaborative filtering systems
[
Cremonesi and Lentini present an evaluation methodology
for collaborative filtering recommender systems [
Konstan et al. summarize findings about the usage of
automated recommender systems for information-seeking tasks
[
Kohavi et al. provide a survey and practical guide for
conducting controlled experiments on the web [
A further valuable source for metrics and measures is the
literature on information retrieval. It is, by its very nature,
concerned with the quality of search results and provides
insight into evaluation methodologies and metrics (e.g.[
The evaluation of recommender systems has emerged as
an important topic as more and more algorithms and
techniques for recommenders systems are presented. Many
different evaluation metrics can be applied in evaluations,
although some (e.g. RMSE, MAE and precision) are more
frequently used than others. However, authors rarely justify
their particular choice and little guidance is o↵ ered on how
to choose a metric for a specific purpose. Some notable
exceptions are the comparison of metrics given by Herlocker et
al. [
The first step of an evaluation should be to define its goal
as precisely as possible. Although this seems rather obvious,
we would like to emphasize this step since it is often not
executed with su cient care in evaluations in general. We can
only choose one or several adequate metrics and interpret
their results, if we have set the purpose of our evaluation
beforehand. The results of popular evaluation metrics such
as RMSE, precision or F1-measure do not necessarily lead us
to the best algorithm for our purpose, as we try to illustrate
later on. Only if the set of metrics accurately reflects the
specific objectives that are connected to the recommender
system in our evaluation and the concrete usage scenario,
can we be sure that the obtained results are useful [
Objectives for Recommender Usage
Having a clear conception of the objectives we want to achieve by designing, implementing or applying a recommender system is a worthwhile e↵ ort, even if we recognize that it may be too di cult or expensive to measure them accurately. In many cases we may discover that only the results of a large scale field study would accurately reflect our evaluation goal. Nevertheless a defined goal will help us in selecting the most appropriate metric or set of metrics that allows us to obtain the most precise measurement of the recommender’s suitability for the usage scenario which can be achieved with a reasonable e↵ ort.
Common reasons for implementing a recommender system
are the desire to improve user satisfaction and to increase
the economic success of a platform. Although both goals
are interrelated they may be competing in some scenarios.
As an example, in e-commerce a recommender may either
determine the top recommendations based on the best
priceperformance ratio for the customer but it may also show the
products that are likely to lead to the highest revenue for
the business. For this purpose commercial recommenders
for web shops often consider a reward attribute for items
that models how much the company profits from a sale of a
certain item. This information can be used e.g. in
combination with classification accuracy metrics (see Section 4.2).
A recommender system in a live environment usually has
to be optimized with regard to various other objectives that
are related to the technical performance and the system’s life
cycle such as responsiveness, scalability, peak load,
reliability, ramp-up e↵ orts, maintainability, extensibility and cost
of ownership [
Analyzing the Recommender System and its Data
In order to further specify our evaluation goal we have to take a closer look at the recommender’s task, its interface and the utilized data.
What is the recommender’s task and how does the system interact with the user? In our opinion the most common usage scenarios for recommender systems are prediction, ranking and classification tasks which manifest themselves in di↵ erent types of user interfaces.
In a prediction task the focus lies on predicting ratings for unrated items (e.g. movies, music or news articles) of a user and showing these predictions to him. For example, a movie rental company might show predicted ratings to users in order to aid them in their decision making.
In a ranking task the recommender tries to determine an order of items, often with the purpose of creating a top-k list of items. This task is particularly common in e-commerce applications, where a top-k or unlimited ordered list of recommended products is shown in a sidebar or on a dedicated page. But also web portals for news, content and multimedia make heavy use of top-k lists. A further interesting ranking task for a recommender is to order a limited set of search results according the user’s predicted preference. In a classification task the recommender determines a set with a limited (often fixed) number of recommended items with no particular order among them implied. E.g. articles or products that are predicted to be of interest to a user are highlighted or annotated on a web page. A top-k list of recommended items can be seen as a classification task as well, especially if the number of items is very small and the order is less prominent e.g. recommended items are arranged in a grid or scrollable in two directions in a circular fashion (cf. Amazon).
Recommender systems are further used to determine similar items or similar users. E.g. in web shops on a product detail page usually a list of similar products is shown or a news web site often lists articles similar to the one currently shown. Recommending similar users is of particular interest for social web applications such as Facebook. These tasks which focus on the items and users themselves instead of the user’s ratings, can be seen as ranking or classification tasks depending on the chosen interface.
Realizing for which of these tasks the recommender system is used and what the user interface looks like is important for choosing the most appropriate metric. The number of displayed recommendations and their visual arrangement (To which extent does it convey a ranking?) should also be considered before choosing a metric. In many usage scenarios more than one of these tasks has to be fulfilled by a recommender system. Therefore it may be sensible to apply one accuracy metric for each of the usage scenarios to find a reasonable compromise or to even consider using di↵ erent recommendation algorithms for each task.
What kind of user preference data is used by the recommender? The preferences of a user can be gathered either through explicit or implicit ratings. While an explicit rating is made as a deliberate statement by a user who has the intention to express his or her opinion, an implicit rating is deducted from actions of the user that had a di↵ erent primary goal. If, for example, a user rates a movie on a web site, this is a direct expression of his or her opinion and preferences, so we consider it an explicit rating. Purchasing a product or clicking on a link to display an article usually has a di↵ erent primary goal than expressing a positive rating, so these actions are considered an implicit rating.
The user ratings are usually collected using either a numerical, a binary or a unary rating scale. Although other preference models such as textual reviews are conceivable they are rarely used in today’s recommender systems. A numerical rating is represented by a number from either a discrete or a continuous rating scale, in most cases with a limited range. Typical examples of a discrete rating scale are ratings on a scale from zero to five stars or Likert response scales that are commonly used in questionnaires. To be precise, these rating scales are actually ordinal scales, a fact which is ignored by predictive accuracy metrics (cf. 4.1) that make intensive use of ratios. An example of a continuous rating scale could be a slider that is set by a user and translated to a real value.
A binary rating scale allows users to assign items to two different classes (like/dislike). YouTube, for example, allows users to rate movies with either thumb up or thumb down. A unary rating, by contrast, allows users to assign items only to a single class, which is in most cases positive (e.g. like). A prominent example of an explicit unary rating is Facebook’s “Like”-button. Implicit unary ratings can be purchased products in a web shop or clicked links on a news page.
The important di↵ erence between binary and unary ratings is that unary ratings o↵ er no distinction between disliked and unrated items. With unary ratings we cannot tell whether a user actually dislikes an item or simply does not know the item or does not bother to rate it. In a large web shop, such as Amazon, a customer will only be aware of a small portion of the product catalog. Furthermore, other factors such as limited budget, limited time, external constraints, and products the user already owns determine whether a customer will actually buy a product he or she likes. Being aware of this di↵ erence and the implied biases is important when conducting an evaluation and interpreting the results of various metrics. 4.
OVERVIEW OF EVALUATION METRICS
Evaluation metrics for recommender systems can be
divided into four major classes [
Predictive Accuracy Metrics
Predictive accuracy or rating prediction metrics embark on
the question of how close the ratings estimated by a
recommender are to the true user ratings. This type of measures
is very popular for the evaluation of non-binary ratings. It
is most appropriate for usage scenarios in which an accurate
prediction of the ratings for all items is of high importance.
The most important representatives of this class are mean
absolute error (MAE), mean squared error (MSE), root mean
squared error (RMSE) and normalized mean absolute error
(NMAE) (cf. [
These predictive accuracy error metrics are frequently used for the evaluation of recommender systems since they are easy to compute and understand. They are well-studied and are also applied in many contexts other than recommender systems. However, they do not necessarily correspond directly to the most popular usage scenarios for recommender systems. There are few cases where users are in fact interested in the overall prediction accuracy. Recommender systems are more commonly used to display a limited list of top ranked items or the set of all items that have been rated above a certain threshold. Many recommendation algorithms are able to provide more accurate statements about a limited set of items that the user either likes or dislikes. The estimations for many other items are rather inaccurate but often also significantly less important to users. This applies in particular to e-commerce applications where we are usually more concerned with suggesting some products to customers that they will like in comparison to estimating the most accurate ratings for the large amount of items that customers would never purchase. 4.2
Classification Accuracy Metrics
Classification accuracy metrics try to assess the successful
decision making capacity (SDMC) of recommendation
algorithms. They measure the amount of correct and incorrect
classifications as relevant or irrelevant items that are made
by the recommender system and are therefore useful for user
tasks such as finding good items. SDMC metrics ignore the
exact rating or ranking of items as only the correct or
incorrect classification is measured [
A comprehensive overview of classic, basic information
retrieval metrics, such as recall and precision and further
improved metrics, is given in the survey by Powers [
The common basic information retrieval (IR) metrics are calculated from the number of items that are either relevant or irrelevant and either contained in the recommendation set of a user or not. These numbers can be clearly arranged in a contingency table that is sometimes also called the confusion matrix (see Table 1).
Precision or true positive accuracy (also confidence in data mining) is calculated as the ratio of recommended items that are relevant to the total number of recommended items: tp precision = tpa = tp + f p This is the probability that a recommended item corresponds to the user’s preferences. The behavior of this (and the fol
tp f p f n tn tp + f n f p + tn
tp + f p f n + tn
N lowing) IR metrics can be observed in Figure 1 which shows a potential ranking of four relevant items in a set of ten items by a recommender. In examples (a) to (j) the length k of the top-k list varies, while the item order remains fixed. Recall or true positive rate (also called sensitivity in psychology) is calculated as the ratio of recommended items that are relevant to the total number of relevant items: recall = tpr =
tp tp + f n This is the probability that a relevant item is recommended. The two measures, precision and recall, are inversely related which we notice if we vary the size of the set of recommendations (cf. Fig. 1). In most cases, increasing the size of the recommendation set will increase recall but decrease precision. Because of this mutual dependence it makes sense to consider precision and recall in conjunction with two other metrics that are called fallout and miss rate.
Fallout or false positive rate is calculated as the ratio of recommended items that are irrelevant to the total number of irrelevant items: fallout = fpr =
f p f p + tn This is the probability that an irrelevant item is recommended.
Miss rate or false negative rate is calculated as the ratio of items not recommended but actually relevant to the total number of relevant items: missRate = fnr =
f n tp + f n This is the probability that a relevant item is not recommended.
Just as precision and recall describe the recommender’s performance regarding the true positives, two similar metrics can be defined that measure how the algorithm behaves regarding true negatives. Since this corresponds to interchanging the true and false values of the underlying data for the precision and recall metric, they are called inverse precision and inverse recall.
Inverse precision or true negative accuracy is calculated as the ratio of items not recommended that are indeed irrelevant to the total number of not recommended items: inversePrecision = tna =
tn f n + tn This is the probability that an item which is not recommended is indeed irrelevant.
Inverse recall or true negative rate (also called specificity) is calculated as the ratio of items not recommended that are really irrelevant to the total number of irrelevant items: inverseRecall = tnr = = 1
fpr tn f p + tn This is the probability that an irrelevant item is indeed not recommended.
The F1-measure and F -measure try to combine precision and recall into a single score by calculating di↵ erent types of means of both metrics. The F1-measure or F1-score is calculated as the standard harmonic mean of precision and recall:
F1 =
2 1 1 precision + recall = 2 · precision · recall precision + recall
Moreover there is a large variety of metrics that are
derived from these basic information retrieval metrics [
AP =
PrN=1(P (r) ⇥ rel(r)) number of relevant documents Afterwards we calculate the arithmetic mean of the average precisions of all users to get the final mean average precision: MAP =
PM u=1 APu
M Various other means including the geometric mean (GMAP), harmonic mean (HMAP) and quadratic mean (QMAP) can be applied instead. All of these measures emphasize true positives at the top of a ranking list. This behaviour can be observed in Figure 2 (a) to (j) where the position of a single relevant item within a ranking of ten items is varied. Further derived measures that we will discuss later are ROC curves and the area under the curve (AUC) measure. 4.3
Rank Accuracy Metrics
A rank accuracy or ranking prediction metric measures the ability of a recommender to estimate the correct order of items concerning the user’s preference, which is called the measurement of rank correlation in statistics. Therefore this type of measure is most adequate if the user is presented with a long ordered list of items recommended to him. A rank prediction metric uses only the relative ordering of preference values so that is independent of the exact values that are estimated by a recommender. For example, a recommender that constantly estimates items’ ratings to be lower than the true user preferences, would still achieve a perfect score as long as the ranking is correct.
For the usage of rank accuracy metrics, it is important to
know whether they measure total or partial orderings. Most
rank accuracy metrics such as Kendall’s tau and Spearman’s
rho compare two total orderings. The problem with these
measures is that in most cases we are not provided with a
full ranking of all items. Many recommendation algorithms
can generate only a partial list of items that are most likely
to be preferred by a user. All other items would have to be
concatenated to the list in an arbitrary order. Furthermore,
the recommendation list can contain groups of items with a
similar rating that can appear in varying orders. The same
applies to the true preferences of a user. In order to create
a full ranking of the items all preference values for the user
have to be known. Since the user might express the same
rating for several items the list will again contain groups
of items that can appear in an arbitrary order. The largest
problem is posed by items for which no user rating is known.
These items could in fact hold an arbitrary place within the
ranking. Sometimes it is assumed that ratings for items
that are preferred are known, so that the unknown items
are concatenated to the end of the list. In general, however,
the unknown items could as well contain items that would
appear within the top ranks if rated by the user [
We think that rank accuracy metrics are very well suited for e-commerce applications if they allow to compare partial rankings in a meaningful way. In our implementation we used a variation of the Kendall’s tau metric for partial rankings on boolean classifications. That is, we compare the partial ranking provided by recommended and not recommended items with the partial ranking of relevant and irrelevant items. This boolean Kendall’s tau is in fact highly correlated to the AUC metric (e.g. Fig. 2).
RECOMMENDER SYSTEMS
A major problem with the frequently used metrics precision, recall and F1-measure is that they su↵ er from severe biases. The outcome of these measures not only depends on the accuracy of the recommender (or classifier) but also very much on the ratio of relevant items. In order to illustrate this we imagine an ideal recommender and its inverse and apply them to generate top-k lists of four items for varying ratios of relevant items (see Fig. 3). The examples show that the informatory value of all three measures varies. In extreme cases with a highly skewed ratio its value can be even misleading (cf. Fig. 3 e – h).
To solve this problem Powers [
Markedness combines precision and inverse precision into a single measure and expresses how marked the classifications of a recommender are in comparison to chance: markedness = precision + inversePrecision 1
tp tn = + 1
tp + f p f n + tn Informedness combines recall and inverse recall into a single measure and expresses how informed the classifications of a recommender are in comparison to chance: informedness = recall + inverseRecall 1
tp tn = + 1
tp + f n f p + tn
Both markedness and informedness return values in the range
[
The Matthews Correlation combines the informedness and markedness measures into a single metric by calculating their geometric mean: correlation = (tp · tn)
(f p · f n) p(tp + f n) · (f p + tn) · (tp + f p) · (f n + tn)
p
= ± informedness · markedness
The range of the Matthews Correlation is [
ROC curves provide a graphical representation for the
performance of a recommender system, an information
retrieval system or any other type of binary classifier. A ROC
curve plots recall (true positive rate) against fallout (false
positive rate) for increasing recommendation set size. An
in-depth discussion of ROC curves can be found in the
paper by Fawcett [
A perfect recommender would yield a ROC curve that goes straight up towards 1.0 recall and 0.0 fallout until all relevant items are retrieved. Afterwards it would go straight right towards 1.0 fallout while the remaining irrelevant items follow. The obvious aim is consequently to maximize the area under the ROC curve. The area under the curve (AUC) can therefore be used as a single measure for the overall quality of a recommender system. However, a frequently uttered point of criticism is that users are often more interested in the items at the top of recommendation lists but that the AUC measure is equally a↵ ected by swaps at the top or the bottom of a recommendation list (cf. Figures 2 and 4 e – h). This may be a disadvantage if we are mainly interested in finding the top ranked items and thus care mostly for the first part of the ROC graph. Therefore, in addition to the standard AUC measure, we implemented a slight variation of the AUC measure. This limited area under the curve (LAUC) measure uses the same approach but instead of measuring the complete area, it takes only the area under the first part of the curve into account. This is the part of the curve which is formed by the first k recommendations.
However, measuring this partial area is not easy due to several problems: • The measure has to be normalized in a certain way to be comparable. • The normalized measure should return one if all relevant items are retrieved first by the recommendation algorithm and fit within the recommendation list. • If the recommendation list contains only relevant items, then the area under the curve is in fact zero. Nevertheless the normalized measure should still return one. • Relevant items that are retrieved at the end of the list with no irrelevant items following do not add to the area under the limited curve.
A good solution for these problems is to generate just a limited recommendation list that only contains a fixed number of items and to assume that all other relevant items will be distributed uniformly over the rest of the ranking list until all items are retrieved. This means that we calculate the AUC measure for the first part of the ROC curve in the standard way until the first k recommendations have been retrieved. Then we take the end point of the ROC curve formed by the first k recommendations and draw a straight line to the upper right corner (see Fig. 5). The area under the curve that is situated to the right of the curve formed by the top-k list thus is a simple trapezoid.
The resulting LAUC measure is very useful for the evaluation of recommender systems that are applied to generate top-k lists of items (cf. Figures 2 and 4): • The measure returns one if all relevant items are retrieved within the top-k list. • The measure becomes minimal if no relevant items are retrieved within the top-k list. If all irrelevant items are retrieved first and fit within the top-k list the measure returns zero. • A top-k list that contains more relevant items will yield a higher score than a list with less relevant items, except if the length of the list is close to the total number of items. In this case the order of relevant and irrelevant items within the recommendation list would have a higher influence on the overall score. • If a relevant item moves towards the top of the list the measure increases. • A swap at the top or bottom of the top-k list has the same e↵ ect on the measure’s value. • All changes beyond the end of the top-k list are ignored by the measure.
A similar variation of the boolean Kendall’s tau, which considers only the top-k recommendations, is another useful and highly correlated metric (e.g. Fig. 2). 6. SOME GUIDELINES FOR CHOOSING A
METRIC
In order to choose an appropriate evaluation metric for a given recommendation scenario, it is helpful to answer the following questions: Is there a distinction between rated and unrated items? If all items are implicitly rated predictive accuracy metrics are not applicable, because there are no unrated items for which we can predict a rating and measure the accuracy.
Are items rated on a numerical or a binary scale? A binary rating scale usually suggests a classification or ranking task.
Are users interested in rating predictions or only in top-ranked items? If users only care about top-ranked (or lowest-ranked) items and not about individual rating scores for items this suggests a classification or ranking task. Although an algorithm may use predicted ratings internally, it has to succeed in estimating the top-ranked (or lowestranked) items and a higher error on rating predictions for items is acceptable as long as top-ranked (or lowest-ranked) items are identified correctly. Therefore an evaluation should focus on classification or ranking metrics.
Is a limited list of top-ranked items shown? If yes, a metric that measures the overall predictive accuracy or overall ranking accuracy is not appropriate. The exact rating predictions and ranking of other items are irrelevant to users and should not be considered by the metric. Do the recommended items have or imply an order? Users will usually consider recommendations in a certain order, in particular if many recommendations are shown. If this is the case, basic information retrieval metrics such as precision, recall, markedness and informedness are not sufficient since they ignore the order among the recommended items. A metric that considers the order of recommended items as well is more appropriate for this purpose. How fast does the user’s interest in lower ranked items decay? The metric should reflect the user’s decay in interest. MAP and GMAP, for example, emphasize the first recommendations in contrast to the AUC or Kendall’s tau measure that weighs swaps in lower and higher ranks in the same way (cf. Fig. 2). If users hardly look at the ranking of lower ranked items, a classification or ranking error for lower ranked items becomes irrelevant. 7.
CONCLUSION
In this paper we analyzed the relation between objectives for applying recommender systems and metrics used for their evaluation. We emphasized the importance of defining a precise goal for the evaluation and discussed how the data used by the recommender as well as its user interface a↵ ect the specific task for the recommender. We gave an overview of the three main classes of accuracy related evaluation metrics and discussed their applicability for di↵ erent types of recommender systems. In order to illustrate the specific advantages and disadvantages of various metrics discussed, we compared them using several informative examples. We proposed to utilize markedness, informedness and Matthews correlation as classification metrics since they are superior to precision, recall and F1-measure for most purposes. We presented a new variation of the area under the curve measure that is particularly suited for top-k recommendations which are used in many e-commerce applications. This limited area under the curve measure combines classification and ranking accuracy to create a better measure for this purpose. Furthermore, we provided some crisp guidelines that help to choose an appropriate evaluation metric for a specific usage scenario.
ACKNOWLEDGMENTS
The work presented in this paper is funded by the European Social Fund and the Free State of Saxony under the grant agreement number 080954843.