The classical strategy to evaluate the performance of a Recommender System is to measure the error in rating predictions. But when focusing on a particular dimension in a recommending process it is reasonable to assume that every prediction should not be treated equally, its importance depends on the degree to which the predicted item matches the deemed dimension or feature. In this paper we shall explore the use of weighted Mean Average Error (wMAE) as an alternative to capture and measure their e ects on the recommendations. In order to illustrate our approach two di erent dimensions are considered, one item-dependent and the other that depends on the user preferences.
Several algorithms based on di erent ideas and concepts
have been developed to compute recommendations and, as
a consequence, several metrics can be used to measure the
performance of the system. In the last years, increasing
efforts have been devoted to the research of Recommender
System (RS) evaluation. According to [
For this purpose, two standard metrics [
But, these metrics consider that the standard deviation of the error term is constant over all the predictions, i.e. each prediction provides equally precise information about the error variation. This assumption, however, does not hold, even approximately, in every recommending application. In this paper we will focus on the weighted Mean Absolute Error, wMAE, as an alternative to measure the impact of a given feature in the recommendations1. Two are the main purposes for using this metric: On the one hand, as an enhanced evaluation tool for better assessing the RS performance with respect to the goals of the application. For example, in the case of recommending books or movies it could be possible that the accuracy of the predictions varies when focusing on past or recent products. In this situation, it is not reasonable that every error were treated equally, so more stress should be put in recent items. On the other hand, it can be also useful as a diagnosis tool that, using a \magnifying lens", can help to identify those cases where an algorithm is having trouble with. For both purposes, di erent features shall be considered which might depend on the items, as for example, in the case of a movie-based RS the genre, the release date, price, etc. But also, the dimension might be user-dependent considering, for example, the location of the user, the users' rating distribution, etc.
This metric has been widely used for evaluation of model
performance in several elds as meteorology or economic
forecasting [
The objective of this paper is to study the use of a
weighting factor in the average error. In order to illustrate its
functionality we will consider simple examples obtained using
four di erent collaborative-based RS (using Mahout
implementations): i) Means, that predicts using the average
rat1A similar reasoning can be used when considering squared
error, which yields to the weighted Mean Root Squared
Error, wRMSE.
ings for each user; ii) LM [
A weighting factor would indicate the subjective importance we wish to place on each prediction, relating the error to any feature that might be relevant from both, the user or the seller point of view. For instance, considering the release date, we can assign weights in such a way that the higher the weight, the higher importance we are placing on more recent data. In this case we could observe that even when the MAE is under reasonable threshold, the performance of a system might be inadequate when analyzing this particular feature.
The weighted Mean Absolute Error can be computed as wM AE =
PiU=1 PjN=i1 wi;j
PiU=1 PjN=i1 wi;j abs(pi;j ri;j ) ; (1) where U represents the number of users; Ni, the number of items predicted for the ith-user; ri;j , the rating given by the ith-user to the item Ij; pi;j , the rating predicted by the model and wi;j represents the weight associated to this prediction. Note that when all the individual di erences are weighted equally wM AE coincides with M AE.
In order to illustrate our approach, we shall consider two factors, assuming that wi;j 2 [0; 1].
Item popularity: we would like to investigate whether the error in the predictions depends on the number of users who rated the items. Two alternatives will be considered: i+ The weights will put more of a penalty on bad predictions when an item has been rated quite frequently (the items has a high number of ratings). We still penalize bad predictions when it has a small number of ratings, but we do not penalize as much as when we have more samples, since it may just be that the limited number of ratings do not provide much information about the latent factors which in uence the users ratings. Particularly, for each item Ii we shall consider its weight as the probability that this item were rated in the training set, i.e. wi = pr(Ii). i{ This is the inverse of the previous criterion, where we put more emphasis on the predictions over those items with fewer ratings. So the weights are wi = 1 pr(Ii).
Rating distribution: It is well known that the users does not rate the items uniformly, they tend to use highvalued ratings. By means of this feature we can measure whether the error depends on the ratings distribution or not. Particularly, we shall consider four di erent alternatives: rS+ Considering the overall rating distribution in the system, putting more emphasis on the error in the predictions on those common ratings. So the weights are wi = prS(ri), ri being the rating given by the user to the item Ii. rS{ Inversely, we assess more weight to the less common ratings, i.e. wi = 1 prS(ri). rU+ Di erent users can use a di erent pattern of rating, so we consider the rating distribution of the user, in such a way that those common ratings for a particular user will have greater weights, i.e. wi = prU (ri). rU{ The last one assigns more weight to the less frequent rating, i.e. wi = 1 prU (ri).
Figures 1-A and 1-B present the absolute values of the MAE and wMAE error for the four RSs considered in this paper. Figure 1-A shows the results where the weights are positively correlated to the feature distribution, whereas Figure 1-B presents the results when they are negatively correlated. In this case, we can observe that by using wMAE we can determine that error is highly dependent on the users' pattern of ratings, and weaker when considering item popularity. Moreover, if we compare the two gures we can observe that all the models perform better when predicting the most common ratings. In this sense, they are able to learn the most frequent preferences and greater errors (bad performance) are obtained when focusing on less frequent rating values. Related to item popularity these di erences are less conclusive. In some sense, the way in which the user rates an item does not depend of how popular the item is. 2.1
Another di erent alternative to explore the bene ts of using the wMAE metric is to consider the ratio between wMAE and MAE. In this sense, denoting as ei;j = abs(pi;j ri;j ), we have that wMAE/MAE is equal to wM AE=M AE =
Pi;j wi;j ei;j = Pi;j ei;j
Pi;j wi;j =N :
Taking into account that we restrict the weights to take its value in the [0; 1] interval, the denominator might represent the average percentage of mass of the items that is related to the dimension under consideration whereas the numerator represents the average percentage of the error coming from this feature. So, when wMAE > MAE we have that the percentage of error coming from the feature is greater than its associated mass, so the the system is not able to predict properly such dimension. When both metrics are equal this implies that the expected error is independent of the feature.
In Figure 1-C we present the values of the wMAE/MAE where, again, we can see that there exists a dependence between the rating and the error. The error associated to the common ratings are less than the relative importance of this feature in the system whereas for less common ratings the system is not able to perform good predictions, being greater the relative error than its associated weights. This situation does not hold when considering item popularity.
Figures 1-D and 1-E present an scatter plot that relates the relative weights (horizontal axis) to the relative error (vertical axis) for each user in the system and for each RS used2. Particularly, in Figure 1-D we are considering rU+ as weighting factor. Note that, since there are 5 possible ratings, the relative weight is equal to 0.2 when all the ratings are equally probable and its value increases with the importance of the most used ratings. In this gure we can see that both percentage of mass and the percentage of error are positively correlated, being wMAE/MAE < 1 for most of the users. Moreover, there is a trend to improve the predictions for those users with higher relative mass (for example, we can see how the regression line for the LM model 2We have included all the users with at least 10 predictions. 0,6 0,5 rrr o e 0,4 e v ilt a e 0r,3 0,2
A D
Means
LM
Slope One
SVD LM Lineal (LM ) SO SVD
i
F 0,10,1 0,2 0r,3elative0,w4eight0,5 0,6 0,7 0,0000,50005 0,0010 0,0015 0,0020 relative weight 0,0025 0,0030 0,0035 rUrS+ rSigets further away3 from the line y=x). In some way we can conclude that recommendation usefulness of the rating distribution is consistent for all the users and RS models. On the other hand, Figure 1-E considers i+ as weights. In this case, although weights and error are positively correlated, there exists signi cant di erences between di erent users. This result is hidden in the global measures. 2.2
Although wMAE might give some information about how the error has been obtained, there is no criterion about what a good prediction is. In order to tackle this situation we propose the use of the relative rather than the absolute error, i.e. the weighted Mean Absolute Percentage Error, wMAPE. Then, given two models, M 1 and M 2, the relative metric is de ned as wM AEM1=wM AEM2. In this metric, the less the value, the greater the improvements. Thus, if we x the model M2 to be a simple model (as the average rating) we obtain the wMAE values in Figure 1-F. From these values, we can obtain some conclusions as for instance that LM ts better the common user's preferences (rU+), whereas Slope One and SVD are more biased toward the overall rating distribution in the system (rS+). Similarly, we found that better improvements, with respect to the average ratings, are obtained when focusing on less frequent ratings. Finally, with respect to item popularity all the models obtain better improvements when considering the most popular items, although these di erences are less signi cant.
In this paper we have explored the use of weighted Mean Average Error as a means to measure the RS's performance by focusing on a given dimension or feature, being able to 3The other models perform similarly, but we have decided to not include these regression lines due to clarity reasons. uncover speci c cases where a recommendation algorithm may be having suboptimal performance. This is a very useful way to know the origin of the errors found in the recommendations and therefore useful for improving the RSs, although its main problem is that it is not absolute as MAE. 4.