In this paper, we propose new metrics to accurately measure the concentration reinforcement of recommender systems and the enhancement of the \long tail". We also conduct a comparative analysis of various RS algorithms illustrating the usefulness of the proposed metrics.
Even though many researchers have focused on developing
e cient algorithms for generating more accurate
recommendations, there is increasing interest in metrics that go beyond
this paradigm [
Several measures have been employed in prior research in
order to measure the concentration reinforcement and
popularity bias of RSes as well as other similar concepts. These
metrics include catalog coverage, aggregate diversity, and
the Gini coe cient. In particular, catalog coverage
measures the percentage of items for which the RS is able to
make predictions [
However, these metrics do not take into consideration the
prior popularity of candidate items and, hence, do not
provide su cient evidence on whether the prior concentration
of popularity is reinforced or alleviated by the RS. Moving
towards this direction, [
To precisely measure the concentration reinforcement (popularity) bias of RSes and alleviate the problems of the aforementioned metrics, we propose a new metric as follows:
X 1 s(i) i2I 2 Pj2I s(j)
0 1 A+ 2 N 1 rN (i) jU j
0 1 A ; where s(i) is the prior popularity of item i (i.e., the number of positive ratings for item i in the training set or correspondingly the number of prior sales of item i), rN (i) is the number of times item i is included in the generated topN recommendation lists, and U and I are the sets of users and items, respectively.1 In essence, following the notion of Jensen-Shannon divergence in probability theory and statistics, the proposed metric captures the distributional divergence between the popularity of each item in terms of prior sales (or number of positive ratings) and the number of times each item is recommended across all users. Based on this metric, a score of zero denotes no change (i.e. the number of times an item is recommended is proportional to its prior popularity) whereas a (more) positive score denotes that the generated recommendations deviate (more) from the prior popularity (i.e., sales or positive ratings) of items.
In order to measure whether the deviation of recommendations from the distribution of prior sales (or positive ratings) promotes long-tail rather than popular items, we also 1Another smoothed version of the proposed metric is: CI@N = Pi2I 21 s%(i) ln 12 s%(is%)+(i12)r%N (i) + 12 r%N (i) ln 12 s%(ri%N)+(i12)r%N (i) ; where s%(i) = Pj2I s(j) and r%N (i) = NrNj(Ui)j .
s(i) propose a measure of \long-tail enforcement" as follows: 0 1 X jIj i2I + (1 where 2 (0; 1) controls which items are considered long-tail (i.e., the percentile of popularity below which a RS should increase the frequency of recommendation of an item). In essence, the proposed metric rewards a RS for increasing the frequency of recommendations of long-tail items while penalizing for frequently recommending already popular items.
To empirically illustrate the usefulness of the proposed
metrics, we conduct a large number of experiments
comparing various algorithms across di erent performance
measures. The data sets we used are the MovieLens 100k
(ML100k), 1M (ML-1m), and \latest-small" (ML-ls), and the
FilmTrust (FT). The recommendations were produced using
the algorithms of association rules (AR), item-based
collaborative ltering (CF) nearest neighbors (ItemKNN),
userbased CF nearest neighbors (UserKNN), CF ensemble for
ranking (RankSGD) [
Figure 1 illustrates the results of the comparative analysis of the di erent algorithms across various metrics. In particular, Fig. 1 shows the relative ranking in performance for each algorithm based on popular metrics of predictive accuracy and dispersion as well as the newly proposed metrics; green (red) squares indicate that the speci c algorithm achieved the best (worst) relative performance among all the algorithms for the corresponding dataset and metric.2
Based on the results, we can see that the proposed metrics capture di erent performance dimensions of an algorithm compared to the relevant metrics of Gini coe cient and aggregate diversity. Comparing the performance based on the proposed concentration bias metric (CI@N ) with the metric of Gini coe cient, we see that even though on aggregate an algorithm might distribute more equally than another algorithm the number of times each item is recommended, it might still achieve this by deviating less from the prior popularity (i.e., number of sales or positive ratings) of each item separately (e.g., green color for Gini coe cient and red color for concentration reinforcement). Nevertheless, the differences among the LT I performance and the other metrics (e.g., aggregate diversity) indicate that even though some algorithms might recommend fewer (more) items than others or distribute how many times each item is recommended less (more) equally among the recommended items, they might achieve this by frequently recommending more (fewer) longtail items rather than more (fewer) popular items (e.g., red color for Gini coe cient and green color for \long-tail enforcement"). Hence, the two proposed metrics should be used in combination in order to evaluate i) how much the recommendations of a RS algorithm deviate from the prior popularity of items and ii) whether this deviation occurs by promoting long-tail rather than already popular items. 2We have reversed the scale of the Gini coe cient for easier interpretation of the results (i.e., the green color corresponds to the most uniformly distributed recommendations).
We propose new metrics to accurately measure the concentration reinforcement and \long-tail enforcement" of recommender systems. The proposed metrics capture di erent performance dimensions of an algorithm compared to existing metrics of RSes as they take into consideration the prior distribution of positive ratings and sales of the candidates items in order to accurately measure the e ect of a RS. We also conduct a comparative analysis of various RS algorithms illustrating the usefulness of the proposed metrics.