Group recommendations are a sub-domain of recommender systems (RS), where the final recommendations should comply with preferences of all members of the group. Usually, group recommendations are built on top of common "single-user" RS via aggregating models or predictions for multiple users with some notions of fairness and relevance in mind. So far, group recommendations were usually evaluated of-line either as a tightly coupled pair with the underlying RS or in a decoupled fashion, where the relevance scores estimated by underlying RS serves as a ground truth. Both evaluation types may sufer from diferent biases that provide illicit advantages to some classes of group recommending strategies. In experimental part, we evaluate several recent group recommendation models and show that the evaluation process itself significantly afects their perceived usability. While coupled evaluation favors group RS that tend to select per-user best items, decoupled evaluation favors strategies aiming to find items with (some degree of) overall agreement. We further evaluate methods w.r.t several variants of inverse propensity based de-biasing scenario in order to reduce the popularity bias of coupled evaluations. Also in this case, if groups of similar users are considered, the magnitude of de-biasing has a determining efect on the ordering of individual methods.
In this paper, we focus on various fairness-preserving group recommendation strategies that
operate on top of classical (single user) recommending systems (e.g. [
We can roughly divide group RS aggregators to item-wise and list-wise. Item-wise approaches
evaluate the utility function for each item separately, irrespective of other items’ scores. Many
utility functions were proposed in the history, e.g. Least Misery, Average, Average without Misery,
Borda count and many more. Item-wise approaches were thoroughly surveyed in [
List-wise approaches argue that item-wise approaches may introduce a systematic bias against
some group members, e.g. if his/her preferences difers from the rest of the group [
The main concern of this paper is the evaluation of group RS aggregators. Having a couple of
underlying (single user) recommender and a group RS aggregator leaves two principal options
to conduct the evaluation. First, one can perform a coupled evaluation, i.e. consider the RS and
the group aggregator as a tightly coupled pair and evaluate their overall performance as in
[
In this paper, we show that both evaluation types may introduce certain biases and provide considerably diferent results. Furthermore, in order to provide bias-free estimation of group RS aggregator’s performance, we utilized coupled evaluation with Self-normalized Inverse Propensity Score (SNIPS) evaluator[12]. Notably, not only the results of coupled and decoupled evaluations difer significantly, but the relative performance of individual aggregators also greatly depend on the magnitude of de-biasing when SNIPS evaluator is employed.
We will continue with a brief description of evaluation protocols followed by the results presentation and a discussion on possible causes and implications of the observed results.
Evaluation protocols of recently proposed group RS aggregators difered considerably. Therefore, in this section, we provide a brief overview of utilized variants.
The first design choice of every evaluation protocol is the selection of underlying RS. While
the choice of RS is in theory orthogonal to the rest of the evaluation protocol, it may have
some interesting implications for both coupled and decoupled evaluation protocols. Most
related papers ([
Next, the groups of users whom the recommendations are to be addressed have to be
assembled. Finding a suitable dataset containing actual groups of users is a prevalent problem. Xiao
et al. [
Other than this, authors resort to artificially constructed groups on standard datasets. One can
either use random users, or their similarity is considered. For instance, Kaya et al. [
Coupled evaluation protocol (see Figure 1 top) closely resembles the standard static of-line
evaluation of RS (with imputed group RS aggregator). Historical user feedback (, ∈ ℛ is first
divided into train set and test set (cross-validation is often applied) and train set is forwarded
to the RS. Recommender system outputs estimated preferences (^,) of individual users and
push them to the group RS aggregator, which provides final recommendations for the whole
group (list of top-20 recommendations in our case). Performance of the overall solution
is evaluated w.r.t. withheld fraction of the user feedback. As such, this evaluation strategy
estimates the performance of RS and group aggregator couple. Kaya et al. [
The obvious disadvantage of this evaluation strategy is its inability to completely disentangle the performance of group aggregators from the performance of underlying RS (i.e., the way how RS recommend may be more/less suitable for individual group RS aggregators). This problem can be to some extent solved by utilizing multiple, suficiently diverse, recommender systems.
The second disadvantage is that in vast majority of datasets, only a small fraction of potentially relevant items is known and the missing feedback is not randomly distributed. Feedback on relevant long-tail items is missing more often than feedback on highly popular items (i.e., missing-not-at-random problem, MNAR [12]). Furthermore, collaborative filtering RS such as variants of matrix factorization tends to exhibit popularity bias [15]. I.e., popular items tend to be recommended more frequently (and on higher positions) than what would be proportional to 2To be more specific, next group member was selected at random from users who have Pearson’s correlation of at least 0.3 (similar) or no more than 0.1 (divergent) to some of the existing group members. their overall popularity. Combined with the MNAR problem and coupled evaluation, popularity biased RS may provide an illicit favor for such group recommendation strategies that often select top recommended items for each user (such as FAI or GFAR) over strategies recommending items that are acceptable (not necessarily best) for the majority of users.
In decoupled evaluation, estimated ratings ^, received from the RS are considered as the ground
truth (see Figure 1 middle). With this, we simulate the situation where all user preferences
are known and we only evaluate the capability of group RS aggregators to efectively combine
those preferences. Malecek et al. [8], Serbos et al. [
As in coupled evaluation, a possible problem of this evaluation scenario is the interference between underlying RS and group aggregation strategies (i.e., the general characteristics of supplied ratings may be more suitable for some approaches than others). For instance, consider a RS that systematically overestimates the true relevance of items. In this case, many items that are actually not preferred would be rendered as (mildly) preferred by users. Approaches seeking overall agreement on items could illicitly benefit from this bias, because they receive a broader space of possible combinations to choose from.
Both of the previously described evaluation approaches may introduce a bias that would provide an illicit advantage to some class of group RS aggregators. In order to compare the extent of these biases as well as to mitigate some of them, we considered the utilization of a de-biasing evaluation strategy. Specifically, we focused on de-biasing the popularity bias in coupled evaluation strategies. In order to do so, we utilized the self-normalized inverse propensity score (SNIPS) approach proposed by Yang et al. [12] (further denoted as de-biasing evaluation strategy). De-biasing evaluation strategy is essentially a coupled one, but the SNIPS score is utilized as a normalization to reduce an impact of items with high overall popularity (see Figure 1 bottom).
For the set of user’s known relevant items (), estimated item’s propensity score , and some scoring metric (, ), the relevance of the list of recommendations for the user is calculated as follows: (1) (2) (3) (, ) = {︃,/|| if ∈
0 otherwise (, ) = {︃,/2((, )) if ∈
0 otherwise Nonetheless, the question is how to estimate the propensity score ,. Yang et al. decomposed , to the probability that item is recommended by RS and the probability that item is interacted with if recommended. Authors further assume user independence, completeness of user’s observation and popularity bias for the probability of being recommended and finally propose the following estimation of the propensity score:
1 ∑︁ (, ) , = ∑︀∈ 1, ∈ ,
For the two metrics considered in this paper, average relevance and discounted cumulative gain at top-20 recommendations, the corresponding scoring functions are as follows: +1 , ∝ 2 where denotes observed popularity of an item (i.e., the volume of known interactions with item ) and power-law exponent is a hyperparameter of the model afecting the propensity distributions over items with various observed popularity levels. Larger values lead to lower propensity scores for long-tail items and higher scores for popular ones.
Our research question is whether the popularity bias indeed afects the results of individual group RS strategies as assumed. In order to answer it, we manipulate the hyperparameter to gradually decrease the efect of popularity bias in evaluations. We hypothesize that by doing this, the de-biasing evaluation results should gradually resemble the results obtained from decoupled evaluation (i.e., that the de-biasing evaluation strategy can serve as a smooth bridge between coupled and decoupled evaluation strategies). Obviously, the transition cannot be perfect, because no counter-measures were applied for the possible biases introduced by the decoupled evaluation protocol.
In order to corroborate our theories on biases introduced by coupled and decoupled evaluation strategies and their impact on various classes of group RS aggregators, we conducted a following experiment. Eight group RS aggregation strategies were evaluated w.r.t. coupled, decoupled and a range of de-biasing evaluation protocols. By observing relative diferences of per-approach results w.r.t. individual evaluation protocols, we can derive conclusions on the presence and efects of said biases. Let us first describe the evaluated group RS strategies and parameters of the experiment.
In the evaluation, we considered one item-wise and seven list-wise group RS aggregation strategies. Their brief description and references follows. In description, we mainly focused on the estimation, whether the considered algorithm is more biased towards finding an overall agreement between users or whether it rather tend to provide a selection of (close to) best items for individual users.
AVG [
FAI [
SPGreedy [
GreedyLM [
XPO [11] considers the concept of Pareto domination w.r.t. per-user ranking3. list of the size is generated from items that are dominated by at most items, where is set as smallest number such that the size of the resulting set of candidates is at least . Finally, is selected via probabilistic approach. A series of weighted averages of per-user ranks is generated (with weights selected at random). For each item, it is calculated how many times it fits into 3I.e. item 1 dominates item 2 if it is better for at least one user and better or equal for all others. top-N best items and items with the highest counts are recommended. Given the pre-selection procedure that minimize the volume of eligible candidates, XPO would tend to propose per-user best items.
GFAR [
FuzzDA [8] approach is based on D’Hondt’s mandates allocation strategy, but extends it with fuzzy candidate-party membership (in the context of group RS aggregation, it allows that an item is preferred by multiple users to a certain degree). Each group member receives certain amount of initial votes and at each step the item with the highest weighted relevance = ∑︀∈ ^, * ¯ is selected. Current per-user votes ¯ are reduced proportionally to the user-item relevance of currently selected item.
EP-FuzzDA [8] approach is based on FuzzDA, but modifies the item selection procedure. For each user and each iteration, EP-FuzzDA calculates the amount of relevance that is missing (,) to have exactly proportional representation of user’s votes in the partially constructed . Then the item with highest (constrained) overall relevance = ∑︀∈ (^,, ,) is selected. Both FuzzDA and EP-FuzzDA tend to select items with higher overall agreement rather than best items for individual users.
The experimental setup utilized in this paper is partially based on [
Coupled and de-biasing evaluation scenarios utilize the test set data. KGRec dataset contains
binary interactions, but ML1M dataset contains graded relevance feedback (1*-5* ratings). In
accordance with [
Two evaluation metrics are considered: normalized discounted cumulative gain (nDCG) and
average relevance score (AR), both evaluated on top-20 recommendations. For each list of
recommendations , we evaluate these metrics for all group members ∈ and collect three
aggregated per-group statistics: mean per-user scores for relevance evaluation, minimal user’s
score (i.e., least misery fairness metric as used in [
Table 1 depicts the results of coupled and decoupled evaluation scenarios for similar groups of size = 8. We can observe that both types of evaluation give us highly diferent ranking of best approaches. For coupled evaluation, GFAR and XPO compete for the best approach (seemingly, GFAR is slightly better). FAI algorithm provide good ratios of minimal vs. maximal per-group performance, but its mean per-group performance is inferior to the previous two examples. Decoupled evaluation provides a diferent view: EP-FuzzDA receives the best minimal per-group scores as well as minimal vs. maximal ratios followed by GreedyLM and AVG algorithms. For the mean scores, naturally, AVG is the best approach followed by EP-FuzzDA and GreedyLM (ML1M dataset), resp. FuzzDA and EP-FuzzDA (KGRec dataset). The diference between coupled and decoupled approach can be further illustrated by the Pearson’s correlation, which is -0.32 and -0.29 for mean AR and mean nDCG on ML1M dataset respectively. Note that although we only depict the results for similar groups of size = 8, results for other sizes of the similar groups exhibited analogical levels of contradiction. As for the divergent groups, the level of agreement between coupled and decoupled evaluations was higher in general (results were positively correlated in most cases), but the disagreement on the best-performing approaches prevails.
Figures 2 and 3 depict results of group RS aggregators w.r.t. coupled, decoupled and all variants of de-biasing scenarios. Because the scale of results w.r.t. individual evaluation scenarios difers greatly, we depict their normalized comparison. For the sake of space, we only show two of the evaluated metrics, additional figures are available from supplementary materials.
Results of similar groups exhibited a clear dependence between algorithm’s tendency to recommend per-user best items or items with overall agreement and the hyperparameter of the de-biasing evaluation. The performance of algorithms preferring per-user best (FAI, GFAR, XPO) decreased with the increasing . On the other hand, algorithms preferring items with overall agreement (AVG, EP-FuzzDA, FuzzDA) gradually improved their relative performance with increasing values. The transition between the de-biasing evaluation with = 4.0 and the decoupled evaluation was not as smooth as for the individual de-biasing scenarios, but remain fairly close (e.g., Kendall score was 0.71, 0.64 and 0.71 for KGRec dataset, (AR, min) metric and group sizes of 2, 4 and 8 respectively). Similar results were obtained for both datasets, and per-group mean and minimal values of both metrics.
Results w.r.t. divergent groups were less clear. In ML1M dataset and mean nDCG (Figure 2), we can observe some of the tendencies from similar groups. For instance, GFAR performance gradually decrease with increasing and the opposite applies for EP-FuzzDA. In contrast, the performance decrease of AVG, FuzzDA and SPGreedy was in contradiction to the results of corresponding similar groups. Also, there was almost complete shift of algorithm’s relative ordering between de-biasing, = 4.0 and decoupled evaluation scenarios (Kendall scores of -0.43, -0.57 and -0.43 for group sizes = 2, 4 and 8 respectively). Similar results were obtained also for divergent groups on ML1M dataset and (AR, mean) metric.
Results w.r.t. per-group minimal AR on KGRec dataset were rather indiferent of the choice of and highly resembles those of the coupled evaluation. Similarly as in the previous case, there was almost complete shift of algorithm’s ordering between de-biasing, = 4.0 and decoupled evaluation scenarios. Similar results were obtained also for divergent groups on ML1M dataset and (AR, min), (AR, min/max) and (nDCG, min) metrics as well as KGRec dataset and (AR, mean), (AR, min/max) and (nDCG, min) metrics.
In this paper, we compared several of-line evaluation protocols for group RS aggregation strategies. First, we showed that two widely adopted strategies, coupled evaluation and decoupled evaluation can lead to highly contradictory results. Then, we utilized de-biasing evaluation with the aim to provide a smooth transition between coupled and decoupled evaluation strategies. Debiasing strategy is coupled in nature, but it introduces an inverse propensity based normalization of results that mitigate one of the major flaws in coupled evaluation: assumption on randomly missing feedback.
This task was partially achieved as long as groups of similar users are considered. We observed increase of performance w.r.t. values for strategies that prefer items with overall agreement rather than per-user bests (AVG, FuzzDA, EP-FuzzDA) and in contrast, performance of strategies that tends to propose per-user best items (FAI, GFAR, XPO) drops with increasing . These results were mostly consistent with the ones given by decoupled evaluation strategy.
We believe that the main factor behind the observed behavior is the combination of MNAR dataset and popularity-biased underlying RS, which boosted the chance that top recommended items were observed (above the relative diference in estimated preferences) and which is gradually penalized by the de-biasing evaluation scenario. This gives a chance to recommend less popular, but more agreeable items.
For divergent groups of users, one possible explanation is that commonly preferred items were simply not present in the test sets.4 Therefore, penalizing recommendations of too popular items as in de-biasing scenario would not change the results much, because there are simply no good alternatives instead of per-user best items.5 Decoupled evaluation, in theory, can bridge this problem as the underlying RS can discover mutually agreeable items that are not present in the historical data. This may be the cause of the large performance jump between the de-biasing = 4.0 and decoupled evaluation scenarios for divergent groups. However, the same efect could be also observed if the underlying RS overestimates the true preferences of users. ALS MF utilized in this work rarely ever provided negative ^, ratings and as no ratings’ post-processing was performed, most of the items were considered as (at least) mildly preferred by users. Many RS would behave in a similar fashion. Therefore, detecting and counter-measuring the overestimation bias is one direction of our future work.
To conclude, the main message of this paper is that performance of group RS strategies can highly depend on the considered evaluation scenario. So, before applying any particular evaluation strategy, authors should carefully consider intended goals of the proposed algorithm and its compliance with the possible biases introduced by the evaluation process.
We consider our work as rather preliminary and there are numerous possible extensions. First, we only evaluated results w.r.t. one underlying RS. We plan to experiment with other recommenders to corroborate our findings. Especially, it would be interesting to observe the results w.r.t. some content-based RS that should be less prone to the popularity bias and e.g. variants of item KNN to decrease the overestimation bias. Also, instead of applying de-biasing evaluation strategy, it is possible to bridge the results of coupled and decoupled evaluation by utilizing only a portion of RS prediction (e.g. only those with the highest estimated relevance). This could be another option to reduce the efect of overestimation bias.
Our long-term future work focus on real-world performance of group recommenders. For 4As the groups were assembled w.r.t. the level of disagreement among users, it is not an unrealistic assumption. Nonetheless, further validation is needed.
5As long as some popularity de-biasing strategy, e.g. [15] is not considered, which was not the case of evaluated approaches. instance, one question risen by this study is whether the real-world user groups are intrinsically rather similar or diverse. This knowledge can greatly afect the success of agreement oriented vs. per-user-best oriented designs of group RS aggregators. Also the relation between coupled/decoupled/de-biasing evaluations and on-line performance is currently unknown. Therefore, another direction of our future work is to conduct a realistic user study that could both provide some insights on user group formations as well as evaluate the on-line performance of group recommenders.
The work on this paper has been supported by Czech Science Foundation project GACR-1922071Y and by Charles University grant SVV-260588. Additional results can be obtained from https://github.com/lpeska/Perspectives-RecSys2021. NY, USA, 2017, p. 107–115. URL: https://doi.org/10.1145/3109859.3109887. doi:10.1145/ 3109859.3109887. [8] L. Malecek, L. Peska, Fairness-preserving group recommendations with user weighting, in: Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 4–9. URL: https://doi.org/10.1145/3450614.3461679. doi:10.1145/3450614. 3461679. [9] L. Quijano-Sanchez, J. A. Recio-Garcia, B. Diaz-Agudo, G. Jimenez-Diaz, Social factors in group recommender systems, ACM Trans. Intell. Syst. Technol. 4 (2013). URL: https: //doi.org/10.1145/2414425.2414433. doi:10.1145/2414425.2414433. [10] L. Quijano-Sánchez, J. A. Recio-García, B. Díaz-Agudo, Modelling hierarchical relationships in group recommender systems, in: E. Hüllermeier, M. Minor (Eds.), Case-Based Reasoning Research and Development, Springer International Publishing, Cham, 2015, pp. 320–335. [11] D. Sacharidis, Top-n group recommendations with fairness, in: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 1663–1670. URL: https://doi.org/10.1145/3297280. 3297442. doi:10.1145/3297280.3297442. [12] L. Yang, Y. Cui, Y. Xuan, C. Wang, S. Belongie, D. Estrin, Unbiased ofline recommender evaluation for missing-not-at-random implicit feedback, in: Proceedings of the 12th ACM Conference on Recommender Systems, RecSys ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 279–287. URL: https://doi.org/10.1145/3240323.3240355. doi:10. 1145/3240323.3240355. [13] I. Pilászy, D. Zibriczky, D. Tikk, Fast als-based matrix factorization for explicit and implicit feedback datasets, in: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 71–78. URL: https://doi.org/10.1145/1864708.1864726. doi:10.1145/1864708.1864726. [14] B. Sarwar, G. Karypis, J. Konstan, J. Riedl, Item-based collaborative filtering recommendation algorithms, in: Proceedings of the 10th International Conference on World Wide Web, WWW ’01, Association for Computing Machinery, New York, NY, USA, 2001, p. 285–295.
URL: https://doi.org/10.1145/371920.372071. doi:10.1145/371920.372071. [15] H. Abdollahpouri, R. Burke, B. Mobasher, Controlling popularity bias in learning-to-rank recommendation, in: Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 42–46. URL: https://doi.org/10.1145/3109859.3109912. doi:10.1145/3109859.3109912. [16] F. M. Harper, J. A. Konstan, The movielens datasets: History and context, ACM Trans.
Interact. Intell. Syst. 5 (2015). [17] S. Oramas, V. C. Ostuni, T. D. Noia, X. Serra, E. D. Sciascio, Sound and music recommendation with knowledge graphs, ACM Trans. Intell. Syst. Technol. 8 (2016). URL: https://doi.org/10.1145/2926718. doi:10.1145/2926718.