Popularity bias is an important issue in recommender systems, as it afects end-users, content creators, and content provider platforms alike. It can cause users to miss out on less popular items that would fit their preference, prevent new content creators from finding their audience, and force providers to pay higher royalties for serving expensive popular content. Over the past years, various approaches to mitigate popularity bias in recommender systems have been proposed. Among them, post-processing methods are widely accepted due to their versatility and ease of implementation. While previous studies have investigated the efects of diferent post-processing techniques on accuracy and fairness of recommendations, the influence of diferent algorithms on diferent user groups have not received much attention in this context. Addressing this research gap, we study the efect of a recent mitigation strategy, Calibrated Popularity, in conjunction with a selection of state-of-the-art recommender algorithms including BPR, ItemKNN, LightGCN, MultiVAE, and NeuMF. We show that these algorithms demonstrate diferent characteristics in terms of the trade-of between accuracy and fairness, both within and between various user groups defined by gender and inclination towards consumption of mainstream items. Finally, we demonstrate how these discrepancies can be exploited to achieve a more efective trade-of between utility and fairness of recommender systems.
While not all biases are harmful—recommendation results need to be biased in the sense of personalization to match
the end user’s preferences—data, algorithmic, and presentation biases may lead to unfair behavior of RSs, i. e., the
recommender “systematically and unfairly discriminates against certain individuals or groups of individuals in favor of
others” [
of the recommendation pipeline. One can distinguish pre-, in- and post-processing methods. The first act before the main RS, often applying transformations to the data the RS is trained on, in an attempt to make its output less biased.
or regularization [
also diferent user groups sufer from it to various extents. These findings lead us to the following research questions we tackle in this work: RQ1: Is the mitigation technique of post-processing equally efective for all algorithms? Do all algorithms show the same character of trade-of between utility and calibration? RQ2: Are all user groups equally afected by the mitigation procedure? Are the optimal mitigation parameters the same for all user groups? RQ3: To which extent can the trade-of between utility and calibration be softened through using specific mitigation parameters for each user group?
algorithms, analyzing the mitigation efectiveness and utility-fairness trade-of for each of them. We also consider the efect of mitigation on diferent user groups, and through this characterize the scoring strategy of each recommendation algorithm investigated. Finally, we conduct an experiment to evaluate potential gains of mitigation approaches tailored specifically to each user group. 2
RELATED WORK
Many studies formalize fairness of a RS on the user level through calibration of a certain item attribute (such as genre)
[
mitigation techniques. They operate on the output of a RS, re-ranking the items, striving to create a list that satisfies both utility and fairness objectives. A big advantage of post-processing is its flexibility to be used with almost any
years. Abdollahpouri et al. [
them on both platform-wide and user-preference levels. They show that CP is preferable for providing fairness on per-user level.
Usually bias mitigation algorithms allow to adjust the weight distribution between the utility and fairness objectives.
da Silva et al. [
interactions with it. We distinguish Popular, Niche, and Mid categories of items. Popular items are represented by items most interacted with and jointly receiving 20% of all user-item interactions. Similarly, Niche items are the least interacted with, receiving 20% of aggregated user-item interactions. The rest of items falls into the category Mid.
ways of user grouping: by users’ gender1 and by their inclination towards consumption of popular items. With the latter,
similar to [
and their proneness to popularity bias. Following previous work, we define the latter on per-user level as Jensen-Shannon
If and are item popularity (probability) distributions of consumption history and recommendation for user , respectively, we calculate Jensen-Shannon Divergence as:
(, ) = 21 ∑︁ ()2 (2)+() () + ∑︁ ()2 (2)+() () (1)
where () is the proportion of items of popularity category in the consumption history of user . can be seen
as symmetrical version of Kullback–Leibler divergence. Note that using 2 we ensure that the value of is bound
!
between 0 and 1 [
tion [
, | |= where ( ) is the sum of relevance scores and ( ) the item popularity distribution of the candidate item list.
constituting ( ) to the interval [
for diferent user groups we introduce a way of selecting an optimal value of parameter taking into account both utility and fairness of the recommendation. For ease of notation, we denote (, ) = 1 − (, ) a fairness measure; similar to NDCG, higher values are better. We define the optimal value of for a user group as follows: = arg max NDCG · (3)
∈ [
(JSF) for the group. The selection is done by conducting a grid search on the interval [
EXPERIMENT SETUP
Datasets. We investigate two datasets, MovieLens-1M (ML-1M) 2 in the movie domain and LFM-2b 3 in the music domain. The former provides ratings for 6K users and almost 4K movies. The latter is a Last.fm music listening dataset, which we modify to fit our experimental setup. Firstly, we consider only listening events in the year 2019 of users with meta-information regarding age, gender and country. Secondly, all user–item interactions with a playcount () of < 2 are removed to reduce the number of spurious interactions and noise. Thirdly, we only consider tracks that were listened to by at least 5 diferent users (constraint 1) and we only consider users who have listened to at least 5 diferent tracks (constraint 2). Lastly, we treat each user–item interaction in a binary way – 1 if the user has listened to a track, 0 otherwise. Furthermore, we sample 100K tracks uniformly-at-random to ensure items of diferent characteristics are equally likely to be included in the final subset. We then reinforce constraints 1 and 2. This ultimately results in almost 10K users retained with a total of 10.7M listening events. See Table 1 for details.
Algorithms and Baselines. We study popular collaborative filtering algorithms (i. e., neighborhood-based, neural
matrix factorization, autoencoders, and graph convolution networks), briefly described in the following. For consistency,
we use the algorithm implementations from the Recbole framework [
LFM-2b
ItemKNN
LightGCN
MultiVAE
NeuMF Exploring Cross-group Discrepancies in Calibrated Popularity for Accuracy/Fairness Trade-of Optimization LightGCN
LightGCN
.14 BPR
.10 BPR .10 BPR .15 .10 .05 .2 .1 0 .1 .25 .20 D JS.15 .10 .2 D S J .1 .2 D S J .1 .20 .15 .10 .05 .2 .1 .2 .1 .24 .25 .13 .14 .15 .09 .10 .11 .04 ItemKNN
MultiVAE
NeuMF
.10
.24
.10
.04
on the basic matrix factorization approach but replaces the inner product with a neural architecture that can learn an
arbitrary function from the interaction data. Variational Autoencoders (MultiVAE) [
train/validation/test groups with a 60-20-20 ratio split. Therefore, 60% of all users’ interactions are used to train the algorithms. We maximize the NDCG @10 metric over the validation set. All results are reported for the test set.
Popularity Bias Mitigation. We conduct a series of tests comparing utility and fairness of recommendation lists produced by the above mentioned algorithms and re-ranked by the CP post-processing mitigation technique with
LowPop MidPop
HighPop .04
NeuMF Male Female .04 D S J .02 .04 D S J.02
BPR .316 BPR .04 .02 .04 .02 .04 .02 .04 .02 .04 .02 .04 .02 ItemKNN
LightGCN
MultiVAE
NeuMF .344 .348
.352 ItemKNN .352 .356 nDCG@10 LightGCN diferent settings. For every algorithm 5, we re-rank the list of top 100 recommendations to create the final list of 10 items for each user. We test it for ten values of the weighting parameter from 0 (no mitigation) to 0.9 (weight 0.9 to the fairness objective and 0.1 to utility) with step size 0.1. We aim to exploit potential diferences in the way various user groups respond to the popularity bias mitigation to achieve a better trade-of between utility and fairness. To this end, we split users uniformly-at-random into train and test sets of the same size striving to ensure all user groups are represented in both. We search for optimal values of for each user group and the whole population using the criterion in Equation 3 on the train set. We then compute the new recommendation lists for the test users, applying mitigation with the weights learned for their corresponding user groups. 5 RESULTS
NDCG (utility measure) against (popularity bias measure) for every recommendation algorithm and the ten values of , respectively, for LFM-2b and ML-1M dataset. On the plots, the opacity of the points corresponds to the value of , such that the palest show the results of = 0. Let us first look at the the top row of the plots in each figure which shows the results achieved on the whole population of the dataset. On LFM-2b we notice diferences in behavior of recommended lists produced by diferent recommendation algorithms. In particular, KNN shows an increase in NDCG combined with an improvement in fairness at = 0.1. BPR and LightGCN show steady progress towards debiased results of lower utility with growing . At the same time, MultiVAE and NeuMF demonstrate a notably larger drop in utility over the first step (from = 0 to 0.1). Considering that every point on each plot corresponds to a new set of items recommended to most of the users, we can examine the quality of the top 100 recommendation lists produced by diferent algorithms. In this regard, a smooth decay of NDCG and signify a better overall quality of the top 100 list as it allows to debias the recommendation gradually without a sudden drop in utility. A sudden drop in both metrics however would mean that the achieved utility to a large degree comes from concentration of the popular items at the
6 the third row of Figure 1 show the results of the users grouped by genders6. Analyzing the results according to the mainstreamness groups for LFM-2b, we observe initially all algorithms provide the best utility to the HighPop group, the group most exposed to popularity bias varies from model to model. BPR, LightGCN and KNN show that LowPop user group benefit from the mitigation method in terms of utility. KNN also shows the same for the MidPop group. In most cases, the HighPop group experiences the largest drop in utility through popularity bias mitigation. These findings support our hypothesis that selecting mitigation parameters separately for each user group potentially improves the trade-of between utility and fairness. On the ML-1M dataset we see that all three groups maintain a certain level of utility, while steadily decreasing the bias metric, showing that all five algorithms on this dataset can successfully achieve good results in terms of bias mitigation while maintaining utility. Considering the results on genders, we do not observe a notable diference in the overall patterns between the genders.
Finally addressing RQ3, Table 2 shows the results of the experiment with group-specific values of . For every algorithm and dataset, we report utility and bias metric under four conditions: = 0 no mitigation, where one optimal parameter value is selected for the whole population, where specific optimal parameter value selected for each popularity inclination user group, and finally indicating the selection of specific parameter value for each gender. We observe that, except for NeuMF on ML-1M, group-specific s provide the best result on bias metric and trade-of between utility and fairness, namely a lower value of together with utility staying on the same level or slightly decreasing. Among all algorithms LightGCN has shown the lowest sensitivity to the mitigation, as its values do not notably drop in either utility or bias metrics. All algorithms show low bias metrics without any mitigation on
slight decrease in utility. 6
CONCLUSION AND FUTURE WORK We explore the efectiveness of a post-processing popularity bias mitigation technique, Calibrated Popularity, applied to an array of state-of-the-art recommendation algorithms. We conduct experiments on two datasets from the music and movie domain, of diferent size and sparsity, considering how various user groups (defined by gender and mainstreaminess) are afected by bias mitigation. The larger music dataset LFM-2b exposes discrepancies in behavior of diferent algorithms. NeuMF and MultiVAE show a notable drop in utility and bias metrics even with light bias mitigation settings. KNN shows an increase in utility with moderate bias mitigation applied. We also show that for BPR, KNN, and LightGCN users least interested in popular items can benefit in terms of utility from popularity bias mitigation as opposed to other users. Our experiments show that diferent user groups respond to the mitigation diferently depending on their inclination towards consumption of popular content. We also found responses of diferent gender groups relatively similar. Finally, we conduct an experiment showing that selecting mitigation parameters individually for every user group (by interest towards popular items) leads to a better trade-of between utility and fairness overall.
LIT-2021-YOU-215.