Typical recommender evaluations treat users as an homogeneous unit. However, user subgroups often difer in their tastes, which can result more broadly in diverse recommender needs. Thus, these groups may have diferent degrees of satisfaction with the provided recommendations. We explore the ofline top- N performance of collaborative filtering algorithms across two domains. We find that several strategies achieve higher accuracy for dominant demographic groups, thus increasing the overall performance for the strategy, without providing increased benefits for other users.
• Information systems → Recommender systems;
Recommender system evaluation—ofline and online —typically
focuses on the system’s efectiveness, in aggregate over the entire user
population. While individual user characteristics are sometimes
taken into account, as in demographic-informed recommendation,
evaluations typically still aggregate over all users [
Our central research question is this: what changes about our assessment of relative or absolute recommender efectiveness when we consider performance for diferent subgroups of users– basically when we consider all subgroups’ satisfaction to be equally important? Does popularity bias exacerbate demographic bias efects? How do popularity bias mitigations afect the demographic bias? 2
We answer these questions with an ofline analysis using LensKit [
Demographic distribution and its impact on evaluation. Figure 1 shows user gender distribution; with the majority of users reporting as male. The age distribution reveals some diferences: the largest block of MovieLens users belong to the [25-35] group, whereas a plurality of LastFM users belong to the [18-24] group.3 1Code and scripts are available at https://doi.org/10.18122/B2ND8P 2Later MovieLens dataset do not include demographic information. 3For consistency, we binned LastFM users into the same groups used in MovieLens-1M.
Gender 0.3 0.2 0.1 R0.0 R M 0.3 0.2 0.1 0.0 DataSet
LastFM
ML−1M sLaF t M M L − 1 M Standard Results. Figure 2 shows the MRR achieved by each algorithm, grouped by demographic group. For each demographic characteristic, All is the accuracy achieved by averaging across all users, and Bucketed is the result of first averaging within each demographic group, and then averaging the groups’ results (thus giving each group equal weight, instead of each user). The results across subgroups are broadly similar for both data sets, though the All analysis tracks most closely with the dominant group. However, if a decision is to be made based on “performs best", then the small diferences become non-trivial, as they will afect the final decisions. One example case emerges from our analysis: on LastFM, II performs better using play counts (“-C”) for some age groups, while the “-B” variant is more efective for other age groups.
While we cannot conclude, based on this ongoing study, which is the right decision, our preliminary analysis demonstrates the need for further exploration from a demographic perspective.
Al Bucketed F M NA
Popularity Bias Mitigating Results. We also seek to
understand how demographic bias interacts with mitigation techniques
for other issues, such as popularity bias. To that end, we performed
a version of our analysis using Bellogin’s UAR technique [
The diferences obtained using UAR or traditional evaluations
show that mitigating popularity bias comes with the cost of
significantly changing the distribution of measured accuracy across user
subgroups. (Analysis using 1R [
Our analysis showed that, unsurprisingly, a number of recommendation strategies achieve moderately higher accuracy metric values for dominant demographic groups. This can cause an algorithm’s performance to increase without delivering benefit to smaller subgroups of the user population. In other words, the perceived satisfaction with a recommender may not be the same for the “cool” users—in the dominant group—as it is for those in smaller groups.
Demographic bias in accuracy metric results also has a complex interaction with mitigation strategies for other ofline evaluation ailments such as popularity bias. A uniform item strategy results in disproportionately higher accuracy values for users in some smaller subgroups. Further work is needed to understand which paradigm maps most closely to actual user experience or response.
Our findings highlight the need for careful and multi-faceted consideration of recommender system behavior across a range of both users and items. As prior work has found that recommenders are not equally good at recommending for all items, we find that recommenders are not equally good for all users in predictable and socially-relevant ways. While the full social and business ramifications of our findings have yet to be explored, we encourage researchers and practitioners to pay attention to which users receive how much benefit from a particular recommender.
We thank Ion Madrazo for helping with analysis, and the People and Information Research Team (PIReT) for their support.