To date, graph collaborative filtering (CF) strategies have outperformed pure CF models in generating accurate recommendations. However, concerns about fairness and potential biases in recommendations have emerged, as unfair recommendations may harm the interests of Consumers and Producers (CP). Recognizing the lack of a thorough evaluation of graph CF on CP-aware fairness measures, we initially assessed the efects of eight state-of-the-art graph models and four pure CF recommenders on CPaware fairness measures. Surprisingly, graph CF solutions do not ensure significant item exposure and user fairness. To unravel this performance puzzle, we propose a taxonomy for graph CF, highlighting diferences in node representation and neighborhood exploration. Through this lens, the experimental outcomes become clear and pave the way for a multi-objective CP-fairness analysis (Codes are available at: https://github.com/sisinflab/ECIR2023-Graph-CF).
Recommender systems (RSs) have evolved from collaborative filtering (CF) to deep learning
(DL) models [
This work aims to bridge the knowledge gap in the literature by studying the efects of state-of-the-art graph strategies on consumer fairness, producer fairness, and system accuracy. We evaluate these dimensions in terms of overall accuracy, user fairness, and item exposure, referring to their combination as CP-fairness when appropriate. I-Exp
U-Fair RP3 I-Exp
I-Exp U-Fair
U-Fair (a) Baby
(b) Boys & Girls (c) Men
Motivating Example. We compare leading graph-based models, such as LightGCN, DGCF, LR-GCCF, and GFCF, against classical CF baselines, BPRMF and RP3 , on three Amazon datasets. Our evaluation (see Figure 1), considering overall accuracy, user fairness, and item exposure, indicates that graph CF is more accurate, but classical CF shows a better item exposure. No clear winner is found regarding user fairness. We aim to answer two research questions (RQs): RQ1. Can we explain the variations observed when testing several graph models on overall accuracy, item exposure, and user fairness separately? We analyze the impact of graph CF strategies for nodes representation and neighborhood exploration on accuracy and CP-fairness. RQ2. How and why do nodes representation and neighborhood exploration algorithms strike a trade-of between overall accuracy, item exposure, and user fairness? Using Pareto optimality [11], we determine the influence of these dimensions in two-objective scenarios, including overall accuracy, item exposure, and user fairness. 2. Nodes Representation and Neighborhood Exploration in
Let = ( , ℐ , R) represent a bipartite, undirected graph connecting users and items. We denote the node features for users ∈ and items ∈ ℐ as embeddings e ∈ ℝ and e ∈ ℝ , where is much smaller than and . Diferent applications use message-passing schema to update the model using . Graph convolutional networks (GCNs), neural graph collaborative ifltering (NGCF), and LightGCN are examples of collaborative filtering (CF) applications that utilize message-passing. The need to account for the varying importance of user-item interactions has led to attention mechanisms and models such as graph attention networks (GATs) and disentangled graph collaborative filtering (DGCF). The recent UltraGCN and GFCF address over-smoothing by surpassing and simplifying the traditional message-passing concept. To better understand graph CF models, we propose a classification system based on:
Node representation: This aspect concerns the strategies used for representing users’ and items’ nodes, including the dimensionality of node embeddings and the weighting of contributions from neighboring nodes.
Neighborhood exploration: This aspect involves the methods for exploring multi-hop neighborhoods to update node representations, including the types of node-node connections and the message-passing schema (either explicit or implicit). We propose (see Table 1) a taxonomy to classify the state-of-the-art graph models. In the following two sections, we will assess eight graphs CF models based on this classification system: GCN-CF [12], GAT-CF [13], NGCF [14], LightGCN [15], DGCF [16], LR-GCCF [17], UltraGCN [18], and GFCF [19].
This section addresses RQ1 (“Can we explain the variations observed when testing several graph models on overall accuracy, item exposure, and user fairness separately?”) by examining how the taxonomy of graph strategies can elucidate recommendation evaluation on CP-Fairness and overall accuracy. We test 48 hyper-parameter configurations on the Amazon Men dataset and top-20 lists (Table 2), focusing on message-passing, explored nodes, edge weighting, and latent representations. We report the best metric result for each <dimension, value> pair.
Message-passing. We study implicit and explicit message-passing strategies. Both approaches perform similarly on accuracy and user fairness, but explicit techniques perform better on item exposure, particularly Gini and APLT.
Explored nodes. We examine four node exploration methods: same and diferent , with 1 and 2 hops. Same-2 and diferent-1 are the most prominent, with diferent-1 outperforming same-2 on overall accuracy and same-2 being the best strategy for item exposure. User fairness does not ofer a reason to choose between same and diferent .
Weighted. We investigate weighted and unweighted graph CF techniques. Unweighted methods provide the best performance on almost all CP-fairness metrics, except for APLT, where weighted GAT-CF performs better.
Latent representations. We compare graph CF techniques with 64, 128, and 256 features. Higher latent representations (128 and 256) result in better performance on all measurements, with 128 being the turning point for stable performance (see Table 1 as a reference).
This section examines the trade-of, through Pareto optimal solutions, between accuracy, item exposure, and user fairness in graph CF baselines. The analysis is conducted on the Amazon Men dataset and focuses on the message-passing and weighting of graph edges. Three categories † indicates the best results on same and diferent configurations. The symbols ↑ and ↓ indicate if high or low values are better. “rank ” and “rat ” stand for UMADrank@k and UMADrat@k.
implicit explicit same-1 same-2 diferent-1 diferent-2 weighted unweighted emb-64 emb-128 emb-256
Recall↑ 0.1222 (GFCF) nDCG↑ 0.0911 (GFCF)
EFD↑ 0.2615 (GFCF)
Gini↑
APLT ↑ rank↓
rat↓
0.2871 0.1808 0.0123 0.0022
(UltraGCN) (UltraGCN) (UltraGCN) (UltraGCN)
0.1223 0.0884
(LR-GCCF) (LR-GCCF)
0.2536 0.5090 0.3823
(LR-GCCF) (LR-GCCF) (GAT-CF)
0.1221†
(LR-GCCF)
0.0884†
(LR-GCCF)
0.2500†
(LR-GCCF)
0.4377
(LR-GCCF)
0.1184 0.0841 0.2380 0.5090†
(LightGCN) (LightGCN) (LightGCN) (LR-GCCF)
0.1222†
(GFCF)
0.1210
(DGCF)
0.1210
(DGCF)
0.1193
(LR-GCCF)
0.1221
(LR-GCCF)
0.0911†
(GFCF)
0.0850
(DGCF)
0.0857
(DGCF)
0.2615†
(GFCF)
0.2428
(DGCF)
0.4093
(NGCF)
0.3240
(DGCF)
0.2407 0.4934†
(LightGCN) (LR-GCCF)
0.0871
(LR-GCCF)
0.2479 0.5090
(LR-GCCF) (LR-GCCF)
0.0883 0.2536 0.5090
(LR-GCCF) (LR-GCCF) (LR-GCCF)
0.1223 0.0884 0.2532
(LR-GCCF) (LR-GCCF) (LR-GCCF)
0.5038
(LR-GCCF)
0.1223 0.0884 0.2536 0.5090 0.3438
(LR-GCCF) (LR-GCCF) (LR-GCCF) (LR-GCCF) (LR-GCCF)
0.3433
(GAT-CF)
0.3823†
(GAT-CF)
0.3424
(GAT-CF)
0.3438†
(LR-GCCF)
0.3823
(GAT-CF)
0.3627
(GAT-CF)
0.3644
(GAT-CF)
0.3823
(GAT-CF)
0.0002
(DGCF)
0.0002†
(DGCF)
0.0002†
(DGCF)
0.0002†
(DGCF)
0.0002†
(DGCF)
0.0002
(DGCF)
0.0101
(GCN-CF)
0.0002
(DGCF)
0.0002
(DGCF)
0.0002
(DGCF)
0.0169
(LightGCN)
0.0022†
(UltraGCN)
0.0209
(NGCF)
0.0022†
(UltraGCN)
0.0388
(LightGCN)
0.0301
(DGCF)
0.0169
(LightGCN)
0.0054
(UltraGCN)
0.0111
(UltraGCN)
0.0022
(UltraGCN)
are studied: (1) implicit message-passing; (2) explicit message-passing with neighborhood
weighting; (3) explicit message-passing without neighborhood weighting. Plots with extensive
results are available in the extended work [
Accuracy/Item Exposure. Explicit/weighted models exhibit a trade-of, maximizing either accuracy or item exposure. Explicit/unweighted models show a balanced trade-of, not prioritizing a single goal. Implicit models prioritize accuracy, at the expense of niche item exposure.
Accuracy/User Fairness. No graph CF strategy emerges as an absolute winner. Every graph CF strategy is insuficient to guarantee adequate fairness among diferent user groups.
Item Exposure/User Fairness. Two groups of baselines are observed: those with poor item exposure and those with acceptable exposure for long-tail items. Explicit/unweighted strategies can generally ensure a satisfactory trade-of between user fairness and item exposure.
We assess the performance of graph CF models on Consumer and Producer (CP)-fairness metrics showing that their superior accuracy capabilities is reached at the expense of user fairness, item exposure, and their combination. By recognizing nodes representation and neighborhood exploration as the two main dimensions of a novel graph CF taxonomy, we study their influence on CP-fairness and overall accuracy separately and simultaneously. The outcomes raise concerns about the efective application of recent approaches in graph CF (e.g., implicit message-passing techniques). On such basis, we are performing further investigations on other datasets and algorithms, and we are working on new graph models balancing accuracy and CP-Fairness. [9] H. Abdollahpouri, R. Burke, B. Mobasher, Controlling popularity bias in learning-to-rank recommendation, in: RecSys, ACM, 2017, pp. 42–46. [10] Y. Deldjoo, V. W. Anelli, H. Zamani, A. Bellogín, T. D. Noia, A flexible framework for evaluating user and item fairness in recommender systems, User Model. User Adapt.
Interact. 31 (2021) 457–511. [11] V. Paparella, Pursuing optimal trade-of solutions in multi-objective recommender systems, in: RecSys, ACM, 2022, pp. 727–729. [12] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: ICLR (Poster), OpenReview.net, 2017. [13] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: ICLR (Poster), OpenReview.net, 2018. [14] X. Wang, X. He, M. Wang, F. Feng, T. Chua, Neural graph collaborative filtering, in: SIGIR,
ACM, 2019, pp. 165–174. [15] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, M. Wang, Lightgcn: Simplifying and powering graph convolution network for recommendation, in: SIGIR, ACM, 2020, pp. 639–648. [16] X. Wang, H. Jin, A. Zhang, X. He, T. Xu, T. Chua, Disentangled graph collaborative filtering, in: SIGIR, ACM, 2020, pp. 1001–1010. [17] L. Chen, L. Wu, R. Hong, K. Zhang, M. Wang, Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach, in: AAAI, AAAI Press, 2020, pp. 27–34. [18] K. Mao, J. Zhu, X. Xiao, B. Lu, Z. Wang, X. He, Ultragcn: Ultra simplification of graph convolutional networks for recommendation, in: CIKM, ACM, 2021, pp. 1253–1262. [19] Y. Shen, Y. Wu, Y. Zhang, C. Shan, J. Zhang, K. B. Letaief, D. Li, How powerful is graph convolution for recommendation?, in: CIKM, ACM, 2021, pp. 1619–1629.