How Neighborhood Exploration influences Novelty and Diversity in Graph

Abstract

Graph convolutional networks (GCNs) have recently been shown to improve the recommendation accuracy of collaborative filtering algorithms. Their message-passing schema refines user and item node representation by aggregating the informative content from the neighborhood. However, noisy contributions can flatten the diferences among nodes after multiple hops, as not all user-item interactions are equally important. This impact is mitigated by (i) restricting the exploration depth in the graph and optionally weighting the neighbor contribution and (ii) going beyond the traditional message propagation at multiple hops. Nevertheless, it remains unclear how these exploration strategies afect the recommendation of novel and diverse products. This study investigates the influence of such GCN techniques on novelty and diversity of recommendations. It also assesses and motivates the impact of the number of exploration hops on the same metrics by analyzing interactions between same-type and diferent-type nodes, such as user-user and user-item. Code and datasets are available at: https://github.com/sisinflab/Novelty-Diversity-Graph. CCS Concepts: • Information systems → Recommender systems; • Computing methodologies → Neural networks. ∗Copyright 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Presented at the MORS workshop held in conjunction with the 16th ACM Conference on Recommender Systems (RecSys), 2022, in Seattle, USA. †Authors are listed in alphabetical order. Corresponding authors: Daniele Malitesta ([e-mail]) and Claudio Pomo ([e-mail]).

1 INTRODUCTION

In the challenge of bridging the gap between supply and demand, popular companies (e.g., Amazon, Booking) have opted to integrate recommendation systems into their online platforms. These algorithms attempt to present customers with personalized lists of preferred products by identifying preference patterns among users and items. Among the existing recommendation paradigms, collaborative filtering (CF) [ 9 ] has long settled as the dominant approach, suggesting that like-minded users could interact with similar items. CF models optimize an objective score function between users and items, where both of them are mapped into embeddings and combined linearly (e.g., inner product [ 21 ]) or non-linearly (e.g., neural networks [ 16 ] and probabilistic models [ 24 ]).

At the same time, the natural representation of users and items in a recommendation system is a bipartite, undirected graph, where users and items are the nodes and recorded interactions are the edges linking them. For this reason, graph convolutional networks (GCNs) [ 20 ] have gained traction in CF-based recommendation, from pioneering works [ 38, 42 ] to more recent solutions [ 15, 25, 29 ].

Graph convolution relies upon the concept of message-passing networks [ 11 ] to refine nodes’ representation, where each ego node embedding is refined by aggregating its’ neighbors node embeddings (i.e., whose contribution is called message). The procedure is performed iteratively over multiple hops, therefore exploring deeper and deeper neighborhoods surrounding the ego node. Diferently from previous CF approaches, the adoption of a message-passing schema helps explicitly incorporate user and item high-order relationships into their embedding representations, therefore efectively distilling the collaborative signal [ 38 ]. Nevertheless, GCN performance has been shown to decrease as the number of explored hops increases since graph convolution indiscriminately aggregates all contributions from the neighbor nodes (even unimportant ones), eventually smoothing the diferences in the neighborhood [ 7, 45 ].

To mitigate this over-smoothing efect, graph-based techniques for collaborative filtering limit the exploration of neighborhood to three hops [ 8, 15, 38 ]. Similar approaches are designed to weight the importance of each neighbor node on its ego node through attention mechanisms [ 36 ], which allows the exploration of even smaller portions of the neighborhood to reach remarkable results [ 39 ].

Conversely, recent works [ 25, 29 ] highlight critical limitations in the adoption of graph convolution to explore users’ and items’ neighborhoods. Starting from the idea described in [ 15 ], they propose alternative reformulations of GCN for the recommendation task, providing simplified and lighter versions which go beyond the traditional concept of multi-hop message-passing. By comparing these latter approaches to the ones described earlier, we might categorize them all into two families, namely, graph recommendation techniques performing explicit (e.g., [ 8, 15, 38, 39 ]) and implicit (e.g., [ 25, 29 ]) message-passing.

Although the literature has widely shown the recommendation accuracy boost of such models to traditional (i.e., nongraph) CF baselines, their ability to produce novel and diverse recommendation lists [ 34, 35 ] remains poorly investigated. While the topic of multi-objective recommendation has been addressed only recently by few works in graph CF [ 32, 44 ], modern recommender systems are more and more required to reach a suficient trade-of between accurate and novel/diverse recommendations [ 23, 31, 41 ], as a renewed need from both user’s and business’s perspectives [ 1, 2, 22 ].

This paper seeks to understand how and why the neighborhood exploration strategy and (optionally) depth may influence novelty and diversity recommendation metrics in graph collaborative filtering. To this aim, we run extensive experiments by training and evaluating six state-of-the-art graph CF models on three popular recommendation datasets.

Our contributions are threefold: (i) to the best of our knowledge, no previous work has evaluated approaches from the two recognized graph recommendation families (i.e., explicit and implicit message-passing) on a grid of accuracy/novelty/diversity recommendation metrics, (ii) to provide a fair comparison, we train all explicit messagepassing models exploring the whole hop range 1-4, which also allows examining the accuracy/novelty/diversity trade-of on the neighborhood size, and (iii) we propose a simple reformulation of the explicit message-passing schema where same-type node connections (e.g., user-user) and different-type node connections (e.g., user-item) are formally highlighted, in an efort to unveil their influence on the metrics’ trade-of. 2

RELATED WORK

This section provides an overview of graph collaborative filtering and novelty/diversity in recommendation. The scope is to underline the contributions of our work to the existing literature. 2.1

Graph Collaborative Filtering

After pioneer works [ 33, 42 ] adopting vanilla GCN [ 20 ] for recommendation, other approaches propose finer neighborhood explorations built upon it. Wang et al. [ 38 ] aggregate the messages from the neighborhood considering the similarity between each neighbor node and its ego node, while the works in [ 8, 15, 18 ] improve accuracy when removing non-linearities and feature transformations. As neighbor nodes are not equally important to their ego node, noisy messages tend to over-smooth the existing node diferences after multiple hops [ 7, 45 ]. To tackle the issue, messages are propagated to a maximum of three hops [ 15, 38 ], optionally leveraging attention mechanisms [ 36 ] to learn the importance of users’ intents on the interacted items [ 39, 40 ]. The above-cited works leverage what we might define as an explicit message aggregation, meaning that it is always possible to derive a formulation where user and item node embeddings are explicitly updated through their multi-hop neighbors. Conversely, following a diferent rationale, more recent approaches take a step further and try to rethink the message-passing schema by allowing theoretically unlimited propagation hops [ 25 ] and revisiting the concept of graph convolution and node embedding smoothness through the lens of graph signal processing [ 29 ]. To distinguish such techniques from the explicit ones, in this work, we introduce the concept of implicit message-passing, where message aggregation is replaced and improved through ad-hoc mathematical proxies.

Contribution 1. We study the influ ence of explicit and implicit message-passing on accuracy/novelty/diversity recommendation trade-of. Additionally, focusing on explicit message-passing, we propose a simple mathematical reformulation of the message aggregation, highlighting same- and different-type node explorations (see later). 2.2

Novelty and Diversity in Recommendation

User experience is becoming crucial on recommendation platforms [ 17, 19, 30 ] as the suggestion of interesting lists of items satisfies users and entices them to remain loyal to the platform, thus increasing profits [ 37 ]. A good user experience requires the recommended items to be nontrivial, as diverse as possible, and possibly unexpected [ 10, 30 ]. However, designing dedicated models is particularly challenging due to the inherent dificulty of evaluating them without a user study. For this reason, researchers have dedicated a considerable efort to the beyond-accuracy dimensions over the past two decades [ 28, 35, 43 ]. While the search for the accuracy/novelty/diversity trade-of has gained momentum in recommendation [ 2, 5, 23, 31, 41 ], to the best of our knowledge, only two studies investigate novelty and diversity dimensions in the field of graph collaborative filtering [ 32, 44 ]. They focus on identifying the accuracy/diversity trade-of by proposing specific models that could achieve competitive performance. However, they do not deepen into analyzing the influence of neighborhood exploration on the highlighted dimensions.

Contribution 2. On the contrary, in this work, we assess the state-of-the-art, most accurate models for graph recommendation and inspect how they behave on novelty and diversity, exploring the potential motivations with a focus on their different neighborhood exploration strategies. 3

REFORMULATING EXPLICIT MESSAGE-PASSING

Starting from the novel model classification for graph collaborative filtering outlined in this work (i.e., neighborhood exploration approaches leveraging explicit or implicit message-passing), in this section we propose a simple (but useful) reformulation for the former family where same- and different-type node interactions (e.g., user-user and user-item, respectively) are formally highlighted. 3.1

Preliminaries

Let U = {1, 2, . . . , } and I = {1, 2, . . . , } be the sets of users and items. Starting from U and I, we consider the bipartite and undirected graph connecting pairs of nodes (i.e., users and items) with an existing interaction among them. User and item node features are the embeddings e ∈ R , ∀ ∈ U and e ∈ R , ∀ ∈ I, respectively. , e(1) = ne(0′) , ∀ ′ ∈ N () o where e(1) and e(1) are the refined embedding versions of user and item after one hop, (·) is the aggregation function (e.g., the summation), while e(0′) = e′ and e(′0) = e′ . To explore deeper and deeper neighborhoods of the ego nodes, aggregation is usually iterated. After two hops, the embeddings of user and item are: o o e(2) = ne(′1) , ∀ ′ ∈ N ()

, e(2) = ne(1′) , ∀ ′ ∈ N () Thus, the general message-passing formulation after hops is:

o e() = ne(′−1) , ∀ ′ ∈ N () , e() = ne(′−1) , ∀ ′ ∈ N () o 3.3

Proposed reformulation

an item-user-item path connecting with ′′, respectively (Figure 1a). Such paths link same-type nodes. In a similar (1) (2) (3) (4) manner, let us apply the general formula from Equation (3) to the three-hop node update:

e(3) = ne(′2) , ∀ ′ ∈ N ()o , e(3) = ne(2′) , ∀ ′ ∈ N ()o which we expand through Equation (4): e(3) = n n ne(′0′′) , ∀| ′′′ ∈ N3{z-(ho′p′) \ { ′′} } o , ∀ ′′ ∈ N ( ′) \ { ′′} o , ∀ ′ ∈ N () o | 2{z-hop } | 1{z-hop } e(3) = n n ne(0′′)′ , ∀| ′′′ ∈ N3{z-h(o′p′) \ { ′′} } o , ∀ ′′ ∈ N ( ′) \ { ′′} o , ∀ ′ ∈ N () | 2{z-hop } | 1{z-hop } o (5) (6) After three hops, the node embeddings of user and item get the contributions of those items ′′′ and users ′′′ for whom there exists a user-item-user-item path connecting with ′′′, and an item-user-item-user path connecting with ′′′, respectively (Figure 1b). In this case, such paths link different-type nodes.

This reformulation outlines two neighborhood exploration types, propagating messages through same- and differenttype nodes after an even and an odd number of hops, respectively. While previous works assess recommendation performance when indistinctly increasing the hop numbers, we provide a finer evaluation based on the type of the explored nodes. In the next sections, we will count hops following the introduced categorization. For example, same-type node explorations after 1 and 2 hops refer to the paths user-item-user and user-item-user-item-user, respectively, while different-type node explorations after 1 and 2 hops refer to the paths user-item and user-item-user-item, respectively.

4 EXPERIMENTS AND DISCUSSION

In the following, we describe datasets, baselines, reproducibility details, evaluation protocol, and results of our work.

4.1 Experimental Setup

Datasets. We adopt Movielens-1M [ 12 ], Amazon Digital Music [ 26 ], and Epinions [ 27 ]. Following a similar approach to [ 4 ], these datasets are binarized by retaining interactions with a score greater than 3 (Epinions already has an implicit version) and filtered through the -core to avoid the cold-start efect [ 13, 14 ] which is out of the scope of this paper. Movielens-1M counts 5,915 users, 2,753 items, and 570,622 interactions, Amazon Digital Music counts 8,328 users, 6,275 items, and 99,400 interactions, and Epinions counts 14,341 users, 13,145 items, and 269,170 interactions. All datasets statistics are fully reported in Table 1.

Baselines. We evaluate graph recommendation models adopting explicit and implicit message propagation. Explicit message-passing • Neural graph collaborative filtering (NGCF) [ 38 ] proposes to refine users’ and items’ collaborative embeddings by using a GCN-like model which explores the neighborhood and the inter-dependencies among ego and neighbor nodes. • Light graph convolutional network (LightGCN) [ 15 ] lightens and improves the NGCF architecture by removing the embedding projections and non-linear activations in each propagation layer. • Disentangled graph collaborative filtering (DGCF) [ 39 ] weights the importance of neighbor nodes on the ego node by disentangling the intents involved in each user/item interaction for the sake of explainability. • Linear residual graph convolutional collaborative filtering (LR-GCCF) [ 8 ] improves the LightGCN approach by introducing a novel residual block in the convolutional layer for the user-item preference prediction. • Ultra simplification of graph convolutional networks (UltraGCN) [ 25 ] introduces additional objective function components to approximate infinite propagation layers and learn useful item-item connections. • Graph filter based collaborative filtering (GFCF) [ 29 ] leverages graph signal processing to formulate a closedform user-item preference prediction based upon the bipartite graph.

Reproducibility. Datasets are split into train/validation/test with the 80/10/10 hold-out. Models are trained by searching the best hyperparameters as in [ 6 ] and setting search spaces according to the original works while fixing the number of epochs to 400 and batch size to 1024. Our implementation is based upon the Elliot framework for reproducible recommender systems [ 3 ]. To foster the future reproduction of our work, datasets, codes, and configuration files are made accessible to a public GitHub repository1.

Evaluation. First, we use the recall ( @) and the normalized discounted cumulative gain ( @) to measure the recommendation Accuracy of the baselines. Then, following [ 34, 35 ], we select the expected popularity complement (@) and the expected free discovery ( @) as Novelty metrics [ 35 ], along with the 1’s complement of the Gini index (@) and the Shannon entropy (@) as Diversity metrics [ 28 ]. Both the @ and the @ account for long-tail items and measure the expected number of recommended unknown and known items, which are also relevant, respectively. The @ and the @ calculate how unequally a recommender system shows diferent items to users. We set the @20 as validation metric to follow the original papers. For each recommendation metric, higher values stand for better performance. 4.2

Results and Discussion

This section shows the recommendation performance of the tested baselines from a general and a finer evaluation of the accuracy/novelty/diversity trade-ofs. All reported results refer to the top-20 recommendation lists. Overall Recommendation Performance. Table 2 depicts recommendation performance on accuracy, novelty, and diversity, when comparing explicit to implicit message-passing graph approaches in their best configuration.

Coherently with the literature, DGCF and LR-GCCF are steadily the best or the second-to-best models on accuracy (e.g., DGCF reaches the second-to-best on Amazon Digital Music, while LR-GCCF obtains the best on Movielens-1M). Approaches with implicit message aggregation (i.e., UltraGCN and GFCF) still compete with the 1https://github.com/sisinflab/Novelty-Diversity-Graph.

As for the accuracy/novelty/diversity trade-of, we see that, independently of the adoption of message-passing, accurate approaches can also produce novel recommendations (e.g., LR-GCCF and DGCF are the best and secondto-best approaches for accuracy and novelty on Movielens-1M, and GFCF and UltraGCN provide superior accuracy performance on Amazon Digital Music and Epinions, respectively, with GFCF outperforming all other baselines on novelty, and UltraGCN getting slightly lower and values than DGCF). Unexpectedly, NGCF settles as the approach producing the most diverse lists of recommended items on all datasets (i.e., see and ) but cannot cope with the other baselines in terms of and (similarly to Random). Other graph models with explicit message-passing (especially DGCF and LR-GCCF) are placed in the best accuracy/diversity trade-of spot, as they are often the second-to-best approaches on diversity, with limited observable drops in the accuracy. Contrarily, techniques with implicit message aggregation always show the lowest diversity.

Observation 1. While the accuracy/novelty trade-of does not depend on the explicit/implicit message-passing, the accuracy/diversity trade-of is preserved only when explicitly propagating messages, at the expense of (limited) recommendation accuracy drops.

A finer trade-ofs evaluation. by varying the message-passing strategy (i.e., explicit and implicit) and neighbor exploration depth only for the former case. Specifically, we use the reformulation from Section 3.3 to separate explicit message propagation results into same- and different-type node explorations at 1/2 hops.

We confirm that, while UltraGCN and GFCF can compete well on the accuracy/novelty trade-of with the other baselines (whatever the explored number of hops and node type), the opposite occurs on the accuracy/diversity trade-of. Indeed, higher accuracy values for UltraGCN and GFCF are obtained at the expense of significant drops in their diversity, even compared to message propagation at 1 hop (e.g., DGCF surpasses them on diversity at the expense of a slightly lower accuracy in the same-node setting).

As for the influence of same- and different-type node explorations, wider explorations of the neighborhood almost always lead to improved accuracy/novelty and accuracy/diversity performance, independently of the explored node types (apart from the same-type settings for NGCF on the and LR-GCCF on the and the ). Noticeably, the exploration of 1 hop in the same-type node setting leads to a better trade-of in accuracy/novelty/diversity than the exploration of 2 hops in the different-node setting (e.g., LightGCN increases the and the without a significant variation of , and DGCF slightly decreases the and the , but improves ).

Observation 2. To confirm observation 1, explicit message propagation (even at 1 hop) can reach a better accuracy/diversity trade-of than implicit propagation; then, same-type node explorations may lead to improved accuracy/novelty and accuracy/diversity trade-ofs.

5 CONCLUSION AND FUTURE WORK

This work studies the accuracy/novelty/diversity trade-of in graph collaborative filtering for diferent neighborhood exploration strategies (i.e., explicit and implicit message-passing) and depths (i.e., number of explored hops). Results for six state-of-the-art graph models on three e-commerce datasets reveal that the accuracy/diversity trade-of is reachable only when explicitly propagating messages. Thanks to a message-passing reformulation, we show that user-user and item-item explorations may improve accuracy/diversity/novelty trade-of. We plan to expand the evaluation to recent graph models which optimize diversity and better investigate the same-type node setting.

ACKNOWLEDGMENTS REFERENCES

The authors acknowledge partial support from the projects PASSEPARTOUT, ServiziLocali2.0, Smart Rights Management Platform, BIO-D, and ERP4.0.

References

[1] Himan Abdollahpouri , Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Augusto Pizzato. 2020 . Multistakeholder recommendation: Survey and research directions. User Model. User Adapt . Interact . 30 , 1 ( 2020 ), 127 - 158 .

[2] Himan Abdollahpouri , Mehdi Elahi, Masoud Mansoury, Shaghayegh Sahebi, Zahra Nazari, Allison Chaney, and Babak Loni . 2021 . MORS 2021: 1st Workshop on Multi-Objective Recommender Systems . In RecSys. ACM, 787 - 788 .

[3] Vito Walter Anelli , Alejandro Bellogín, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021 . Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation . In SIGIR. ACM , 2405 - 2414 .

[4] Vito Walter Anelli , Alejandro Bellogín, Tommaso Di Noia, Dietmar Jannach, and Claudio Pomo . 2022 . Top-N Recommendation Algorithms: A Quest for the State-of-the-Art . In UMAP. ACM , 121 - 131 .

[5] Vito Walter Anelli , Alejandro Bellogín, Tommaso Di Noia, and Claudio Pomo . 2021 . Reenvisioning the comparison between Neural Collaborative Filtering and Matrix Factorization . In RecSys. ACM , 521 - 529 .

[6] James Bergstra , Rémi Bardenet, Yoshua Bengio, and Balázs Kégl . 2011 . Algorithms for Hyper-Parameter Optimization . In NIPS . 2546 - 2554 .

[7] Deli Chen , Yankai Lin , Wei Li , Peng Li , Jie Zhou , and Xu Sun . 2020 . Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View . In AAAI. AAAI Press, 3438 - 3445 .

[8] Lei Chen , Le Wu , Richang Hong, Kun Zhang, and Meng Wang . 2020 . Revisiting Graph Based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approach . In AAAI. AAAI Press, 27 - 34 .

[9] Michael D. Ekstrand , John Riedl, and Joseph A. Konstan . 2011 . Collaborative Filtering Recommender Systems . Found. Trends Hum. Comput. Interact. 4 , 2 ( 2011 ), 175 - 243 .

[10] Alireza Gharahighehi and Celine Vens . 2021 . Diversification in session-based news recommender systems . Personal and Ubiquitous Computing (29 Jul 2021 ). https://doi.org/10.1007/s00779-021-01606-4

[11] Justin Gilmer , Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl . 2017 . Neural Message Passing for Quantum Chemistry . In ICML (Proceedings of Machine Learning Research , Vol. 70 ). PMLR, 1263 - 1272 .

[12] F. Maxwell Harper and Joseph A. Konstan . 2016 . The MovieLens Datasets: History and Context . ACM Trans. Interact. Intell. Syst. 5 , 4 ( 2016 ), 19 : 1 - 19 : 19 .

[13] Ruining He and Julian J. McAuley . 2016 . Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering . In WWW. ACM , 507 - 517 .

[14] Ruining He and Julian J. McAuley . 2016 . VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback . In AAAI. AAAI Press, 144 - 150 .

[15] Xiangnan He , Kuan Deng, Xiang Wang, Yan Li , Yong-Dong Zhang , and Meng Wang . 2020 . LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation . In SIGIR. ACM , 639 - 648 .

[16] Xiangnan He , Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua . 2017 . Neural Collaborative Filtering . In WWW. ACM , 173 - 182 .

[17] Dietmar Jannach , Lukas Lerche, Iman Kamehkhosh, and Michael Jugovac . 2015 . What recommenders recommend: an analysis of recommendation biases and possible countermeasures . User Model. User Adapt. Interact . 25 , 5 ( 2015 ), 427 - 491 .

[18] Weiguang Jiang , Su Wang, Jun Zheng , and Wenxin Hu . 2021 . LGACN: A Light Graph Adaptive Convolution Network for Collaborative Filtering . In ICANN (3) (Lecture Notes in Computer Science , Vol. 12893 ). Springer, 113 - 126 .

[19] Marius Kaminskas and Derek Bridge . 2017 . Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems . ACM Trans. Interact. Intell. Syst. 7 , 1 ( 2017 ), 2 : 1 - 2 : 42 .

[20] Thomas N. Kipf and Max Welling . 2017 . Semi-Supervised Classification with Graph Convolutional Networks . In ICLR (Poster) . OpenReview.net.

[21] Yehuda Koren , Robert M. Bell , and Chris Volinsky . 2009 . Matrix Factorization Techniques for Recommender Systems . Computer 42 , 8 ( 2009 ), 30 - 37 .

[22] Mounia Lalmas . 2020 . Personalising and Diversifying the Listening Experience . In ICTIR. ACM , 3 .

[23] Alfonso Landin , Javier Parapar, and Álvaro Barreiro . 2020 . Novel and Diverse Recommendations by Leveraging Linear Models with User and Item Embeddings . In ECIR (2) (Lecture Notes in Computer Science , Vol. 12036 ). Springer, 215 - 222 .

[24] Dawen Liang , Rahul G. Krishnan, Matthew D. Hofman , and Tony Jebara . 2018 . Variational Autoencoders for Collaborative Filtering . In WWW. ACM , 689 - 698 .

[25] Kelong Mao , Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang , and Xiuqiang He . 2021 . UltraGCN: Ultra Simplification of Graph Convolutional Networks for Recommendation . In CIKM. ACM , 1253 - 1262 .

[26] Julian J. McAuley , Christopher Targett , Qinfeng Shi , and Anton van den Hengel. 2015 . Image-Based Recommendations on Styles and Substitutes . In SIGIR. ACM , 43 - 52 .

[27] Matthew Richardson , Rakesh Agrawal, and Pedro M. Domingos . 2003 . Trust Management for the Semantic Web . In ISWC (Lecture Notes in Computer Science , Vol. 2870 ). Springer, 351 - 368 .

[28] Guy Shani and Asela Gunawardana . 2011 . Evaluating Recommendation Systems . In Recommender Systems Handbook . Springer, 257 - 297 .

[29] Yifei Shen , Yongji Wu, Yao Zhang, Caihua Shan, Jun Zhang, Khaled B. Letaief , and Dongsheng Li . 2021 . How Powerful is Graph Convolution for Recommendation? . In CIKM. ACM , 1619 - 1629 .

[30] Thiago Silveira , Min Zhang, Xiao Lin, Yiqun Liu, and Shaoping Ma. 2019 . How good your recommender system is? A survey on evaluations in recommendation . Int. J. Mach. Learn. Cybern . 10 , 5 ( 2019 ), 813 - 831 .

[31] Dusan Stamenkovic , Alexandros Karatzoglou, Ioannis Arapakis, Xin Xin, and Kleomenis Katevas . 2022 . Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning . In WSDM. ACM , 957 - 965 .

[32] Jianing Sun , Wei Guo, Dengcheng Zhang, Yingxue Zhang, Florence Regol, Yaochen Hu, Huifeng Guo, Ruiming Tang, Han Yuan, Xiuqiang He , and Mark Coates . 2020 . A Framework for Recommending Accurate and Diverse Items Using Bayesian Graph Convolutional Neural Networks . In KDD. ACM , 2030 - 2039 .

[33] Rianne van den Berg, Thomas N. Kipf , and Max Welling . 2017 . Graph Convolutional Matrix Completion . CoRR abs/1706 .02263 ( 2017 ).

[34] Saúl Vargas . 2014 . Novelty and diversity enhancement and evaluation in recommender systems and information retrieval . In SIGIR. ACM , 1281 .

[35] Saul Vargas and Pablo Castells . 2011 . Rank and relevance in novelty and diversity metrics for recommender systems . In RecSys. ACM , 109 - 116 .

[36] Petar Velickovic , Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio . 2018 . Graph Attention Networks . In ICLR (Poster) . OpenReview.net.

[37] Sanne Vrijenhoek , Mesut Kaya, Nadia Metoui, Judith Möller, Daan Odijk, and Natali Helberger . 2021 . Recommenders with a Mission: Assessing Diversity in News Recommendations . In CHIIR. ACM , 173 - 183 .

[38] Xiang Wang , Xiangnan He, Meng Wang , Fuli Feng , and Tat-Seng Chua . 2019 . Neural Graph Collaborative Filtering . In SIGIR. ACM , 165 - 174 .

[39] Xiang Wang , Hongye Jin , An Zhang, Xiangnan He, Tong Xu , and Tat-Seng Chua . 2020 . Disentangled Graph Collaborative Filtering . In SIGIR. ACM , 1001 - 1010 .

[40] Junkang Wu , Wentao Shi, Xuezhi Cao, Jiawei Chen, Wenqiang Lei, Fuzheng Zhang, Wei Wu, and Xiangnan He . 2021 . DisenKGAT: Knowledge Graph Embedding with Disentangled Graph Attention Network . In CIKM. ACM , 2140 - 2149 .

[41] Wenzhuo Yang , Jia Li , Chenxi Li , Latrice Barnett , Markus Anderle, Simo Arajärvi, Harshavardhan Utharavalli, Caiming Xiong, and Steven C. H. Hoi . 2021 . On the Diversity and Explainability of Recommender Systems: A Practical Framework for Enterprise App Recommendation . In CIKM. ACM , 4302 - 4311 .

[42] Rex Ying , Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton , and Jure Leskovec . 2018 . Graph Convolutional Neural Networks for Web-Scale Recommender Systems . In KDD. ACM , 974 - 983 .

[43] ChengXiang Zhai , William W. Cohen, and John D. Laferty . 2003 . Beyond independent relevance: methods and evaluation metrics for subtopic retrieval . In SIGIR. ACM , 10 - 17 .

[44] Yu Zheng , Chen Gao , Liang Chen, Depeng Jin, and Yong Li . 2021 . DGCN: Diversified Recommendation with Graph Convolutional Networks . In WWW. ACM / IW3C2 , 401 - 412 .

[45] Kaixiong Zhou , Xiao Huang, Yuening Li , Daochen Zha , Rui Chen, and Xia Hu . 2020 . Towards Deeper Graph Neural Networks with Diferentiable Group Normalization . In NeurIPS.