Adopting Knowledge Graphs (KGs) in recommender systems has engendered the emergence of sophisticated techniques, such as path reasoning, designed to navigate KGs and model complex relationships. KGs enable the representation of intricate connections, while path reasoning approaches adeptly learn to traverse these graphs, constructing detailed user-product relationships by discerning reasoning paths linking recommended products with those previously experienced by users. These identified paths are subsequently converted into well-articulated textual explanations, facilitating a deeper and more comprehensive understanding for the users. Despite its potential, the field is hindered by disparate and insuficient evaluation protocols, complicating eforts to assess the impact of existing methodologies. In this paper, we summarize our previous work on replicating and evaluating three state-of-the-art path reasoning recommendation approaches, originally presented at prestigious conferences, using a standardized protocol based on two publicly available datasets and benchmarking them against other knowledge-aware techniques. Our analysis encompasses recommendation utility, explanation quality, and fairness considerations for both consumers and providers. This investigation ofers a comprehensive overview of the progress in the field, emphasizing key challenges and potential avenues for future exploration. Source code is available at https://github.com/giacoballoccu/rep-path-reasoning-recsys.
Recommender systems (RS) have become a prevalent approach for facilitating personalized
user experiences. Traditional RSs are trained using historical data, such as browsing activity
and ratings, as well as product characteristics, such as their textual description. To augment
product information, Knowledge Graphs (KGs) have been employed as an additional data
source [
Path reasoning methods stand out among KG-based recommendation techniques, as they
utilize high-order relations between users and products to guide RS training and generate
explanations [
Numerous KG- and path-reasoning-based methods have been proposed for recommendation
and explanation generation [
In this paper, we summarize our prior work [17] on conducting a replicability study of path reasoning methods, focusing on unexplored evaluation perspectives. We initially reviewed top-tier conference and journal proceedings, identifying seven relevant papers, but only three methods were replicable using the released source code. We then established a common evaluation protocol, encompassing two public datasets with two user sensitive attributes, and sixteen metrics across four perspectives. Path reasoning methods were assessed under this protocol and compared to other knowledge-aware methods. Our results indicate that despite similar utility, these methods difer in achieving other recommendation goals. This study highlights the need for broader evaluation and responsible adoption of path reasoning methods.
Paper Collection. To gather existing path reasoning methods, we systematically reviewed proceedings from top-tier information retrieval events and journals published by leading publishers. We employed keywords combining technical and non-technical terms and identified papers that addressed recommendation methods augmented with KGs, and produce reasoning paths. We excluded papers on other domains or tasks, as well as knowledge-aware methods that could not generate reasoning paths. Seven relevant papers were selected for our study. Methods Replicability. For each relevant paper, we analyzed the rationale of the proposed method and the components of the experimental setting, as summarized in Table 1. We then attempted to replicate each method using the original source code, making necessary changes to accommodate diferent datasets and extract recommendations and reasoning paths. Out of the seven papers, three (PGPR, UCPR, CAFE) were replicable with reasonable efort, while EKAR, PLM-Rec, MLR and TAPR were not reproducible due to the unavailability of the original source code and, in the case of MLR, due to unavailability of external dependencies. Evaluation Protocol. We combined replication and reproduction to create a unified evaluation framework for assessing path reasoning methods on two public datasets, MovieLens-1M (ML1M) and LastFM-1M (LFM1M). We relied on original source codes, our data processing, and computed evaluation metrics based on returned recommendations and paths. The datasets were chosen due to the availability of demographic attributes (Gender, Age) for fairness assessment.
We preprocessed and prepared the data, and performed a temporal-based training-validationtest split for each dataset. Additionally, we compared path reasoning methods against nonexplainable knowledge-aware models (namely, CKE, CFKG, and KGAT), which were replicated and evaluated under the same protocol. Hyperparameters were fine-tuned using grid search.
The evaluation metrics computed for each model include recommendation utility (NDCG and MRR [21]), beyond utility objectives (COVerage, DIVersity, SERendipity, and NOVelty) and both consumer and provider fairness. For explanation path quality, based on [14], we measured perspectives related to recency (R), popularity (P), and diversity (D) of diferent path portions, named Linked Interaction (LI), Shared Entity (SE), and Path Type (PT). The detailed procedure, including metric definitions and implementation details, can be found in our original work [ 17].
Trading Recommendation Goals for Explanation Power (RQ1). First, we compared path
reasoning methods (PGPR, CAFE, UCPR) with knowledge-aware but non-explainable methods
(KGAT, CKE, CFKG) in terms of recommendation utility, beyond utility objectives, and fairness.
The goal is to assess if any trade-of between explainable power (i.e., being able to produce
a textual explanation) and other goals exists. The results, reported in Table 2, showed that
path reasoning methods achieved comparable recommendation utility to knowledge-aware
methods in the ML1M dataset, but lower utility in the LFM1M dataset. However, path reasoning
methods generally outperformed knowledge-aware methods in terms of serendipity, diversity,
and provider fairness. Overall, path reasoning methods sacrificed recommendation utility and
coverage for increased explanation power, particularly in the LFM1M dataset.
Producing Explanations for All Recommended Products (RQ2). We then investigated
the ability of path reasoning methods (PGPR, UCPR, CAFE) of producing explanations for
recommended products across diferent list sizes. To this end, as a metric, we consider fidelity
[
Diferences on Explanation Quality (RQ3). Lastly, we explored how the quality of selected paths and resulting explanations varies based on path characteristics in path reasoning methods (PGPR, UCPR, CAFE). We considered seven explanation path quality perspectives and found that the methods often yield substantially diferent paths in terms of recency, popularity, and diversity. However, no remarkable disparate impacts on explanation quality were observed. This analysis highlights the importance of understanding the specific characteristics of each path reasoning method when selecting one for a particular application.
In this paper, we compared path reasoning methods and knowledge-aware non-explainable
baselines in terms of utility, beyond utility perspectives, and fairness objectives. Results indicate
that path reasoning methods slightly divergence in terms of utility and coverage but result in
higher serendipity and diversity. We also investigated the ability of path reasoning methods
to produce explanations across various recommended list sizes and found that model design
choices can influence this capability. Lastly, we examined the quality of reasoning paths and
empirically show that not all goals can be met simultaneously, aligning to [
In the next steps, we plan to explore in detail the impact of KG characteristics on the considered perspectives, as well as devise novel path reasoning methods robust to the KG structure and efective on multiple objectives. especially on the quality of the provided textual explanations. [13] Y. Zhao, X. Wang, J. Chen, Y. Wang, W. Tang, X. He, H. Xie, Time-aware path reasoning on knowledge graph for recommendation, ACM Trans. Inf. Syst. (2022). Just Accepted. [14] G. Balloccu, L. Boratto, G. Fenu, M. Marras, Reinforcement recommendation reasoning through knowledge graphs for explanation path quality, Knowledge-Based Systems 260 (2023) 110098. [15] C. Upadhyay, H. Abu-Rasheed, C. Weber, M. Fathi, Explainable job-posting recommendations using knowledge graphs and named entity recognition, in: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2021, pp. 3291–3296. doi:10.1109/SMC52423.2021.9658757. [16] F. Zhang, N. J. Yuan, D. Lian, X. Xie, W.-Y. Ma, Collaborative knowledge base embedding for recommender systems, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 353–362. [17] G. Balloccu, L. Boratto, C. Cancedda, G. Fenu, M. Marras, Knowledge is power, understanding is impact: Utility and beyond goals, explanation quality, and fairness in path reasoning recommendation, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2023, pp. 3–19. [18] S. Geng, Z. Fu, J. Tan, Y. Ge, G. de Melo, Y. Zhang, Path language modeling over knowledge graphsfor explainable recommendation, in: Proceedings of the ACM Web Conference 2022, WWW ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 946–955. [19] G. Linden, B. Smith, J. York, Amazon.com recommendations: Item-to-item collaborative ifltering, IEEE Internet Comput. 7 (2003) 76–80. [20] Y. Cao, X. Wang, X. He, Z. Hu, C. Tat-seng, Unifying knowledge graph learning and recommendation: Towards a better understanding of user preference, in: WWW, 2019. [21] N. Craswell, Mean Reciprocal Rank, Springer US, 2009, pp. 1703–1703. [22] G. Balloccu, L. Boratto, G. Fenu, M. Marras, Post processing recommender systems with knowledge graphs for recency, popularity, and diversity of explanations, in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 646–656.