Recommender systems, a specialised subfield within information retrieval, are crucial for identifying items that align with users' preferences. A knowledge graph-based recommender system can excel in the task of making recommendations due to the rich semantic information inherent in knowledge graphs. In this paper, our central focus is to investigate the impact of the semantic richness of knowledge graphs on the efectiveness of such recommender systems. To explore this research topic, we focus on the movie recommendation domain. For this, we create seven movie ontologies with varying levels of semantic richness, and combine these ontologies with movie data from the MovieLens 1M dataset and augmented using additional open linked data derived from Wikidata to produce seven diferent movie knowledge graphs. We provide the ontologies and knowledge graphs in an open-source repository. We then conduct experiments using two diferent approaches, consisting of nine Knowledge Graph Based Recommending (KGBR) methods and four Link Prediction (LP) methods based on Knowledge Graph Embeddings (KGE). The results demonstrate that richness of the knowledge graph does not impact the performance of KGBR methods significantly, but has a considerable impact on the KGE-based LP methods. We furthermore compare the best performing KGBR methods with the KGE-based LP methods, showing that the LP methods outperform all other recommendation methods when paired with the most extensive knowledge graph. From this, we conclude that the richness of the knowledge graph does not have a significant impact if the method already integrates other recommending approaches, but can heavily impact the LP methods employing the KGE approach, which interprets relationships as translations in embedding spaces. This supports the idea that using extended knowledge graphs is an efective approach for successful recommender systems.
Recommender systems play a critical role in a world where individuals are subject to more and more digital information and must make choices whether to purchase goods or access information. Recommendations can be applicable to many scenarios, such as suggesting which movie to watch, which book to read, which restaurant to go or which products to buy. The central purpose of recommender systems is to meet the need or preference of the people or group in question, therefore personalised decision support is needed for a successful recommendation system. The recommending process is implemented by algorithms able to analyse a user’s data and profile, as well as the landscape of possibilities, and thus to predict items that the user might be interested in.
There are many types of recommender systems, whose algorithms are based on techniques such as
content-based filtering, collaborative filtering, hybrid filtering, or popularity metrics, each with their
own strengths [
The main contributions of this paper are as follows: 1) Seven modular movie ontologies with varying levels of semantic richness. 2) Seven movie knowledge graphs based on the ontologies, combined with data from MovieLens 1M dataset, Wikidata and data produced from additional statistical analysis. 3) An extensive comparison between nine Knowledge Graph Based Recommender Systems and four Link Prediction approaches. To measure the impact of the knowledge graph richness on the recommender system performance, we design an evaluation task using existing evaluation metrics. To support future reuse, we provide the ontologies, knowledge graphs, our code and dataset used in the experiments as open access in our repository1.
The rest of this paper is structured as follows: in Section 2 we describe the background theory on recommendation systems, and more specifically, knowledge graph based recommender systems. In Section 3 we describe our method of ontology creation, knowledge graph generation and recommending model selection. In Section 4 we describe our experiments and in Section 5 we note our results and analysis. Finally, in Section 6 we conclude our paper and discuss possible future avenues.
Recommender systems are a type of information filtering systems designed to predict a user’s ratings or preferences for items, thereby enhancing user experience and engagement. The fundamental function of these systems can be expressed as ˆ = (, , ; ), where ˆ predicts the user ’s interest in item , provides additional contextual parameters, and represents the model’s parameters. These systems are categorised into: Content-Based Filtering, which recommends items similar to those a user has previously liked, based on item features (ˆ = x⊤ y), where x and y are feature vectors representing user preferences and item attributes, respectively; Collaborative Filtering, which makes recommendations using the ratings behaviour of other users (ˆ = p⊤ q), with p and q as the latent factor vectors for users and items, respectively; and Hybrid Approaches, combining both methods to leverage their strengths (ˆ = · x⊤ y + (1 − ) · p⊤ q), where is a parameter balancing the contribution of each method. These methods allow recommender systems to tailor responses based on individual user preferences, thereby improving the eficiency and user satisfaction.
Ontologies formally represent concepts and relationships within a specific domain, which can be instantiated through Knowledge Graphs. Such knowledge graphs are used to represent a network of realworld entities — such as objects, events, situations, or concepts — and illustrates the interrelationships between them. It efectively organises information in a way that facilitates not only retrieval but also inference, making it an invaluable tool for enhancing the capabilities of recommender systems and other artificial intelligence applications. A knowledge graph can be modelled as a set of triples, denoted as (, , ), where stands for the subject, the predicate (or relation), and the object. This triple format encapsulates the relationships within the data, allowing systems to leverage structured semantic information for advanced reasoning and query processing.
Knowledge Graph Based Recommender (KGBR) systems use such complex graph-structured networks
of entities and their interrelationships to enhance recommendation accuracy and relevance to context
1https://github.com/XuWangDACS/Rich_KGRS
by leveraging semantically rich information inherent in the graphs. Wang et al. introduce the
RippMKR model, a novel multitask feature learning framework that leverages the benefits of knowledge
graph embeddings through RippleNet to enhance recommender systems, demonstrating superior
performance over existing methods in various recommendation scenarios [
To build the required knowledge graphs, we use three data sources: MovieLens 1M, Wikidata and additional statistical data. For the purpose of testing diferent levels of semantic richness, we create seven ontologies in RDF and RDFS, each with additional levels of information density and linking. These ontologies vary both in size as well as structure. Four ontologies, each designed with a gradually increasing level of semantic complexity, are set up as follows:
O1 General Information Extension, which includes additional general attributes.
O2 Movie Information Extension, which includes attributes specifically regarding the movies. O3 User-Movie Linking Extension, which includes links between Users and Movie attributes.
We create an additional ‘plus’ version of each ontology, which holds the extended attributions as concepts instead of literals, to support further extensions. These ontologies are called O1+, O2+ and O3+. All of the ontologies (with the namespace mr: <http://ontology.tno.nl/movie-rating-ontology/>) are included in our repository in the TTL file format 1.
Ontology O0 (Figure 1) serves as the base ontology, which holds just three concepts for a User, which provides a Rating about a Movie. These concepts are linked with each other through the mr:hasRating, mr:isAboutMovie, mr:providesRating, and mr:providedBy relations.
Continuing on top of O0, ontologies O1 and O1+ (Figure 2) serve as extensions that also include semantic properties regarding MovieLens. They include concepts that belong to the Users: mr:Gender, mr:Zipcode, mr:MovieLensID, mr:Occupation, concepts that belong to the Ratings: mr:Timestamp and mr:AmountOfStars, as well as concepts that belong to the Movies: mr:Title and mr:Genre.
On top of O1+, we build ontologies O2 and O2+ (Figure 3). These extensions also contain information from IMDb through Wikidata. They include concepts for Movies’ mr:IMDbID, mr:Duration, mr:ReleaseDate, and associated parties: mr:Actor, mr:Producer, mr:Director and mr:Writer.
Finally, on top of O2+, we build ontologies O3 and O3+ (Figure 4). These extensions also convey extended information based on statistical information about the Users and the Movies they have rated. They include concepts for a User Liking Movies and Genres (defined by having the Rating >2.5), Having Favourite Movies and Genres (defined by highest Rating) and Having Top 5 Favourite Movies and Genres (defined by top 5 defined by highest Rating). Furthermore, it includes relations when the User Likes movies that involve certain Actors, Directors, Producers and Writers (defined by having the Rating of a Movie with these Persons involved >2.5). Finally, it includes additional links between Actors, Directors, Producers and Writers that have worked together with each other, and even works often with each other (defined by having an above average cooperation rate over all of the movies).
Based on the ontology concepts and relations we have previously defined, we generate a movie-user knowledge graph by integrating the MovieLens datasets with Wikidata’s open data and mapping this data to the ontologies. The initial step involves mapping the MovieLens datasets to Wikidata. The MovieLens datasets, specifically the 1M dataset used in this research, provide basic information about users, movies, and user interactions. However, these datasets lack semantic depth. To enhance this, we extract additional movie data from Wikidata. Notably, the MovieLens 1M dataset can be easily linked to the MovieLens 20M dataset as they share the same MovieLens movie IDs. The 20M dataset includes links to IMDb IDs, which are widely used in IMDb’s non-commercial data2. Given that movie entities in Wikidata are also linked to these IMDb IDs, we can map movies from the MovieLens 1M dataset to corresponding entries in Wikidata using IMDb IDs.
After establishing the necessary mappings, we enrich the knowledge graph with detailed movie data retrieved from Wikidata that is not available in the MovieLens datasets. In constructing the knowledge graph, we use predicate URIs that are not only defined in our custom ontology but also those mapped from Wikidata. The entity URIs within the knowledge graph leverage Wikidata URIs for the data sourced therefrom, and we create additional URIs consistent with the namespace defined in our ontology.
We compare the following nine KGBR systems: CKE integrates collaborative filtering and knowledge
base embeddings to enrich the feature representation of users and items, leveraging both user-item
interactions and semantic knowledge graph data [
In addition to these nine methods, we examine the impact of semantic richness on the efectiveness of recommendation systems by framing the recommendation task as a link prediction challenge. The goal is to predict potential recommendation links between users and movies. For this purpose, we employ the TransE [16], DistMult [17], RotatE [18], and SimplE [19] models, to learn vector representations of entities and relations to capture and infer relationships within a knowledge graph, each using diferent mathematical approaches to model these interactions.
We run 91 experiments (13 recommender methods on seven knowledge graphs) to research the impact of richness of knowledge graph on the efectiveness of recommender systems. These experiments are marked as KG0 through KG3+, as shown in Table 1. The source code and datasets of our experiments can be found at our repository1.
We use the RecBole [20] Python library in our experiments. RecBole provides many existing recommender system implementations, including the knowledge based methods we evaluate in our experiments. We use the seven generated knowledge graphs (based on mapping the data sources on the created ontologies) as discussed in Section 3 as the knowledge background for RecBole’s knowledge based methods. We use the PyKEEN [21] Python library to train the TransE embedding of the knowledge graph. After training, we use the cosine similarity between embeddings of user and movie to predict the potential link between them.
Four metrics are used to evaluate the performance of recommendation: MRR@k (Mean Reciprocal Rank at k): Computes the average of reciprocal ranks of the first relevant item in the top-k results, emphasising the importance of higher-ranked relevant items. NDCG@k (Normalised Discounted Cumulative Gain at k): Evaluates the ranking quality of the top-k recommendations by considering the position of relevant items and applying a logarithmic discount based on rank [22]. Hit@k (Hit Rate at k): Calculates the proportion of queries for which at least one relevant item appears among the top-k recommendations, efectively measuring the model’s ability to retrieve relevant items within the top results. And finally, Precision@k: Assesses the fraction of relevant items among the top-k recommendations made by a model. For RecBole, these metrics are already included. For the Link Prediction based methods, we use the ranx [23] Python library to get these measures. In this paper, we choose to set the k of these metrics as 10. This provides a good indication, because the top 10 recommendations is usually of the most importance. The top 10 recommended items is likely to be the most visible to users, especially on platforms where the first page or screen carries high importance.
Table 1 shows the results of the evaluation metrics of all recommender methods over the seven experiment settings (KG1 until KG3+). Highest scores are marked with bold text, second highest are underlined, and each of the metrics the best performing KGBR and LP method are highlighted. As evident from the Table, the best performing methods are KGIN for KGBR methods and and SimplE for LP methods, with SimplE gaining a significant impact on KG 3+.
By analysing Table 1, we can conclude the following: 1) For KGBR methods, the improvement on semantic richness of knowledge graph does not significantly impact the performance of recommendation, on the final experiment, which uses KG 3+, the knowledge graph with the highest semantic richness. Therefore, on the first knowledge graphs with less semantic richness, KGBR methods perform better, while LP methods thrive based on richer knowledge graphs. improves after introducing more links from KG2+ on. With the introduction of more semantic links in the knowledge graphs, SimplE surpasses the KGBR methods in performance with KG3 on Precision and
Regarding the dataset, while MovieLens is a hugely useful because of its user data and the most popular dataset in recent research [24], it is released in 2003 and therefore does not take current movies into account. As movie recommendation is seen in popular everyday household streaming services such as Netflix [ 25, 26], it is interesting to further extend this research with more current data. Our focus on only the MovieLens dataset is a limitation, which could be expanded upon. Popular social platforms where users track their favourite items, such as Letterboxd3 for movies or Goodreads4 [27] for books, could be examined for this, and can provide more recent information. Furthermore, it would be possible to enrich our generated knowledge graphs with other data sources through Wikidata, as Wikidata links movie entities to many other data sources.
One limitation of this work is that both the ontology creation and knowledge graph mapping is a manual, intricate process. However, automatically providing such mappings is still the subject of ongoing research in the field of Ontology Mapping [ 28, 29]. To ensure the ontologies were focused, we purposefully did not model the entire movie domain. To ensure relevancy, concepts and relations that we did include are based on MovieLens and Wikidata, but the ontologies (as provided in our repository1) could be expanded upon with further concepts and relations.
Furthermore, it is only when we introduce additional links in the transition to ontology O3+ that there is a noticeable impact in LP performance. This might be the reason why the Link Prediction methods have such a large increase in performance in KG3+. Future research could be done into extending the ontology and subsequent knowledge graph even more with additional information, with a specific focus on the amount of semantic links to see which parts of ontology extensions have the most impact on recommendations. Furthermore, our research focuses on the movie domain as popular example, but further steps would need to be taken to generalise this to other domains.
Regarding the recommendation methods, as the LP methods show the most promising outcomes, it would be a valuable future step to research other LP approaches more deeply. Other LP methods, such as path based or neighbourhood based methods could therefore provide further insight in the efect of this ontology based approach.
Regarding the results, we find that the performance of the KGBR methods stayed relatively unchanged without being afected by the change of semantic richness of the knowledge graph. This stability is indicative that KGBR methods maintain consistent performance, irrespective of varying knowledge graph richness. Notably, most methods do not exhibit a consistent trend of improvement or deterioration, suggesting that the modifications introduced in each experiment have a balanced impact on performance.
KTUP shows a significant decline in all metrics from KG 0 to KG3+, indicating potential issues with adapting to changes over the experiments. SimplE, in contrast, exhibits dramatic fluctuations, especially pronounced on KG3+. Given that SimplE primarily relies on embedding similarities, the enriched semantic content of the knowledge graph can directly influence its embeddings, enhancing its performance. Furthermore, KGIN consistently shows the best performance among the KGBR methods we evaluated. This underscores its efectiveness in leveraging the features of the knowledge graph across diverse experimental setups.
Most LP methods outperformed KGBR methods on all metrics on KG3+, based on Table 1. For the MRR score, they achieve more than 0.55, which means that these recommender systems could relative-quickly surface the most relevant items. Their NDCG scores reach 0.3, which means that they cannot correctly put the most relevant results on the top-ranked items of the recommendation list. Hit and Precision scores reach 0.8 and 0.35, respectively, which means that they can always recommend at least one relevant result on top-10 list (based on Hit@10) and can recommend more than three relevant results on this list (based on Precision@10). The similar performance of DistMult, SimplE, and TransE in link prediction tasks comes from their reliance on linear operations and their ability to model simple relationships. These models perform well on datasets where these assumptions are valid, leading to comparable results, specifically for KG 3+. However, the RotatE method underperformed. RotatE is designed to model complex relational structures using rotations in a complex vector space, which allows it to handle more intricate and non-linear relationships better than the simpler models. RotatE’s ability ability to manage a wider range of relational patterns, makes it a stronger choice for knowledge graphs with more complex and diverse relationships. Methods like KGIN and RippleNet show very marginal changes, indicating stability in their performance despite the enhanced richness of KG3+. This suggests that simply adding more semantic richness to a knowledge graph does not universally translate to improved recommendation performance.
In this paper, we explored the impact of the semantic richness of a knowledge graph for knowledge based recommending methods. To test this, we focused on the movie recommendation domain. We created seven movie ontologies which are used to build seven knowledge graphs with varying semantic richness and linking. We enriched the ontology-based knowledge graphs with MovieLens 1M data, information from wikidata and supplemental statistical data. We then designed and conducted 91 experiments, and compared the performance of state-of-the-art knowledge based recommending methods and link prediction methods on four existing ranking evaluation metrics. The results indicate that increasing the semantic richness of the knowledge graph does not bring significant impact on the knowledge based recommending methods, but does significantly impact the link prediction approaches.
This paper built its enrichment approach using manually created movie ontologies. However, future research could explore the potential advantages of a learning based approach to ontology enrichment, to further tailor the recommendations to the specific needs of the user. By using machine learning algorithms, researchers could study the process of automated adaption or enhancement of such ontologies and subsequent knowledge graphs. This shift could lead to more dynamic and comprehensive recommender systems, as learning-based systems are capable of continuously updating and refining their knowledge base from vast amounts of data. Finally, increasing the semantic strength can impact the understandability of the provided recommendendations. This paper does not further consider the impact of semantic richness of knowledge graphs on explainable recommender systems, particularly knowledge based or graph based explainable recommender systems [30, 31, 32]. Increasing semantic richness of a knowledge graph could also increase more paths between users and movies in the knowledge graph, which could bring more possible explanations in the recommendations. More research into combining extensive semantic knowledge graphs and explainable recommender systems could therefore be useful to even further strengthen the recommender systems.
This work is funded by the HORIZON EUROPE project “EU-FarmBook: supporting knowledge exchange between all AKIS actors in the European Union” (Grant ID: 101060382). [15] H. Wang, F. Zhang, J. Wang, M. Zhao, W. Li, X. Xie, M. Guo, Ripplenet: Propagating user preferences on the knowledge graph for recommender systems, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, ACM, 2018.
URL: http://dx.doi.org/10.1145/3269206.3271739. doi:10.1145/3269206.3271739. [16] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko, Translating embeddings for modeling multi-relational data, in: C. Burges, L. Bottou, M. Welling, Z. Ghahramani, K. Weinberger (Eds.), Advances in Neural Information Processing Systems, volume 26, Curran Associates, Inc., 2013. URL: https://proceedings.neurips.cc/paper_files/paper/2013/file/ 1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf. [17] B. Yang, W.-t. Yih, X. He, J. Gao, L. Deng, Embedding entities and relations for learning and inference in knowledge bases, 2014. URL: https://arxiv.org/abs/1412.6575. doi:10.48550/ARXIV. 1412.6575. [18] Z. Sun, Z.-H. Deng, J.-Y. Nie, J. Tang, Rotate: Knowledge graph embedding by relational rotation in complex space, 2019. URL: https://arxiv.org/abs/1902.10197. doi:10.48550/ARXIV.1902.10197. [19] S. M. Kazemi, D. Poole, Simple embedding for link prediction in knowledge graphs, 2018. [20] L. Xu, Z. Tian, G. Zhang, J. Zhang, L. Wang, B. Zheng, Y. Li, J. Tang, Z. Zhang, Y. Hou, X. Pan, W. X. Zhao, X. Chen, J. Wen, Towards a more user-friendly and easy-to-use benchmark library for recommender systems, in: SIGIR, ACM, 2023, pp. 2837–2847. [21] M. Ali, M. Berrendorf, C. T. Hoyt, L. Vermue, S. Sharifzadeh, V. Tresp, J. Lehmann, PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings, Journal of Machine Learning Research 22 (2021) 1–6. URL: http://jmlr.org/papers/v22/20-825.html. [22] K. Järvelin, J. Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Trans. Inf.
Syst. 20 (2002) 422–446. [23] E. Bassani, ranx: A blazing-fast python library for ranking evaluation and comparison, in: ECIR (2), volume 13186 of Lecture Notes in Computer Science, Springer, 2022, pp. 259–264. doi:10.1007/ 978-3-030-99739-7\_30. [24] Z. Sun, D. Yu, H. Fang, J. Yang, X. Qu, J. Zhang, C. Geng, Are we evaluating rigorously? benchmarking recommendation for reproducible evaluation and fair comparison, in: Proceedings of the 14th ACM Conference on Recommender Systems, 2020, pp. 23–32. [25] C. A. Gomez-Uribe, N. Hunt, The netflix recommender system: Algorithms, business value, and innovation, ACM Transactions on Management Information Systems (TMIS) 6 (2015) 1–19. [26] M. Jenner, Netflix and the Re-invention of Television, Springer, 2018. [27] M. Thelwall, K. Kousha, Goodreads: A social network site for book readers, Journal of the
Association for Information Science and Technology 68 (2017) 972–983. [28] Y. He, J. Chen, D. Antonyrajah, I. Horrocks, Bertmap: a bert-based ontology alignment system, in:
Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 5684–5691. [29] Y. Kalfoglou, M. Schorlemmer, Ontology mapping: the state of the art, The knowledge engineering review 18 (2003) 1–31. [30] X. Wang, D. Wang, C. Xu, X. He, Y. Cao, T.-S. Chua, Explainable reasoning over knowledge graphs for recommendation, Proceedings of the AAAI Conference on Artificial Intelligence 33 (2019) 5329–5336. URL: http://dx.doi.org/10.1609/aaai.v33i01.33015329. doi:10.1609/aaai. v33i01.33015329. [31] Z. Kurt, T. Köllmer, P. Aichroth, An explainable knowledge graph-based news recommendation system, in: Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, SCITEPRESS - Science and Technology Publications, 2023. URL: http://dx.doi.org/10.5220/0012161300003598. doi:10.5220/ 0012161300003598. [32] L. Xie, Y. Jin, Z. Cui, L. Wang, Research on explainable recommendation model based on knowledge graph, in: 2022 IEEE International Conference on Networking, Sensing and Control (ICNSC), IEEE, 2022. URL: http://dx.doi.org/10.1109/icnsc55942.2022.10004187. doi:10.1109/icnsc55942. 2022.10004187.