In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on ofline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study for four leading DL-RecSys models in the movie domain. We test how diferent DL-RecSys models perform in personalized recommendation generation by conducting a survey study with 445 real active users. We ifnd some DL-RecSys models to be superior in recommending novel and unexpected items but weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction compared to classic collaborative filtering (CF) methods. Qualitatively, we confirm with real user quotes that accuracy plus at least one other attribute is necessary to ensure good user experience, while their demands for transparency and trust cannot be neglected. Based on our findings, we discuss future human-centric DL-RecSys design and optimization strategies.
For modern recommender systems (RecSys), deep learning (DL) models are commonly recognized as
state-of-the-art (SOTA) solutions, usually credited to their high accuracy scores (e.g., RMSE, HR, recall,
MRR, NDCG) on benchmark datasets. However, how well such standards transfer to end user-related
values, such as recommendation transparency [
Referring to prior works that evaluated RecSys algorithms beyond ofline scores from users’ perspective on collaborative filtering or matrix factorization models [ 8, 9], we design this study to comprehensively assess the performance of four representative types of DL models from an open-sourced recommender model leaderboard. Our human-centric framework consists of both ofline benchmark dataset measurement and personalized online recommendation feedback collected from 445 real users of an online movie recommender. Inspired by previous human-centric approaches [8, 9], we focus on seven human-centric metrics including Novelty, Diversity, Serendipity, Accuracy, Trustoworthiness, Transparency, and their overall Satisfaction in real user evaluation.
With this study design, we aim to answer the following two research questions:
RQ1: How do diferent DL-RecSys models perform compared with each other and classic CF methods, as evaluated by our human-centric framework? RQ2: What are some common human-centric value requests from users that future DL-RecSys designers should consider?
In the rest of the paper, we first discuss related work, then provide a high-level overview of our research methods and the four deep learning models. After that, we share the model-wise performance comparison based on ofline and real-user input results, also the qualitative data analysis of real user evaluation data. Finally, we consolidate and discuss all findings and propose human-centric design implications for future DL-RecSys studies.
Deep Neural Networks were pervasively used in the RecSys domain for its nature to abstract useritem interaction patterns and efectively learn the representation of large amount of input data [ 10]. Depending on the choice of optimization, there exists a diverse set of architectural paradigms. Multilayer Perceptron (MLP) [11] that served as a basic approximation technique relies on a feed-forward neural network that consists of hidden layers of non-linear transformations of features. Graph Convolutional Neural Network (GNN) is one step further as it can model real-world network structure, such as social networks or user-item relationships with its graph structure and conduct link prediction for recommendation tasks [12]. Recurrent Neural Network (RNN) is appropriate when it comes to sequential recommendation [13] for its ability to remember former computations in memory. Transformers work well in session-based recommendation tasks [14] with its self-attention mechanism. As Dacrema et al. points out, systematic studies are needed for a fair evaluation of DL-RecSys models to truly assess the progress they bring to the field of recommender systems [ 15]. In prior literature, DL-RecSys model benchmarking was primarily ofline, centering around prediction accuracy [ 16] and training time [17].
User experience has been a critical aspect in evaluating the success of recommendations ever since the ifeld started. Konstan and Riedl suggested that the only reliable way to measure the RecSys behavior in a natural context is through a long-term field experiment [ 18]. Munawar et al. identified that subjective recommender system aspects, such as perceived quality and efectiveness, can be significant factors in user satisfaction [19]. Similarly, Knijnenburg and Willemsen and Pu et al. also proposed a usercentric evaluation framework to assess recommender systems with user experiments and statistical analysis [20, 8]. Kunkel et al. evaluated diferences of trustworthiness between personal and impersonal recommendations with real human explanations [21]. In industry, RecSys practitioners mainly evaluated user values from their engagement [22], long-term satisfaction [23], and privacy [24] as complements to accuracy or monetization metrics.
Our overall study design is split into two parts: 1) In phase one, we reproduced the four DL-RecSys models with the MovieLens-1M (ML-1M) dataset and evaluated their performances of novelty, diversity, and serendipity based on formulas defined in previous literature; 2) In phase two, we generated personalized recommendation with each DL-RecSys model for over 3000 active users of an online movie recommender. We then designed a survey with top recommendation lists from each model and sent to users for subjective evaluation including both Likert-scale questions and free-form text input. Detailed user evaluation flow can be found in Fig. 1.
We select four distinct DL models from the SOTA leaderboard based on their accuracy performances with the major benchmark dataset MovieLens-1M (ML-1M)1:
Personalized recommendations
Evaluation for current list Metric Novelty
• Neural Collaborative Filtering (NCF) [25] is one of the early seminal works introducing DL
methods to recommender systems. It employs deep learning for collaborative filtering (CF) by
replacing the inner product with a neural network.
• BERT4Rec [
For each model, we take the code from the oficial open-source repository provided by the original authors and train the model on the dataset we collect for real users (detailed in section 3.3). For fair comparison, we adopt the optimal hyperparameters for ML-1M dataset reported in their original papers. All code repository links and hyperparameters can be found in the Appendix. The two baseline models we choose are k-nearest neighbor user-based collaborative filtering (with min k = 2) and funk SVD(with factors=10 and epochs=20) models.2
Based on previous empirical analyses on human-centric evaluation [
Novelty() = 1 1 − log2 || · | | ∈ ∑︁ − log2 |{ ∈ , ̸= ∅}| | | (1)
• Diversity as the average pairwise distance between items’ Tag Genome information [
Diversity() =
∑︀∈ ∑︀∈∖{} dist(, )
||(|| − 1)
• Serendipity as the content-based surprise metric that measures the average minimal distance
between each candidate’s Tag Genome [
Serendipity() = 1 ∑︁ min dist(, ) || ∈ ∈ (2) (3)
We then calculate the human-centric performances of each DL model with the formula above. To ensure convergence, we train each model for 300 epochs and plot their performance every 10 epochs. As a comparable baseline, we select the classical collaborative filtering (CF)-based funk singular value decomposition (SVD) with factors as 10 and epochs as 20. We apply 5-fold cross-validation to the baseline training and tested with only unrated movies for each user. The final score is calculated as the mean of all 5 folds. For computational eficiency, all metrics are just evaluated with the top 8 items on the list of recommendations.
The participants recruited for this study were from MovieLens (https://movielens.org/), an academicrunning online movie recommender system with thousands of active users. Due to the strict user privacy policy of the website, we select participants mainly based on their activity level (logged in over 12 times and rated over 20 movies) in the year of 2022. For the training set, we then collect those active users’ ratings in the three calendar years before the experiment (from 2020-01-01 until 2022-12-31). To ensure the minimal popularity of movies, we also filter out those movies with less than 20 user ratings. The dataset contains 3,537 users, 7,462 movies, and 983,376 ratings. With the new real user dataset, we generate a personalized list containing the top-recommended movies for each user using the four DL models and two baseline CF models, respectively. To avoid making the questionnaire too long to exceed the general user’s attention span, we only randomly assign 3 DL-generated recommendation lists and 1 CF list to each user.
Every user gets a Google form survey of 6 pages. The first page describes the purpose of this study
("test diferent personalization models"). After that, users proceed to 4 pages of recommendations that
each contains a list of the top 12 movies (we ran pilot tests and considered that to be an adequate number
for users to consume and make judgments) generated by one of the three DL models or one CF model.
Each recommendation item contains the movie name, release year, and detailed MovieLens link that
users can visit to check for details. After reviewing each page of recommendations, users are asked to
ifll in 7 human-centric questions outlined in Fig. 1. Our design and phrasing of questions was inspired
by a series of previous evaluation and user study works [
All questions are designed on a 5-point Likert scale except for free text responses. Specifically, the first question on the third list of recommendations is designed as an attention check. It is a reverse-scaled 5-point Likert question asking the same Satisfaction question in a diferent phrase: "How much do you like the list of recommendations?" In the analysis stage, we compare the reversed response between this check question and the actual satisfaction question on the third list of recommendations from all users with a Mann-Whitney-Wilcoxon test and confirm they are not statistically diferent ( = 0.664), meaning that general user attention during the survey is high.
We sent out surveys via email to 3,537 qualified active users in 7 batches between April to June 2023, with one week between consecutive batches. In the email, we emphasized that filling in this survey was voluntary with no incentives, and users had the right to exit at any time during their participation. We then collected user responses after two weeks of the survey distribution. Overall, 3,172 out of 3,537 surveys were successfully delivered to users’ email inboxes, and 445 of them replied, making the final response rate as 14.03%.
In Fig. 2, we see that in terms of novelty, only the Bert4Rec model constantly beat the baseline SVD performance, with NCF results being the lowest. As for diversity, most of the DL models beat the baseline model, with Bert4Rec leading the board. Serendipity-wise, all DL models were not hitting the baseline SVD performance. Among all models, GLocal-K performs the most volatile and has no clear pattern, while the rest are generally stable among all epochs, which is expected due to the fact that they were not designed to optimize for the human-centric metrics. With this preliminary observation, we extend the human evaluation to invite real users to share their judgments on the recommendation quality of these DL models. 0.07 0.06 s leu0.05 a V 0.04 0.03 0.60 0.55
We report the mean and standard deviation (STD) along with pairwise statistical significant diference of all model-wise user evaluation questions (see Fig. 1) in Table 1. We observe that GLocal-K is the top performer in Novelty, while SSE-PT wins in Serendipity. Without CF baselines, NCF perfroms the best in terms of delivering diverse, trustworthy, transparent, accurate recommendations, while SSE-PT and BERT4Rec perform worst on Trustworthiness, Transparency, Accuracy, and Satisfaction.
RQ1: How do diferent DL-RecSys models perform compared with each other and classic CF methods, as evaluated by our human-centric framework?
With both ofline and real user evaluation comparison, we see that some DL models such as GLocal-K outperform in recommending novel items, and SSE-PT produced a good set of unexpected items for real users. While the ofline evaluation indicates a good diversity performance for most of DL models, they are worse at producing a diverse set of items in real user evaluation. Moreover, real users report those models struggle with gaining user trust or transparency, and most importantly, matching user personalized interest and achieving high satisfaction.
In the last section of real user evaluation, we look into users’ commonly requested recommender
properties and the balance between them. In total, we have 294 users who shared their preferences in
the free response. Then, three researchers followed the grounded theory method (GMT) to conduct open
coding, inductive thematic analyses, and afinity map building of diferent clusters on codes with similar
meanings [
RQ2: What are some common human-centric value requests from users that future DL-RecSys designers should consider?
Accuracy + X. The largest theme under this cluster is about the balance between accuracy and novelty (N=54), like P28 shared: "I use it primarily to find out about movies I hadn’t considered that closely match the kinds of movies I like." The next is surrounding accuracy with diversity (N=12). For example, P39 said: ""I want the recommender to find movies I’m not familiar with that it thinks I will like. I want broad recommendations across lots of genres, and time (both old and new movies)." The third one blends accuracy with serendipity (N=10) – P234: "I want a recommender that would challenge my tastes without ofending me."
Trust builder. Another big theme users discuss is to gain well-grounded recommendations (N=21). As P290 shares, "I want to get reliable recommendations of movies that I wouldn’t have come across otherwise." One step further, many users also mention other properties they relate to building trust with the recommender, such as accuracy (N=10), "I want the recommender to be adapted to my tastes so I can have a big level of confidence that I will enjoy the movies listed (P46)" . P275 claim that their trust is built upon serendipity, "I would even go as far to say I ’trust’ or enjoy MovieLens specifically because I can’t tell where the recommendation came from. I think when it’s traceable that’s what feels fake or mechanical. "
Demand for transparency. The third cluster we identify for preferred recommendation value is transparency (N=27). Some users appreciate more explanation about the recommendation, like P286 says, "...I would also love to see just a tiny bit more info on the films themselves, most importantly Trustworthiness, "I want a recommender that I can rely on to pick movies to watch. I want to see it recommend a variety of diferent Diversity genres and styles that introduce me to new movies I wouldn’t have come across on my own but that I enjoy or can learn from or otherwise appreciate." "Usually I’ve diferent moods at that situation and not that many movies will be the fit, so I guess the best would be getting a good variety of movies that are really not similar between them" "If I am shown something that appears to be very diferent for me, I’m interested to understand why the movie was selected for me - that can help me decide if I agree and want to watch it." "A black box that spits out movies I’ll enjoy. When I don’t have a good movie in mind, I sort the MovieLens list by predicted rating, and pick one of the top ones that looks interesting" "To recommend movies for me to watch. I don’t care how it decides this. I want to watch great stories with good actors that are well-directed." "I use the recommenders as double-check devices: I find a movie; if I find it interesting I check the rating given by the recommender. If it’s good, I watch it. But I wouldn’t trust the recommender on ’blind’ recommendations." Accuracy, "I like to see movies recommended to me which I have already seen. This mostly just shows me that the algorithm Trustworthiness, is in track and I can trust the movies it’s shown me that I don’t know. It would also be good to see a diferent set of Diversity recommendations each time I visited the website." "I’m looking for something to be able to make sense of the reasons why I like the movies that I like, and to not be afraid of recommending me strange niche movies. But at the same time, if possible, to not recommend me things that I clearly do not watch, while keeping variety." "I’m expecting a recommender to find movies that have good ratings from other users who’s ratings are similar to mine.
I don’t want recommendations based on information about the movie. Maybe someday AI will be capable enough to watch movies and guess what I like about them, but till then only other users have that information." P6 P18 P34 P43 P68 P96 P146 P186 P86
In this section, we consolidate our findings and discuss the state of DL-RecSys models from a humancentered perspective as well as opportunities for future design.
While DL-RecSys models perform well in ofline accuracy metrics, in our study, we find that they are not as stable when evaluated with pre-defined Novelty, Diversity, or Serendipity measures. As for real users, only NCF performs relatively equal in terms of generating transparent, accurate, and trustworthy recommendations compared to baseline CF models. It also has the highest performance in terms of general user satisfaction among all DL models. Since the other three DL models we assessed did not get close to CF in most metrics other than novelty and serendipity, there are still a lot of work to do with DL-RecSys user-centric improvement.
In particular, though some DL models have superior performance in terms of serendipity, prior
work by Kotkov et al. suggested that higher serendipity in movies did not necessarily lead to higher
user satisfaction [
Overall, it is challenging to give specific optimization plans since diferent DL models have subtle and
complex design choices, but by looking into some specific user cases, we summarize some generalized
strategies researcher can consider for future development. One major demand from users asking for
more transparency in algorithms can benefit from generating item descriptions with more personalized
context, like P132 shares: "I expect the recommender to recommend me movies that I haven’t seen
but would enjoy. Pointing out why it was licked would help me select a movie to be watched". Thus,
incorporating personalized recommendation explanations with new technique such as large language
models [
We want to clarify again that since the focus of our study is a human-centric evaluation of existing DL-RecSys models, we used the optimal values reported by the authors in the corresponding paper or repository and did not tune the hyperparameters of chosen DL models to optimize for ofline metrics. Given that this study uncovers critical human-centric metrics previously overlooked in conventional measurement approaches, we believe our findings carry significant meanings in terms of future optimization directions in DL-RecSys.
We recognize some limitations of this study: 1) We only chose DL-based models based on their performance of movie recommendation and collected real user feedback in the movie domain, without further expansion to other application fields; 2) We only tested personalized recommendations with active users, and did not generalize how those DL-RecSys models perform with cold-start users; 3) More comprehensive analysis can be done to further explore the latent relationships between diferent user perception values in DL-RecSys. We believe all the above points can be interesting future works. For example, researchers can conduct cross-domain studies, select both new and old users as test subjects, and run more complicated path analysis or structural equation modeling (SEM) to quantify how users’ perception of each recommendation attributes impact each other.
In this study, we investigate how four DL-RecSys models perform under multi-dimensional humancentric evaluation with a movie recommendation case study. By conducting both ofline objective metric evaluation on a benchmark dataset and producing personalized top recommendation lists and evaluating them with real user feedback, we find that sequential and kernel-based DL-RecSys models are superior in recommending novel and serendipitous items while underperforming classic CF models in user-perceived diversity, accuracy, trust, transparency, and general satisfaction. We also analyze users’ qualitative input, revealing their requests for beyond-accuracy recommendation attributes and diferent elements they use to build trust with the system. We hope this case study can serve as a new perspective on evaluating and optimizing future DL-RecSys models under a human-centric framework. ADBIS 2009, Riga, Latvia, September 7-10, 2009. Revised Selected Papers 13, Springer, 2010, pp. 121–128. [5] T. T. Nguyen, F. Maxwell Harper, L. Terveen, J. A. Konstan, User personality and user satisfaction with recommender systems, Information Systems Frontiers 20 (2018) 1173–1189. [6] P. Li, M. Que, Z. Jiang, Y. Hu, A. Tuzhilin, Purs: personalized unexpected recommender system for improving user satisfaction, in: Proceedings of the 14th ACM Conference on Recommender Systems, 2020, pp. 279–288. [7] Z. Fu, X. Niu, M. L. Maher, Deep learning models for serendipity recommendations: A survey and new perspectives, ACM Computing Surveys (2023). [8] P. Pu, L. Chen, R. Hu, Evaluating recommender systems from the user’s perspective: survey of the state of the art, User Modeling and User-Adapted Interaction 22 (2012) 317–355. [9] B. P. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, C. Newell, Explaining the user experience of recommender systems, User modeling and user-adapted interaction 22 (2012) 441–504. [10] S. Zhang, L. Yao, A. Sun, Y. Tay, Deep learning based recommender system: A survey and new perspectives, ACM computing surveys (CSUR) 52 (2019) 1–38. [11] M. W. Gardner, S. Dorling, Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences, Atmospheric environment 32 (1998) 2627–2636. [12] R. v. d. Berg, T. N. Kipf, M. Welling, Graph convolutional matrix completion, arXiv preprint arXiv:1706.02263 (2017). [13] T. Donkers, B. Loepp, J. Ziegler, Sequential user-based recurrent neural network recommendations, in: Proceedings of the eleventh ACM conference on recommender systems, 2017, pp. 152–160. [14] M. Ludewig, N. Mauro, S. Latifi, D. Jannach, Empirical analysis of session-based recommendation algorithms: A comparison of neural and non-neural approaches, User Modeling and User-Adapted Interaction 31 (2021) 149–181. [15] M. F. Dacrema, P. Cremonesi, D. Jannach, Are we really making much progress? a worrying analysis of recent neural recommendation approaches, in: Proceedings of the 13th ACM Conference on Recommender Systems, 2019. [16] J. Zhu, K. Mao, Q. Dai, L. Su, R. Ma, J. Liu, G. Cai, Z. Dou, X. Xiao, R. Zhang, Bars: Towards open benchmarking for recommender systems, in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022. [17] Z. Sun, H. Fang, J. Yang, X. Qu, H. Liu, D. Yu, Y. Ong, J. Zhang, Daisyrec 2.0: Benchmarking recommendation for rigorous evaluation, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (2022) 8206–8226. [18] J. A. Konstan, J. Riedl, Recommender systems: from algorithms to user experience, User modeling and user-adapted interaction 22 (2012) 101–123. [19] Z. Munawar, N. Suryana, Z. B. Sa’aya, Y. Herdiana, Framework with an approach to the user as an evaluation for the recommender systems, in: 2020 Fifth International Conference on Informatics and Computing (ICIC), IEEE, 2020, pp. 1–5. [20] B. P. Knijnenburg, M. C. Willemsen, Evaluating recommender systems with user experiments, in:
Recommender systems handbook, Springer, 2015, pp. 309–352. [21] J. Kunkel, T. Donkers, L. Michael, C.-M. Barbu, J. Ziegler, Let me explain: Impact of personal and impersonal explanations on trust in recommender systems, in: Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–12. [22] I. Kangas, M. Schwoerer, L. J. Bernardi, Recommender systems for personalized user experience: lessons learned at booking. com, in: Proceedings of the 15th ACM Conference on Recommender Systems, 2021, pp. 583–586. [23] H. Steck, L. Baltrunas, E. Elahi, D. Liang, Y. Raimond, J. Basilico, Deep learning for recommender systems: A netflix case study, AI Magazine 42 (2021) 7–18. [24] M. Bartlett, F. Morreale, G. Prabhakar, Analysing privacy policies and terms of use to understand algorithmic recommendations: the case studies of tinder and spotify, Journal of the Royal Society of New Zealand 53 (2023) 119–132. [25] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, T.-S. Chua, Neural collaborative filtering, in: Proceedings
Deep Learning Model Repositories: • NCF : https://github.com/yihong-chen/neural-collaborative-filtering • SSE-PT : https://github.com/lizli502/SSE-PT • BERT4Rec: https://github.com/jaywonchung/BERT4Rec-VAE-Pytorch • GLocal-K: https://github.com/fleanend/TorchGlocalK
For all model reproduction, we mostly use the default optimized parameter claimed by the repository on the ml-1m dataset, detailed below. Specifically, we ran 301 epochs for each model and select the top 12 recommendations from the best performed eopch (based on its NDCG value) for each user. gmf_config mlp_config
neumf_config num_epoch batch_size optimizer adam_lr latent_dim num_negative l2_regularization layers pretrain
Dataloader
NegativeSampler
Trainer
Model
None None 0 600 200 0.5