Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features Luca Luciano Costanzo Yashar Deldjoo Maurizio Ferrari Dacrema Politecnico di Milano, Italy Polytechnic University of Bari, Italy Politecnico di Milano, Italy lucaluciano.costanzo@mail.polimi.it deldjooy@acm.org maurizio.ferrari@polimi.it Markus Schedl Paolo Cremonesi Johannes Kepler University Linz, Politecnico di Milano, Italy Austria paolo.cremonesi@polimi.it markus.schedl@jku.at ABSTRACT additional information sources (aka side information) beyond the In order to improve the accuracy of recommendations, many rec- user rating matrix [25]. A prominent example—and the one we focus ommender systems nowadays use side information beyond the on—is item content. In the movie domain, for instance, a variety of user rating matrix, such as item content. These systems build user content features have been considered, such as metadata or features profiles as estimates of users’ interest on content (e.g., movie genre, extracted directly from the core audio-visual signals. Metadata- director or cast) and then evaluate the performance of the rec- based movie recommender systems typically use genre [10, 13, 26] ommender system as a whole e.g., by their ability to recommend or user-generated tags [18, 31, 34] over which user profiles are relevant and novel items to the target user. The user profile mod- built, assuming that these aspects represent the semantic content elling stage, which is a key stage in content-driven RS is barely of movies. In contrast, audio-visual signals represent the low-level properly evaluated due to the lack of publicly available datasets content (e.g., color, lighting, spoken dialogues, music, etc.) [4, 6, that contain user preferences on content features of items. 7, 9, 10]. Some approaches try to infer semantic concepts from To raise awareness of this fact, we investigate differences be- low-level representations, e.g., via word2vec embeddings [2], deep tween explicit user preferences and implicit user profiles. We create neural networks [30, 33], fuzzy logic [28], or genetic algorithms [20]. a dataset of explicit preferences towards content features of movies, For these reasons, it is evident that item content plays a key role which we release publicly. We then compare the collected explicit in building hybrid or content-based filtering (CBF) models and, user feature preferences and implicit user profiles built via state-of- furthermore, it is important to correctly distinguish and weight the-art user profiling models. Our results show a maximum average the item features by their estimated relevance for a target user, to pairwise cosine similarity of 58.07% between the explicit feature better model his or her tastes. preferences and the implicit user profiles modelled by the best in- In Figure 1, we illustrate a simplified diagram that shows our vestigated profiling method and considering movies’ genres only. research contributions. Standard recommendation based on content For actors and directors, this maximum similarity is only 9.13% (CBF or hybrid) is structured in three main steps: (i) extraction of and 17.24%, respectively. This low similarity between explicit and item content, consisting of building a feature vector that describes implicit preference models encourages a more in-depth study to each item i; (ii) building the profile of the target user pu , i.e., a investigate and improve this important user profile modelling step, structured representation of the user’s preference over item content which will eventually translate into better recommendations. features; (iii) matching the user profile pu against the feature vector of each item fi to produce the list of recommended items most ACM Reference Format: similar to the target user’s tastes. Luca Luciano Costanzo, Yashar Deldjoo, Maurizio Ferrari Dacrema, Markus A shortcoming of typical RS evaluation is that the user profiling Schedl, and Paolo Cremonesi. 2019. Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features. In Proceedings of stage, which is a key part of the RS, is barely evaluated. Usually, Joint Workshop on Interfaces and Human Decision Making for Recommender only the performance of the entire RS, which is composed of several Systems (IntRS ’19). CEUR-WS.org, 5 pages. components, is assessed and how effectively the user profiling step functions remains an open question. We argue that it is important 1 INTRODUCTION to investigate the user profiling stage and compare performance of different profile modelling methods (see upper part of Figure 1). The performance of collaborative filtering (CF) recommendation The goal of this work is therefore to investigate the difference models have reached a remarkable level of maturity. These models between explicit user ratings on individual movie content features are now widely adopted in real-world recommendation engines (e.g., genre, actors, or directors) and implicit models inferred via because of their state-of-the-art recommendation quality. In recent state-of-the-art user modelling techniques from explicit ratings of years, a number of recommendation scenarios have emerged, which the whole movies. To this end, we (i) create (and make publicly have encouraged the research community to consider using various available) a varied dataset of explicit ratings both on movies and Copyright ©2019 for this paper by its authors. Use permitted under Creative Com- content features and (ii) evaluate different user profiling methods mons License Attribution 4.0 International (CC BY 4.0). IntRS ’19: Joint Workshop and compare their resulting implicit models against the true feature on Interfaces and Human Decision Making for Recommender Systems, 19 Sept 2019, Copenhagen, DK. ratings provided in the collected dataset. IntRS ’19, Sept 16–20, 2019, Copenhagen, DK L.Costanzo et al. OUR FOCUS: Expl. user USER PROFILING EVALUATION profile p'u COSINE/JACCARD User profiling Rated SIMILARITY evaluation features sim(pu , p'u ) Impl. user Item profile pu contents USER PROFILE Rated MODELLING Items RECOMMENDER Recommended SYSTEM MODEL items Feature Item vector fi contents ITEM CONTENT All Items EXTRACTION STANDARD RECOMMENDATION BASED ON CONTENT Figure 1: Main steps involved in a recommendation system leveraging content information, highlighting our contributions. 2 RELATED WORK Zhang method. Zhang et al. [32] build the user profile based With respect to previous research, to the best of our knowledge, the on item ratings or explicit feature ratings. Let U and I denote the only work that evaluates implicit user profiles against true ratings set of users and items, respectively, and F the set of all features of on content features is [21]. Nasery et al. compare actually rated the items. In case of binary ratings (like in our dataset), this method features with the ones implicitly derived from rated movies, but assigns relevance weight hu, f equal to 1 for each feature f in F no concrete user profiling methods are investigated. Instead, the that applies to items with which the target user u interacted with, number of times each feature is explicitly rated and the number 0 otherwise. The obvious limitation of this method is that it assigns of times it appears in the content of all rated movies is counted, only weights 0 or 1 to the features, without distinguishing their and these counts are compared. The authors create a dataset of relevance for the user. movies’ feature ratings (genres, actors/cast, and directors), dubbed Li method. Li et al. [17], unlike Zhang et al., differentiate the rel- PoliMovie,1 through a survey web application they built. Their ap- evance of features contained in an item by assigning scalar weights. proach, using limited survey questions and a fixed reduced dataset Their method furthermore ignores items with low ratings by using of top popular movies and features, extracted from IMDb,2 tends a threshold value. In case of binary ratings, the threshold rating to push users to limited and convergent preferences. In contrast, rτ is 0 and the relevance weight hu, f of each feature f in F for we systematically investigate 4 methods to model implicit user the target user u becomes the percentage of occurrences of f in profiles and we compare them with explicit user profiles obtained the items u interacted with: hu, f = Nu,f /Mu , where Nu,f is the by feature ratings. Another contribution of the work at hand is number of items rated by user u containing feature f and Mu is the creation of a dataset that includes ratings on movie content the total number of items rated by user u. features. Other datasets commonly used in movie recommender Symeonidis method. Symeonidis et al. [27] adopt an approach systems research, but which do not contain such feature ratings, similar to TF-IDF to compute feature relevance weights, but define include MovieLens 20M (ML-20M) [11], IMDB Movies Dataset [16], them in the vector space of user profiles. The rationale of using The Movies Dataset [3], MMTF-14K and MVCD-7K [5, 8] and the TF-IDF is to increase the relevance of rare features contained in Netflix Prize dataset [22]. less user profiles. Symeonidis et al. also use a fixed rating threshold to consider only the most relevant items. In case of binary ratings, 3 USER PROFILE MODELLING TECHNIQUES the threshold rating rτ is set to 0 and the relevance weight hu,f of each feature f in F for the target user u is computed as: hu,i = To create a user profile, we adopt the vector profile representation, F F (u, f ) · IUF (f ) , where F F (u, f ) is the feature frequency, i.e., the consisting of weighted attributes measuring the user’s taste on each number of times feature f occurs in movies rated by u, and IUF (f ) feature [6, 14], because it is best suited for our evaluation in terms | | of similarity functions. Formally, the user profiling methods we is the inverse user frequency of feature f . IUF (f ) = log UFU(f ) , investigate build the user profile pu as a vector whose attributes where UF (f ) is the user frequency of f, i.e., the number of users are the relevance weight of each feature f for the target user u, whose rated movies contain feature f at least once. denoted as hu, f . TF-IDF method. After having reviewed the 3 state-of-art meth- We analyze 3 state-of-the-art methods from literature to model ods described above, we decided to investigate another variant of user profiles and we refer to them according to the first author of TF-IDF as a user profiling method. The Symeonidis method above is the corresponding publication, for simplicity and a 4th method that similar to TF-IDF, but it is user-centric because it considers the vec- applies the TF-IDF (term frequency–inverse document frequency) tor space of user profiles. Instead, our proposed TF-IDF method is term weighting idea, which is widely used in CBF and, in general, item-centric as it considers the vector space of items (movies). First, | | in information retrieval [19, 23, 29]. we compute the IDF of each feature f as: IDF (f ) = log nIf , where 1 PoliMovie: http://bit.ly/polimovie n f denotes the number of items in I in which feature f occurs at 2 Internet Movie Database (IMDB): www.imdb.com least once. Then, for each user u, we compute the relevance weight Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features IntRS ’19, Sept 16–20, 2019, Copenhagen, DK hu, f of a feature f as: hu,f = T F (u, f ) · IDF (f ) , where T F (u, f ) is We collected a total 4,109 favourites (movies and content fea- equivalent to F F (u, f ) of the Symeonidis method (i.e., number of tures) selected by participants, including 1,212 unique elements, times feature f occurs in items rated by user u). In contrast to the i.e., favourites selected by at least one user. In the following experi- method by Symeonidis et al., IDF (f ) is computed in relation to all ments, we include only favourites of reliable users, that are 3,341 the existing items in which feature f appears, not related to user (81%), including 1,737 favourite movies, 461 genres, 698 actors, 198 profiles. As will be shown in Section 5.2, our TF-IDF method yields directors, 74 production companies, 92 production countries, 39 better results than Symeonidis et al.’s. producers, 17 screenwriters, 21 release years, and 4 sound crew members. The dataset is available on Kaggle5 . 4 DATA ACQUISITION The dataset we use to evaluate user profiling methods has been 5 RESULTS AND DISCUSSION collected through a web application we implemented, which can be 5.1 Initial statistical analysis navigated on a variety of stationary and mobile devices. It provides An initial statistical analysis highlights main differences between access to a large catalogue of more than 450K movies and any related the set of all explicitly rated features and the set of all implicit content feature. This vast breadth of choice is possible thanks to features extracted from rated movies. In Tables 1 and 2, we present the fact that we retrieve up-to-date information on-the-fly from a comparison between the explicit and implicit sets of features, in TMDb3 via APIs. We developed the application with the idea of a percentage of common attributes (features), focusing on the k most completely free user experience, instead of making it like a survey frequently selected attributes, respectively, for genre, actor, and application, so that users are not forced in any way during their director. These tables generally highlight a low overlap between the selections. explicitly preferred features and the implicitly estimated ones (de- To acquire the needed data, we asked users to select a set of rived from favourite movies), in particular for actors and directors. “favourites”, which included at least 5 movies, 2 genres, 3 actors, The only exception is the genre attribute, which reveals a maximum and 1 director. Users were, nevertheless, free to select more than overlap of 94.74% when considering all 19 genres. These results these numbers of elements. We also asked users to provide some generally confirm the previous findings in [21] regarding existing demographics information: age range, gender, and country of resi- gaps between explicitly selected features and implicitly estimated dence. The collection of data was divided into two phases, the first ones, with a different dataset containing more up-to-date movies one involved the volunteer users, which are the ones invited to and not limited to the most popular movies as used in [21]. freely contribute (friends, family, acquaintances, and colleagues of the authors), while the second phase involved users recruited by Table 1: Common genres in the most selected k attributes, either the crowdsourcing platform MTurk,4 which have been paid between explicitly or implicitly 20 and 50 US cents for their contribution. To assess the participants’ reliability, we also asked them to complete a final consistency test k No. common genres % common genres which required to select again all (and only) the favourites they 5 3 60.00% remember to have added (from a list of movies, genres, and actors 10 8 80.00% of random popular elements). A user’s reliability is then estimated 15 13 86.67% by means of the precision score computed on the re-selection of All genres (19) 18 94.74% correct favourites. Finally, in order to explore a catalogue of existing features needed We further provide a finer-grained analysis of the gap between for user profiling evaluation, we retrieved The Movies Dataset [3] explicit and implicit preferences of users according to their gender. containing the content of 45,3K movies scraped from TMDb. Then, In Tables 3 and 4, we compare the 5 most frequently selected genres, we extended this dataset by scraping the content of missing movies actors, and directors, by male and female users, respectively. We that were added as favorites by users on our web application. notice a substantial difference between between male and female Dataset characteristics. We have collected the preferences of users with the exception of genre. 194 users, 180 (93%) of whom have added the minimum number Investigating the results, it is surprising that in both Tables 3 and 4 of required favourites. Among all users, 81 (42%) are volunteers Stan Lee is among the top implicitly preferred actors even if he and 113 (58%) are paid ones. We consider users reliable if they are barely acted as a main character in any movie. The most probable either volunteers that have completed the required favourites or reason is that even though he has not been selected explicitly as crowdsourced users who scored at least 50% of precision during favourite actor by study participants, he appeared in all Marvel the consistency test (see above). The reliable volunteers are 67 (83% movies (in small “cameo roles”), so he is included in the implicit of all volunteers), while the crowdsourcing ones are 88 (78% of all profiles. Furthermore, it is surprising that the genre “action” is crowdsourcing), hence a total of 155 reliable users (80% of all users). highly ranked by female users. This could be due to the fact that Regarding users’ gender, 115 users (59%) are male, 66 are female the genre tastes of young women might be changing nowadays, (34%), and 13 (7%) did not specify gender. 53% of the users are especially because many popular action movies, like the Marvel between 24 and 30 years old. We received registrations from users ones, are liked by many people (especially under 30, i.e., the largest coming from 10 different countries, mainly from Italy (40%), India age group in our dataset), irrespective of gender. Nonetheless, the (31%), and United States (19%). other differences between male and female users suggest to embed 3 The Movie Database (TMDb): www.themoviedb.org gender information in a recommender system. 4 Amazon Mechanical Turk (MTurk): www.mturk.com 5 https://www.kaggle.com/lucacostanzo/mints-dataset-for-recommender-systems IntRS ’19, Sept 16–20, 2019, Copenhagen, DK L.Costanzo et al. Table 2: Common quota of either actors or directors between the Table 5: Average pairwise similarity between explicit and implicit most selected k attributes, either explicitly or implicitly user profiles, for all the methods. k % of common actors % of common directors Similarity Feature Zhang Li Symeonidis TF-IDF 10 10.00% 30.00% Genre 48.52% 58.07% 42.00% 53.08% 20 20.00% 20.00% Cosine Actor 7.03% 9.13% 6.50% 7.24% 40 22.50% 27.50% Director 15.17% 17.24% 15.32% 16.14% 60 16.67% 28.33% Genre 27.49% 36.19% 18.54% 33.36% Jaccard Actor 0.97% 5.73% 2.87% 4.64% Director 5.22% 10.24% 6.30% 8.17% Table 3: Most selected 5 features, either explicitly (R fe x p ) or implic- imp itly (R f ), by male users; Pos. Explicit selection exp Rf Implicit selection imp Rf yields better results than Symeonidis even if they are intrinsically similar, hence the item-centric TF-IDF approach outperforms the 1 Action 51 Action 86 2 Drama 31 Adventure 83 user profle-based one. In general, the average pairwise similarities Genres 3 Adventure 30 Drama 80 4 Thriller 28 Science Fiction 76 are remarkably low, even for the best investigated method, i.e., Li. 5 Science Fiction 28 Thriller 74 The overlap between explicit and implicit profiles increases if we 1 Robert Downey Jr. 16 Samuel L. Jackson 64 2 Johnny Depp 15 Stan Lee 56 consider only genres; the reason is that the catalogue of all possible Actors 3 4 Jason Statham Leonardo Di- 10 10 Bradley Cooper Paul Bettany 51 47 genres in the dataset is rather limited (19) compared to actors (567K) Caprio and directors (58K). The Jaccard measure yields lower similarities 5 Tom Hardy 8 Vin Diesel 47 because it can be applied only to vectors composed of binary at- 1 Quentin Tarantino 11 Hajar Mainl 42 2 Steven Spielberg 9 Chris Castaldi 41 tributes while our tested profiling methods compute scalar weights Directors 3 Joe Russo 7 Mark Rossini 41 4 M. Night Shya- 6 Lori Grabowski 41 (except for Zhang); hence we had to cut-off some feature weights malan by considering only the k most relevant features in the implicit 5 Christopher Nolan 6 Eli Sasich 41 profile of each user considered, in which k is the number of explicit features rated by that user. Table 4: Most selected 5 features, either explicitly (R fe x p ) or implic- The presented results underline the low effectiveness of the imp itly (R f ), by female users; investigated user profiling methods to model real user tastes. This finding gives rise to the need of further research on this important Pos. Explicit selection Rf exp Implicit selection Rf imp user profiling step when devising recommender systems. If user 1 Drama 26 Drama 52 profiles are not properly modelled before applying any RS technique, 2 Action 22 Adventure 48 the accuracy of the final recommendations will likely be affected Genres 3 Adventure 14 Action 47 4 Comedy 14 Fantasy 45 and lowered by an inaccurate representation of the user’s tastes. 5 Thriller 13 Science Fiction 43 1 Robert Downey Jr. 12 Stan Lee 27 . Actors 2 Leonardo Caprio Di- 7 Samuel L. Jackson 26 6 CONCLUSION AND FUTURE WORKS 3 Jennifer Lawrence 5 Bradley Cooper 23 4 Chris Hemsworth 5 Djimon Hounsou 21 In this paper, we analyzed the user profiling modelling by studying 5 Bruce Willis 4 James McAvoy 21 the differences between explicit user preferences and implicit user 1 Joe Russo 4 Anthony Russo 16 profiles. We evaluated different user profiling methods and showed 2 Christopher Nolan 4 Joe Russo 16 Directors 3 Steven Spielberg 4 Bryan Singer 15 that even the best profiling method that we tested provided low 4 Martin Scorsese 2 Hajar Mainl 14 5 Ridley Scott 2 Chris Castaldi 14 pairwise similarities between explicit and implicit profiles. This finding can be explained by the fact that when a user rates a movie, he is implicitly rating only some characteristics of the item that 5.2 Evaluation of user profiling methods impacted on her (but not all). Also, it could happen that a user may We study the user profiling step in-depth by investigating the 4 select a movie but she only loved some part of it (e.g., very good user profiling methods described in Section 3. Our aim is to analyze director but bad actors), and this can result in the introduction of the similarity (i.e., the overlap) between the implicitly modelled some noise in the learning process. Overall, our study encourages a user profiles and the real explicit tastes of users. For each target more in-depth on ways we can obtain reliable feedbacks on features user u, we built his or her explicit profile pu′ as vector composed and study the optimization of the user profile modelling step in of relevance weights equal to 1, for all the features explicitly rated RS, which will eventually allow to produce more accurate recom- by u, and weight 0 for the ones not rated. Then we computed the mendations. Furthermore, we publicly provide the dataset that we pairwise similarity between the explicit user profiles and implicit collected and used for evaluation, which includes ratings on movies profiles pu produced by each method, using cosine similarity and and on corresponding content features. Jaccard similarity. The highest is this similarity, the most accurate In the future, we plan to investigate the generalizability of find- is the implicit user profile modelled. ings in this work on other domains where the exist a wide variety The average pairwise similarity sim(pu , pu′ ) between implicit of item content features and personalization on these features is user profile pu and explicit one pu′ is shown in Table 5. As revealed paramount, in domains including but not limited to fashion [12], in the table and already anticipated in Section 3, the TF-IDF method music domain [24], tourism [1, 15] and so forth. Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features IntRS ’19, Sept 16–20, 2019, Copenhagen, DK REFERENCES [20] Juergen Mueller. 2017. Combining aspects of genetic algorithms with weighted [1] Jens Adamczak, Gerard-Paul Leyson, Peter Knees, Yashar Deldjoo, Far- recommender hybridization. In Proceedings of the 19th International Conference shad Bakhshandegan Moghaddam, Julia Neidhardt, Wolfgang Wörndl, and on Information Integration and Web-based Applications & Services, iiWAS 2017, Philipp Monreal. 2019. Session-Based Hotel Recommendations: Challenges and Salzburg, Austria, December 4-6, 2017, Maria Indrawan-Santiago, Matthias Stein- Future Directions. arXiv preprint arXiv:1908.00071 (2019). bauer, Ivan Luiz Salvadori, Ismail Khalil, and Gabriele Anderst-Kotsis (Eds.). ACM, [2] Fahad Anwaar, Naima Iltaf, Hammad Afzal, and Raheel Nawaz. 2018. HRS-CE: 13–22. DOI:http://dx.doi.org/10.1145/3151759.3151765 A hybrid framework to integrate content embeddings in recommender systems [21] Mona Nasery, Mehdi Elahi, and Paolo Cremonesi. 2015. PoliMovie: a feature- for cold start items. J. Comput. Science 29 (2018), 9–18. DOI:http://dx.doi.org/10. based dataset for recommender systems. DOI:http://dx.doi.org/10.13140/RG.2.2. 1016/j.jocs.2018.09.008 20636.49286 [3] Rounak Banik. 2017. The Movies Dataset. Dataset on Kaggle. (2017). https: [22] Netflix. 2009. Netflix Prize Data. Dataset on Kaggle. (2009). https://www.kaggle. //www.kaggle.com/rounakbanik/the-movies-dataset com/netflix-inc/netflix-prize-data [4] Yashar Deldjoo, Mihai Gabriel Constantin, Hamid Eghbal-Zadeh, Bogdan Ionescu, [23] Diego Sánchez-Moreno, María N. Moreno García, Nasim Sonboli, Bamshad Markus Schedl, and Paolo Cremonesi. 2018. Audio-visual encoding of multimedia Mobasher, and Robin Burke. 2018. Inferring User Expertise from Social Tag- content for enhancing movie recommendations. In Proceedings of the 12th ACM ging in Music Recommender Systems for Streaming Services. In Hybrid Artificial Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October Intelligent Systems - 13th International Conference, HAIS 2018, Oviedo, Spain, June 2-7, 2018, Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O’Donovan 20-22, 2018, Proceedings (Lecture Notes in Computer Science), Francisco Javier (Eds.). ACM, 455–459. DOI:http://dx.doi.org/10.1145/3240323.3240407 de Cos Juez, José Ramón Villar, Enrique A. de la Cal, Álvaro Herrero, Héctor [5] Yashar Deldjoo, Mihai Gabriel Constantin, Bogdan Ionescu, Markus Schedl, and Quintián, José Antonio Sáez, and Emilio Corchado (Eds.), Vol. 10870. Springer, Paolo Cremonesi. 2018. MMTF-14K: a multifaceted movie trailer feature dataset 39–49. DOI:http://dx.doi.org/10.1007/978-3-319-92639-1_4 for recommendation and retrieval. In Proceedings of the 9th ACM Multimedia [24] Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Systems Conference. 450–455. DOI:http://dx.doi.org/10.1145/3204949.3208141 Elahi. 2018. Current challenges and visions in music recommender systems [6] Yashar Deldjoo, Paolo Cremonesi, Markus Schedl, and Massimo Quadrana. 2017. research. International Journal of Multimedia Information Retrieval 7, 2 (2018), The effect of different video summarization models on the quality of video recom- 95–116. mendation based on low-level visual features. In Proceedings of the 15th Interna- [25] Yue Shi, Martha Larson, and Alan Hanjalic. 2014. Collaborative filtering beyond tional Workshop on Content-Based Multimedia Indexing, CBMI 2017, Florence, Italy, the user-item matrix: A survey of the state of the art and future challenges. ACM June 19-21, 2017. ACM, 20:1–20:6. DOI:http://dx.doi.org/10.1145/3095713.3095734 Computing Surveys (CSUR) 47, 1 (2014), 3. [7] Yashar Deldjoo, Maurizio Ferrari Dacrema, Mihai Gabriel Constantin, Hamid [26] Márcio Soares and Paula Viana. 2017. The Semantics of Movie Metadata: Enhanc- Eghbal-zadeh, Stefano Cereda, Markus Schedl, Bogdan Ionescu, and Paolo Cre- ing User Profiling for Hybrid Recommendation. In Recent Advances in Information monesi. 2019. Movie genome: alleviating new item cold start in movie rec- Systems and Technologies - Volume 1 [WorldCIST’17, Porto Santo Island, Madeira, ommendation. User Model. User-Adapt. Interact. 29, 2 (2019), 291–343. DOI: Portugal, April 11-13, 2017]. (Advances in Intelligent Systems and Computing), http://dx.doi.org/10.1007/s11257-019-09221-y Álvaro Rocha, Ana Maria R. Correia, Hojjat Adeli, Luís Paulo Reis, and San- [8] Yashar Deldjoo and Markus Schedl. 2019. Retrieving Relevant and Diverse Movie dra Costanzo (Eds.), Vol. 569. Springer, 328–338. DOI:http://dx.doi.org/10.1007/ Clips Using the MFVCD-7K Multifaceted Video Clip Dataset. In Proceedings of 978-3-319-56535-4_33 the 17th Int. Workshop on Content-Based Multimedia Indexing. [27] Panagiotis Symeonidis, Alexandros Nanopoulos, and Yannis Manolopoulos. 2007. [9] Fuhu Deng, Panlong Ren, Zhen Qin, Gu Huang, and Zhiguang Qin. 2018. Lever- Feature-Weighted User Model for Recommender Systems. In User Modeling 2007, aging Image Visual Features in Content-Based Recommender System. Scientific 11th International Conference, UM 2007, Corfu, Greece, June 25-29, 2007, Proceedings Programming 2018 (2018), 5497070:1–5497070:8. DOI:http://dx.doi.org/10.1155/ (Lecture Notes in Computer Science), Cristina Conati, Kathleen F. McCoy, and 2018/5497070 Georgios Paliouras (Eds.), Vol. 4511. Springer, 97–106. DOI:http://dx.doi.org/10. [10] Ralph Jose Rassweiler Filho, Jonatas Wehrmann, and Rodrigo C. Barros. 2017. 1007/978-3-540-73078-1_13 Leveraging deep visual features for content-based movie recommender systems. [28] Pooja Bhatt Vashisth, Purnima Khurana, and Punam Bedi. 2017. A fuzzy hybrid In 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, recommender system. Journal of Intelligent and Fuzzy Systems 32, 6 (2017), AK, USA, May 14-19, 2017. IEEE, 604–611. DOI:http://dx.doi.org/10.1109/IJCNN. 3945–3960. DOI:http://dx.doi.org/10.3233/JIFS-14538 2017.7965908 [29] Donghui Wang, Yanchun Liang, Dong Xu, Xiaoyue Feng, and Renchu Guan. 2018. [11] F. Maxwell Harper and Joseph A. Konstan. 2016. The MovieLens Datasets: History A content-based recommender system for computer science publications. Knowl.- and Context. TiiS 5, 4 (2016), 19:1–19:19. DOI:http://dx.doi.org/10.1145/2827872 Based Syst. 157 (2018), 1–9. DOI:http://dx.doi.org/10.1016/j.knosys.2018.05.001 [12] Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual [30] Jian Wei, Jianhua He, Kai Chen, Yi Zhou, and Zuoyin Tang. 2017. Collaborative evolution of fashion trends with one-class collaborative filtering. In proceedings filtering and deep learning based recommendation system for cold start items. of the 25th international conference on world wide web. International World Wide Expert Syst. Appl. 69 (2017), 29–39. DOI:http://dx.doi.org/10.1016/j.eswa.2016.09. Web Conferences Steering Committee, 507–517. 040 [13] Tae-Gyu Hwang, Chan-Soo Park, Jeong-Hwa Hong, and Sung Kwon Kim. 2016. [31] Shouxian Wei, Xiaolin Zheng, Deren Chen, and Chaochao Chen. 2016. A hybrid An algorithm for movie classification and recommendation using genre correla- approach for movie recommendation via tags and ratings. Electronic Commerce tion. Multimedia Tools Appl. 75, 20 (2016), 12843–12858. DOI:http://dx.doi.org/10. Research and Applications 18 (2016), 83–94. 1007/s11042-016-3526-8 [32] Chenyi Zhang, Ke Wang, Ee-Peng Lim, Qinneng Xu, Jianling Sun, and Hongkun [14] Ameni Kacem. 2017. Personalized Information Retrieval based on Time-Sensitive Yu. 2015. Are Features Equally Representative? A Feature-Centric Recommenda- User Profile. (Recherche d’Information Personalisée basée sur un Profil Utilisateur tion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Sensible au Temps). Ph.D. Dissertation. Paul Sabatier University, Toulouse, France. January 25-30, 2015, Austin, Texas, USA., Blai Bonet and Sven Koenig (Eds.). AAAI https://tel.archives-ouvertes.fr/tel-01707423 Press, 389–395. http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/ [15] Peter Knees, Yashar Deldjoo, Farshad Bakhshandegan Moghaddam, Jens Adam- 9287 czak, Gerard-Paul Leyson, and Philipp Monreal. 2019. RecSys Challenge 2019: [33] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep Learning Based Session-based Hotel Recommendations. In Proceedings of the Thirteenth ACM Recommender System: A Survey and New Perspectives. ACM Comput. Surv. 52, Conference on Recommender Systems (RecSys ’19). ACM, New York, NY, USA, 2. 1 (2019), 5:1–5:38. https://dl.acm.org/citation.cfm?id=3285029 DOI:http://dx.doi.org/10.1145/3298689.3346974 [34] Zhou Zhao, Qifan Yang, Hanqing Lu, Tim Weninger, Deng Cai, Xiaofei He, and [16] Orges Leka. 2016. IMDB Movies Dataset. Dataset on Kaggle. (2016). https: Yueting Zhuang. 2017. Social-Aware Movie Recommendation via Multimodal //www.kaggle.com/orgesleka/imdbmovies Network Learning. IEEE Transactions on Multimedia (2017). [17] Qing Li and Byeong Man Kim. 2004. Constructing User Profiles for Collabo- rative Recommender System. In Advanced Web Technologies and Applications, 6th Asia-Pacific Web Conference, APWeb 2004, Hangzhou, China, April 14-17, 2004, Proceedings (Lecture Notes in Computer Science), Jeffrey Xu Yu, Xuemin Lin, Hongjun Lu, and Yanchun Zhang (Eds.), Vol. 3007. Springer, 100–110. DOI: http://dx.doi.org/10.1007/978-3-540-24655-8_11 [18] Haibo Liu, Shi Feng, and Ge Yu. 2017. An interest propagation based movie rec- ommendation method for social tagging system. In 2017 International Conference on Machine Learning and Cybernetics, ICMLC 2017, Ningbo, China, July 9-12, 2017. IEEE, 130–135. DOI:http://dx.doi.org/10.1109/ICMLC.2017.8107754 [19] Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. 2011. Content-based Recommender Systems: State of the Art and Trends. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.). Springer, 73–105. DOI:http://dx.doi.org/10.1007/978-0-387-85820-3_3