Addressing the New User Problem with a Personality Based User Similarity Measure Marko Tkalčič, Matevž Kunaver, Andrej Košir, and Jurij Tasič University of Ljubljana Faculty of electrical engineering, Tržaška 25, 1000 Ljubljana, Sovenia {marko.tkalcic,matevz.kunaver,andrej.kosir,jurij.tasic}@fe.uni-lj.si http://ldos.fe.uni-lj.si Abstract. The new user problem is a recurring problem in memory based collaborative recommender systems (MBCR). It occurs when a new user is added to the system and there are not enough information to make a good selection of the user’s neighbours. As a consequence, the recommended items have poor correlation with the user’s interests. We addressed the new user problem by observing the user similarity measure (USM). In this paper we present two novelties that address the new user problem : (i) the usage of a personality based USM to alleviate the new user problem and (ii) a method for establishing the boundary of the cold start period. We succesfully used a personality based USM that yielded significantly better recommender performance in the period where the new user problem occurs. Furthermore we presented a new methodology for assessing the boundary of the period where the new user problem occurs. Keywords: memory based collaborative recommender system, new user problem, personality based user similarity measure 1 Introduction The new user problem is an important issue in memory based collaborative rec- ommender systems [Adomavicius and Tuzhilin, 2005]. It occurs when a new user joins the system and there are no (or there are too few) overlapping ratings to calculate good estimates of user similarities with rating-based user similar- ity measures (USM). We will denote this initial period as the cold start period (CSP). The consequences of being in the CSP are bad rating predictions for unseen items and thus poor quality of the recommender system. Usually, the new user problem (NUP) has been addressed by introducing content-based ap- proaches which resulted in hybrid systems [Adomavicius and Tuzhilin, 2005, Ahn, 2008]. Once the system has enough overlapping items it is not in the CSP and rating based USM can be used. We introduced a personality-based USM using the five factor model (FFM) in Tkalčič et al. [2009]. The same approach was later used by Hu and Pu [2010] for the NUP in a music recommender system. In this paper we present (i) the 2 Marko Tkalčič, Matevž Kunaver, Andrej Košir, and Jurij Tasič results of the proposed USM in a CF recommender system for images and (ii) a methodology for assessing the boundary of the cold start period. The proposed approach to use a personality-based USM in the NUP allows us to calculate user similarities immediately, without waiting for the user to rate several items. The underlying assumption for choosing personality as the basis for the proposed user similarity measure is that people with similar personalities have similar tastes for products. In psychology, personality is described as a set of factors that account for the majority of between-user variance in emotive, interpersonal, experiential and attitudinal styles [John and Srivastava, 1999]. The second novelty is a statistical method for determining at which point the new user problem stops occurring. A review of literature showed that authors either (i) did not set limits for the CSP [Schein et al., 2002] or (ii) provided limits without further argumentation, e.g. Massa and Bhattacharjee [2004] defined cold start users as the users who have expressed less than 5 ratings. We propose to determine the boundary of the CSP with a statistical approach, as the number of ratings where the recommender’s performance stops being significantly lower than the performance with higher number of ratings given by the user. 2 The new user problem The new user problem in collaborative filtering recommenders is described as the period from the moment when a user joins the system to the moment when there are enough ratings to yield a stable list of neighbours (i.e. users with simi- lar preferences). We rewrote this description from various sources [Adomavicius and Tuzhilin, 2005, Schein et al., 2002, Ahn, 2008]. To the best of the authors’ knowledge no formal definition of the new user problem period is available. In this section we define the boundary of the CSP. Let us have a user u joining the system. The user starts using the system and gives ratings r(u, h) to items h ∈ H where H ⊂ {h1 . . . hJ }, a set of J items. At any given moment the user has given n ratings to n different items which yields the set Run = {r1u . . . rnu } (1) The boundary of the new user problem period (the CSP) for the selected user is the number of ratings NuCS after which the system starts to yield stable sets of users. The consequence of a stable set of users is a stable confusion matrix of recommended items. We define that the confusion matrix is stable if a sequence of F -measure values, has statistically equivalent means at different n. We choose the F measure as a scalar measure of the confusion matrix. We denote the F measure when n ratings have been used to calculate neighbours as F n . We define the CSP boundary as the point N where the means of F values of the sets RuN J = {F N . . . F J } Ru(N −1)J = {F (N −1) . . . F J } (2) are significantly different. Addressing the New User Problem with Personality 3 In Tkalčič et al. [2009] we presented a user similarity measure that takes two vectors bi = (bi1 . . . bi5 ) and bj = (bj1 . . . bj5 ) containing the personality values of two users ui and uj and yields the scalar similarity value v u 5 uX dW (bi , bj ) = t wl (bil − bjl )2 (3) l=1 We use the proposed user similarity measure to calculate similarities in the CSP. 3 Experiment The goal of the experiment was to (i) assess the CSP boundary and to (ii) see whether the personality-based USM performs better in the CSP. We conducted two experiments with the CF recommender system: (i) one with the personality based USM and (ii) one with the rating based USM. We used the LDOS-PerAff-1 [Tkalčič et al., 2010] dataset which contained all data necessary to carry out our experiments. The dataset provided the usage history (i.e. the log of users’ interactions) of 52 users consuming 70 content items (a subset of images from the IAPS1 dataset) and giving explicit ratings to each item. The users’ task was to assess the images for their computer’s wallpaper. The users’ personalities vectors b were assessed using the IPIP502 questionnaire. We used the personality based USM as defined in Eq. 3 to calculate the distances between the users. We calculated the predicted ratings based on the neighbours’ ratings using the adjusted Pearson’s coefficient as defined in Kunaver et al. [2007]. We then compared the predicted ratings with the ground truth ratings which yielded the confusion matrix. We calculated the rating based USM d(ui , uj ) between two arbitrary users ui and uj based on their respective ratings e(u, h) of the overlapping items hm , where m is the index of the items that both the users have rated s X d(ui , uj ) = (e(ui , hm ) − e(uj , hm ))2 (4) m The dataset used in our experiments had a full ratings-items table without missing values with I users and K ratings. To simulate the CSP we determined a usage history path in the form of a random sequence of ratings, for each user separately. We iterated through cold start stages s from one (the user has given only one rating) to K (the user has rated all items) for each user separately. At each stage 1 ≤ s ≤ K we performed the recommender procedure and calculated the confusion matrix for the observed user u at the observed stage s. We chose the F measure as the performance measure of the recommender system. The 1 http://csea.phhp.ufl.edu/media.html 2 http://ipip.ori.org/New_IPIP-50-item-scale.htm 4 Marko Tkalčič, Matevž Kunaver, Andrej Košir, and Jurij Tasič experimental procedure thus yielded a table of F values at different stages s ∈ {1..K} and for each user u ∈ U . 3.1 Evaluation methodology We compared the performance of the rating based USM and the personality based USM by testing the hypothesis H0 : µR = µB5 at different cold start stages using the t-test. The value µR represents the mean F values using the rating based USM and µB5 represents the mean F values using the personality based USM. We determined the position of the CSP boundary by testing the hypothesis H0 : µs = µs−K where µs represents the mean F value at stage s and µs−K represents the mean F value from stages s+1 to K, where K is the last observed cold start stage. 4 Results When seeking for the CSP boundary we calculated the p values which are shown in Fig. 1. On the dataset used we observed that p < 0.05 occurs when the cold start stage is s < 6. We analyzed the CSP by graphing the quality rate of the recommender(F ) versus the number of ratings used. At each cold start stage s the F measures for each user were calculated. The results of the t test showed that, on the dataset used, the personality based USM yields a significantly higher mean of F values than the rating based USM when the number of ratings taken in account for the calculation of the neighbours is lower than 50 (see Fig. 2). When the number of ratings is higher than 50 the means of F values for both similarity measures are not significantly different at α = 0.05. 5 Discussion and conclusion Experimental results showed that the personality based USM performs signif- icantly better than the rating based USM in cold start conditions. A positive outcome is also the fact that the personality USM is statistically equivalent to the rating based USM which makes it a good candidate for a complete replacement of the rating based USM. However, the results presented here were verified only on the specific dataset and we don’t have any ground to conclude that the presented approach is useful also in other domains. We do speculate that hedonistic-content domains would benefit from the presented approach but this should be verified as future work. The main drawback of the personality based USM is the difficulty of acquisi- tion of end users’ personality parameters. There are two main obstacles in this: (i) it is annoying for the end user to fill in questionnaires and (ii) the acquisition Addressing the New User Problem with Personality 5 0 10 −1 10 −2 10 0 10 20 30 40 50 60 70 Fig. 1. p values of the t test for the CSP boundary. On the dataset used the CSP occurs when s < 6. t test p values: personality based USM vs. rating based USM 0 10 −1 10 −2 10 −3 10 −4 10 −5 10 −6 10 0 10 20 30 40 50 60 70 Fig. 2. p values of the t test of the comparison of the personality based USM and rating based USM. of personality data raises ethical and privacy issues that need to be addressed first. The progress beyond the state of the art here is the knowledge that per- sonality does account for between-users variance in entertainment applications. In the lack of existing methodologies for assessing the boundaries of the new user problem we chose a statistical approach. We acknowledge that further in- vestigations should be conducted to determine how to test for the CSP boundary and that these investigations might conclude that a different approach is more suitable. 6 Marko Tkalčič, Matevž Kunaver, Andrej Košir, and Jurij Tasič We provided a methodology for the assessment of the new user boundary. The results presented should not be taken for granted and several repetitions of the procedure should be carried out on different datasets. In this paper we have evaluated a personality based USM under cold start conditions. The results showed that the personality based USM performed signif- icantly better than the rating based USM. Furthermore we described a method- ology for the assessment of the CSP border. Both novelties are important in the field of memory based collaborative filtering recommender systems and should be further explored. References G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans- actions on Knowledge and Data Engineering, 17(6):734–749, 2005. H.J. Ahn. A new similarity measure for collaborative filtering to alleviate the new user cold-starting problem. Information Sciences, 178(1):37–51, 2008. R. Hu and P. Pu. Using Personality Information in Collaborative Filtering for New Users. Recommender Systems and the Social Web, page 17, 2010. Oliver P. John and Sanjay Srivastava. The big five trait taxonomy: His- tory, measurement, and theoretical perspectives. In Lawrence A. Pervin and Oliver P. John, editors, Handbook of Personality: Theory and Research, pages 102–138. Guilford Press, New York, second edition, 1999. URL http: //www.uoregon.edu/\~{}sanjay/pubs/bigfive.pdf. Matevž Kunaver, Tomaž Požrl, Matevž Pogačnik, and Jurij Tasič. Optimisation of combined collaborative recommended systems. International Journal of Electronic Communications, 61:433–443, 2007. P. Massa and B. Bhattacharjee. Using Trust in Recommender Systems: An Experimental Analysis. In Trust management: second international confer- ence, iTrust 2004, Oxford, UK, March 29-April 1, 2004: proceedings, page 221. Springer-Verlag New York Inc, 2004. A.I. Schein, A. Popescul, L.H. Ungar, and D.M. Pennock. Methods and metrics for cold-start recommendations. In Proceedings of the 25th annual interna- tional ACM SIGIR conference on Research and development in information retrieval, pages 253–260. ACM, 2002. Marko Tkalčič, Matevž Kunaver, Jurij Tasič, and Andrej Košir. Personality based user similarity measure for a collaborative recommender system. In C. Peter, E. Crane, L. Axelrod, H. Agius, S. Afzal, and M. Balaam, editors, Proceedings of the 5th Workshop on Emotion in Human-Computer Interaction - Real world challenges, pages 30–37. Fraunhofer Verlag, September 2009. Marko Tkalčič, Jurij Tasič, and Andrej Košir. The LDOS-PerAff-1 Corpus of Face Video Clips with Affective and Personality Metadata. In Michael Kipp, editor, Proceedings of the LREC 2010 Workshop on Multimodal Corpora: Ad- vances in Capturing, Coding and Analyzing Multimodality, 2010.