User Evaluation of Fusion-based Approach for Serendipity-oriented Recommender System Kenta Oku Fumio Hattori College of Information Science and Engineering, College of Information Science and Engineering, Ritsumeikan University Ritsumeikan University 1-1-1 Nojihigashi, Kusatsu City 1-1-1 Nojihigashi, Kusatsu City Shiga, Japan Shiga, Japan oku@fc.ritsumei.ac.jp fhattori@is.ritsumei.ac.jp ABSTRACT 1. INTRODUCTION In recent years, studies have focused on the development of In recent years, several studies have focused on the de- recommender systems that consider measures that go be- velopment of recommender systems that consider measures yond simply the accuracy of the system. One such measure, beyond simply the accuracy of the system, such as the nov- serendipity, is dened as a measure that indicates how the elty, diversity, and serendipity [1][2]. This is because these recommender system can nd unexpected and useful items studies have found that users are not always satised with for users. We have previously proposed a fusion-based rec- recommender systems with only high accuracythey desire ommender system as a serendipity-oriented recommender for the systems to consider various other viewpoints, too. system. In this study, we improve upon this system by con- In an attempt to satisfy this need, in this study, we fo- sidering the concept of serendipity. Our system possesses cus on the serendipity. Serendipity means "the ability to mechanisms that can cause extrinsic and intrinsic accidents, make unexpected and valuable discoveries by accident." We and it enables users to derive some value from such acci- thus dene a serendipitous item as something unexpected dents through their sagacity. We consider that such mech- and valuable, and we believe that such an item can diver- anisms are required for the development of the serendipity- sify users' interest regardless of their experiences, thus mak- oriented recommender system. The key idea of this system is ing their lives richer. This study therefore aims to develop the fusion-based approach, through which the system mixes a serendipity-oriented recommender system that provides two user-input items to nd new items that have the mixed users with serendipitous items. features. The contributions of this paper are as follows: First, it is necessary to gain some insight into the origi- providing an improved fusion-based recommender system nal meaning of the word "serendipity." The word "serendip- that adopts a fusion-based approach to improve serendip- ity" originated from a story called "The Three Princes of ity; practically evaluating the recommender system through Serendip" [3], which tells the story of three princes. These user tests using a real book data set from Rakuten Books; princes discovered a series of novel things during the course and showing the eectiveness of the system compared to of various and unexpected events on their journeys, which recommender systems on websites such as Amazon from the they attributed to their luck. Horace Walpole, who read viewpoint of serendipity. this story, stated that "the princes were always making dis- coveries, by accidents and sagacity," to describe which he coined the word "serendipity," which means "the ability to Categories and Subject Descriptors make unexpected discovery by accidents and sagacity" [4]. H.3.3 [Information Search and Retrieval]: Information Fil- In light of Walpole's denition, we believe that a serendipity- tering oriented recommender system should possess an interface that has mechanisms that output "unexpected discoveries" General Terms based on the input of "accidental events" experienced by the users and the sagacity of the users. Experimentation In addition, [4] states that accidents are of two types: "ex- trinsic" and "intrinsic." For example, a well-known serendip- Keywords itous discovery is that of gravityit is stated that "Newton Recommender systems, Serendipity-oriented recommender had an inspiration of the notion of universal gravitation at systems, Serendipity the sight of an apple that fell from a tree"[5]. In this event, the apple falling from the tree can be considered an "ex- trinsic accident," that is, one that occurs regardless of the action of a person. Another example of a serendipitous dis- covery is that made by Koichi Tanaka, which won him the Nobel Prize in Chemistry in 2002. Although he realized that he had accidentally used glycerin instead of acetone Copyright is held by the author/owner(s). Workshop on Recommendation as a sample, he continued his experiments in order to ob- Utility Evaluation: Beyond RMSE (RUE 2012), held in conjunction with serve the results. This led to him discovering an unknown ACM RecSys 2012. September 9, 2012, Dublin, Ireland. phenomenon. In this event, the discovery of the unknown 39 phenomenon can be considered an "intrinsic accident," that is, one that results from the action of a person with the positive expectation of something. It is of great importance to derive some value from these accidents. In this light, a person's sagacity plays a crucial role. The above-described examples suggest that a serendipity- oriented recommender system should have an interface con- sisting of the following mechanisms: (a) A mechanism that causes extrinsic accidents. (b) A mechanism that causes intrinsic accidents. (c) A mechanism that enables users to derive some value from accidents through their sagacity. In this study, we have proposed a fusion-based recommender system to satisfy these requirements. The key idea of this system is adopting a fusion-based approach for discovering serendipitous items by mixing two user-input items together. As described at the beginning of this section, we dene a serendipitous item as an unexpected and valuable item. Specically, the following items are relevant to serendipitous items: Figure 1: Interface of Fusion-based Recommender System. • Items that can excite the user's interest for the rst The user can also select an interesting book as a material time although he/she does not know about them and from the displayed books based on his/her sagacity, and then he/she would not be able to discover them by him- drag-and-drop it into a base book, which is also selected by self/herself. the user. The system then provides the user with books pos- sessing mixed features of the two books. Although the user • Items that can excite the user's interest for the rst can select books to mix with some expectation, some book time although he/she thought that he/she was not in- combinations may yield unexpected results. This may cause terested in them. intrinsic accidents. The user can repeatedly and creatively • Items that can attract the user's interest after being use the system to see various mixing results until he/she displayed by the system. is satised with the results. In this process, serendipitous items are interactively provided to the user. We also dene a high-serendipity recommender system that We have already developed a predecessor to the proposed can recommend more serendipitous items to users. fusion-based recommender system[6]. In this study, we have By using our proposed fusion-based recommender system, improved upon the system interface and internal process- a user can mix two items together in the system interface to ing based on the deeper idea of serendipity, and we have create something new from something existing in a manner evaluated this system from the viewpoint of practical use. analogous to mixing colors, ingredients, or sounds. The act The contributions of this study are as follows: of mixing also entails the following: • developing the improved fusion-based recommender sys- a) We can intuitively expect mixed results from a combi- tem that adopts a fusion-based approach for improving nation of inputs. On the other hand, some combina- the serendipity; tions can yield unexpected results. • experimentally evaluating the practical usability of the b) Because our curiosity may be aroused by the intu- recommender system using a real book data set from itive comprehensibility and unexpectedness of the act Rakuten Books; of mixing, we might feel like being creative and mixing various combinations of inputs. • showing the eectiveness of the system compared to recommender systems on websites such as Amazon Characteristic (a) corresponds to the mechanism that causes from the viewpoint of serendipity. intrinsic accidents because unexpected results may be pro- duced by mixing materials together with the expectation of some positive results. Characteristic (b) corresponds to the 2. RELATED WORK mechanism that enables us to derive some value from ac- Herlocker et al. [1] suggested that recommender systems cidents through our sagacity in that we can select valuable with high accuracy do not always satisfy users. Therefore, inputs from among the given inputs. they suggested that recommender systems should be eval- Figure 1 shows the interface of the fusion-based recom- uated not only by their accuracy but also by various other mender system for book recommendation. When the user metrics such as novelty, diversity, and serendipity. clicks [Random], [Search], [Popular], and [New] buttons, the Several studies have already focused on serendipity in the system randomly provides the user with corresponding books context of recommendation. Ziegler et al. [7][8] suggested from the book database. Randomly providing books corre- that diversifying recommendation lists improves user satis- sponds to the mechanism that causes extrinsic accidents. faction. Toward this end, they proposed topic diversication 40 based on an intra-list similarity metric. Sarwar et al. [9] we describe the system interface and the user interactions suggested that serendipity might be improved by removing related to the above mechanisms. Finally, in Section 3.3, we obvious items from recommendation lists. Berkovsky et al. show fusion methods as the internal processing of the fusion. [10] proposed group-based recipe recommendations. They suggested that recipes loved by a group member are likely 3.1 Book database to be recommended to others, which may increase serendip- In this study, we consider books as the recommendation ity. content; in the future, of course, we intend to apply the sys- Hijikata et al. [11] and Murakami et al. [2] proposed tem to various contents such as music, movies, and recipes. recommendation methods that predict novelty or unexpect- We collected Japanese book data using Rakuten Books book 1 2 edness. The former study proposed collaborative ltering, search API from Rakuten Books . We obtained data for which predicts unknown items for a target user based on 667,218 books between Dec. 27, 2011, and Feb. 10, 2012. known/unknown proles explicitly acquired from the user, The book data consists of the attributes of isbn , title , and showed that such ltering can improve novelty by pro- sub _title , author , sales _date , item _url , review _count , viding unknown items to the user. The latter study pro- review _average , books _genre _id . We created a book table posed a method that implicitly predicts unexpectedness consisting of these attributes, in addition to the following based on a user's action history. They introduced a pref- tables: erence model that predicts items the user likes and a habit • book − phrase(isbn, phrase, idf ) model that predicts items habitually selected by the user. The method estimates the unexpectedness of recommended • book − author (isbn, author ) items by considering the dierences between the models. The disadvantage of these methods is that they need to ob- • book − genre(isbn, genre _id ) tain models or proles for an individual user. Our proposed Here, the book − phrase table contains phrases from book .title system, however, does not have these requirements. It can and book .sub _title for each book. In Section 3.1.1, we ex- instantly recommend serendipitous items based on items the plain how phrases are extracted. The book − author table user has just selected. contains the authors of each book. The book − genre table Murakami et al. [2] and Ge et al. [12] introduced mea- contains the genre id of each book. Rakuten Books has 800 sures for evaluating the unexpectedness and serendipity of genres such as "novels and essays" and "sciences, medical recommender systems. sciences, and technologies," each of which consist of four- The former study assumed that unexpectedness is the dis- level categories. The genre id is a unique id that corresponds tance between the results produced by the system to be eval- to each genre. uated and those produced by primitive prediction methods. Here, primitive methods include recommendation methods 3.1.1 Phrase extraction from book data based on user proles or action histories. Based on this no- 3 The system extracts phrases using Chasen , a Japanese tion, they proposed unexpectedness for measuring the unex- morphological analyzer, from book .title and book .sub _title pectedness of recommendation lists and unexpectedness _r for each book. We heuristically selected "nouns," "verbs," to take into account the rankings in the lists. The latter "adjectives," "adverbs," and "unknown words" as target study also propose unexpectedness following the notion of parts of speech. Here, the system extracts also compound the former study. words such as "cognitive psychology" that are treated as one In our previous study[6], we evaluated our recommender phrase. system based on Murakami et al.'s evaluation metrics. How- ever, we did not evaluate the system through tests involving 3.2 System interface real users to determine its serendipity. In contrast, in this Figure 1 shows the interface of the proposed system, which study, we evaluate our proposed fusion-based recommender implements mechanisms (a), (b), and (c) mentioned above. system through experiments involving real users. (a) Mechanism that causes extrinsic accidents. The system implements [random], [search], [popular], and 3. FUSION-BASED RECOMMENDER SYS- [new] buttons, which cause extrinsic accidents. When the TEM user clicks each button, the system randomly searches for In this section, we describe our proposed fusion-based rec- k corresponding books from the book database. The books ommender system. This system has an interface that con- are displayed in input item views I, II, and III in Figure 1. sists of the aforementioned mechanisms for recommending Table 1 lists the processes that are called when each button serendipitous items (Figure 1). is clicked. As shown in Figures 1 , a user selects a base item from When the user moves the mouse cursor over the books dis- items displayed in views and drags-and-drops another ma- played in the views, the book information ("title," "sub ti- terial item onto the base item. Then, the system mixes tle," "authors," "publication date," and "genres") are shown these two items and outputs recommended items that have in a pop-up window. When the user right-clicks the books, features of both, which we dene as fusion. The user can he/she can view detailed information from the site of Rakuten repeatedly perform fusion by reselecting the base items and Books through an external browser. researching the material items until he/she obtains accept- 1 Rakuten Books book search API (in Japanese): able results. During this process, the user may interactively http://webservice.rakuten.co.jp/api/booksbooksearch/ discover serendipitous items. 2 Rakuten books (in Japanese): In Section 3.1, we describe the book database used as http://books.rakuten.co.jp/book/ 3 the recommendation content in this study. In Section 3.2, Chasen (in Japanese): http://chasen.naist.jp/hiki/ChaSen/ 41 Table 1: Search processing by each button. Button Processing Target view Random Searching k books from the book Input item database at random. view I Search Searching k books at random from Input item books whose title or sub _title includes view II keywords input in the text box. New Searching k books at ran- Input item dom from books satisfying view III review _count × review _average ≥ θ . Popular Searching k books at random from Input item books saled last one month. view III (b) Mechanism that causes intrinsic accidents. The system implements a fusion mechanism as an inter- face that causes intrinsic accidents. The user can select a base item by double-clicking a book from among the books in the input item view or recommen- dation item view. The base item is considered as the basis when performing fusion. The user can select a material item from among the books in the same two views. The material item is used for per- forming fusion with the base item. When the user drags- and-drops the material item onto the base item, fusion of the two items is performed. The system then displays the items outputted by the fusion in the recommendation item view. In Section 3.3, we dene three fusion methods. The system displays items outputted by each fusion method in the corresponding recommendation item view I, II, or III. (c) Mechanism that enables users to derive some value Figure 2: Example of each fusion method. from accidents through their sagacity. of m books whose book .title or book .sub _title includes at In mechanism (b), the user can select a base item and a least one phrase from the phrase list bookA.phraseList in material item from among the books deemed interesting in bookA and whose book .genre _id corresponds to at least one the views. Such intuitive selection of books may correspond genre from the genre list bookB .genre _idList in bookB . The to his/her sagacity. searched books are shown in recommendation item view II. Here, the type of book that can be selected depends on Figure 2 (b) shows an example of the fusion of bookA the user. When performing fusion, the user can select items "Management"and bookB "Equation loved by a doctor." that are suitable for his/her preferences as well as items that In this case, the system displays "If a female student who is a are considered interesting. manager of high-school baseball team reads `Management'," whose book .title or book .sub _title includes "management" 3.3 Fusion method and whose book .genre _id corresponds to bookB .genre _id As shown in Section 3.2 (b), fusion is performed using (i.e., "[novels and essays  Japanese novels]"). the base and the material item when the user drags-and- drops the material item onto the base item. We dene the phrase − author fusion. following three methods as fusion methods. In this section, The phrase − author fusion method searches for a maxi- bookA, bookB , and book denote the base item, material item, mum of m books whose book .title or book .sub _title includes and recommended item, respectively. at least one phrase from the phrase list bookA.phraseList in bookA and whose book .author corresponds to at least one phrase − phrase fusion. author from the author list bookB .authorList in bookB . The The phrase − phrase fusion method searches for a maxi- searched books are shown in recommendation item view III. mum of m books whose book .title or book .sub _title includes Figure 2 (c) shows an example of the fusion of bookA at least one phrase from the phrase list bookA.phraseList in "Neuroscience of language"and bookB "Excitement of bookA and at least one phrase from the phrase list science by Kenichiro Mogi." In this case, the system displays bookB .phraseList in bookB . The searched books are shown "Neuroscience class we want to take the best in the world," in recommendation item view I. Figure 2 (a) shows an exam- whose book .title or book .sub _title includes "neuroscience" ple of fusion for bookA"Equation loved by a doctor"and and whose book .author corresponds to bookB .author (i.e., bookB "Magic for cleaning up giving palpitations of life." "[Kenichiro Mogi]"). In this case, the system displays "Magic doctor" based on "doctor" in bookA and "magic" in bookB . 4. EXPERIMENTS phrase − genre fusion. In this section, we show the experimental results of user The phrase −genre fusion method searches for a maximum tests of our proposed fusion-based recommender system. We 42 implemented this system using Java and Processing as the Table 2: Questions for recommended books. evaluation system. In the experiments, we selected books No. Question as recommendation contents and created the book database Q1 I did not know this book. described in Section 3.1 using MySQL. Q2 I have been interested in this book before the system presented it to me. 4.1 Experimental method Q3 This book excited my interest for the rst time. Q4 I think that I could not nd this book by myself. Nine subjects (eight males and one female) participated in our study. Their age is from 20 to 23. They had av- erage computer skills and used the Internet regularly (ev- ery day/nearly every day). They also used online shopping websites such as Amazon very rarely (a few times so far) or rarely (a few times a month). They read books rarely (a few times a month) or moderately (once to three times a week). The experimental procedure is as follows: (1) We explain the recommender system to be used to each subject and provide them with the task "Find three books you want to read on holidays." (2) Each subject carries out the task using the assigned system (without time limitation). (3) If the subject nds suitable books, he/she marks them (at most 3 books). We call these books the main rec- ommended books. Figure 3: Separate evaluation of sub-recommended book. (4) If the subject nds books that are not suitable but are We considered two types of systemsAmazon search and interesting, he/she marks them (any number of books). recommend (A-RS) and Amazon ranking (A-Rank)as base- We call these books the sub-recommended books. line systems. In this section, we explain the utilization of the baseline systems and the proposed system. (5) The subject nishes the task when he/she nds three main recommended books. However, he/she can nish Amazon search and recommend (A-RS). the task if he/she is satised or satiated with even less The subjects are allowed to only use keyword and genre than three books. search method on the Amazon site, following which they can use the recommendation list (a list shown under "Customers (6) After the task is nished, the subject answers all the Who Bought This Item Also Bought"). We encouraged the questions listed in Table 2 for each recommended book. subjects to refer to the entire recommendation list because toward the end, the list potentially includes unexpected but (7) The subject performs the same steps for each recom- interesting books. Amazon's recommendation method is im- mender system. plemented by item-based collaborative ltering[13]. Section 4.2 discusses the recommender systems used in the experiments. Each subject uses the various recommender Amazon ranking (A-Rank). systems in a dierent order to cancel any eect that might The subjects are allowed to only refer to the ranking of otherwise be produced. "Best Sellers" and "New Releases." They are also allowed Table 2 lists the questions about the recommended books. to refer to the ranking in each category. Here, the subjects answered Q1 using a three-level scale {3:unknown, 2:known but never read, 1:have been ever read}, Fusion-based recommender system (F-RS). and Q2 to Q4 using a ve-level scale {5:strongly agree, We explained the system interface, described in Section 4:agree, 3:neither agree nor disagree, 2:disagree, 1:strongly 3.2, and how it is used to the subjects in advance. How- disagree}. With regard to "by myself" in Q4, we explained ever, we did not explain the details of the internal process- to the subjects that "if you think that you can easily nd ing of the fusion method, described in Section 3.3, because the book by using existing search engines (e.g., Google, Ya- we would like to observe whether the subjects can gradually hoo!) or by using a genre or keyword search at online/real understand the same through trial and error. book stores or libraries by yourself, the book is regarded as Here, we used k = 4, θ = 1000, and m = 3, as mentioned `ndable book by myself '." in Section 3.2 and Section 3.3. After all tasks were nished, the subjects answered ques- tions, this system excited my interest and enabled me to 4.3 Results discover somthing new, which is related to serendipity of the recommender systems using the same ve-level scale. 4.3.1 Evaluation of sub-recommended books 4.2 Comparative systems We analyzed what type of books were marked as sub- 4 recommended books. Figure 3 shows the overall results of We choose Amazon , a large online store with recom- the subjects' ratings for Q1Q4 from Table 2 about sub- mender systems, for comparison with our proposed system. recommended books. The gure shows the averages of the 4 amazon.co.jp (Japanese site): http://www.amazon.co.jp/ ratings for each recommender system. 43 As described in Section 1, the rst denition of serendip- intrinsic accidents and enables users to derive some value itous items is "items that can excite the user's interest for from accidents through their sagacity. The key idea of the the rst time although he/she does not know about them system is the fusion-based approach, through which the sys- and he/she would not be able to discover them by him- tem mixes two user-input items to nd new items that have self/herself." From this viewpoint, we evaluated the systems the mixed features. based on not only the discoverability but also whether the We experimentally evaluated the fusion-based recommended items excited the users' interest. Therefore, recommender system through user tests using a real book from the viewpoint of serendipity, we analyzed how many data set from Rakuten Books. The experimental results items that satised the conditions of books that "Q1: I did showed the eectiveness of this system compared with the not know this book," "Q4: I think that I could not nd this recommender systems used on the Amazon website from the book by myself," and "Q3: This book excited my interest viewpoint of serendipity. We would like to enhance its in- for the rst time" could be found by each system. If the terfaces and make the fusion methods more intuitive and rating of a book for Q1 = 3, Q4 ≥ 4, and Q3 ≥ 4, we assign understandable for the users. it a score of "1," otherwise we assign a score of "0." Figure 3 shows the averages. We found signicant dierences be- 6. ACKNOWLEDGEMENT tween the average of F-RS and those of A-RS and A-Rank This work was supported by a Grant-in-Aid for Young by a t-test with a signicance level of 5%. Scientists (B) (23700132). The second denition of serendipitous items is "items that can excite the user's interest for the rst time although he/she thought that he/she was not interested in them." 7. REFERENCES [1] J.L. Herlocker, J.A. Konstan, L.G. Terveen, and J.T. Riedl. From this viewpoint, we analyzed how many items that sat- Evaluating collaborative ltering recommender systems. ACM ised the conditions of books that "Q2: I have not been Transactions on Information Systems (TOIS), Vol. 22, No. 1, interested in this book" and "Q3: This book excited my in- pp. 553, 2004. terest for the rst time" could be found by each system. If [2] Tomoko Murakami, Koichiro Mori, and Ryohei Orihara. Metrics for evaluating the serendipity of recommendation lists. the rating of a book for Q2 ≤ 2 and Q3 ≥ 4, we assign it New Frontiers in Articial Intelligence, pp. 4046, 2008. a score of "1," otherwise we assign a score of "0." Figure [3] Elizabeth Jamison Hodges. Three Princes of Serendip. 3 shows the averages. We found signicant dierences be- Atheneum, 1964. tween the average of F-RS and that of A-RS by a t-test with [4] Shigekazu Sawaizumi, Osamu Katai, Hiroshi Kawakami, and Takayuki Shiose. Use of serendipity power for discoveries and a signicance level of 1%. In addition, we found signicant inventions. Intelligent and Evolutionary Systems, Studies in dierences between the average of F-RS and that of A-Rank Computational Intelligence, Vol. 187, pp. 163169, 2009. with a signicance level of 5%. [5] Royston M. Roberts. Serendipity: Accidental Discoveries in Although A-RS recommends books related to the browsed Science. Wiley, 1989. [6] Kenta Oku and Fumio Hattori. Fusion-based recommender book through item-based collaborative ltering, there is lit- system for improving serendipity. In Proceedings of the tle possibility of the recommended book being largely against Workshop on Novelty and Diversity in Recommender Systems the user's interest because of its high accuracy. Meanwhile, (DiveRS 2011), at the 5th ACM International Conference on Recommender Systems (RecSys 2011), pp. 1926, 2011. because A-Rank recommends popular books, the user may [7] Cai-Nicolas Ziegler, Georg Lausen, and Lars Schmidt-Thieme. already know the recommended books if they belong to gen- Taxonomy-driven computation of product recommendations. In res the user is interested in. On the other hand, the fusion- Proceedings of the Thirteenth ACM conference on Information based recommender system can recommend books that are and knowledge management - CIKM '04, p. 406, New York, New York, USA, 2004. ACM Press. occasionally against the user's interest depending on the se- [8] C.N. Ziegler, S.M. McNee, J.A. Konstan, and Georg Lausen. lection of the material item. This is why the system showed Improving recommendation lists through topic diversication. high discoverability, although this involves some risks. In In Proceedings of the 14th international conference on World Wide Web, pp. 2232, New York, New York, USA, 2005. ACM. addition, because the recommended books are still relevant [9] Badrul Sarwar, George Karypis, Joseph Konstan, and John to the base item, the user may be interested in them. This Reidl. Item-based collaborative ltering recommendation is why the fusion-based recommender system was superior algorithms. In Proceedings of the 10th international conference from the viewpoint of serendipity. on World Wide Web, pp. 285295, New York, New York, USA, 2001. ACM. [10] Shlomo Berkovsky and Jill Freyne. Group-based recipe 4.3.2 Evaluation of systems recommendations: analysis of data aggregation strategies. In We focus on the question about serendipity, "this system Proceedings of the fourth ACM conference on Recommender excited my interest and enabled me to discover something systems, pp. 111118. ACM, 2010. [11] Y. Hijikata, T. Shimizu, and S. Nishida. Discovery-oriented new." The average of the subjects' ratings were 3.00 for A- collaborative ltering for improving user satisfaction. In RS, 2.67 for A-Rank, and 4.22 for F-RS. From this view- Proceedings of the 13th international conference on Intelligent point, the proposed system signicantly outperformed A-RS user interfaces, pp. 6776. ACM, 2009. and A-Rank with a signicance level of 5%. This result in- [12] Mouzhi Ge, C. Delgado-Battenfeld, and D. Jannach. Beyond accuracy: evaluating recommender systems by coverage and dicates that the proposed system can provide serendipitous serendipity. In Proceedings of the fourth ACM conference on items related to "items that can attract the user's interest Recommender systems, pp. 257260. ACM, 2010. after being displayed by the system," which is one of the [13] G. Linden, B. Smith, and J. York. Amazon.com recommendations: Item-to-item collaborative ltering. IEEE denitions of serendipitous items. Internet Computing, Vol. 7, No. 1, pp. 7680, January 2003. 5. CONCLUSION In this study, we improved upon our fusion-based rec- ommender system based on the deeper idea of serendipity. This system possesses mechanisms that cause extrinsic and 44