Enhancing Accuracy of Hybrid Recommender Systems through Adapting the Domain Trends Fatih Aksel Ayşenur Birtürk Department of Computer Engineering Department of Computer Engineering METU METU fatih.aksel@ceng.metu.edu.tr birturk@ceng.metu.edu.tr ABSTRACT Content-based and collaborative filtering are the two major Hybrid recommender systems combine several algorithms recommendation techniques that have come to dominate the based on their hybridization strategy. Prediction algorithm current recommender system area. Content-based recom- selection strategy directly influence the accuracy of the hy- mender system uses the descriptions about the content of the brid recommenders. Recent research has mostly focused items (such as meta-data of the item), whereas collaborative on static hybridization schemes which are designed as fixed filtering system tries to identify users whose tastes are simi- combinations of prediction algorithms and do not change lar and recommends items they like. Other recommendation at run-time. However, people’s tastes and desires are tem- technologies include hybrid techniques, knowledge-based ap- porary and gradually evolve. Moreover, each domain has proaches etc [6]. The popular Amazon item-to-item system unique characteristics, trends and unique user interests. In [7] is one of the well-known recommender system that uses this paper, we propose an adaptive method for hybrid rec- collaborative filtering techniques. NewsWeeder [12] and In- ommender systems, in which the combination of algorithms foFinder [11] are the pure content-based recommender sys- are learned and dynamically updated from the results of tems that analyze the content of items, in their recommen- previous predictions. We describe our hybrid recommender dation process. system, called AdaRec, that uses domain attributes to un- derstand the domain drifts and trends, and user feedback Previous research in this area, has shown that these tech- in order to change it’s prediction strategy at run-time, and niques suffer from various potential problems-such as, spar- adapt the combination of content-based and collaborative sity, reduced coverage, scalability, and cold-start problems algorithms to have better results. Experiment results with [1, 6, 23]. For example; collaborative filtering techniques datasets show that our system outperforms naive hybrid rec- depend on historical ratings of across users that have the ommender. drawback, called cold start problem - an item cannot be rec- ommended until it has been rated by a number of existing users. The technique tends to offer poor results when there Categories and Subject Descriptors are not enough user ratings. Content-based techniques can H.3.3 [Information Storage and Retrieval]: Informa- overcome the cold start problem, because new items can be tion Search and Retrieval - Information filtering, Retrieval recommended based on item features about the content of models, Selection process; I.2.6 [Artificial Intelligence]: the item with existing items. Unfortunately, content-based Learning approaches require additional information about the content of item, which may be hard to extract (such as, movies, mu- General Terms sic, restaurants). Every recommendation approach has its Design, Experimentation, Algorithms own strengths and weaknesses. Hybrid recommender sys- tems have been proposed to gain better results with fewer drawbacks. Keywords Hybrid Recommender Systems, Switching Hybridization, De- Most of the recommender system implementations focuses cision Tree Induction, Hybrid Recommender Framework, on hybrid systems that use mixture of recommendation ap- Adaptive Recommender Systems proaches [6]. This helps to avoid certain limitations of content- based and collaborative filtering systems. Previous research 1. INTRODUCTION on hybrid recommender system has mostly focused on static hybridization approaches (strategy) that do not change their hybridization behavior at run-time. Fixed strategy may be suboptimal for dynamic domains&user behaviors. Moreover they are unable to adapt to domain drifts. Since people’s tastes and desires are transient and subject to change, a good recommender engine should deal with changing con- sumer preferences. Copyright is held by the author/owner(s). Workshop on the Practical Use of Recommender Systems, Algorithms and Technologies (PRSAT 2010), held in conjunction with RecSys 2010. September 30, 2010, Barcelona, Spain. In this paper, we describe an Adaptive Hybrid Recommender System, called AdaRec, that modifies its switching strat- each movie as a text document. User and item profiles are egy according to the performance of prediction techniques. built by using a Naive Bayes classifier that can handle vec- Our hybrid recommender approach uses adaptive prediction tors of bags of words; where each ’bag-of-words’ corresponds strategies that determine which prediction techniques (al- to a movie-feature (e.g. title, cast, etc.). The Naive Bayes gorithms) should be used at the moment an actual predic- classifier is used to approximate the missing entries in the tion is required. Initially we used manually created rule- user-item rating matrix, and a user-based collaborative fil- based strategies which are static. These static hybridization tering is applied over this dense matrix. schemes have drawbacks. They require expert knowledge and they are unable to adapt to emerging trends in the do- In our system, we choice the Duine Framework2 for our rec- main. We now focus on prediction strategies that learn by ommendation engine component, which is an open-source themselves. hybrid recommendation system [20]. The Duine framework allows users to develop their own prediction engines for rec- The paper is organized as follows. Related work is described ommender systems. The framework contains a set of rec- in Section 2. In Section 3, we present the adaptive predic- ommendation techniques, ways to combine these techniques tion strategy model for hybrid recommenders. We then de- into recommendation strategies, a profile manager, and it scribe our experimental recommender systems’ architecture allows users to add their own recommender algorithm to & learning module that dynamically adjusts recommenda- the system. It uses switching hybridization method in the tion strategy in response to the changes in domain. An ini- selection of prediction techniques. The result of a Duine pre- tial evaluation of our approach, based on MovieLens dataset, diction engine is the retrieved set of information with added is presented in Section 6. data about how interesting each piece of information is for the user [20, 17]. 2. RELATED WORK Personalization techniques have been investigated extensively 3. ADAREC: AN ADAPTIVE HYBRID REC- in several areas of computer science. Especially in the do- OMMENDER SYSTEM mains of recommender systems, personalization algorithms Hybrid recommendation systems combine multiple algorithms have been developed and deployed on various types of sys- and define a switching behavior (strategy) among them. This tems [1]. strategy decides which technique to choose under what cir- cumstances for a given prediction request. Recommender There have been several research efforts to combine different system’s behavior is directly influenced by the prediction recommendation technologies. The BellKor system [4], stat- strategy. The construction of accurate strategy that suits ically combines weighted linear combination of more than a in all circumstances is a difficult process. A well-designed hundred collaborative filtering engines. The system uses the adaptive prediction strategy offers advantages over the tra- model based approach that first learns a statistical model ditional static one. in an offline fashion, and then uses it to make predictions and generate recommendations. The weights are learned In our approach we use the switching hybridization in or- by using a linear regression on outputs of the engine. The der to decide which prediction technique is most suitable STREAM [3] recommender system, which can be thought of to provide a prediction. Prediction techniques, also called as a special case of the BellKor system, classifies the recom- the predictors are combined into an upper prediction model mender engines in two levels: called level-1 and level-2 pre- that is called prediction strategy. The central concept in dictors. The hybrid STREAM system uses run-time metrics combining multiple predictors using the switching hybridiza- to learn next level predictors by linear regression. However tion method is the prediction strategy. Prediction Strategy, combining many engines level by level results performance which defines the algorithm selection strategy, changes the problems at run-time. Our approach combines different al- behavior of the recommender engines at run-time. gorithms on a single hybrid engine with an adaptive strategy. Most of the currently available personalized information sys- Some hybrid recommenders choice the best suited recom- tems focus on the use of a single selection technique or a fixed mender engine for a specific case (user, item, input etc.). combination of techniques [20, 14]. However, application For example, the Daily Learner system [5], which is a per- domains are dynamic environments. Users are continuously sonal web-based agent, selects the best recommender engine interacting with domain, new concepts and trends emerge according to the confidence levels. But in order to han- each day. Therefore, user interests might change dynami- dle different engines in a common point, confidence scores cally over time. It does not seem possible to adapt trends should be comparable. by using a static approach (static prediction strategy). In- stead of static methods dynamic methods that can adapt to The use of machine learning algorithms for user modeling change on domains, could be more effective. purposes has recently attracted much attention. In [13], the authors proposed a hybrid recommender framework to rec- Different design approaches might be used for the predic- ommend movies to users. The system uses a content-based tion strategy adaptation. Rule based, case based, artificial predictor to enhance existing user data, and then provides neural networks or Bayesian are some of the learning tech- personalized suggestions through collaborative filtering. In niques. Each technique has its own strengths and weak- the content-based filtering part of the system, they get ex- nesses. In this paper, we introduce self adaptive prediction tra information about movies from the IMDB1 and handle strategy learning module which employs a strategy based on 1 2 http://www.imdb.com/ http://duineframework.org/ its attached learning technique. Learning module initializes prediction engine according to the specified machine learn- ing technique. This prediction strategy adapts itself to the current context by using the previous performance results of the techniques. Different machine learning algorithms that induce decision trees or decision rules could be attached to our experimental design. 4. PREDICTION STRATEGY LEARNING Duine Recommender offers extensive options for configuring various recommender algorithms. It provides a sample of most common recommendation algorithms that can be com- bined in algorithm strategies. In the Duine Recommender the knowledge that designs the selection strategy is provided manually by experts [8, 21]. However, the combination of different techniques in a dynamic, intelligent and adaptive way can provide better prediction results. The combina- tion of techniques should not be fixed within a system and that the combination ought to be based on knowledge about Figure 1: The Duine Recommender’s prediction strengths and weaknesses of each technique and that the strategy depicted as decision tree. Grey nodes rep- choice of techniques should be made at the moment a pre- resent prediction algorithms and arrows represent diction is required. the attributes&threshold values. Hybridization of a recommender system employs using the leaf nodes represent the decisions of the tree [20]. best prediction technique from the available ones. The main purpose of a prediction strategy is to use the most appropri- Adaptive strategy can be designed using by rule sets or trees ate prediction technique in a particular context. Adaptive that contain the knowledge on decisions. Decision rules can prediction strategy depicts the selection rules of prediction also be expressed as decision trees. Experts often describe techniques. Figure 1 shows a sample prediction strategy their knowledge about a process using decision trees and that decides when to use which predictors (gray nodes) by decision rules as they are easy to interpret by other people. using the threshold values (arrows). The decision trees are a good interface between experts and the recommender systems. Prediction strategy employs the selection rules of the avail- able prediction techniques. Our experimental hybrid rec- Adaptive prediction strategy is in the form of a decision ommender system has plenty of pre-defined prediction tech- tree. Another way to represent decision tree is using decision niques. These prediction techniques are implemented by us- rules. Decision rules generally take the form of IF . . . THEN ing different paradigms. Content-based and collaborative fil- . . . rules, i.e. IF attribute1 = value1 AND attribute2 = tering are the two principal paradigms for computing recom- value2 THEN result2 . mendations [23]. In our system we have used content-based, collaborative filtering, knowledge based and case-based pre- Depending on the nature of the domains (movie, music, book diction techniques. etc.) different attribute-value combinations can be used for prediction strategy design. In our proposed system, since To make decisions about which predictor is suitable for the we tested on MovieLens dataset, we choice these specific current context, threshold values, predictors’ state and users attributes that have meaningful correlations between movie feedback are used by the adaptive prediction strategy. The domain and prediction techniques. We believe that, by mea- state of a predictor is described by the amount and at the suring the changes on these attributes, we can capture the quality of knowledge that is available to the predictor. In domain drifts and trends other words, the knowledge that is used by the prediction technique is the basis of its predictions. One of the objectives of the prediction strategy is to select the right prediction technique according to the current states of the predictors. 1. item ratings count, the number of ratings that the cur- The initial strategy is defined by using the expert knowledge. rent item has. System starts with an initial prediction strategy. Later on 2. item similar user count, similar users count that have the Learning Module adjusts the prediction strategy to the already rated the current item. current systems’ domain. 3. similar item count, the number of similar items count Decision trees and decision rules are model based strategies. according to similarity measures. Each node in a decision tree represents some attributes and each branch from a node corresponds to a possible value for 4. main genre interest, main genre interest certainty of that attribute. When trying to classify a certain instance, the current item among the users items. as seen in the Figure 1, one starts at the root of the decision tree and move down the branches of the tree according to 5. sub genre interest, sub genre interest certainty of the the values of the attributes until a leaf node is reached. The current item among the users items. of two core parts, Recommender Engine and Learning Mod- ule. Recommender Engine is responsible for generating the pre- dictions of items based on the previous user profiles and item contents. It attempts to recommend information items, such as movies, music, books, that are likely to be of interest to the user. The recommender generate the predictions by us- ing its attached prediction strategy. The implementation here uses the open source Duine Framework for the recom- mender engine. Learning Module handles the new prediction strategy cre- ation upon the previous instances and performance results of the prediction techniques on each learning cycle. It al- lows the building of new decision trees/decision rules based on the previous recorded instances. Figure 2: Overall architecture of the AdaRec Sys- tem. Learning cycle is a logical concept that represents the re- design frequency of the prediction strategy. Each instance, based on the indicated count, and prediction algorithm’s Decision trees/decision rules are constructed using the com- performance results are collected between two learning cy- bination of the above five attributes. As shown in Figure cles. The learning module modifies the values of attributes 1 at each node of the decision tree a attribute is compared in decision rules, which is also called threshold values accord- with a value, which is called threshold value. These five at- ing to the gathered results of the prediction techniques per- tributes are used to classify the prediction algorithms. formance. The old prediction strategy is modified by using recommender engines’ machine learning algorithm (rule tun- The Recommender System needs to work with the best suited ing, rule adaptation, decision tree induction etc.). The mod- prediction technique for its domain and users. In our system, ification of the threshold values allows recommender system we used and tested different prediction techniques. These to analyze&adapt the nature of the users and the domain. are; topNDeviation, userAverage, socialFiltering, CBR (Case Based Reasoning), mainGenreLMS, subGenreLMS, infor- The learning module first tests the accuracy of the each pre- mationFiltering. Because of the dynamic nature of the do- dictor in the system. Than the prediction strategy is re- main, these attributes create different forms of decision trees. designed by the learning module in order to improve proper use of predictors. Adaptive prediction strategy improves its’ Each domain has unique characteristics including user be- prediction accuracy by learning better when to use which haviors, emerging trends etc. Recommender engine able to predictors. The learning module adapts the hybrid recom- adapt itself to the changes in the domain by analyzing the mender system to the current characteristics of domain. changes. Also the system can capture the trends in the do- main able to re-design its’ attached prediction strategy. In Previous predictions and user feedbacks are fed to the train- our system, we used and tested different prediction tech- ing set of the next learning cycle. Inductive learning is used niques. in learning from the training set. In our experiments we tested different (1K, 1.5K, 2K and 3K) instance sizes for The quality of a decision tree depends on both the classifi- training sets. The training set contains the instances from cation accuracy and the size of the tree[10]. After reviewing the previous learning cycle results. There are quite a few and testing many of the options, we decided to use two de- inductive learning techniques to choose from, including in- cision tree classifiers; J48 (pruned C4.5 decision tree)[18], formation theoretic ones (e.g. Rocchio classifier), neural which is the WEKA’s3 implementation of the decision tree networks (e.g. back-propagation), instance-based methods learner based on the C4.5 decision tree algorithm, and BF- (e.g. nearest neighbour), rule learners (e.g. RIPPER), deci- Tree (best first-decision tree), which is a decision tree learner sion trees (e.g. C4.5) and probabilistic classifiers (e.g. naive that uses a best first method of determining its branches. Bayes) [14]. Also in order to compare the rule-based and tree induction methods we plugged and tested the Conjunctive Rules clas- The User Profile, is the representation of the user in the sifier, which is a simple rule learner that learns a set of simple system. For each active user a user model is stored in the conjunctive rules. user profile. User profile holds the knowledge (such as pref- erences, feedbacks, feature weights etc.) about users in a structured way. The Recommender Shell, encapsulates the 5. OVERVIEW OF THE ADAREC SYSTEM Recommender Engine’s interaction with other modules. The Figure 2 depicts the architectural overview of the proposed shell serves the created prediction strategies to the engine. AdaRec system. Our experimental framework is an exten- The Prediction Parser, produces the performance results of sion of the open-source Duine Framework. System consists the prediction algorithms based on the analyzing of the col- 3 lected predictions & feedbacks. This module handles the Waikato Environment for Knowledge Analysis: decomposition of the prediction results and generates the http://www.cs.waikato.ac.nz/ml/weka/ training set of sample instances with current attributes. The aim of the experiments is to examine how the recom- mendation quality is affected by our proposed learning mod- User feedbacks and MAE (Mean Absolute Error) are the ule. The present model of the Duine Framework is non adap- main criteria, which describe the trends in the domain. Adap- tive but it supports predictor level learning. This original tive prediction strategy learns its domain trends over time state of the framework is referred to baseline. As shown via unobtrusive monitoring and relevance feedback. In our in the Figure 1, Duine recommender uses a static prediction proposed system, we focused self adaptive prediction strat- strategy as its’ hybridization scheme, which does not change egy that classifies according to its’ attached machine learn- at run-time. We want to compare the prediction quality ob- ing technique. This prediction strategy adapts itself to the tained from the framework’s baseline (non adaptive) to the current context by using the previous performance results of quality obtained by our proposed experimental framework the techniques. Different machine learning algorithms that (adaptive). The approach will be considered useful if the induce decision trees or decision rule sets could be attached prediction accuracy is better than the baseline. At the first to our experimental design.The architecture is open and flex- iterations both systems are initialized with the same strat- ible enough to attach different machine learning algorithms. egy, which is the default strategy of the Duine Framework. 6. EXPERIMENTS The validation process is handled using the following proce- In this section we present a brief discussion of our experimen- dure: tal dataset, evaluation metric followed by the experimental results and discussion. 1. The ratings provided by the dataset are fed to the system one by one, in the logical order of the system In order to assess the impact of our proposed adaptive rec- (ordered by timestamps). ommender and different machine learning algorithms, we calculated prediction accuracy (MAE) of the system using 2. When a rating is provided during validation, predic- different configurations of the machine learning schemes. tion strategy is invoked to provide a prediction for the Different MovieLens datasets are examined during the ex- current user and the current item. The average pre- periments. diction error can be used a performance indicator of the attached prediction strategy. The datasets are divided into temporal subsets according to their time-stamp values. Natural domain trends and changes 3. After the error has been calculated, the given rating in user interests are handled by using the subsets of the is provided as feedback to the recommender system. dataset. The adaptive system collects the feedback as well as the current attributes of the system as instances. 6.1 Datasets We used data for our recommender system from MovieLens4 , 4. Whenever the collected instances reached the learning which is a web-based research recommender system that de- cycle’s instance count (1000 instances for example), buted in Fall 1997 [15]. the prediction strategy of the system will be redesigned by the adaptive system according to the instances. In our experiments MovieLens one million ratings dataset is used, with 6040 users and 3900 movies pertaining to 19 genres. MovieLens dataset contains explicit ratings about This way, when the next learning cycle is processed, the movies and has a very high density. In order to train the adaptive system has learned from all the previously pro- recommender system, the MovieLens dataset is divided in to cessed ratings. This process is repeated for all ratings at different temporal sets based on their distribution in time. both adaptive and non-adaptive (baseline) systems in the When testing the ratings of first sets are used for recom- test set. At the end MAE is calculated by averaging abso- mender engine training [15, 9, 16]. lute errors within the baseline and the adaptive system, as described above. 6.2 Experimental Setup Recommender systems researchers use a number of differ- 6.3 Results & Discussion ent measures for evaluating the success of the recommenda- Each prediction provided by the two different systems are tion or prediction algorithms [19, 22]. For our experiments, examined. The prediction accuracy and the prediction error we use a widely popular statistical accuracy metric named (MAE) are recorded. In experimenting with the MovieLens Global Mean Absolute Error (MAE), which is a measure dataset, we considered both the proposed method, called of the deviation of recommendations from their true user- adaptive system, and the existing method, called baseline. specified values. The MAE is defined as the average differ- ence between the predicted ratings and the real user ratings, In the experiments, different number of instances, such as as defined within the test sets. Formally, MAE can be de- 1K, 2K and 3K, are used. The purpose of different number fined as: of instances was to compare the influence of the instance PN size on algorithms at the same domain. In the adaptive i=1 |pi , ri | M AE = system, C4.5, BF-Tree and Conjunctive Rules classifier al- N gorithms are attached to the learning module and its results where pi is the predicted value for item i and ri is the user’s are recorded. We tuned the algorithms to optimize and con- rating. figured to deliver the highest quality prediction without con- 4 http://www.movielens.umn.edu cern for performance. MAE). In case of other algorithms it is expected that in- creasing the number of instances would mean small MAE values. The same trend is observed in the case of 2K and 3K instances. Figure 5 presents the average MAE of hundred runs for 1K instance size. In this experiment we evaluate the impact of more runs for 1K instance size. It can be observed from the chart that changes in the MAE show the similar trends for both the baseline, adaptive and best systems. A harmony is achieved through time. The curves are similar in such a way that if one of them has a good prediction accuracy in one run, the others also have the good accuracy for that run. Figure 3: Quality of prediction (MAE) using In order to determine the impact of the instance size, we AdaRec (attached J48, BF-Tree and Conjunctive carried out an experiment where we varied the value of in- Rules with 1000 instances) vs Baseline & Best. stance size (1K, 1.5K, 2K and 3K). For each of these training set/instance size values we run our experiments. Figure 6 presents the whole picture of the adaptive system’s perfor- mance results. From the plot we observe that the quality of MAE increases as we increase the instance size. Figure 3 presents the accuracy results of all used ML tech- niques for 1K instance size. It can be observed from the fig- ure that the J48 attached AdaRec system performs better than the other systems. Also BF-Tree seems good enough to compete against the naive hybrid system. But BF-Tree algorithm needs some domain depended configuration ad- justments. From the figure we also observe that the Con- junctive Rules algorithm underperforms among other ML algorithms. The rule learner algorithm seems not stable as the decision tree learners. The results also show that, when using well tuned algo- Figure 4: Prediction accuracy comparison of the rithms, the adaptive system is stable (better than the base- AdaRec (attached J48, BF-Tree and Conjunctive line) in obtaining the average prediction accuracy. This Rules) vs Baseline & Best for different instance durability, which can be called the impact of learning, is sizes. established by the learning module. In order to adapt rec- ommender engine to the current trends, the learning module re-designs the prediction strategy. The learning ability sup- We also plot the result of the best MAE (less is better) of the ports the recommender system adaptation to the changes. hybrid recommender in the current context at each iteration. The best MAE is referred as best at the charts. Therefore it is possible to compare the performance of the algorithms 7. CONCLUSION & FUTURE WORK and the best possible result. In this paper, we introduced an adaptive hybrid recom- mender system, called AdaRec, that combines several rec- Figure 6 presents the prediction quality (average MAE) re- ommender algorithms in which the combination parame- sults of our experiments for the adaptive system as well as ters are learned and dynamically updated from the results the original system referred to baseline. J48 algorithm is of previous predictions. Research study shows that tradi- used in these experiments. In this chart, prediction quality tional static hybrid recommender systems suffer from chang- is plotted for each of the iterations. On each iteration adap- ing user preferences. In order to improve the recommenda- tive system re-designs its prediction strategy according to tion performance, we handle domain drifts in our approach. the previous iteration’s performance result (feedbacks and The Learning Module re-designs its prediction (switching) results). It can be inferred from chart that, the predic- strategy according to the performance of prediction tech- tion quality of the adaptive system performs better than niques based on user feedbacks. As a result, the system the baseline. It can also be observed from the chart that the adapts to the application domain, and the performance of adaptive system adapts itself to the changes in the domain recommendation increases as more data are accumulated. In and users. the MovieLens dataset, the proposed adaptive system out- performs the baseline (naive hybrid system). Figure 4 presents the prediction accuracy results of the dif- ferent instance sizes. In the figure, we also plot the overall Initial experimental results show its potential impacts. There- performance of ML algorithms as average. It can also be ob- fore, for the next step, we plan to further testing the learning served from the charts that as we increase the instance size module with various heterogeneous datasets. It would be in- of algorithms the quality tends to be superior (decreased teresting to examine the different domains other than movie Figure 5: Quality of prediction using J48 (pruned C4.5), Baseline and Best according to 100 iterations. Previous 1000 instances are used for learning on each iteration. Figure 6: Comparison of the different instance sizes. 1K, 1.5K, 2K and 3K instances are used for learning cycle. 3K learning cycles’ prediction quality yields better results than the others (such as music, book, news etc. ). Also, our future work [12] K. Lang. Newsweeder: Learning to filter netnews. In will explore the effectiveness of other machine learning tech- In Proceedings of the Twelfth International Conference niques for use in learning module. In our experiments we on Machine Learning, 1995. fixed the used attributes for domain monitoring. It would [13] P. Melville, R. Mooney, and R. Nagarajan. be also interesting to use dynamic attributes, which means Content-boosted collaborative filtering for improved to use different attributes on different iterations. We believe recommendations. In Proceedings of the National that with this adaptive learning module, a traditional hybrid Conference on Artificial Intelligence, pages 187–192. recommender should have higher chance to allow its users Menlo Park, CA; Cambridge, MA; London; AAAI to efficiently obtain an accurate and confident decision. Press; MIT Press; 1999, 2002. [14] S. E. Middleton. Capturing knowledge of user 8. ACKNOWLEDGMENTS preferences with recommender systems. PhD thesis, We would like to acknowledge the anonymous reviewers of University of Southampton, May 2003. HaCDAIS’2010 and PRSAT’2010 workshops for their useful [15] Movielens dataset. comments. An earlier version of this study has appeared as http://www.grouplens.org/node/73. a short paper in the proceedings of HaCDAIS 2010 workshop [16] Netflix dataset. http://www.netflixprize.com. [2]. [17] Novay. Duine recommender — telematica instituut-novay, 2010. [Online; accessed 1-July-2010]. 9. REFERENCES [18] S. Salzberg and A. Segre. Book review: ”c4.5: [1] G. Adomavicius and A. Tuzhilin. Toward the next Programs for machine learning” by j. ross quinlan. In generation of recommender systems: A survey of the Machine Learning. morgan kaufmann publishers, inc, state-of-the-art and possible extensions. IEEE 1994. Transactions on Knowledge and Data Engineering, [19] B. M. Sarwar, J. A. Konstan, A. Borchers, 17(6):734–749, 2005. J. Herlocker, B. Miller, and J. Riedl. Using filtering [2] F. Aksel and A. Birturk. An Adaptive Hybrid agents to improve prediction quality in the grouplens Recommender System that Learns Domain Dynamics. research collaborative filtering system. pages 345–354. International Workshop on Handling Concept Drift in ACM Press, 1998. Adaptive Information Systems: Importance, [20] M. V. Setten. Supporting People in Finding Challenges and Solutions (HaCDAIS-2010) at the Information- Hybrid Recommender Systems and Goal European Conference on Machine Learning and Based Structuring. Telematica instituut fundamental Principles and Practice of Knowledge Discovery in research series no:016, Telematica Instituut, November Databases 2010 (ECML PKDD 2010), page 49, 2010. 2005. [3] X. Bao, L. Bergman, and R. Thompson. Stacking [21] M. V. Setten, M. Veenstra, and A. Nijholt. Prediction recommendation engines with additional strategies: Combining prediction techniques to meta-features. In Proceedings of the third ACM optimize personalization. In Proceedings of the conference on Recommender systems, pages 109–116. workshop Personalization in Future TV, volume 2. ACM, 2009. Citeseer, 2002. [4] R. Bell, Y. Koren, and C. Volinsky. The bellkor [22] U. Shardanand and P. Maes. Social information solution to the netflix prize. KorBell Team’s Report to filtering: algorithms for automating “word of mouth”. Netflix, 2007. In CHI ’95: Proceedings of the SIGCHI conference on [5] D. Billsus and M. Pazzani. User modeling for adaptive Human factors in computing systems, pages 210–217, news access. User Modeling and User-Adapted New York, NY, USA, 1995. ACM Interaction, 10(2):147–180, 2000. Press/Addison-Wesley Publishing Co. [6] R. Burke. Hybrid recommender systems: Survey and [23] C. Ziegler. Towards Decentralized Recommender experiments. User Modeling and User-Adapted Systems. Freiburg i. br, Albert-Ludwigs-Universität Interaction, 12(4):331–370, 2002. Freiburg,Germany, June 2005. [7] A. Gunawardana and C. Meek. A unified approach to building hybrid recommender systems. In RecSys ’09: Proceedings of the third ACM conference on Recommender systems, pages 117–124, New York, NY, USA, 2009. ACM. [8] J. Herlocker and J. Konstan. Content-independent task-focused recommendation. IEEE Internet Computing, pages 40–47, 2001. [9] The internet movie database. http://www.imdb.com/. [10] D. Jensen and P. R. Cohen. Multiple comparisons in induction algorithms. In Machine Learning, pages 309–338, 1998. [11] B. Krulwich and C. Burkey. Learning user information interests through extraction of semantically significant phrases. In Proceedings of the AAAI spring symposium on machine learning in information access, pages 100–112, 1996.