An Architecture for a General Purpose Multi-Algorithm Recommender System Jose C. Cortizo, Francisco M. Carrero and Borja Monsalve BrainSins http://www.brainsins.com Madrid, Spain {josecarlos.cortizo, francisco.carrero, borja.monsalve}@brainsins.com ABSTRACT those games with none or few ratings and a social-based rec- Although the actual state-of-the-art on Recommender Sys- ommender system [3]. We also needed to adapt the resulting tems is good enough to allow recommendations and person- system to other domains, beginning with image recommen- alization along many application fields, developing a gen- dations to be integrated in FlickrBabel2 , our multilingual eral purpose multi-algorithm recommender system is a tough multimedia search engine. And, finally, we decided to use a task. In this paper we present the main challenges involved ”Software as a Service” (SaaS) model to separete rommen- on developing such system and a system’s architecture that dations from the rest of the platform. allows us to face this challenges. Although there exist several commercial approaches to gen- eral SaaS recommender systems, like Strands3 or Directed- Categories and Subject Descriptors Edge4 there exist no literature focusing on the system’s as- H.4 [Information Systems Applications]: Miscellaneous pects of general recommender systems. In this paper we de- scribe our experience developing our general purpose multi- General Terms algorithm recommender system, which is currently being Design, Algorithms used to personalize our products and services at BrainSins and will be used as experimental platform to compare and evaluate our further research on recommender systems. Keywords General purpose recommendations, System architecture, API, In the next section we describe the main challenges we found Multi-algorithm in order to develop a general purpose multi-algorithm recom- mender system. In section 3 we describe the general archi- 1. INTRODUCTION tecture of the system focusing on the elements that allowed There is a lot of literature on Recommender Systems for us to solve the main issues, and in section 4 we focus on our specific online domains like social software items [4], mu- recommender system API, which enables all our products sic [1], queries (in search engines) [8], news [6], e-commerce and services to interoperate with the recommender system. [7] or even for non online domains such as [5]. Those rec- Section 5 describes the next research works we will face, and ommender systems employ specific techniques for specific section 6 concludes the paper. domains in order to produce the most accurate systems for each single domain. 2. MAIN CHALLENGES FOR A GENERAL We wanted to integrate a recommender system in Wipley , 1 PURPOSE MULTI-ALGORITHM RECOM- our social network for videogamers launched by the end of MENDER SYSTEM 2009. With this purpose, we worked on a recommender sys- When designing a general purpose multi-algorithm recom- tem for videogames using a collaborative filtering approach mender system, we found several challenges that had to be with multidimensional and contextual features to fit this addressed in order to develop a useful system. particular domain. After that, we wanted to improve the recommender with a content-based recommender system for 1 http://www.wipley.es • Interoperability: Recommender systems are usually created to access a specific database that uses a well- known data structure. However, since our system had to offer the possibility of being integrated into several different existing platforms, we had to deal with the problem of accessing sources with different data struc- tures. PRSAT 2010. Copyright is held by the author/owner(s). Workshop on the Practical Use of Recommender Systems, Algorithms and Technologies 2 http://www.flickrbabel.com (PRSAT 2010), held in conjunction with RecSys 2010. September 30, 2010, 3 Barcelona, Spain. http://recommender.strands.com 4 http://www.directededge.com • Configurability and easy of use: As one of the main goals of this recommender system was to be able to manage several recommender algorithms to provide a particular recommendation, the system had to be highly configurable. In these cases usability may be- come a problem difficult to solve. • Performance: Recommendations must be served in real time, but users do not tolerate an increase in page downloading time. However, when a system is de- signed with a high degree of configurability, there are always some issues that slow down performance. In our particular case, the new system had to be as effective as the recommender we already had on production. Figure 1: The request processor evaluates if a cer- tain API request needs to be run online or it can be • Disk usage: A highly configurable system also presents batched. disk space problems, due to the fact that data repre- sentation cannot be optimized. • Batch requests, which may be stored and processed • Scalability: When applied to the web, the number of only at given time periods. Now requests processing items to be recommended and the number of users can be delayed and attended when the system is not to receive those recommendations often grow expo- at full capacity. These requests are used to upload the nentially. Therefore, a recommender system needs to initial data from a client and also to update informa- be scalable in order to grow at the same rate. How- tion concerning users with a wide user profile. ever, being scalable also means that the system should somehow hide the way it scales, so there should be no need to code or rewrite configuration parameters in Figure 1 shows how API requests are being processed. Each the products and services that access the recommender API request generates an HTTP request to a certain end- system. point where the Request Processor evaluates it and deter- mines whether it must be processed at that moment or it can be delayed until more requests reach the system (for a After analyzing possible solutions for those challenges and more optimum processing) or until certain batch process is start designing and developing the system, we thought that programmed to be run. a key fact to achieve most of them was to conveniently sep- arate recommendation process from the interface with the Requests can also be classified as update or retrieval: clients, so interoperability became our first focus. Our goal was to design a way to easily integrate the recommender sys- • Retrieval requests just ask the system to return some tem while keeping recommender complexity hidden from its kind of information, such as a recommendation. clients, so we designed a REST API. REST APIs are easy to use and to integrate in any product and service on the web, • Update requests have the objective to update the pro- and many developers are familiar to it. Furthermore, REST file of the source user. API helps to hide the complexity of the recommender sys- tem, and allows to transform the REST petitions into more complex data structures needed to maintain the performance When an update request begins to be processed there are and scalability of the recommender system. two steps that must be taken to produce recommendations for the user. As we wanted to be able to process several To face configuration and performance issues we designed a types of recommendations (collaborative filtering, content back-end architecture that enables us to define recommen- based, social recommendations), the system had to be gen- dation algorithms as software modules that can be adapted eral enough to process data in several ways. So we defined to any domain required by clients. The REST API also al- those steps in a way that enabled the use of any possible lowed us to introduce configurability elements as optional recommender algorithm (see Fig. 2): parameters in the petition. • Update user profile. This can be done by re-calculating 3. GENERAL ARCHITECTURE simmilarity with other users, re-calculating trust or The input to our system are API requests, which can be updating a content-based profile. classified as online or batch: • Update user recommendations. This step uses the val- ues obtained from the previous task as input for the recommendation algorithm and produces a new rank • Online requests, which must be handled in real time. of recommendations for the user. Their processing can’t be delayed, because users are waiting for a response. They are also utilized to update the profile for new users and begin to provide them As a first approach we coded 3 different recommender algo- with recommendations. rithms: Figure 4: Relationships describes actions and be- Figure 2: Main modules of the recommender sys- haviors. tem. Figure 3: Entities describes every possible data used Figure 5: Integration of the recommendations in by the recommender algorithms. Wipley, our social network for videogamers. • Collaborative filtering. A standard method that pro- A client request is described by an HTTP request to a cer- duces collaborative recommendations when using a col- tain resource. For instance, for one of our domains, we may laborative similarity measure, or social recommenda- want videogames recommendations for a certain user (user tions when using trust. id=2). With our REST API, the client must make an HTTP • Information retrieval based recommender used for con- request to a certain URI5 . The first part of the URI de- tent based recommendations. scribes the endpoint where our API server is running6 . Next part shows that a recommendation (and the recommenda- • Machine learning based recommender also used for con- tion’s type) is requested (’/recomm’) for a certain entity type tent based recommendations. (’/player’) considering a relation of ownership with another entity type (’/has/videogame’). The user id is ’2’ and the response format is ’.xml’. The server must return an XML From the data point of view, the system only stores entities file containing the recommendations. and relationships among entities. Each possible data point (user, group, content or product), is represented by an en- Figure 5 shows our integration on Wipley, our social network tity (see Fig. 3). Relationships (see Fig. 4) among entities for videogamers, where XML responses are processed with represent actions (ratings, comments, reviews) or behaviors PHP in order to generate a more visual interface. (buy a product, pageview, joining a group). This data rep- resentation allows enough degree of data abstraction for the In Wipley we have configured our system to serve differ- interoperability, and it also stands near enough to data rep- ent types of recommendations for a user, such as products, resentation used by the different recommender algorithms. users and groups. They can be requested just modifying the URI. In order to update the information that the rec- 4. INTEROPERABILITY: A GENERAL API ommender system manages internally, we have designed a A Representational State Transfer (REST) API [2] is a style XML based specification, which allows to update informa- of software architecture for distributed systems like the Web tion about any entity (data points, see Fig. 6) or action where clients initiate requests and the servers process re- (relationships among entities, see Fig. 7). quests and return appropriate responses. Requests and re- sponses are built around the transfer of resources. REST is 5 http://webservices.brainsins.com/api/recomm/player/has/ described in the context of HTTP, although it can be used videogame/2.xml 6 in other contexts. http://webservices.brainsins.com/api There are no references in the recommender systems litera- ture describing a general purpose recommender system be- ing used to test several recommender algorithms within sev- eral domains with the aim to produce an extensive exper- imental comparison of recommender algorithms. We have developed a general recommender system which encapsu- lates several recommender algorithms (collaborative filter- ing, content based recommender, and social recommender) with the main purpose of producing an extensive compari- son of recommender algorithms in a real environment. We Figure 6: Example XML containing a description of have also integrated this general recommender system in a an entity (videogame). real social network that represents an experimental setup with real users in real conditions. 6. ACKNOWLEDGMENTS The research described in this paper has been partially sup- ported by the Madrid autonomous region, IV PRICIT, S- 0505/TIC/0267. 7. REFERENCES [1] O. Celma and P. Lamere. If you like the beatles you might like...: a tutorial on music recommendation. In Figure 7: Example XML containing a description of MM ’08: Proceeding of the 16th ACM international an action (user rating a videogame). conference on Multimedia, pages 1157–1158, New York, NY, USA, 2008. ACM. [2] R. T. Fielding. Architectural Styles and Architectural 5. FUTURE WORK Styles and the Design of Network-based Software We have developed this general purpose multi-algorithm rec- Architectures. PhD thesis, University of California, ommender system and we have integrated it into Wipley Irvine, 2000. with great results, given that videogames recommendations [3] J. Golbeck. Tutorial on using social trust for are considered by our users as one of the top features. Our recommender systems. In RecSys ’09: Proceedings of future work is defined in several lines: the third ACM conference on Recommender systems, pages 425–426, New York, NY, USA, 2009. ACM. [4] I. Guy, N. Zwerdling, D. Carmel, I. Ronen, E. Uziel, • Test the performance of the general recommender sys- S. Yogev, and S. Ofek-Koifman. Personalized tem in terms of running time and disk usage. Our first recommendation of social software items based on impressions make us think that generalizing the rec- social relations. In RecSys ’09: Proceedings of the third ommender system does not introduces computational ACM conference on Recommender systems, pages overhead, since processing the REST requests repre- 53–60, New York, NY, USA, 2009. ACM. sents only a 3-4% of the total running time of a rec- [5] J. F. McCarthy. The challenges of recommending digital ommendation, but we need to run a larger set of ex- selves in physical spaces. In RecSys ’07: Proceedings of periments to evaluate the final performance. the 2007 ACM conference on Recommender systems, pages 185–186, New York, NY, USA, 2007. ACM. • Scalability. As the REST API give us a lot of in- [6] O. Phelan, K. McCarthy, and B. Smyth. Using twitter dependence of the clients from the implementation, to recommend real-time topical news. In RecSys ’09: we are actually re-coding the recommender algorithms Proceedings of the third ACM conference on through a map-reduce perspective using Apache Ma- Recommender systems, pages 385–388, New York, NY, hout7 , which will allow the platform to be really scal- USA, 2009. ACM. able. [7] J. B. Schafer, J. Konstan, and J. Riedi. Recommender systems in e-commerce. In EC ’99: Proceedings of the • Experimental platform. One of our main goals is to 1st ACM conference on Electronic commerce, pages obtain an experimental platform to test the perfor- 158–166, New York, NY, USA, 1999. ACM. mance of our recommender systems implementations. [8] Z. Zhang and O. Nasraoui. Mining search engine query We are starting to measure several metrics related to logs for query recommendation. In L. Carr, D. D. web analytics such as CTR. We expect to obtain real Roure, A. Iyengar, C. A. Goble, and M. Dahlin, editors, feedback about what recommender algorithms and set- WWW, pages 1039–1040. ACM, 2006. tings works better for different domains. We are also developing a web based backend which will allow us to define the experiments and measure several aspects related to the effectivity of the recommendations. 7 http://lucene.apache.org/mahout/