A Novel Interface for the Explanation of Group Recommendations using Augmented Reality ? Juan A. Recio-Garcia and Guillermo Jimenez-Diaz Department of Software Engineering and Artificial Intelligence Universidad Complutense de Madrid, Spain email: jareciog@ucm.es, gjimenez@ucm.es Abstract. This paper describes our novel approach that applies Aug- mented Reality (AR) for the explanation of recommendations in the movie domain. Our goal is to use augmented reality in order to explain the recommendation of a movie either to a single user or a group of friends that are creating a joint plan. The presented system reuses our past results in group recommendation and AR to include explanation capabilities in a mobile scenario where users can receive situated recom- mendations (in a bus stop or in front of a movie marquee), and to join friends in a plan through facial recognition. 1 Introduction Augmented Reality (AR) is the combination of real and virtual imagery, interac- tive in real time [3]. A few years ago the requirements to use AR were expensive. However, the situation is completely different thanks to the current mobile de- vices, which provide all the elements needed for an AR experience: a screen to display virtual elements over the reality; a digital camera that takes information about the real world; a powerful processor to run the AR software; and other features like GPS, gyroscopes, compasses, or optic sensors, among others. However, the combination of AR interfaces and Recommender Systems is uncommon in research literature. The work in [2] describes an application for restaurant recommendations that employs AR for localizing and placing recom- mended restaurants around the user. Another mobile recommender in tourism domain uses AR to show physical reproductions of points of interest using the user location [6]. Recently, a study assesses the influence of the AR interfaces in the user perception of the recommendation system [7]. It reveals that the ratings and user trust in the recommender system are improved using more innovative interfaces like AR or interactive 3D. In this paper we present a novel approach that employs AR for the explana- tion of recommendations in the movie domain. Initially, existing recommenders ? Supported by the Complutense University of Madrid (Group 910494) and Spanish Committee of Economy and Competitiveness (TIN2017-87330-R) and the funding provided by Banco Santander in UCM (CT42/18-CT43/18). Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). were focused on individual users [5, 10]. But nowadays, the rise of the collabora- tive Web (a.k.a. Web 2.0) has promoted the organization of different activities performed by groups of people, like watching a movie, going to a restaurant, listening to a radio station or traveling with friends. To address this issue, the number of recommender systems that deal with the challenge of making rec- ommendations for groups of people has increased [15, 14]. Our goal is to use augmented reality in order to explain the recommendation of a movie either to a single user or a group of friends. This paper runs as follows. Next Section presents the background and related work. Section 3 describes our system and the use of AR in order to explain recommendations. Finally, Section 4 concludes the paper. 2 Background The main contribution of this work is the combination of group recommenda- tion strategies and the use of AR in order to explain such recommendations. Therefore, next we introduce both areas. 2.1 Group recommendation methods In our previous works [17, 18], we have proposed a group recommendation method based on preference aggregation approaches. These approaches [13, 15] aggregate the individual ratings predicted for every user u given an item i -denoted as pred(u, i)- to obtain an prediction for the group: G gpred(G, i) = pred(u, i) (1) ∀u∈G Here G is a group of users, which user u belongs to, and pred(u, i) is the in- dividual prediction for user u and item i returned by the individual estimation module. There are several aggregation functions –represented with the t symbol– that can be chosen to obtain the group prediction. These functions provide an aggregated value that predicts the group preference for a given item i. Then, our group recommender proposes the k items with the highest estimated group scoring. A wide set of aggregation functions has been devised to combine individ- ual preferences. Choosing the aggregation function that performs best is a key element for providing good group recommendations. Here we explain the func- tions that we have previously studied and can be used by the FilmAR group recommendation engine: – Average Satisfaction: Refers to the common arithmetic mean, which is a method to derive the central tendency of a sample space. – Borda Count: The Borda count is a single-winner election method in which users rank candidates in order of preference. The Borda count determines the winner of an election by giving each candidate a certain number of points Fig. 1: Examples of the application of AR in TV news. corresponding to the position in which she is ranked by each voter. Once all votes have been counted the candidate with the most points is the winner. – Copeland Rule: Alternatives are ordered by the number of pairwise victo- ries, minus the number of pairwise defeats. – Approval Voting: This is a single-winner voting system used for elections. Each voter may vote for (approve of) as many of the candidates as they wish. The winner is the candidate receiving the most votes. – Least Misery: This strategy follows the idea that, even if average satisfac- tion is high, a solution that leaves one or more members very dissatisfied is likely to be considered undesirable. This strategy considers that a group is as happy as its least happy member. The final list of ratings is the minimum of each of the individual ratings. – Most Pleasure Strategy: It is the opposite of the previous strategy, Least Misery; it chooses the highest rating for each item to form the final list of predicted ratings. – Average Without Misery: Assigns a preference to the average of the weights in the individual ratings. The difference here is that those items that have predicted ratings under a certain threshold will not be considered. 2.2 Augmented Reality AR systems are employed for multiple purposes, like marketing and advertising, education, medicine, or entertainment, among others [3]. Its becoming an popu- lar technology with a very high interest from the point of view of the consumers. This way, nowadays there are many examples of the use of AR and it is being introduced into our routine from several media, as shown in Figure 1. Our previous interest in AR interfaces focused on the cultural heritage field and its use in museums [4]. Applying AR techonology to museum contents is an innovative way to attract young people to these spaces. It increases the amount of contents and information that the museum provides to visitors without modify- ing it. AR also improves the user experience both inside and outside the museum. Finally, AR provides interactivity to the visitor activities: tourists are engaged with the museum contents, adding a new value to the cultural heritage [1]. Some AR methods require the existence of points of interest to locate the virtual elements in the real world. Points of interest can be defined as markers presented in the real world like as pictures, QR or BIDI codes, text, simple 3D objects (cubes or cylinders) or even complex 3D objects with an unknown geom- etry. Points of interest are employed as a reference to reconstruct a coordinate system for positioning the virtual objects in the real world. This reference will be updated with the instruments that recognize movement (such as compasses and gyroscopes) if the AR systems is running on a mobile device. Regarding the technological solutions for the integration of AR, nowadays we can found several alternatives. ARCore, is a solution provided by Google that supports several platforms and mobile devices [8]. ARKit is the proposal by Apple but it is only compatible with iOS devices [12]. Another option is Wikitude [11], although it is mostly focused on geolocalization. Finally, one of the most popular solutions is Vuforia [9]. It supports several platforms, and its development process is simple thanks to the use of markers. This is the option chosen to implement our system as explained in the following section. 3 FilmAR FilmAR is a mobile application for the management of plans to watch a movie with friends or people who have a similar tastes. The recommender engine inside FilmAR uses a collaborative filtering algorithm implemented with Mahout [16], using the movie ratings provided by IMDB1 . One of the main challenges when developing this kind of applications is the integration of the recommender engine and the AR capabilities. The choice of Vuforia for the AR capabilities allow us to employ its cloud storage in order to recognize the movie posters. This way, the server side of the application is responsible for the Mahout recommender system and the communication with Vuforia cloud recognition. Then, the client side combines Vuforia and the native Android SDK. This architecture is described in Figure 2. Thanks to augmented reality, we will be able to recognize movie posters with our camera device, and it even allows facial recognition, so we can obtain explanations about what we are recognizing in real time and so be able to interact with it through augmented reality. According to its AR capabilities, FilmAR provides two main functionalities: the explanation of a movie recommendation and the explanation of a recom- mended plan with friends. Next, we detail both features. 3.1 AR interface for movie recommendation Most of movie platforms like Netflix or IMDB provide users with personalized recommendations in a proactive way, displaying a list of recommended movies when the user logs in the platform. However, when the user wants to know whether a concrete film would fit her tastes, only some platforms like Movielens provide searching capabilities and recommendation functionalities to predict the 1 https://www.imdb.com Fig. 2: Global architecture of the FilmAR system user rating for this movie. Using Augmented Reality, FilmAR supports situated recommendations (in a bus stop or in front of a movie marquee), where the user could check if a movie could be interesting for her. If a user focuses the mobile camera on a movie poster, FilmAR recognizes the film and displays the interface shown in Figure 3. The information displayed over the poster should be enough for the user to decide whether the film fits her tastes: – The rating predicted by the recommender system is displayed on top. – The YouTube icon in the middle of the poster is a link to watch the movie trailer. – The icon on the bottom right provides access to a movie excerpt (extracted from IMDB) and a link to save this movie in our favorites, in order to consume it in a future plan. Once a movie is saved in favourites, the user can create a future plan to watch that movie in the cinema with friends and rate the movie after that. Here rating is an important step as the system obtains the feedback required to improve future recommendations. In order to provide this rating we use a visual gauge representation. Gauge diagrams display the value of a single mea- sure in a simple way. A gauge is easy to read and understand and, in the case of movie rating, gives an instant feedback of the rating. We have implemented a dynamic gauge visualization that changes color according to the rating given by the user, from red to green, as presented in Figure 4 We have chosen this visualization, not only to rate movies but also to pro- vide an instant indication easy to understand when we have to provide a group recommendation that combines the ratings predicted for several users that will share a plan. Next, we will explain this case of use. Fig. 3: AR interface for a movie 3.2 AR interface for recommending a plan Any user in FilmAR can create many plans and then invite other users to join in the plan. Plans can be created using AR from the movie poster as explained in the previous subsection. Then, they are stored in the application as shown in Figure 5. Initially, the only member of the plan is the creator, that must invite users to the plan. A traditional interface should provide a browsing functionality to look for a user and navigate over a list of plans, searching an interesting movie and inspecting which other people has joined the plan. However, this task can be done in a more straightforward way using augmented reality. Through the use of AR we can enhance this process and help users in the process of choosing the best plan according to the current members, and, more importantly, to explain the proposed plan. When the user who wants to join a plan –joining user– focuses the mobile camera on a friend –organizer user– FilmAR recognizes that user and the rec- ommender system looks for the plans that best fit the tastes of the target and any other friends that previously joined the plan. In this sense, the recommen- dation system predicts a group rating of the movies contained in the plans of the organizer and selects the best rated ones. This prediction follows the method presented in Section 2.1. The recommended plans are displayed using AR over the organizer using the interface depicted in Figure 6. Fig. 4: Visual ratings: a gauge on the bottom right corner represents a use rating The global predicted rating for the group is used to sort the movie posters being presented. The movie with the highest predicted rating for the group is presented in the middle, whereas the second and third are positioned on the left and right side, respectively. Then, the interface shows over each movie poster all the information needed by the joining user to choose the best plan among the ones associated to the organizer: – A photo of the joining user appears on the top left corner and it is surrounded by a gauge that represents her individual predicted rating for this movie. – The organizer appears on the top right corner. This photo is surrounded by a gauge that represents the individual predicted rating of the organizer user for this movie. – Below, the interface shows a photo of every user that is enrolled in this plan. As in the previous ones, each photo is surrounded by a gauge that represents the rating predicted by the recommender system by each user. Due to space limitations, the interface only displays up to six additional users enrolled in the plan. The joining user can navigate over the recommended plans in order to find the most suitable for the group and easily join it from the own AR interface (using the Join button). Fig. 5: Example of a list of plans created by the user. 4 Conclusions and Future Work FilmAR is a mobile application for supporting the creation of plans for group of friends who want to go together to watch a movie. The application assists the users in choosing the plan that best fits her tastes using a recommendation engine that provides both individual predictions and group recommendations. In this paper we have depicted the novel interfaces that employ Augmented Reality to explain the results provided by the recommender system. Despite the novelty of this interface, we believe that these interfaces can en- hance user experience and trust in recommender systems. During the application design, several users participated in informal evaluation sessions with interface prototypes, that led us in the development of understandable and easy to use AR interfaces. However, a more formal user evaluation must be performed, in the same way proposed in [7], in order to validate its effect on the user perception of the recommendation system. Additionally, face recognition was prototyped in FilmAR using Augmented Reality technology over user photos. It is not easy to integrate both intelligent face recognition and AR techniques due to the high computing needs of both technologies. Despite the controversial issues that face recognition implies, we think that it could be interesting to continue the research on its use in the explanation of personalized recommendations. Fig. 6: AR interface for a common plan Acknowledgement. FilmAR is an application developed by the following students as their final degree project under the supervision of the authors of this paper: Diego Acuña Berger, Daniel Calle Sánchez, Carlos Gómez Cereceda and Zihao Hong. References 1. Anastassia Angelopoulou, Daphne Economou, Vassiliki Bouki, Alexandra Psar- rou, Li Jin, Chris Pritchard, and Frantzeska Kolyda. Mobile augmented reality for cultural heritage. In Mobile Wireless Middleware, Operating Systems, and Appli- cations, pages 15–22. Springer, 2012. 2. Marco Balduini, Irene Celino, Daniele Dell’Aglio, Emanuele Della Valle, Yi Huang, Tony Lee, Seon-Ho Kim, and Volker Tresp. Bottari: An augmented reality mobile application to deliver personalized and location-based recommendations by con- tinuous analysis of social media streams. Journal of Web Semantics, 16:33 – 41, 2012. The Semantic Web Challenge 2011. 3. Julie Carmigniani, Borko Furht, Marco Anisetti, Paolo Ceravolo, Ernesto Dami- ani, and Misa Ivkovic. Augmented reality technologies, systems and applications. Multimedia Tools and Applications, 51(1):341–377, 2011. 4. Marta Caro-Martı́nez, David Hernando-Hernández, and Guillermo Jiménez-Dı́az. Racma o cómo dar vida a un mapa mudo en el museo de américa. In CoSECivi, pages 80–89, 2015. 5. Michael D. Ekstrand, John Riedl, and Joseph A. Konstan. Collaborative filtering recommender systems. Foundations and Trends in Human-Computer Interaction, 4(2):175–243, 2011. 6. Damianos Gavalas, Charalampos Konstantopoulos, Konstantinos Mastakas, and Grammati Pantziou. Mobile recommender systems in tourism. Journal of network and computer applications, 39:319–333, 2014. 7. Brandon Huynh, Adam Ibrahim, Yun Suk Chang, Tobias Höllerer, and John O’Donovan. A study of situated product recommendations in augmented real- ity. In 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pages 35–43. IEEE, 2018. 8. Micheal Lanham. Learn ARCore-Fundamentals of Google ARCore: Learn to build augmented reality apps for Android, Unity, and the web with Google ARCore 1.0. Packt Publishing Ltd, 2018. 9. Jonathan Linowes and Krystian Babilinski. Augmented Reality for Developers: Build practical augmented reality applications with Unity, ARCore, ARKit, and Vuforia. Packt Publishing Ltd, 2017. 10. Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Content-based recom- mender systems: State of the art and trends. In Recommender Systems Handbook, pages 73–105. 2011. 11. Lester Madden. Professional augmented reality browsers for smartphones: pro- gramming for junaio, layar and wikitude. John Wiley & Sons, 2011. 12. Molly Maskrey and Wallace Wang. Understanding arkit. In Pro iPhone Develop- ment with Swift 4, pages 389–418. Springer, 2018. 13. Judith Masthoff and Albert Gatt. In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems. User Modeling and User-Adapted Interaction, 16(3-4):281–319, 2006. 14. Joseph F. McCarthy and Theodore D. Anagnost. MusicFX: An arbiter of group preferences for computer aupported collaborative workouts. In CSCW ’98: Pro- ceedings of the 1998 ACM conference on Computer supported cooperative work, pages 363–372. ACM, 1998. 15. Mark O’Connor, Dan Cosley, Joseph A. Konstan, and John Riedl. Polylens: a recommender system for groups of users. In ECSCW’01: Proceedings of the seventh conference on European Conference on Computer Supported Cooperative Work, pages 199–218, Norwell, MA, USA, 2001. Kluwer Academic Publishers. 16. Sean Owen, Robin Anil, Ted Dunning, and Ellen Friedman. Mahout in Action. Manning Publications Co., 2011. 17. Lara Quijano-Sanchez, Derek G Bridge, Belen Diaz-Agudo, and Juan A Recio- Garcia. Case-based aggregation of preferences for group recommenders. In ICCBR, volume 7466 of Lecture Notes in Computer Science, pages 327–341. Springer, 2012. 18. Lara Quijano-Sanchez, Juan A Recio-Garcia, Belen Diaz-Agudo, and Guillermo Jimenez-Diaz. Happy movie: A group recommender application in facebook. In FLAIRS Conference, pages 419–420. AAAI Press, 2011.