=Paper=
{{Paper
|id=Vol-2567/paper12
|storemode=property
|title=A Novel Interface for the Explanation of Group Recommendations using Augmented Reality
|pdfUrl=https://ceur-ws.org/Vol-2567/paper12.pdf
|volume=Vol-2567
|authors=Juan A. Recio-Garcia,Guillermo Jimenez-Diaz
|dblpUrl=https://dblp.org/rec/conf/iccbr/Recio-GarciaJ19
}}
==A Novel Interface for the Explanation of Group Recommendations using Augmented Reality==
https://ceur-ws.org/Vol-2567/paper12.pdf
A Novel Interface for the Explanation of Group
Recommendations using Augmented Reality ?
Juan A. Recio-Garcia and Guillermo Jimenez-Diaz
Department of Software Engineering and Artificial Intelligence
Universidad Complutense de Madrid, Spain
email: jareciog@ucm.es, gjimenez@ucm.es
Abstract. This paper describes our novel approach that applies Aug-
mented Reality (AR) for the explanation of recommendations in the
movie domain. Our goal is to use augmented reality in order to explain
the recommendation of a movie either to a single user or a group of
friends that are creating a joint plan. The presented system reuses our
past results in group recommendation and AR to include explanation
capabilities in a mobile scenario where users can receive situated recom-
mendations (in a bus stop or in front of a movie marquee), and to join
friends in a plan through facial recognition.
1 Introduction
Augmented Reality (AR) is the combination of real and virtual imagery, interac-
tive in real time [3]. A few years ago the requirements to use AR were expensive.
However, the situation is completely different thanks to the current mobile de-
vices, which provide all the elements needed for an AR experience: a screen to
display virtual elements over the reality; a digital camera that takes information
about the real world; a powerful processor to run the AR software; and other
features like GPS, gyroscopes, compasses, or optic sensors, among others.
However, the combination of AR interfaces and Recommender Systems is
uncommon in research literature. The work in [2] describes an application for
restaurant recommendations that employs AR for localizing and placing recom-
mended restaurants around the user. Another mobile recommender in tourism
domain uses AR to show physical reproductions of points of interest using the
user location [6]. Recently, a study assesses the influence of the AR interfaces in
the user perception of the recommendation system [7]. It reveals that the ratings
and user trust in the recommender system are improved using more innovative
interfaces like AR or interactive 3D.
In this paper we present a novel approach that employs AR for the explana-
tion of recommendations in the movie domain. Initially, existing recommenders
?
Supported by the Complutense University of Madrid (Group 910494) and Spanish
Committee of Economy and Competitiveness (TIN2017-87330-R) and the funding
provided by Banco Santander in UCM (CT42/18-CT43/18).
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
were focused on individual users [5, 10]. But nowadays, the rise of the collabora-
tive Web (a.k.a. Web 2.0) has promoted the organization of different activities
performed by groups of people, like watching a movie, going to a restaurant,
listening to a radio station or traveling with friends. To address this issue, the
number of recommender systems that deal with the challenge of making rec-
ommendations for groups of people has increased [15, 14]. Our goal is to use
augmented reality in order to explain the recommendation of a movie either to
a single user or a group of friends.
This paper runs as follows. Next Section presents the background and related
work. Section 3 describes our system and the use of AR in order to explain
recommendations. Finally, Section 4 concludes the paper.
2 Background
The main contribution of this work is the combination of group recommenda-
tion strategies and the use of AR in order to explain such recommendations.
Therefore, next we introduce both areas.
2.1 Group recommendation methods
In our previous works [17, 18], we have proposed a group recommendation method
based on preference aggregation approaches. These approaches [13, 15] aggregate
the individual ratings predicted for every user u given an item i -denoted as
pred(u, i)- to obtain an prediction for the group:
G
gpred(G, i) = pred(u, i) (1)
∀u∈G
Here G is a group of users, which user u belongs to, and pred(u, i) is the in-
dividual prediction for user u and item i returned by the individual estimation
module. There are several aggregation functions –represented with the t symbol–
that can be chosen to obtain the group prediction. These functions provide an
aggregated value that predicts the group preference for a given item i. Then,
our group recommender proposes the k items with the highest estimated group
scoring.
A wide set of aggregation functions has been devised to combine individ-
ual preferences. Choosing the aggregation function that performs best is a key
element for providing good group recommendations. Here we explain the func-
tions that we have previously studied and can be used by the FilmAR group
recommendation engine:
– Average Satisfaction: Refers to the common arithmetic mean, which is a
method to derive the central tendency of a sample space.
– Borda Count: The Borda count is a single-winner election method in which
users rank candidates in order of preference. The Borda count determines
the winner of an election by giving each candidate a certain number of points
Fig. 1: Examples of the application of AR in TV news.
corresponding to the position in which she is ranked by each voter. Once all
votes have been counted the candidate with the most points is the winner.
– Copeland Rule: Alternatives are ordered by the number of pairwise victo-
ries, minus the number of pairwise defeats.
– Approval Voting: This is a single-winner voting system used for elections.
Each voter may vote for (approve of) as many of the candidates as they
wish. The winner is the candidate receiving the most votes.
– Least Misery: This strategy follows the idea that, even if average satisfac-
tion is high, a solution that leaves one or more members very dissatisfied is
likely to be considered undesirable. This strategy considers that a group is
as happy as its least happy member. The final list of ratings is the minimum
of each of the individual ratings.
– Most Pleasure Strategy: It is the opposite of the previous strategy, Least
Misery; it chooses the highest rating for each item to form the final list of
predicted ratings.
– Average Without Misery: Assigns a preference to the average of the
weights in the individual ratings. The difference here is that those items
that have predicted ratings under a certain threshold will not be considered.
2.2 Augmented Reality
AR systems are employed for multiple purposes, like marketing and advertising,
education, medicine, or entertainment, among others [3]. Its becoming an popu-
lar technology with a very high interest from the point of view of the consumers.
This way, nowadays there are many examples of the use of AR and it is being
introduced into our routine from several media, as shown in Figure 1.
Our previous interest in AR interfaces focused on the cultural heritage field
and its use in museums [4]. Applying AR techonology to museum contents is an
innovative way to attract young people to these spaces. It increases the amount of
contents and information that the museum provides to visitors without modify-
ing it. AR also improves the user experience both inside and outside the museum.
Finally, AR provides interactivity to the visitor activities: tourists are engaged
with the museum contents, adding a new value to the cultural heritage [1].
Some AR methods require the existence of points of interest to locate the
virtual elements in the real world. Points of interest can be defined as markers
presented in the real world like as pictures, QR or BIDI codes, text, simple 3D
objects (cubes or cylinders) or even complex 3D objects with an unknown geom-
etry. Points of interest are employed as a reference to reconstruct a coordinate
system for positioning the virtual objects in the real world. This reference will
be updated with the instruments that recognize movement (such as compasses
and gyroscopes) if the AR systems is running on a mobile device.
Regarding the technological solutions for the integration of AR, nowadays
we can found several alternatives. ARCore, is a solution provided by Google
that supports several platforms and mobile devices [8]. ARKit is the proposal
by Apple but it is only compatible with iOS devices [12]. Another option is
Wikitude [11], although it is mostly focused on geolocalization. Finally, one of
the most popular solutions is Vuforia [9]. It supports several platforms, and its
development process is simple thanks to the use of markers. This is the option
chosen to implement our system as explained in the following section.
3 FilmAR
FilmAR is a mobile application for the management of plans to watch a movie
with friends or people who have a similar tastes. The recommender engine inside
FilmAR uses a collaborative filtering algorithm implemented with Mahout [16],
using the movie ratings provided by IMDB1 .
One of the main challenges when developing this kind of applications is the
integration of the recommender engine and the AR capabilities. The choice of
Vuforia for the AR capabilities allow us to employ its cloud storage in order
to recognize the movie posters. This way, the server side of the application is
responsible for the Mahout recommender system and the communication with
Vuforia cloud recognition. Then, the client side combines Vuforia and the native
Android SDK. This architecture is described in Figure 2.
Thanks to augmented reality, we will be able to recognize movie posters
with our camera device, and it even allows facial recognition, so we can obtain
explanations about what we are recognizing in real time and so be able to interact
with it through augmented reality.
According to its AR capabilities, FilmAR provides two main functionalities:
the explanation of a movie recommendation and the explanation of a recom-
mended plan with friends. Next, we detail both features.
3.1 AR interface for movie recommendation
Most of movie platforms like Netflix or IMDB provide users with personalized
recommendations in a proactive way, displaying a list of recommended movies
when the user logs in the platform. However, when the user wants to know
whether a concrete film would fit her tastes, only some platforms like Movielens
provide searching capabilities and recommendation functionalities to predict the
1
https://www.imdb.com
Fig. 2: Global architecture of the FilmAR system
user rating for this movie. Using Augmented Reality, FilmAR supports situated
recommendations (in a bus stop or in front of a movie marquee), where the user
could check if a movie could be interesting for her.
If a user focuses the mobile camera on a movie poster, FilmAR recognizes
the film and displays the interface shown in Figure 3. The information displayed
over the poster should be enough for the user to decide whether the film fits her
tastes:
– The rating predicted by the recommender system is displayed on top.
– The YouTube icon in the middle of the poster is a link to watch the movie
trailer.
– The icon on the bottom right provides access to a movie excerpt (extracted
from IMDB) and a link to save this movie in our favorites, in order to
consume it in a future plan.
Once a movie is saved in favourites, the user can create a future plan to
watch that movie in the cinema with friends and rate the movie after that.
Here rating is an important step as the system obtains the feedback required
to improve future recommendations. In order to provide this rating we use a
visual gauge representation. Gauge diagrams display the value of a single mea-
sure in a simple way. A gauge is easy to read and understand and, in the case
of movie rating, gives an instant feedback of the rating. We have implemented
a dynamic gauge visualization that changes color according to the rating given
by the user, from red to green, as presented in Figure 4
We have chosen this visualization, not only to rate movies but also to pro-
vide an instant indication easy to understand when we have to provide a group
recommendation that combines the ratings predicted for several users that will
share a plan. Next, we will explain this case of use.
Fig. 3: AR interface for a movie
3.2 AR interface for recommending a plan
Any user in FilmAR can create many plans and then invite other users to join
in the plan. Plans can be created using AR from the movie poster as explained
in the previous subsection. Then, they are stored in the application as shown in
Figure 5. Initially, the only member of the plan is the creator, that must invite
users to the plan.
A traditional interface should provide a browsing functionality to look for
a user and navigate over a list of plans, searching an interesting movie and
inspecting which other people has joined the plan. However, this task can be
done in a more straightforward way using augmented reality. Through the use
of AR we can enhance this process and help users in the process of choosing the
best plan according to the current members, and, more importantly, to explain
the proposed plan.
When the user who wants to join a plan –joining user– focuses the mobile
camera on a friend –organizer user– FilmAR recognizes that user and the rec-
ommender system looks for the plans that best fit the tastes of the target and
any other friends that previously joined the plan. In this sense, the recommen-
dation system predicts a group rating of the movies contained in the plans of
the organizer and selects the best rated ones. This prediction follows the method
presented in Section 2.1.
The recommended plans are displayed using AR over the organizer using the
interface depicted in Figure 6.
Fig. 4: Visual ratings: a gauge on the bottom right corner represents a use rating
The global predicted rating for the group is used to sort the movie posters
being presented. The movie with the highest predicted rating for the group is
presented in the middle, whereas the second and third are positioned on the left
and right side, respectively. Then, the interface shows over each movie poster all
the information needed by the joining user to choose the best plan among the
ones associated to the organizer:
– A photo of the joining user appears on the top left corner and it is surrounded
by a gauge that represents her individual predicted rating for this movie.
– The organizer appears on the top right corner. This photo is surrounded by
a gauge that represents the individual predicted rating of the organizer user
for this movie.
– Below, the interface shows a photo of every user that is enrolled in this plan.
As in the previous ones, each photo is surrounded by a gauge that represents
the rating predicted by the recommender system by each user. Due to space
limitations, the interface only displays up to six additional users enrolled in
the plan.
The joining user can navigate over the recommended plans in order to find
the most suitable for the group and easily join it from the own AR interface
(using the Join button).
Fig. 5: Example of a list of plans created by the user.
4 Conclusions and Future Work
FilmAR is a mobile application for supporting the creation of plans for group
of friends who want to go together to watch a movie. The application assists
the users in choosing the plan that best fits her tastes using a recommendation
engine that provides both individual predictions and group recommendations. In
this paper we have depicted the novel interfaces that employ Augmented Reality
to explain the results provided by the recommender system.
Despite the novelty of this interface, we believe that these interfaces can en-
hance user experience and trust in recommender systems. During the application
design, several users participated in informal evaluation sessions with interface
prototypes, that led us in the development of understandable and easy to use AR
interfaces. However, a more formal user evaluation must be performed, in the
same way proposed in [7], in order to validate its effect on the user perception
of the recommendation system.
Additionally, face recognition was prototyped in FilmAR using Augmented
Reality technology over user photos. It is not easy to integrate both intelligent
face recognition and AR techniques due to the high computing needs of both
technologies. Despite the controversial issues that face recognition implies, we
think that it could be interesting to continue the research on its use in the
explanation of personalized recommendations.
Fig. 6: AR interface for a common plan
Acknowledgement.
FilmAR is an application developed by the following students as their final degree
project under the supervision of the authors of this paper: Diego Acuña Berger,
Daniel Calle Sánchez, Carlos Gómez Cereceda and Zihao Hong.
References
1. Anastassia Angelopoulou, Daphne Economou, Vassiliki Bouki, Alexandra Psar-
rou, Li Jin, Chris Pritchard, and Frantzeska Kolyda. Mobile augmented reality for
cultural heritage. In Mobile Wireless Middleware, Operating Systems, and Appli-
cations, pages 15–22. Springer, 2012.
2. Marco Balduini, Irene Celino, Daniele Dell’Aglio, Emanuele Della Valle, Yi Huang,
Tony Lee, Seon-Ho Kim, and Volker Tresp. Bottari: An augmented reality mobile
application to deliver personalized and location-based recommendations by con-
tinuous analysis of social media streams. Journal of Web Semantics, 16:33 – 41,
2012. The Semantic Web Challenge 2011.
3. Julie Carmigniani, Borko Furht, Marco Anisetti, Paolo Ceravolo, Ernesto Dami-
ani, and Misa Ivkovic. Augmented reality technologies, systems and applications.
Multimedia Tools and Applications, 51(1):341–377, 2011.
4. Marta Caro-Martı́nez, David Hernando-Hernández, and Guillermo Jiménez-Dı́az.
Racma o cómo dar vida a un mapa mudo en el museo de américa. In CoSECivi,
pages 80–89, 2015.
5. Michael D. Ekstrand, John Riedl, and Joseph A. Konstan. Collaborative filtering
recommender systems. Foundations and Trends in Human-Computer Interaction,
4(2):175–243, 2011.
6. Damianos Gavalas, Charalampos Konstantopoulos, Konstantinos Mastakas, and
Grammati Pantziou. Mobile recommender systems in tourism. Journal of network
and computer applications, 39:319–333, 2014.
7. Brandon Huynh, Adam Ibrahim, Yun Suk Chang, Tobias Höllerer, and John
O’Donovan. A study of situated product recommendations in augmented real-
ity. In 2018 IEEE International Conference on Artificial Intelligence and Virtual
Reality (AIVR), pages 35–43. IEEE, 2018.
8. Micheal Lanham. Learn ARCore-Fundamentals of Google ARCore: Learn to build
augmented reality apps for Android, Unity, and the web with Google ARCore 1.0.
Packt Publishing Ltd, 2018.
9. Jonathan Linowes and Krystian Babilinski. Augmented Reality for Developers:
Build practical augmented reality applications with Unity, ARCore, ARKit, and
Vuforia. Packt Publishing Ltd, 2017.
10. Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Content-based recom-
mender systems: State of the art and trends. In Recommender Systems Handbook,
pages 73–105. 2011.
11. Lester Madden. Professional augmented reality browsers for smartphones: pro-
gramming for junaio, layar and wikitude. John Wiley & Sons, 2011.
12. Molly Maskrey and Wallace Wang. Understanding arkit. In Pro iPhone Develop-
ment with Swift 4, pages 389–418. Springer, 2018.
13. Judith Masthoff and Albert Gatt. In pursuit of satisfaction and the prevention of
embarrassment: affective state in group recommender systems. User Modeling and
User-Adapted Interaction, 16(3-4):281–319, 2006.
14. Joseph F. McCarthy and Theodore D. Anagnost. MusicFX: An arbiter of group
preferences for computer aupported collaborative workouts. In CSCW ’98: Pro-
ceedings of the 1998 ACM conference on Computer supported cooperative work,
pages 363–372. ACM, 1998.
15. Mark O’Connor, Dan Cosley, Joseph A. Konstan, and John Riedl. Polylens: a
recommender system for groups of users. In ECSCW’01: Proceedings of the seventh
conference on European Conference on Computer Supported Cooperative Work,
pages 199–218, Norwell, MA, USA, 2001. Kluwer Academic Publishers.
16. Sean Owen, Robin Anil, Ted Dunning, and Ellen Friedman. Mahout in Action.
Manning Publications Co., 2011.
17. Lara Quijano-Sanchez, Derek G Bridge, Belen Diaz-Agudo, and Juan A Recio-
Garcia. Case-based aggregation of preferences for group recommenders. In ICCBR,
volume 7466 of Lecture Notes in Computer Science, pages 327–341. Springer, 2012.
18. Lara Quijano-Sanchez, Juan A Recio-Garcia, Belen Diaz-Agudo, and Guillermo
Jimenez-Diaz. Happy movie: A group recommender application in facebook. In
FLAIRS Conference, pages 419–420. AAAI Press, 2011.