=Paper=
{{Paper
|id=Vol-1953/healthRecSys17_paper_7
|storemode=property
|title=PHARA: a Personal Health Augmented Reality Assistant to Support Decision-Making at Grocery Stores
|pdfUrl=https://ceur-ws.org/Vol-1953/healthRecSys17_paper_7.pdf
|volume=Vol-1953
|authors=Francisco Gutierrez,Bruno Cardoso,Katrien Verbert
|dblpUrl=https://dblp.org/rec/conf/recsys/GutierrezCV17
}}
==PHARA: a Personal Health Augmented Reality Assistant to Support Decision-Making at Grocery Stores==
https://ceur-ws.org/Vol-1953/healthRecSys17_paper_7.pdf
PHARA: a personal health augmented reality assistant to
support decision-making at grocery stores
Francisco Gutiérrez Bruno Cardoso Katrien Verbert
Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science
KU Leuven KU Leuven KU Leuven
Celestijnenlaan 200A Celestijnenlaan 200A Celestijnenlaan 200A
Heverlee, Belgium 3001 Heverlee, Belgium 3001 Heverlee, Belgium 3001
francisco.gutierrez@cs.kuleuven.be bruno.cardoso@cs.kuleuven.be katrien.verbert@cs.kuleuven.be
ABSTRACT to provide estimates on individuals’ basal metabolic rates and daily
Poor diet and physical inactivity are important factors that con- kilo-calorie requirements. Although we can find relevant litera-
tribute to the obesity outbreak. Therefore, healthy eating habits ture covering several aspects of food recommendation [1], there
are crucial for physical well-being. In this paper, we present the is not so much research on the HCI factors of the actual delivery
concept design and an early stage evaluation of PHARA, a personal of food recommendations to users. Holding to the principles of
health augmented reality assistant that recommends healthy and just-in-time knowledge management (JITKM)[3], we believe that
similar products to people in their everyday lives. We evaluated delivering the right information at the right context has a positive
a content-based recommender system in a desktop environment impact on people’s food buying decisions and, ultimately, will help
(n = 15) to measure the perceived quality, as well as behavioral them accomplish their health goals. Therefore, we are interested in
intentions of users. In addition, we evaluated the user interface providing information to people when it matters the most - in the
and measured participants’ perceptions of usefulness and ease of moment of decision: when they hold a product in their hands at
use. Whereas perceived usefulness and perceived ease of use are the grocery store. Grocery stores provide an interesting setting for
good, more work is required towards improving the accuracy and our research, as people there make many food buying decisions. To
diversity of recommendations. this end, nutrition labels may play a key role in promoting healthy
food choices [8], as improving people’s diet begins by improving
CCS CONCEPTS the nutritional quality of the food choices they make. On the other
hand, crowdsourced databases such as Open Food Facts1 provide a
•Information systems →Decision support systems; Recom-
big source of nutritional information for food products from around
mender systems; •Human-centered computing →Mixed / aug-
the world, opening a new spectrum of possibilities for relevant rec-
mented reality;
ommendations. We want to bring these concepts together along
with the potential that augmented reality (AR) technologies have to
KEYWORDS
offer while providing a context-aware, continuous AR experience
Decision Making, Recommender Systems, Augmented Reality [5].
ACM Reference format:
Francisco Gutiérrez, Bruno Cardoso, and Katrien Verbert. 2017. PHARA: a 2 RELATED WORK
personal health augmented reality assistant to support decision-making at
grocery stores. In Proceedings of the Second International Workshop on Recommender Systems. van Pinxteren, Youri et al.[12] implemented
Health Recommender Systems co-located with ACM RecSys 2017, Como, a similarity measure for recipes based on a collaborative filtering
Italy, August 2017 (RecSys’17), 4 pages. approach. The similarity measure can be used to promote new
recipes that fit into people’s lifestyle according to their preferences.
Shekar, Sangeetha et al.[10] propose a traditional content-based
implementation using a phone grocery shopping assistant that
1 INTRODUCTION recommends products based on the user profile and a database
Nowadays, recommender systems are increasingly used to support of products. Achananuparp, Palakorn, and Ingmar Weber [1] ex-
decision-making in health-related contexts, suggesting users how plored healthy food recommendations by finding food substitutes
to improve their eating, exercising or sleeping behavior. An ex- in similar contexts using a crowdsourced service.
ample is the use of recommendations as a basis to algorithmically Visualization. The work of C. Siawsolit et al. [11] illustrates the
derive balanced meal plans that meet nutritional guidelines for the use of a simple star-rating visualization to suggest healthy products
user [4]. These recommendations are generally based on personal and how a nutritional-based recommender system may be useful
profiles that include gender, height, weight and physical activity, to people who are motivated to eat healthy but have no time to
International Workshop on Health Recommender Systems, August 2017, Como, Italy. ©
compare products. Many alternatives rely on a simplified nutrition
2017. Copyright for the individual papers remains with the authors. Copying permitted labelling system to help consumers make healthier food choices.
for private and academic purposes. This volume is published and copyrighted by its A prominent example is the five-colour nutrition label/nutri-score
editors.
1 https://world.openfoodfacts.org/
HealthRecSys’17, August 2017, Como, Italy Francisco Gutiérrez, Bruno Cardoso, and Katrien Verbert
User Interface. The interface of PHARA is inspired by the work
of Heun, Valentin et al [6] in smarter objects, and we also consid-
ered following Matthew Kay[7] suggestions for personal health
visualization design. In Figure 2 we present an early digital pro-
totype of our interface that uses an AR card component layout to
show: a) a visual component of Similar Products recommendations;
b) an Impact on Health component that gives a prediction of the
impact of the product on user’s health; and c) an overview visual-
ization of the product that shows a description of the nutrients in
an intuitive visualization.
Figure 1: Architecture of PHARA that illustrates the differ-
4 USER STUDY DESIGN
ent components of the system and the communication be-
tween server and clients. In this section, we present the design of our user studies. Results
are discussed in the next section. We recruited 15 participants (3F,
12M; mean age: 27.6, SD: 5.88) via word-of-mouth. We asked them
to participate in two studies to collect their thoughts towards the
based on the British Food Standards Agency Nutrient Profiling Sys- recommender system, and an early paper prototype to understand
tem (FSA-NPS), which is an score calculated for each 100g amount how PHARA can be used in real world settings. A short offline eval-
of energy, sugar, saturated fatty acid, sodium, fibres, proteins, and uation was conducted afterwards to test recommendation accuracy
fruits and vegetables. and diversity.
Augmented Reality. The most comparable system to PHARA is Study 1: Recommender System. We built our system using
the work of Ahn, Junho et al.[2] who developed a handheld AR sys- a crowdsourced database from Open Food Facts, using a content-
tem that uses color-based AR tagging to support people in finding based approach and a similarity index to estimate similarities be-
healthy food products in supermarkets. Their application uses an tween food products. We generated recommendations based on
in-store navigation system to guide participants to the products similarities of products to provide recommendations about Similar
and shows a single color-based label to highlight information over Products, Healthy Products and Based on your Profile. Participants
the product. While this system may be the most similar to PHARA, were presented with a Web browser application where they were re-
we are interested in the usability of immerse scenarios in which quired to create a profile with their personal data (allergies, height,
users have both hands free to interact and get information from the weight, age and activity level). Afterwards, they were asked to
system, in a just-in-time fashion. On the other hand, in contrast select 10 favorite products to train the system. After creating a
with the work of Ahn, Junho et al, we are interested in more deep profile, a dashboard was shown (see Figure 3) were they could see
recommendations related to the context of both the product and the a list of recommended products. The task was as follows: Select any
user. To this end, item-item content-based recommenders have the products that you would like to have for dinner.
potential to leverage personal data while letting users explore the The system showed a visualization of ”MyPlate”, with suggested
relationships between food products and health-goal achievement. indications for a balanced meal, based on Healthy Eating Plate
created by experts at Harvard School of Public Health and Harvard
3 SYSTEM DESIGN
The design of PHARA is inspired by related work, JITKM goals, per-
sonal health tracking challenges and the opportunities presented by
recent advances in AR head-mounted display (HMD) technologies,
such as Microsoft Hololens2 . In this section, we describe the system
architecture and the design of PHARA’s user interface.
Architecture. The architecture of PHARA is shown in Figure 1.
We envision an immersive system where users can wear the HMD
freely and execute tasks using input such as voice commands or
hand gestures. With a reactive design3 in mind, data is streamed on
demand from the servers to ensure system responsiveness. Com-
puting of recommendations and predictions take place in the server,
releasing the HMD from any other computation other than tracking
and recognizing printed labels and barcodes to identify products. A
Web Application also is served in a separate client, through which
users can introduce and synchronize their devices with personal
data.
Figure 2: Early digital prototype that illustrates the visual
2 https://www.microsoft.com/en-us/hololens
card components of the application a) Recommendations, b)
3 http://www.reactivemanifesto.org/ Health Impact Prediction, c) Product Information.
PHARA: A personal health augmented reality assistant to support HealthRecSys’17, August 2017, Como, Italy
decision making at grocery stores
Medical School4 . Participants had to add products until they were
happy with their selection. When clicking on a product from the
list, the system showed the product information and three lists,
Similar Products, Healthy Products and Based on your Profile, each
including six recommendations. After finishing their selection for a
healthy plate, we asked users to fill out the ResQue[9] questionnaire
to understand the user experience with the recommender system.
Study 2: Paper Prototype. We presented packages of different
food products from the local supermarket to the participants on
a table. We reused the design of the components from Study 1’s
application and printed them on paper, see Figure 4. Participants
were informed about the AR system and the concept in general. We
also created cards with recommendations from the system, based on Figure 4: Participants during the studies using the desktop
similar products and healthy alternatives. Participants were asked application to evaluate the recommender system and using
to think-aloud their ideas during the experiment, and to imagine the paper prototype.
the following scenario you are in the grocery store and you want
to get a selection of food products you would likely buy. The task
started with participants picking up any product from the table 5 RESULTS AND DISCUSSION
and then, as PHARA would (automatically) do, the corresponding
Recommender system. We summarize the results of Study 1 in
detail component card (Figure 2c) was manually attached to the
Figure 5. Participants indicated they perceived the system as useful
front-side of the product package. When participants asked for
(Median = 4) on a 5-point Likert scale (one: strongly disagree,
recommendations of healthier or similar products the correspond-
five: strongly agree). They also indicated that they were familiar
ing recommendations card (Figure 2a) was attached to the package
with the products in the database (Median = 4). The interface
and we asked them to pick one of the recommended products from
that was shown to them appeared to be adequate (Median = 4).
the table. When they did so, and while holding both products,
Other factors such as confidence, novelty, satisfaction and behav-
we attached the corresponding product detail card (Figure 2c) to
ioral intentions tended to be rated lower (Median = 3). The main
the newly picked product. Afterwards, we asked participants to
comments of the participants at the end of the session were related
compare the products using both the printed information in the
to the quality of the data which is reflected in their behavioral
box and PHARA’s detail cards and chose the one they would likely
intentions (Median = 3), indicating that some products were not
buy. We repeated this procedure until participants declared they
easy to find due to missing information about the product. There
were satisfied with their selection. At the end of the task, we asked
were also comments related to the diversity of recommendations
participants to fill out a technology acceptance questionnaire [13].
(Median = 3.5), indicating that similar products appeared too often
Study 3: Offline Evaluation. An offline experiment was con-
in the recommendations.
ducted by collecting activity data of the users that used the Web
Paper prototype. Results are presented in Figure 6. Partici-
application recommender system. Using this data, we calculated
pant feedback tended to be positive in general (Median >= 6) on
the proportion of the recommendations that were actually suitable
a 7-point Likert scale (one: unlikely, seven: likely), indicating that
for the user (precision) and the variation of items in the recommen-
they found the system to be intuitive, easy to use and learn. How-
dations (diversity) of the system.
ever, some participants were concerned about the flexibility of use
4 https://www.hsph.harvard.edu/nutritionsource/healthy-eating-plate/ (Median = 5), given the HMD hardware that they would wear with
Figure 3: Components of the Web Application used to Eval-
uate the usability of the recommender system. a) Product
Navigator, b) My Plate, and c) Product details and recommen- Figure 5: Perceived qualities of the recommender system,
dations. and users’ behavioral intentions.
HealthRecSys’17, August 2017, Como, Italy Francisco Gutiérrez, Bruno Cardoso, and Katrien Verbert
refine the prototype and explore its effects in larger and more di-
verse audiences and settings. We will explore different strategies
to improve diversity and precision of food recommendations in
future studies and different visualizations will be evaluated on how
to effectively communicate personal health data to users.
ACKNOWLEDGEMENTS
The research has been partially financed by the KU Leuven Research
Council (grant agreement no. C24/16/017).
REFERENCES
[1] P. Achananuparp and I. Weber. Extracting Food Substitutes From Food Diary
via Distributional Similarity. 2016.
[2] J. Ahn, J. Williamson, M. Gartrell, R. Han, Q. Lv, and S. Mishra. Supporting
Figure 6: Participants’ perceptions of usefulness and per- healthy grocery shopping via mobile augmented reality. ACM Transactions on
ceived ease of use of PHARA. Multimedia Computing, Communications, and Applications (TOMM), 12(1s):16,
2015.
[3] N. Y. Conteh and G. A. Forgionne. A Simulation Study of Just-in-time Knowledge
Management (JITKM), pages 187–197. Springer London, London, 2006.
[4] D. Elsweiler, M. Harvey, B. Ludwig, and A. Said. Bringing the” healthy” into food
the real HDM device. Some participants had comments during the recommenders. In DMRS, pages 33–36, 2015.
think-aloud session “The information cards look intuitive and easy [5] J. Grubert, T. Langlotz, S. Zollmann, and H. Regenbrecht. Towards pervasive
augmented reality: Context-awareness in augmented reality. IEEE transactions
to read, however more details on the object would be useful”. When on visualization and computer graphics, 23(6):1706–1724, 2017.
asked to compare the information printed on the box with the aug- [6] V. Heun, S. Kasahara, and P. Maes. Smarter objects: using ar technology to
mented component, a participant mentioned that “Actually, this program physical objects and their interactions. In CHI’13 Extended Abstracts on
Human Factors in Computing Systems, pages 961–966. ACM, 2013.
component complements the information in the box”. When asked [7] M. Kay. Challenges in personal health tracking: The data isn’t enough. XRDS:
to hold both products and compare them, a participant mentioned: Crossroads, The ACM Magazine for Students, 21(2):32–37, 2014.
“In this way, it is easy to compare both products, the information is [8] E. V. Kleef and H. Dagevos. The growing role of front-of-pack nutrition profile
labeling: a consumer perspective on key issues and controversies. Critical reviews
consistent. Sometimes reading the package can be misleading and/or in food science and nutrition, 55(3):291–303, 2015.
difficult to read”. [9] P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender
systems. In Proceedings of the fifth ACM conference on Recommender systems,
Offline Evaluation. Our initial test showed that recommenda- pages 157–164. ACM, 2011.
tions had a low precision and recall, (see Figure 7). However, the [10] S. Shekar, P. Nair, and A. S. Helal. iGrocer- A Ubiquitious and Pervasive Smart
numbers for Healthy Alternatives also illustrate the interest of par- Grocery Shopping System. Sac 2003, 5:645–652, 2003.
[11] C. Siawsolit, S. Seepun, J. Choi, A. Do, and Y. Kao. Persuasive Technology: Devel-
ticipants in healthy products. These metrics illustrate participant’s opment and Implementation of Personalized Technologies to Change Attitudes
feedback in study 1, where they indicated that recommendations and Behaviors. 10171:95–106, 2017.
of products seemed to be useful, but factors such as accuracy and [12] Y. van Pinxteren, G. Geleijnse, and P. Kamsteeg. Deriving a recipe similarity mea-
sure for recommending healthful meals. In Proceedings of the 16th international
diversity were an issue. This was noted particularly when partici- conference on Intelligent user interfaces, pages 105–114. ACM, 2011.
pants faced incomplete descriptions of food products, or missing [13] V. Venkatesh and H. Bala. Technology acceptance model 3 and a research agenda
on interventions. Decision sciences, 39(2):273–315, 2008.
values. Participants tended to stick with familiar or similar products
that they felt confident with. Crowdsourced databases present a lot
of opportunities by providing a large set of items supported by a
devoted community of users. However, uncertainty in data quality
is a challenging factor in terms of processing and presentation to
the end user to be addressed.
6 CONCLUSIONS AND FUTURE WORK
We have introduced PHARA, an AR system to support decision-
making at grocery stores and described the results of three pre-
liminary user studies. The obtained results are encouraging and
suggest that PHARA’s concept is likely a viable way to promote
the adoption of healthy food buying behaviors. Future work will
Figure 7: Precision and diversity of recommendations.