PHARA: a personal health augmented reality assistant to support decision-making at grocery stores Francisco Gutiérrez Bruno Cardoso Katrien Verbert Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science KU Leuven KU Leuven KU Leuven Celestijnenlaan 200A Celestijnenlaan 200A Celestijnenlaan 200A Heverlee, Belgium 3001 Heverlee, Belgium 3001 Heverlee, Belgium 3001 francisco.gutierrez@cs.kuleuven.be bruno.cardoso@cs.kuleuven.be katrien.verbert@cs.kuleuven.be ABSTRACT to provide estimates on individuals’ basal metabolic rates and daily Poor diet and physical inactivity are important factors that con- kilo-calorie requirements. Although we can find relevant litera- tribute to the obesity outbreak. Therefore, healthy eating habits ture covering several aspects of food recommendation [1], there are crucial for physical well-being. In this paper, we present the is not so much research on the HCI factors of the actual delivery concept design and an early stage evaluation of PHARA, a personal of food recommendations to users. Holding to the principles of health augmented reality assistant that recommends healthy and just-in-time knowledge management (JITKM)[3], we believe that similar products to people in their everyday lives. We evaluated delivering the right information at the right context has a positive a content-based recommender system in a desktop environment impact on people’s food buying decisions and, ultimately, will help (n = 15) to measure the perceived quality, as well as behavioral them accomplish their health goals. Therefore, we are interested in intentions of users. In addition, we evaluated the user interface providing information to people when it matters the most - in the and measured participants’ perceptions of usefulness and ease of moment of decision: when they hold a product in their hands at use. Whereas perceived usefulness and perceived ease of use are the grocery store. Grocery stores provide an interesting setting for good, more work is required towards improving the accuracy and our research, as people there make many food buying decisions. To diversity of recommendations. this end, nutrition labels may play a key role in promoting healthy food choices [8], as improving people’s diet begins by improving CCS CONCEPTS the nutritional quality of the food choices they make. On the other hand, crowdsourced databases such as Open Food Facts1 provide a •Information systems →Decision support systems; Recom- big source of nutritional information for food products from around mender systems; •Human-centered computing →Mixed / aug- the world, opening a new spectrum of possibilities for relevant rec- mented reality; ommendations. We want to bring these concepts together along with the potential that augmented reality (AR) technologies have to KEYWORDS offer while providing a context-aware, continuous AR experience Decision Making, Recommender Systems, Augmented Reality [5]. ACM Reference format: Francisco Gutiérrez, Bruno Cardoso, and Katrien Verbert. 2017. PHARA: a 2 RELATED WORK personal health augmented reality assistant to support decision-making at grocery stores. In Proceedings of the Second International Workshop on Recommender Systems. van Pinxteren, Youri et al.[12] implemented Health Recommender Systems co-located with ACM RecSys 2017, Como, a similarity measure for recipes based on a collaborative filtering Italy, August 2017 (RecSys’17), 4 pages. approach. The similarity measure can be used to promote new recipes that fit into people’s lifestyle according to their preferences. Shekar, Sangeetha et al.[10] propose a traditional content-based implementation using a phone grocery shopping assistant that 1 INTRODUCTION recommends products based on the user profile and a database Nowadays, recommender systems are increasingly used to support of products. Achananuparp, Palakorn, and Ingmar Weber [1] ex- decision-making in health-related contexts, suggesting users how plored healthy food recommendations by finding food substitutes to improve their eating, exercising or sleeping behavior. An ex- in similar contexts using a crowdsourced service. ample is the use of recommendations as a basis to algorithmically Visualization. The work of C. Siawsolit et al. [11] illustrates the derive balanced meal plans that meet nutritional guidelines for the use of a simple star-rating visualization to suggest healthy products user [4]. These recommendations are generally based on personal and how a nutritional-based recommender system may be useful profiles that include gender, height, weight and physical activity, to people who are motivated to eat healthy but have no time to International Workshop on Health Recommender Systems, August 2017, Como, Italy. © compare products. Many alternatives rely on a simplified nutrition 2017. Copyright for the individual papers remains with the authors. Copying permitted labelling system to help consumers make healthier food choices. for private and academic purposes. This volume is published and copyrighted by its A prominent example is the five-colour nutrition label/nutri-score editors. 1 https://world.openfoodfacts.org/ HealthRecSys’17, August 2017, Como, Italy Francisco Gutiérrez, Bruno Cardoso, and Katrien Verbert User Interface. The interface of PHARA is inspired by the work of Heun, Valentin et al [6] in smarter objects, and we also consid- ered following Matthew Kay[7] suggestions for personal health visualization design. In Figure 2 we present an early digital pro- totype of our interface that uses an AR card component layout to show: a) a visual component of Similar Products recommendations; b) an Impact on Health component that gives a prediction of the impact of the product on user’s health; and c) an overview visual- ization of the product that shows a description of the nutrients in an intuitive visualization. Figure 1: Architecture of PHARA that illustrates the differ- 4 USER STUDY DESIGN ent components of the system and the communication be- tween server and clients. In this section, we present the design of our user studies. Results are discussed in the next section. We recruited 15 participants (3F, 12M; mean age: 27.6, SD: 5.88) via word-of-mouth. We asked them to participate in two studies to collect their thoughts towards the based on the British Food Standards Agency Nutrient Profiling Sys- recommender system, and an early paper prototype to understand tem (FSA-NPS), which is an score calculated for each 100g amount how PHARA can be used in real world settings. A short offline eval- of energy, sugar, saturated fatty acid, sodium, fibres, proteins, and uation was conducted afterwards to test recommendation accuracy fruits and vegetables. and diversity. Augmented Reality. The most comparable system to PHARA is Study 1: Recommender System. We built our system using the work of Ahn, Junho et al.[2] who developed a handheld AR sys- a crowdsourced database from Open Food Facts, using a content- tem that uses color-based AR tagging to support people in finding based approach and a similarity index to estimate similarities be- healthy food products in supermarkets. Their application uses an tween food products. We generated recommendations based on in-store navigation system to guide participants to the products similarities of products to provide recommendations about Similar and shows a single color-based label to highlight information over Products, Healthy Products and Based on your Profile. Participants the product. While this system may be the most similar to PHARA, were presented with a Web browser application where they were re- we are interested in the usability of immerse scenarios in which quired to create a profile with their personal data (allergies, height, users have both hands free to interact and get information from the weight, age and activity level). Afterwards, they were asked to system, in a just-in-time fashion. On the other hand, in contrast select 10 favorite products to train the system. After creating a with the work of Ahn, Junho et al, we are interested in more deep profile, a dashboard was shown (see Figure 3) were they could see recommendations related to the context of both the product and the a list of recommended products. The task was as follows: Select any user. To this end, item-item content-based recommenders have the products that you would like to have for dinner. potential to leverage personal data while letting users explore the The system showed a visualization of ”MyPlate”, with suggested relationships between food products and health-goal achievement. indications for a balanced meal, based on Healthy Eating Plate created by experts at Harvard School of Public Health and Harvard 3 SYSTEM DESIGN The design of PHARA is inspired by related work, JITKM goals, per- sonal health tracking challenges and the opportunities presented by recent advances in AR head-mounted display (HMD) technologies, such as Microsoft Hololens2 . In this section, we describe the system architecture and the design of PHARA’s user interface. Architecture. The architecture of PHARA is shown in Figure 1. We envision an immersive system where users can wear the HMD freely and execute tasks using input such as voice commands or hand gestures. With a reactive design3 in mind, data is streamed on demand from the servers to ensure system responsiveness. Com- puting of recommendations and predictions take place in the server, releasing the HMD from any other computation other than tracking and recognizing printed labels and barcodes to identify products. A Web Application also is served in a separate client, through which users can introduce and synchronize their devices with personal data. Figure 2: Early digital prototype that illustrates the visual 2 https://www.microsoft.com/en-us/hololens card components of the application a) Recommendations, b) 3 http://www.reactivemanifesto.org/ Health Impact Prediction, c) Product Information. PHARA: A personal health augmented reality assistant to support HealthRecSys’17, August 2017, Como, Italy decision making at grocery stores Medical School4 . Participants had to add products until they were happy with their selection. When clicking on a product from the list, the system showed the product information and three lists, Similar Products, Healthy Products and Based on your Profile, each including six recommendations. After finishing their selection for a healthy plate, we asked users to fill out the ResQue[9] questionnaire to understand the user experience with the recommender system. Study 2: Paper Prototype. We presented packages of different food products from the local supermarket to the participants on a table. We reused the design of the components from Study 1’s application and printed them on paper, see Figure 4. Participants were informed about the AR system and the concept in general. We also created cards with recommendations from the system, based on Figure 4: Participants during the studies using the desktop similar products and healthy alternatives. Participants were asked application to evaluate the recommender system and using to think-aloud their ideas during the experiment, and to imagine the paper prototype. the following scenario you are in the grocery store and you want to get a selection of food products you would likely buy. The task started with participants picking up any product from the table 5 RESULTS AND DISCUSSION and then, as PHARA would (automatically) do, the corresponding Recommender system. We summarize the results of Study 1 in detail component card (Figure 2c) was manually attached to the Figure 5. Participants indicated they perceived the system as useful front-side of the product package. When participants asked for (Median = 4) on a 5-point Likert scale (one: strongly disagree, recommendations of healthier or similar products the correspond- five: strongly agree). They also indicated that they were familiar ing recommendations card (Figure 2a) was attached to the package with the products in the database (Median = 4). The interface and we asked them to pick one of the recommended products from that was shown to them appeared to be adequate (Median = 4). the table. When they did so, and while holding both products, Other factors such as confidence, novelty, satisfaction and behav- we attached the corresponding product detail card (Figure 2c) to ioral intentions tended to be rated lower (Median = 3). The main the newly picked product. Afterwards, we asked participants to comments of the participants at the end of the session were related compare the products using both the printed information in the to the quality of the data which is reflected in their behavioral box and PHARA’s detail cards and chose the one they would likely intentions (Median = 3), indicating that some products were not buy. We repeated this procedure until participants declared they easy to find due to missing information about the product. There were satisfied with their selection. At the end of the task, we asked were also comments related to the diversity of recommendations participants to fill out a technology acceptance questionnaire [13]. (Median = 3.5), indicating that similar products appeared too often Study 3: Offline Evaluation. An offline experiment was con- in the recommendations. ducted by collecting activity data of the users that used the Web Paper prototype. Results are presented in Figure 6. Partici- application recommender system. Using this data, we calculated pant feedback tended to be positive in general (Median >= 6) on the proportion of the recommendations that were actually suitable a 7-point Likert scale (one: unlikely, seven: likely), indicating that for the user (precision) and the variation of items in the recommen- they found the system to be intuitive, easy to use and learn. How- dations (diversity) of the system. ever, some participants were concerned about the flexibility of use 4 https://www.hsph.harvard.edu/nutritionsource/healthy-eating-plate/ (Median = 5), given the HMD hardware that they would wear with Figure 3: Components of the Web Application used to Eval- uate the usability of the recommender system. a) Product Navigator, b) My Plate, and c) Product details and recommen- Figure 5: Perceived qualities of the recommender system, dations. and users’ behavioral intentions. HealthRecSys’17, August 2017, Como, Italy Francisco Gutiérrez, Bruno Cardoso, and Katrien Verbert refine the prototype and explore its effects in larger and more di- verse audiences and settings. We will explore different strategies to improve diversity and precision of food recommendations in future studies and different visualizations will be evaluated on how to effectively communicate personal health data to users. ACKNOWLEDGEMENTS The research has been partially financed by the KU Leuven Research Council (grant agreement no. C24/16/017). REFERENCES [1] P. Achananuparp and I. Weber. Extracting Food Substitutes From Food Diary via Distributional Similarity. 2016. [2] J. Ahn, J. Williamson, M. Gartrell, R. Han, Q. Lv, and S. Mishra. Supporting Figure 6: Participants’ perceptions of usefulness and per- healthy grocery shopping via mobile augmented reality. ACM Transactions on ceived ease of use of PHARA. Multimedia Computing, Communications, and Applications (TOMM), 12(1s):16, 2015. [3] N. Y. Conteh and G. A. Forgionne. A Simulation Study of Just-in-time Knowledge Management (JITKM), pages 187–197. Springer London, London, 2006. [4] D. Elsweiler, M. Harvey, B. Ludwig, and A. Said. Bringing the” healthy” into food the real HDM device. Some participants had comments during the recommenders. In DMRS, pages 33–36, 2015. think-aloud session “The information cards look intuitive and easy [5] J. Grubert, T. Langlotz, S. Zollmann, and H. Regenbrecht. Towards pervasive augmented reality: Context-awareness in augmented reality. IEEE transactions to read, however more details on the object would be useful”. When on visualization and computer graphics, 23(6):1706–1724, 2017. asked to compare the information printed on the box with the aug- [6] V. Heun, S. Kasahara, and P. Maes. Smarter objects: using ar technology to mented component, a participant mentioned that “Actually, this program physical objects and their interactions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems, pages 961–966. ACM, 2013. component complements the information in the box”. When asked [7] M. Kay. Challenges in personal health tracking: The data isn’t enough. XRDS: to hold both products and compare them, a participant mentioned: Crossroads, The ACM Magazine for Students, 21(2):32–37, 2014. “In this way, it is easy to compare both products, the information is [8] E. V. Kleef and H. Dagevos. The growing role of front-of-pack nutrition profile labeling: a consumer perspective on key issues and controversies. Critical reviews consistent. Sometimes reading the package can be misleading and/or in food science and nutrition, 55(3):291–303, 2015. difficult to read”. [9] P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems, Offline Evaluation. Our initial test showed that recommenda- pages 157–164. ACM, 2011. tions had a low precision and recall, (see Figure 7). However, the [10] S. Shekar, P. Nair, and A. S. Helal. iGrocer- A Ubiquitious and Pervasive Smart numbers for Healthy Alternatives also illustrate the interest of par- Grocery Shopping System. Sac 2003, 5:645–652, 2003. [11] C. Siawsolit, S. Seepun, J. Choi, A. Do, and Y. Kao. Persuasive Technology: Devel- ticipants in healthy products. These metrics illustrate participant’s opment and Implementation of Personalized Technologies to Change Attitudes feedback in study 1, where they indicated that recommendations and Behaviors. 10171:95–106, 2017. of products seemed to be useful, but factors such as accuracy and [12] Y. van Pinxteren, G. Geleijnse, and P. Kamsteeg. Deriving a recipe similarity mea- sure for recommending healthful meals. In Proceedings of the 16th international diversity were an issue. This was noted particularly when partici- conference on Intelligent user interfaces, pages 105–114. ACM, 2011. pants faced incomplete descriptions of food products, or missing [13] V. Venkatesh and H. Bala. Technology acceptance model 3 and a research agenda on interventions. Decision sciences, 39(2):273–315, 2008. values. Participants tended to stick with familiar or similar products that they felt confident with. Crowdsourced databases present a lot of opportunities by providing a large set of items supported by a devoted community of users. However, uncertainty in data quality is a challenging factor in terms of processing and presentation to the end user to be addressed. 6 CONCLUSIONS AND FUTURE WORK We have introduced PHARA, an AR system to support decision- making at grocery stores and described the results of three pre- liminary user studies. The obtained results are encouraging and suggest that PHARA’s concept is likely a viable way to promote the adoption of healthy food buying behaviors. Future work will Figure 7: Precision and diversity of recommendations.