Nudging Towards Health in a Conversational Food Recom mender System Using Multi-Modal Interactions and Nutrition Labels

Abstract

Humans engage with other humans and their surroundings through various modalities, most notably speech, sight, and touch. In a conversation, all these inputs provide an overview of how another person is feeling. When translating these modalities to a digital context, most of them are unfortunately lost. The majority of existing conversational recommender systems (CRSs) rely solely on natural language or basic click-based interactions. This work is one of the first studies to examine the influence of multi-modal interactions in a conversational food recommender system. In particular, we examined the efect of three distinct interaction modalities: pure textual, multi-modal (text plus visuals), and multi-modal supplemented with nutritional labeling. We conducted a user study ( =195) to evaluate the three interaction modalities in terms of how efectively they supported users in selecting healthier foods. Structural equation modelling revealed that users engaged more extensively with the multi-modal system that was annotated with labels, compared to the system with a single modality, and in turn evaluated it as more efective.

-

1. Introduction and Context Conversational recommender systems (CRSs) represent a hotly debated area of study in the field of information dation algorithms with conversational strategies. Using multi-turn conversations, CRSs are able to collect users’ nuanced and dynamic preferences in more depth, which can enhance recommendation outcomes and user experience. CRSs are utilized in a variety of domains, including medical diagnosis [ 3 ], e-commerce [ 4 ], and entertainment [ 5, 6 ]. Only a few studies have investigated their merit for food recommendation [ 7 ], and in particular for encouraging users to make healthier food decisions.

Over 60% of all deaths are caused by communicable diseases, which are preventable by

LGOBE 0000-0002-7478-5811 (A. E. Majjodi); 0000-0002-6767-358X (Y. Deldjoo); 0000-0002-9255-3256 (F. Narducci); © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License intake [ 8 ].

While our food decisions are driven by

our overall preferences, the food selection process is extremely contextual and influenced by a variety of Moreover, many of the decisions are made spontaneously and consumers’ judgments are influenced by factors unrelated to the food content, such as their perception of the food’s visual characteristics [ 9 ]. For instance, the packaging of items with nutritional labels can serve to highlight the nutritious nature of the food (cf. [ 10 ]). Moreover, people generally prefer food that has a more visually appealing presentation, such as food that is presented in an attractive way [ 11 ]. People are tastefully/attractively organized, and restaurants strive to generate Instagram-friendly photographs by enhancing the color composition of their plates.

To surface efective and healthy food recommendations it is crucial to understand these underlying decision factors. Regrettably, the large majority of existing conversational recommender systems [ 12, 13 ] only consider a single type of interaction, such as natural language or click-based interaction, thereby neglecting a wealth of information in the actual imaging of meals [14]. The goal of the present work at hand is to employ a new conversational model for food recommendation that permits more natural, multi-modal user-system interaction.

non- are willing to pay extra for food whose ingredients

To attain this goal, this paper introduces a multi- presentation time, healthiness of recipes chosen and a modal conversational food recommender system (MM- user’s level of choice satisfaction and experienced system CFRS). It implements diferent user-system interaction efectiveness. modes, along with nutrition labelling in order to assist the user in making dietary decisions. Our objective is to examine the efects of three distinct interaction modes: pure 2. System Design textual, multi-modal (text plus visuals), and multi-modal smuopdpalel mcoennvteerdsawtiiothnanluintrfiotiromnaatliolanbseelienkgi.ngW( MhiMleCmISu)ltiis- Itniotnhailsfsoeocdtiroencowmemdeesncdreibresythsteefme a,twurheischofsuopuprocrotnsvuesresrasgaining attention by the research in the RecSys/IR/HCI in making healthier choices.1 communities [ 15, 1, 16 ], only a few practical studies have We designed a system-driven conversation in which the been published that focus on topics other than food and system requires user feedback (response/input) to conhealth, such as conservational systems on tourism [17] tinue. The main steps of the conversational flow are and fashion [18, 19]. In the field of food recommendation, shown in Figure 1. Users can interact with the system usElsweiler et al. [20] provide a good frame of reference for ing both buttons and textual messages2. The main steps recent advances in the field of food recommender sys- of the interaction are reported below: tems in general. Specifically for conversational systems, Barko-Sherif et al. [21] investigate the possibility for conversational preference elicitation in a food recommender environment, using a Wizard of Oz study design (see also [22]). Using a between-groups approach, they compare spoken and text-input chat interfaces and reported that such interfaces are useful for users to describe their needs and preferences. In other studies, Samagaio et al. [23] present a RASA-based chatbot that can recognize and categorize user intentions in the conversation aimed to elicit food preferences for recommendation purposes. Another study of Samagaio et al. [24] applies more knowledgebased elements based on word embedding to optimize conversational ingredient retrieval. These studies, however, focus less on aspects pertaining to health, health labelling, or elicitation modalities. In a non-conversational recommender context, El Majjodi et al. [25] recently indicated that nutritional labels can reduce user’s choice dificulty in non-conversational context. The primary distinction between our work and previous studies is the lack of multiple modalities (typically only text is used), as well as that only a few studies (e.g., [25]) have used nutrition labelling.

To summarize, the goal of this study is to compare the impact of three user-system interaction and explanation modalities (textual, multi-modal, and multi-modal with nutritional labels) on both behavioral aspects (what type of recipe is chosen? How healthy is that recipe?) and evaluation aspects (how does the user evaluate the system or their chosen recipe?). Using a mediation analysis (structural equation modelling), we answer the following research question: • Food category acquisition: The user was presented with a choice of four diferent food categories that were considered in this work: Pasta, Salad,

Dessert, and Snack. • User constraints acquisition: The user was then prompted to indicate any potential dietary constraints. Initially, the system used an interface with a single checkbox for each of the most prevalent intolerances and allergies: Lactose, Meat, Alcohol, Seafood, Reflux, Cholesterol, Diabetes. Afterwards, the system asked the user to disclose a list of ingredients she could not consume. • Preference elicitation: According to the constraints specified by the user, the user was prompted to submit preferences for five of the dishes proposed by the system. Each dish was accompanied with two buttons: “Like” and “Skip”.

The skip option was provided to encourage users to inspect an addition dish, which was retrieved from the randomly sorted menu. The retrieval was based on a random active learning strategy.

This way, users were encouraged to like five dishes they were interested in, after which the user profile was built by the system. • Processing: The system constructed the user proifle by analyzing the user’s five preferences from the previous stage. The cosine similarity was computed between the user profile and each of the available foods in the catalog, to provide a list of dishes from which recommendations would be selected. The algorithm also provided a list of dishes ranked according to their healthiness (based on their FSA score; see Section 3). • RQ: To what extent do diferent interaction modalities afect a user’s recipe choices and evaluation in a conversational food recommendation scenario?

To address this question, we consider diferent dimen

sions of analysis. This includes system interaction length,

1Code and recipe data used for implementing the chatbot are avail

able at https://github.com/giocast/MMCFRS 2A video demo of the three versions of our system is available at https://tinyurl.com/mtzxr2sw

manner by displaying the name and image of each dish throughout the dialogue. However, the supplied explanation remains textual. For the first dish, the explanation can be like ”I recommend these dish because I know that you have diet constraints due to: meat, zucchini.

The first dish I proposed contains ingredients that you might like: carrot, lemon, tuna, olive oil”. For the second recommendation, the explanation further provides information about macro nutrients quantities of the two recommended dishes and can be in the form of ”The second dish I proposed has less calories (54 Kcal) than the first one (123 Kcal) and has less fats than the first one. The third version MM-Label (MM + MM) likewise employs a multi-modal interaction approach, but it also makes use of nutritional explanations in the form of a

Three diferent interaction modes were implemented front-of-package nutrition label with FSA’s Multiple Trafby modifying the values associated with the two manipu- ifc Lights (MTL) [ 25]. MTL nutrition labels depicted the lated variables: interaction and explanation , according intake adequacy of a dish in terms of energy and nutrito Table 1. tional content, along five dimensions: energy (kcal), fat,

In the Pure text version (T + T), the system communi- saturates, sugars, and salt. This adequacy, per serving cates with the user solely through text, displaying sim- and per 100g, was depicted using the colors green, yellow ply the dish titles and ofering textual explanations of and red, where green indicated a dish to adhere to the the food recommendations. In the Multi-modal version nutritional intake guideline, while red indicated that the (MM + T), the system engages the user in a multi-modal content was unacceptable. These labels were generated for each dish by following the directives of Food Standard the recommendations, we provide the user with an explaAgency and UK department of health [26]. nation that helps her comprehend the health benefits of

Figure 2 depicts a snapshot of the chatbot prototype, the second alternative above the first, which is the dish visualizing the diferent interaction phases. that best matches her preferences. This is accomplished

In the Textual (T) version, the user received recom- either by text (T and MM variants) or a multiple trafic mendations identified by only the names of the dishes light nutritional label (MM-Label). (e.g., Cupcake Princess’ Vanilla Cupcakes, Floating Island The user can accept one of the two dishes proposed or II). The recommendations were followed by textual ex- can ask for another recommendation. planations, based on the ingredients in the dish that the user likes. A comparative analysis of the nutritional facts (e.g., ‘less sugars’) would also be provided. In the Multi- 3. Experimental Evaluation modal (MM) version, the system additionally provided To evaluate the extent to which diferent versions of the images of the recommended dishes. The explanation was chatbot afected users’ evaluations and decisions, we resimilar to the one presented in the T version. Finally, cruited 195 participants from Amazon MTurk to use our the Multi-modal with labels (MM-Label) version provided system. Participants had to have a hit rate of 95% at least nutritional labels that were annotated to the depicted and were compensated with 2 dollars. On average, user images (e.g., Sugar 2.3g, Fat 10.7g, etc.) presented with required around 15 minutes to complete the study.3 Users red, yellow, and/or green colors according to the FSA score. As stated previously, following the presentation of

3The research conformed to the ethical standards of the Norwegian

Centre for Research Data (NSD). The collected data is available in performed the processes outlined in Section 2, interact- tion duration was significantly longer ( < 0.05 ) than in ing with our chatbot for preference elicitation, evaluating the text-based condition . This indicated that the usage recipe recommendations, selecting one recipe, and evalu- of nutrition labels afected conversation time, on top of ating the experience. A user’s experience was evaluated the other modalities. through choice satisfaction and system efectiveness, us- The duration of the conservation afected, in turn, the ing questionnaire items that were evaluated on 5-point evaluation of the user. Inferred from our confirmatory Likert scales. factor analysis (cf. Table 2), users who interacted with

Chosen recipes were evaluated according to their the chatbot for longer periods of time indicated greater healthiness. This was evaluated using the FSA score [27]. levels of system efectiveness ( < 0.01 ). This indicated Each recipe was scored between 4 and 12, where 4 indi- that an extended engagement did not frustrate users. Incated that all four nutrients (sugar, fat, saturated fat, salt) stead, it indicated that they were enthusiastic about using adhered to nutritional guidelines per 100g [ 9, 28 ], while the system. Figure 3 also shows that the healthiness of 12 would indicate that a recipe was unhealthy because chosen recipes was not significantly related to any of the of all nutritional contents being too high. other aspects or factors. Note that the MM-Label condi

The responses to the evaluation questionnaire item tion led the healthiest recipe choices, but the diferences were submitted to a confirmatory factor analysis (CFA; with the other conditions were not significant. see Table 2). Unfortunately, we could not infer a reliable construct for choice satisfaction, as the variance explained by the questionnaire items was too low, while 4. Conclusion and Future Work Cronbach’s Alpha was only acceptable (0.60). Other items were dropped from the system efectiveness aspect because of low factor loadings.

We organized the diferent factors (e.g., conversation time, condition factors) and aspects (i.e., system efectiveness) into a path model using Structural Equation Modelling. Figure 3 depicts the resulting model, which had decent fit statistics: 2(17) = 28.064, < 0.05 , = 0.969 , = 0.954 , = 0.058 , 90% − : [0.009, 0.095].

The relevant AVEs of the aspects was suficiently high to form a path model [29].

Our analysis revealed that the MM-Label condition with nutrition labels (MM-label) stood out in terms of how long users interacted with our chatbot. Figure 3 illustrates this, while the use of multi-modal approaches alone had no efect on the interaction or evaluation factors considered. For MM-Label, our mediation analysis suggested that in the MM-Label condition, the conversaWe have presented a novel chatbot-like recommender system that introduces multi-modality in interaction with user, presentation of results and explanation of the recommendations with nutrition labels in a conversational scenario. We have designed and analyzed the impact of three distinct version of our chatbot: pure textual, multi-modal (use of text and images), and multi-modal supplemented with nutritional labels.

Our experimental evaluation reveals that our chatbot is the most efective when accompanied by explanatory labels. This is indicated by the length of conversation, as well as by the user’s evaluation of the system efectiveness.

Limitations to this study could be viewed from different viewpoints. In terms of analysis, we have been unable to infer the choice satisfaction evaluation aspect.

Other research have demonstrated that decision satisfaction is a good predictor of post-interaction engagement with selected item, such as for household energy conthe project’s GitHub repository. servation [30]. Moreover, rather than relying solely on system-driven interaction, it might be intriguing and natural to investigate user-driven scenarios in which users might query the system with an image and textual query.

The food categories considered in this work (pasta, salad, dessert, snack) could additionally be expanded to include more meal categories and their combinations, such as to create a complete meat (first dish, second dish and vegetables). On top of that, the distinctions between various label modalities are an additional intriguing topic we wish to investigate more in-depth [31]. vanced Applications, Springer, 2018, pp. 817–820. ingredient retrieval, in: 3rd Conference on Lan[13] J. Schafer, T. Hollerer, J. O’Donovan, Hypothetical guage, Data and Knowledge (LDK 2021), Schloss recommendation: A study of interactive profile ma- Dagstuhl-Leibniz-Zentrum für Informatik, 2021. nipulation behavior for recommender systems, in: [25] A. El Majjodi, A. D. Starke, C. Trattner, Nudging The Twenty-Eighth International Flairs Conference, towards health? examining the merits of nutrition 2015, pp. 507–512. labels and personalization in a recipe recommender [14] Y. Deldjoo, M. Schedl, P. Cremonesi, G. Pasi, Rec- system, in: Proceedings of the 30th ACM Conferommender systems leveraging multimedia content, ence on User Modeling, Adaptation and PersonalACM Computing Surveys (CSUR) 53 (2020) 1–38. ization, 2022, pp. 48–56. [15] Y. Deldjoo, J. R. Trippas, H. Zamani, Towards multi- [26] Department of Health and Social Care UK, Front modal conversational information seeking, in: Pro- of Pack nutrition labelling guidance, 2016. URL: ceedings of the 44th International ACM SIGIR con- https://www.gov.uk/government/publications/ ference on research and development in Informa- front-of-pack-nutrition-labelling-guidance. tion Retrieval, 2021, pp. 1577–1587. [27] D. of Health UK, F. S. Agency, Guide [16] R. G. Sousa, P. M. Ferreira, P. M. Costa, P. Azevedo, to creating a front of pack (fop) nutriJ. P. Costeira, C. Santiago, J. Magalhaes, D. Semedo, tion label for pre-packed products sold R. Ferreira, A. I. Rudnicky, et al., ifetch: Multimodal through retail outlets (2016). URL: https: conversational agents for the online fashion market- //assets.publishing.service.gov.uk/government/ place, in: Proceedings of the 2nd ACM Multimedia uploads/system/uploads/attachment_data/file/ Workshop on Multimodal Conversational AI, 2021, 566251/FoP_Nutrition_labelling_UK_guidance.pdf. pp. 25–26. [28] C. Trattner, D. Elsweiler, Investigating the healthi[17] L. Liao, L. H. Long, Z. Zhang, M. Huang, T.-S. Chua, ness of internet-sourced recipes: implications for Mmconv: an environment for multimodal conversa- meal planning and recommender systems, in: Protional search across multiple domains, in: Proceed- ceedings of the 26th international conference on ings of the 44th International ACM SIGIR Confer- world wide web, ACM, New York, NY, USA, 2017, ence on Research and Development in Information pp. 489–498.

Retrieval, 2021, pp. 675–684. [29] B. P. Knijnenburg, M. C. Willemsen, Evaluating [18] S. Moon, S. Kottur, P. A. Crook, A. De, S. Pod- recommender systems with user experiments, in: dar, T. Levin, D. Whitney, D. Difranco, A. Beirami, Recommender systems handbook, Springer, 2015, E. Cho, et al., Situated and interactive multimodal pp. 309–352. conversations, arXiv preprint arXiv:2006.01460 [30] A. Starke, M. Willemsen, C. Snijders, Efective user (2020). interface designs to increase energy-eficient behav[19] Y. Yuan, W. Lam, Conversational fashion image ior in a rasch-based energy recommender system, retrieval via multiturn natural language feedback, in: Proceedings of the eleventh ACM conference in: Proceedings of the 44th International ACM SI- on recommender systems, 2017, pp. 65–73. GIR Conference on Research and Development in [31] Y. Deldjoo, M. Schedl, B. Hidasi, Y. Wei, X. He, MulInformation Retrieval, 2021, pp. 839–848. timedia recommender systems: Algorithms and [20] D. Elsweiler, H. Hauptmann, C. Trattner, Food challenges, in: Recommender systems handbook, recommender systems, in: Recommender Systems Springer, 2022, pp. 973–1014.

Handbook, Springer, 2022, pp. 871–925. [21] S. Barko-Sherif, D. Elsweiler, M. Harvey, Conversational agents for recipe recommendation, in: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, 2020, pp. 73–82. [22] A. Steinfeld, O. C. Jenkins, B. Scassellati, The oz of wizard: simulating the human for interaction research, in: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, 2009, pp. 101–108. [23] Á. Mendes Samagaio, H. Lopes Cardoso, D. Ribeiro,

A chatbot for recipe recommendation and preference modeling, in: EPIA Conference on Artificial

Intelligence, Springer, 2021, pp. 389–402. [24] Á. M. Samagaio, H. Lopes Cardoso, D. Ribeiro, Enriching word embeddings with food knowledge for

References

[1] H. Zamani , J. R. Trippas , J. Dalton , F. Radlinski , Conversational information seeking, arXiv preprint arXiv:2201.08808 ( 2022 ).

[2] D. Jannach , A. Manzoor , W. Cai , L. Chen , A survey on conversational recommender systems , ACM Computing Surveys 54 ( 2022 ) 1 - 36 . doi:1 0 . 1 1 4 5 / 3 4 5 3 1 5 4 .

[3] P. Cordero , M. Enciso , D. López , A. Mora , A conversational recommender system for diagnosis using fuzzy rules , Expert Systems with Applications 154 ( 2020 ) 113449 . doi:1 0 . 1 0 1 6 / j . e s w a . 2 0 2 0 . 1 1 3 4 4 9 .

[4] D. Griol , J. Milina , From voicexml to multimodal mobile apps: development of practical conversational interfaces , ADCAIJ Adv. Distrib. Comput. Artif. Intell. J . 5 ( 2016 ) 43 .

[5] F. Narducci , P. Basile , M. de Gemmis, P. Lops, G. Semeraro, An investigation on the user interaction modes of conversational recommender systems for the music domain, User Model . User Adapt. Interact . 30 ( 2020 ) 251 - 284 . URL: https:// doi.org/10.1007/s11257-019 -09250-7. doi:1 0 . 1 0 0 7 / s 1 1 2 5 7 - 0 1 9 - 0 9 2 5 0 - 7 .

[6] A. Iovine , F. Narducci , G. Semeraro, Conversational recommender systems and natural language: : A study through the converse framework , Decis. Support Syst . 131 ( 2020 ) 113250 . URL: https: //doi.org/10.1016/j.dss. 2020 . 113250 . doi:1 0 . 1 0 1 6 / j . d s s . 2 0 2 0 . 1 1 3 2 5 0 .

[7] C. Trattner , D. Elsweiler , Food recommendations, in: Collaborative recommendations: Algorithms, practical challenges and applications , World Scientific, 2019 , pp. 653 - 685 .

[8] R. Y. Toledo , A. A. Alzahrani , L. Martinez , A food recommender system considering nutritional information and user preferences , IEEE Access 7 ( 2019 ) 96695 - 96711 .

[9] A. D. Starke , M. C. Willemsen , C. Trattner , Nudging healthy choices in food search through visual attractiveness , Frontiers in Artificial Intelligence 4 ( 2021 ) 621743 .

[10] E. J. Van Loo , C. Grebitus , J. Roosen , Explaining attention and choice for origin labeled cheese by means of consumer ethnocentrism , Food Quality and Preference 78 ( 2019 ) 103716 .

[11] Y. Peng , J. B. Jemott III , Feast for the eyes: Efects of food perceptions and computer vision features on food photo popularity ., International Journal of Communication (19328036) 12 ( 2018 ).

[12] C. Zhou , Y. Jin , K. Zhang , J. Yuan , S. Li , X. Wang , Musicrobot: Towards conversational contextaware music recommender system , in: International Conference on Database Systems for Ad-