1 ‘Keep the user in mind!’ Persuasive Effects of Social Robot as Personalized 1 Nutritional 2 coach. 3 Berardina De Carolis , Francesca D’Errico , Nicola Macchiarulo 1 Department of Computer Science, University of Bari, Italy berardina.decarolis@uniba.it 2 Fil.Co.Spe Department, Roma Tre University, Italy francesca.derrico@uniroma3.it 3 Exprivia SpA, Italy nicola.macchiarulo@exprivia.com Abstract. In this paper we investigate the use of a social robot as an interface for providing personalized information about nutrition. In particular, we evalu- ated the effect of message adaptation to some demographic traits that are auto- matically recognized by a social robot, Pepper in this case. The proposed ap- proach is based on soft biometrics and it can estimate several traits simultane- ously, such as gender and age of people in the field of view of the robot. Our hypothesis is that adding this capability to a social robot improves the persua- sive effects, in terms of perceived informational quality, motivational strength and social believability, and the recall of the message. We performed a prelimi- nary experimental study and its results seem to support the persuasiveness of the personalized nutritional coach. Keywords: Personalization, Social Robot, Soft Biometrics, Persuasive technol- ogies 1 Introduction Social robots are physically embodied, autonomous agents that communicate and interact with humans on a social and emotional level. They represent an emerging field of research focused on developing a “social intelligence” to maintain the illusion of dealing with a human being [1, 2]. They are being applied in several domains such as elderly care [3], autism therapy [4], education [5, 6], public places [7], domestic and work environments [8]. To be believable, social robots must exhibit social intelli- gence and adapt their behavior to the situation, therefore they should be endowed with a model of the environment and of the user that may include their profile, emotions, personality and past interactions [9]. In this regard, robots capable of exhibiting so- 1 Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons Li- cense Attribution 4.0 International (CC BY 4.0) 2 ciability and achieving widespread societal acceptance are being used more and more often in human-centered environments. Due to their ability to enable natural interac- tion, social robots have a great potential for helping people in their daily activities, especially in tasks that need one-to-one interaction, as is the case of elderly people or children. Recently, due to the importance of following healthy eating habits, they are being employed in teaching and informing people about nutrition [10, 11]. It has been shown that a correct diet is related with a decreased risk of cancer and cardiovascular diseases [12], diabetes and obesity [13; 14]. Social Robots have been employed in this context especially for playing serious games with children [15]. The long-term goal of this research is to develop a persuasive social robot, play- ing the role of a nutritional coach, able to induce a behavior change in the health and wellness domains. Such a robot may be especially effective for patients and elderly who want to remain active and live a better and more independent life. Social robots that counsel users on dietary behavior represent an especially promising application in this area. Fruit and vegetable consumption alone plays a protective role in a large number of cancers, and is associated with reduced risk for heart disease, stroke and hypertension, yet only a small percentage of adults meet the guidelines for daily wa- ter, fruit and vegetable consumption [16]. In this context, adaptation of both social behavior and message content to the user is necessary [17] and, as a first phase of our research, we aimed at making the robot aware of users characteristics and testing whether customizing the robot message to them was an effective way to increase the perception of the information quality and motivational strength of the conveyed message, the degree of recall and the perceived social believability of the robot. This is at the basis of a more complex adaptation of the robot behavior in which persuasive and argumentation strategies will be adapted to the user [29]. In this paper, we describe our experience with the design and implementation of a dialog module that allows Pepper talking with users about nutrition. To endow the robot with the capability of adapting the dialog to the user a soft biometrics module, able to estimate gender and age, was used [18]. Then, we performed a preliminary study in which we evaluated the persuasive effects of the adaptation to these traits in terms of perceived 1) informational quality and 2) persuasive effects in terms of robot reliability and the information recalling after the interaction. Preliminary results seem to support the persuasive impact of a personalized nutritional coach. The paper is organized as follows: in Section 2 a brief overview on soft biomet- rics analysis, together with the method employed in our system, is provided. Section 3 describes a preliminary study aiming at showing the effects of adaptation on the user perception of the message information content, motivational strength and believabil- ity of the social robot. Finally, Section 4 is aimed at a final discussion and proposal for future development. 2 3 2 Soft-Biometrics Analysis Soft biometric traits refer to physical and behavioral traits, such as gender, age, height and weight, which are not unique to a specific subject, but are useful for identifica- tion, and description of human subjects [19]. Among soft biometric traits, gender and age have been extensively studied. Among the most robust and accurate approaches to gender and age classification, we can find some based on the analysis of texture patterns. Many texture features have been used like LBP, Histogram of Oriented Gradients (HOG) and they usually em- ploy Support Vector Machine (SVM) or k-nearest neighbor as classifiers [20,21]. Gunay and Nabiyev [22] used LBP feature as an efficient face descriptor. They divid- ed the faces into small regions from which the LBP histograms are extracted and con- catenated into a feature vector. They got 80% age classification rates in FERET data- base [23]. Although such local descriptors achieve higher results than holistic meth- ods, their performance is affected by variations in expression, pose, illumination and occlusion. Recently deep learning approaches are being used successfully in this domain. They mainly use the CNN which is a type of feed-forward artificial neural networks in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex [24]. Yan et al. [25] proposed an approach which uses the CNN to extract the facial features. Their network has 7 layers and it gives as output 4096 features. For the classification part, they use SVM to classify the face into one of thirteen age groups. Levi and Hassner [26] proposed a network architecture for both age and gender classification. Rothe et al. [27] won the ChaLearn LAP 2015 challenge on apparent age estimation, their proposed CNN uses the VGG-16 architec- ture. They proposed an approach in which first the face is detected from the input image and then extracts the CNN predictions from an ensemble of 20 networks on the cropped face. In the context of our research, we developed a soft-biometrics module able to rec- ognize age, gender, eyeglasses and beard presence [18]. In addition, it also recognizes the color of eyes and hairs. For each soft biometric trait, a specific software module has been implemented. In this application, we use only the age estimation and the gender recognition modules. In particular, for age and gender estimation, our system focuses on automatic gender and age classification using deep CNNs. We use a fine- tuned version of the VGG-16 neural network and, on both tasks, the approach reaches a satisfactory accuracy using unconstrained image dataset2. In particular, the gender recognition has been performed with an accuracy of 85% while age estimation reached an accuracy (+/- 1 year) of 84% on the previously mentioned dataset. In an- other experiment we tested the performance in real time in the wild having an accura- cy of 87.5% for gender recognition and 62.5% for age estimation in which the main confusion problems were in estimating the age of middle-age women [18]. 2 https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/ 3 4 3 Personalized Information by the Social Robot Pepper We performed an experiment in real-time with the aim of evaluating the efficacy and the motivational impact of the information about nutrition provided by Pepper. In particular, we tested the difference in the user perception of the information quality and motivational strength of the informative content in case of personalization intro- duced by adapting the communication of the robot to the recognized soft biometric traits of the subjects. Moreover, we measured the degree of recalling on the informa- tive content and we observed the engagement of the users in interacting with the ro- bot. 3.1 Participants Forty-five people (23 males and 22 females) aged between 18 and 60 y.o (st.dev=9.09), equally distributed by gender and background were recruited for the experiment. They were motivated in participating to the study by the fact that they would receive vegetables (carrots) and fresh water after the experiment. 3.2 Study Design In the healthy eating scenario, the subjects interacted with the robot in two different settings: a) no customization (a generic message); b) customization of the message to gender and age. As far as age estimation is concerned, we split in intervals the age of the population typical of the scenario of the experiment and implemented the adapta- tion of the interaction and the messages accordingly. In particular, we distinguished among: Young Adults (ages 18-35 years), Adults (ages 36-55 years), and Older Adults (aged older than 55 years). Then, the 45 partici- pants were divided into 2 groups equally distributed in gender and age, of 22 and 23 participants each. The first group interacted with Pepper in the not customized condi- tion (NCC) while the second one in the customized one (CC). The second group was further split into the three groups described above with the following distribution: • Young Adults: 5 females and 6 males • Adults: 4 females and 4 males • Older Adults: 2 females and 2 males 3.3 Preparation of Material The message was generated according to strategies suggested by a human coach ex- pert in nutrition. In this study we refer to user profiles based on age and gender for which we made some assumptions about their presumed goal’s value, consistently with their gender and age. Then, we prepared the scripts for generated the four kinds of messages (Table 1). The first one is a generic and not customized. The second has 4 5 been customized in order to address the presumed goals of young subjects (i.e. physi- cal appearance, social life, ...). Table 1. The messages prepared for the four different scenarios. Type Message Hello! + . I’m here to provide you with information about the importance of Not hydrating your body. customized … Message: Drinking is important for a healthy body. It is recommended to drink at least two liters of water per day to hydrate the body. Proper hydration slows down the body’s aging process. Our organs, in fact, need water to work best. When our bodies become dehydrated, our organs have to work harder, and this can lead to an acceleration in the aging process. Eating fruit and vegetables also helps keeping the body well hydrated. Moreover, these foods give to the body vitamins and minerals. Hi!+< fist bump gesture> + I’m here to talk with you about the importance of hydrating your body. Customized … Young Message: Drinking during the day is very important. You must drink at least two liters of water a day. Proper (presumed goal: hydration slows down the body's aging process. improve Our organs, in fact, need water to work best. Appropriate hydration, about three and half liters a day, physical can also help keeping the muscles well hydrated, strong and energetic, therefore it is important to appearance) drink during physical activity. Remember that the processes that lead to be in a good shape may only occur in a perfectly hydrated body. So, drinking is very important to look in shape! Appropriate hydration, about two and half liters a day, helps fight water retention which is the antechamber of cellulite. In fact, water retention does not mean having "too much" water in the body but suffering from an imbalanced distribution. People who suffer from it often do not drink enough. To improve this situation, it’s important to drink more and more. So drinking may help you in passing the bikini test! Good morning/evening! + + I’m here to talk with you about the importance of Customized hydrating your body. Adult … Message: (presumed goal: Drinking during the day is very important. You must drink at least two liters of water a day. Proper prevent hydration slows down the body's aging process and improves its functioning. Drinking water and eating fruits and vegetable may help at reducing several risks for your health since your organ will health prob- be more hydrated. lems) Water will help to remove toxins and your skin will look smoother and more radiant. Water will help you to prevent urinary and prostate problems. Adopt a healthier lifestyle … it is known “prevention is better than cure”. Good morning/evening! + + Customized I’m here to provide you with information about the importance of hydrating your body. Older Adult … Message: (presumed goal: After the age of 50 the thirst stimulus fades. The body has no water reserves and for this it is neces- improve sary to rehydrate our body by drinking a lot and eating fruits and vegetables, which contain a signif- icant amount of water. Furthermore, an American study has proved that drinking a good amount of health) water can decrease the risk of a heart attack. Health authorities and others encourage people to consume 2 or more liters of water a day, that is the right dose for an adult. To this quantity must be added the liquids supplied to the body from fruit and vegetables, for a total of 3 liters per day. It is best to drink calcium water that prevents osteoporosis. The third one is directed to middle-age people, caring more for health and disease prevention, and the fourth to older adults. In particular, the second and third messages have been further distinguished between females and males in order to select the right arguments. 5 6 In order to assess the user background knowledge and beliefs about correct nutri- tion a pre-test questionnaire concerning some control variables (like level of infor- mation on nutrition and on the importance of adopting dietary behaviors) was pre- pared. We prepared also a post-test questionnaire aiming at evaluating some charac- teristics of the provided message (i.e. information quality, motivational strength), the level of recall of the information provided in the message and the social believability of the robot during the interaction. Questions were expressed as statements and evalu- ated using a 5 points Likert scale. 3.4 Procedure A realistic scenario was created in one of the open community spaces of our Depart- ment for performing the experiment. We set up a kind of healthy eating stand in which Pepper was acting as an expert, thus providing information to people in the domain. To control the interaction and compare results we focus on the importance of drinking and hydrating our body. In this scenario Pepper was running for two days according to two different settings described before. We decided to dedicate one day to the not-customized condition and one day to the customized one. After receiving a short explanation describing the purpose of the experiment, sub- jects willing to participate to the experiment signed up an informed consent and filled out the pre-test questionnaire. Then they started to interact with Pepper and received the message when asking about suggestions. After the message and the interaction with Pepper they were asked to fill out the post-test questionnaire. a) G: Male -Age: 22 b) G: Female -Age: 45 c) G: Female -Age: 20 Fig. 1. Three examples of interaction with the Pepper robot. The robot recognizes soft biomet- rics traits of the interacting subject and customizes its behavior and the information content of the provided message. In the customized condition, after having detected the face of a person entering in its field of view, using its RGB camera in the forehead with a resolution of 1280x960, Pepper estimated gender and age group by running the soft-biometrics module in the Pepper Department Cloud. In this setting, Pepper adapted the greetings by changing the level of formality/friendliness of the employed language because of the recog- nized age and the style and motivational content of the provided informative messag- es. Figure 1 shows three examples of interaction. On the left-side column the pictures 6 7 taken from the Pepper camera are shown together with the annotation of the soft- biometrics traits. As you can notice in the right-side picture, Pepper adapts the greet- ing gesture to the estimated age of the subject. 3.5 Results and Discussion The analysis of the pre-test questionnaires confirmed our assumptions about the hypo- thetical users and ensured that differences in the post-text were not due to differences in the healthy eating knowledge of the participants. To compare the motivational impact of the message in the two output conditions, we analyzed the results of the post-test questionnaire data using a t-test. Results (Ta- ble 2) show that the customized interaction and message conveyed through Pepper received an overall better evaluation. Significant differences occurred in terms of satisfaction and helpfulness. Moreover, the customized condition was perceived as significantly more persuasive and reliable. There were no significant differences in terms of easiness in the understanding and validity of the message content. Table 2. Results of t-test with α =0.05. Ratings are on a Likert scale from 1 to 5. Not Cus- T-test Customized tomized p (α =0.05) Mean 3.571 4.095 Satisfaction 0.037 Stdev 0.810 0.768 Mean 3.714 4.285 Information Helpfulness 0.018 Quality Stdev 0.783 0.717 Mean 4.047 4.285 Easiness 0.218 Stdev 0.589 0.643 Persuasiveness Mean 3.476 4.142 0.003 Stdev 0.679 0.727 Motivational Mean 3.476 4 Reliability 0.031 Strength Stdev 0.749 0.774 Mean 3.714 4 Validity 0.201 Stdev 0.7171 0.707 Recall Mean 4 4.32 0.106 Stdev 0.91 0.72 Social Be- Mean 3.476 4.047 0.009 lievability Stdev 0.813 0.497 About the evaluation of the social believability of the robot we compared the re- sults of the two groups. In the first condition the rating was lower than in the second 7 8 condition, in which greetings were adapted to the soft-biometric traits and, according to the t-test result (see Table 2), this difference is significant thus showing that the robot is perceived as more believable when it is aware of the characteristics of the type of person present in the environment. However, the recalling degree (in terms of correct answers given by subjects to the 5 questions) does not present a significant difference but a promising tendency that can be reinforced in larger sample (Table 2). 4 Conclusions Our long-term research goal is to develop a Persuasive Social Robot in domains such as nutrition, wellbeing and health [28] for helping people in changing their wrong attitudes and habits [29]. A first step in this direction is represented by the work de- scribed in this paper. To this aim, we endowed a social robot, Pepper in this case, with the capability of analyzing soft-biometrics features of people and use this awareness to customize the information provided to users in the domain of nutrition and healthy eating. To test the efficacy of the proposed approach, we set an experiment in a real- world scenario in which Pepper was providing information about correct nutrition. Even if performed on a small number of subjects, the experiment was carried out in the wild and the results show that, when the dialog was customized to the characteris- tics of the person interacting with the robot, it was better perceived in terms of satis- faction, helpfulness and persuasiveness. Moreover, the robot itself was perceived as more socially believable in the customized condition. These results are in line with classical studies on persuasion which pointed out how a persuasive source can be reliable when it demonstrates to know who is the user and which are his/her needs, by exploiting the empathic and engaging side of persuasion. [30; 31] In our expectation, adapting the content of the dialog can also affect the retention of the learned information and results seem to go in this direction since the recalling degree (in terms of correct answers given by subjects to the 5 questions) present a promising tendency (Table 2) in this sense. Participants in customized condition recall more than in a not customized one (4.32 vs 4; p=.106). In a future experiment, we can better explore this result with a larger sample and more ad hoc measures, by also putting in relation the motivational strength, social believability and informational quality with the level of recalling and other behavioral measures. In this sense in fact a customized robot can affect individual healthy atti- tude simply because users feel a technological ‘trust’ [32] that in this case can be based on persuasive and affective processes of being acknowledged. As far as soft- biometric is concerned, we plan to add new traits recognition from the body for mak- ing the robot aware of the type of person (i.e. slim vs. fat) who is asking for infor- mation and to investigate whether this improve the efficacy of the dialog. However, since it is possible to misclassify age and gender of the user, we plan to investigate also on the effects of wrong customization to these factors. 8 9 References 1. Al-Qaderi, Mohammad, and Ahmad Rad. "A multi-modal person recognition system for social robots." Applied Sciences 8.3 (2018): 387.Faria, D. R. , Vieira, M., Faria, F. C. , Premebida, C. "Affective facial expressions recognition for human-robot interaction", RO-MAN'17: IEEE International Symposium on Robot and Human Interactive Communi- cation, 2017. 2. Broekens, J.; Heerink, M.; Rosendal, H. Assistive social robots in elderly care: review. Gerontechnology 2009, 8, 94–103. 3. Palestra, G., De Carolis, B., Esposito, F. Proceedings of the Workshop on Artificial Intel- ligence with Application in Health co-located with the 16th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, November 14, (2017). 4. Mubin, O.; Stevens, C.J.; Shahid, S.; Al Mahmud, A.; Dong, J.J. A review of the applica- bility of robots in education. J. Technol. Educ. Learn. 2013, 1. 209–0015. 5. Saerbeck, M.; Schut, T.; Bartneck, C.; Janse, M.D. Expressive robots in education: Vary- ing the degree of social supportive behavior of a robotic tutor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM 2010), Atlanta, GA, USA, 10–15 April 2010; pp. 1613–1622. 6. Leite, I.; Martinho, C.; Paiva, A. Social robots for long-term interaction: A survey. Int. J. Soc. Robot. 2013, 5, 291–308. 7. Ahmad, M. I., Mubin, O., Orlando, J. A Systematic Review of Adaptivity in Human- Robot Interaction, Multimodal Technologies and Interaction, 2017, 3(14), 1-25 8. Kidd C. D. and Breazeal, C. “Robots at home: Understanding longterm human-robot inter- action,” in International Conference on Intelligent Robots and Systems. IEEE, 2008, pp. 3230–3235. 9. Short, E., Swift-Spong, K. Greczek, J. Ramachandran, A. Litoiu, A., Grigore, E. C. Feil- Seifer, D., Shuster, S., Lee, J. J., Huang phet, S. al., "How to train your dragonbot: Social- ly assistive robots for teaching children about nutrition through play," in The 23rd IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 2014, p. 924--929. 10. Nicklas, T. A., Baranowski, T., Cullen, K. W. and G. Berenson, “Eating patterns, dietary quality and obesity,” Journal of the American College of Nutrition, vol. 20, no. 6, pp. 599– 608, 2001. 11. Reedy, J. Krebs-Smith, S. M., Miller, P. E. , Liese, A. D. , Kahle, L. L., Park, Y. and Sub- ar, A. F. “Higher diet quality is associated with decreased risk of all-cause, cardiovascular disease, and cancer mortality among older adults,” The Journal of nutrition, vol. 144, no. 6, pp. 881–889, 2014. 12. Faccio, E., Centomo, C., & Mininni, G. “Measuring up to Measure” Dysmorphophobia as a Language Game. Integrative Psychological and Behavioral Science, 45(3), 304, 2011. 13. Hingle, M. Macias-Navarro, L., Rezaimalek, A. and S. Going, “The use of technology to promote nutrition and physical activity behavior change in youth: A review,” The Re- search Dietetic Practice Group Digest, pp. 1–10, 2013. 14. Leyzberg, D., Spaulding, S., Toneva, M. and B. Scassellati, “The physical presence of a robot tutor increases cognitive learning gains,” 2012. 9 10 15. Li R, Serdula M, Bland S, et al. Trends in fruit and vegetable consumption among adults in 16 US states: behavioral risk factor surveillance system, 1990–1996. Am J Public Health 2000; 90:777–81. 16. de Rosis, F., Novielli, N., Carofiglio, V., Cavalluzzi, A., De Carolis, B.: User Modeling And Adaptation In Health Promotion Dialogs With An Animated Character. International Journal of Biomedical Informatics, 514–531 (2006) 17. De Carolis, B., Macchiarulo, N., & Palestra, G. (2019). Soft Biometrics for Social Adap- tive Robots. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (pp. 687-699). Springer, Cham. 18. Dantcheva, A., Velardo, C., D’angelo, A, JL Dugelay, Bag of soft biometrics for person identification: New trends and challenges, Multimedia Tools and Applications, 51(2):739- 777, (2011) 19. Sun, N., Zheng, W., Sun, C., Zou, C., & Zhao, L.. Gender classification based on boosting local binary pattern. In Advances in Neural Networks-ISNN 2006 (pp. 194-201). Springer Berlin Heidelberg (2006). 20. Lian, H. C., & Lu, B. L. Multi-view gender classification using local binary patterns and support vector machines. In Advances in Neural Networks-ISNN 2006 (pp. 202-209). Springer Berlin Heidelberg (2006). 21. Günay A, NabIyev VV. Automatic age classification with LBP. Proceeding of the 23rd In- ternational Symposium on Computer and Information Sciences (ISCIS ’08); October 2008; Istanbul, Turkey. pp. 1–4. 22. Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.J.: The feret database and evaluation pro- cedure for face-recognition algorithms. Image and vision computing 16(5), 295{306 (1998). 23. Simonyan K. and Zisserman, A. “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, (2014). 24. Yan, C. Lang, C., Wang, T. Du, X. and C. Zhang, “Age estimation based on convolutional neural network,” in Proceedings of the 15th Pacific-Rim Conference on Advances in Mul- timedia Information Processing — PCM 2014 - Volume 8879, (New York, NY, USA), pp. 211–220, Springer-Verlag New York, Inc., (2014). 25. Levi G. and Hassner, T. “Age and gender classification using convolutional neural net- works,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June (2015). 26. Rothe, R. Timofte, R. and Gool, L. V. “Dex: Deep expectation of apparent age from a sin- gle image,” in 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 252–257, Dec (2015). 27. Berardina De Carolis, Irene Mazzotta. A user-adaptive persuasive system based on 'a- rational' theory. Int. J. Hum. Comput. Stud. 108: 70-88. (2017) 28. Mininni, G., & Manuti, A. La costruzione discorsiva della cura di sé: culture a confronto sull'idea di benessere. Psicologia della salute. pp.101; (2004) 29. D’Errico, F., & Poggi, I. Tracking a leader’s humility and its emotions from body, face and voice. In Web Intelligence (Vol. 17, No. 1, pp. 63-74). IOS Press.(2009). 30. D'Errico, F., Paciello, M., & Cerniglia, L. (2016). When emotions enhance students’ en- gagement in e-learning processes. Journal of e-Learning and Knowledge Society, 12(4). 31. Castelfranchi, C., & Falcone, R. Trust theory: A socio-cognitive and computational model (Vol. 18). John Wiley & Sons (2010). 10