Exploring how users across cultures design and perceive multimodal robot emotion - Abstract Mathieu DePaul1,* , Dagoberto Cruz-Sandoval2 and Alyssa Kubota1 1 San Francisco State University, School of Engineering, 1600 Holloway Ave, San Francisco, CA 94132, USA 2 University of California San Diego, Computer Science and Engineering, 9500 Gilman Dr, La Jolla, CA 92093, USA Abstract As robots enter more human-centered spaces, such as homes, and engage with more diverse populations, they will need to interact with people in a culturally appropriate manner. This interaction plays an important role in maintaining engagement over long periods of time to maximize efficacy for applications, such as delivering health interventions. In our work, we seek to understand how a user’s cultural background influences how they design expressions to convey different emotions on robots, as well as how they perceive those emotions. We explore how cultural factors impact how people perceive robot emotions composed of different modalities, including sounds (verbal and non-verbal expressions) and color. Our proposed work will contribute towards design considerations to make robots more culturally sensitive and inclusive. Keywords Human-robot interaction, Cross-cultural perception, Multimodal robot expression 1. Introduction Perception of robot expressions may vary widely across cultures and contexts [1, 2], as cultural values influence users’ perception, acceptance, and trust of robots [3]. However, inappropriate design of these expressions may have the potential to perpetuate cultural biases and stereotypes particularly if designers are not familiar with the culture of the intended end users [4]. Understanding these factors together may help reduce the perpetuation of cultural stereotypes and biases in robots while promoting social equity [3]. However, it is unclear how robots can leverage multiple modalities to most effectively convey emotion across cultures. The lack of universality of perceptions of robot emotion and social cues across cultures presents new design considerations for researchers seeking to increase the quality of human-robot interactions [5]. Furthermore, robot expressions are typically designed by roboticists rather than the intended end users of these systems, possibly leading to misalignments in an intended robot emotion and how users perceive them. Our work explores synthesizing multimodal robot expressions, focusing on sound and color, which effectively communicate robot emotion [6]. We aim to identify how combining these modalities affect human perceptions of robot emotion across cultures and how these perceptions may impact design consideration for culturally aware robots, with the long-term goal of supporting autonomous personal- ization. We propose a mixed-measures study in which participants from various cultural backgrounds will design different robot expressions that they perceive to convey specific emotions. We will leverage an online tool for the Cognitively Assistive Robot for Motivation and Neurorehabilitation (CARMEN) [7] to enable participants to design personalized, multimodal robot expressions on a simulated robot (see Figure 1). We will evaluate how participants perceive expressions designed by other participants from different cultures. ALTRUIST, BAILAR, SCRITA, WARN 2024: Workshop on sociAL roboTs for peRsonalized, continUous and adaptIve aSsisTance, Workshop on Behavior Adaptation and Learning for Assistive Robotics, Workshop on Trust, Acceptance and Social Cues in Human-Robot Interaction, and Workshop on Weighing the benefits of Autonomous Robot persoNalisation. August 26, 2024, Pasadena, USA * Corresponding author. $ mdepaul1@sfsu.edu (M. DePaul); dcruzsandoval@ucsd.edu (D. Cruz-Sandoval); akubota@sfsu.edu (A. Kubota)  0000-0002-5936-7240 (D. Cruz-Sandoval) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 1 Mathieu DePaul et al. CEUR Workshop Proceedings 1–5 Figure 1: We will employ CARMEN, a cognitively assistive robot that leverages multiple modalities including sound, color, and movement, to communicate. We anticipate two main contributions from our proposed work. First, we will provide insight into how various modalities of expression and their combinations affect the perception of emotions and social behaviors across cultures. Second, we will propose design considerations for multimodal expression that researchers can leverage to make socially assistive robots more culturally sensitive and synthesize higher quality interactions between users and robots. Our research also extends to maintaining longitudinal engagement with robot-delivered interventions in the home, where multimodal robot expression may provide more quality and engaging interactions between users and robots across cultures [6, 8, 9, 10]. Ultimately, our work seeks to create more effective care that promotes inclusiveness across different cultures. 2. Related Work 2.1. Robot Emotion and Social Cues The design of robot expressions can convey complex information to users such as different emotions or social cues [11, 6]. Researchers have found that robot expressions of emotion and social cues have a positive effect on user perception and how accurately they can be recognized [6, 1, 8, 5, 12, 13]. For instance, utilizing colored lights for expression significantly improves participants’ accuracy when identifying a robot’s internal state and improves trust towards a robot [13]. Other research on vocal expression revealed that intonation, pitch, and timbre are the primary sound parameters which impact its perception [14]. Nonverbal sounds may also convey important information to users about robot emotion or social cues [5, 15]. However, questions remain on how the combinations of these modalities impacts users’ perception of robot expressions across cultures. Thus, we plan to enable participants to design these expressions to better understand how cultural backgrounds affect the perception of robot emotion. 2.2. Cross-Cultural Emotion Perception Researchers have identified that multimodal robot expression may provide more quality and engaging interactions between users and robots across cultures [6, 8, 9, 10]. Research on the cross-cultural impact 2 Mathieu DePaul et al. CEUR Workshop Proceedings 1–5 of vocal expression shows that human emotion and social cue expressions manifest as subtle, nuanced patterns for expressing and perceiving such expressions [2]. A related study [16] provided various human vocal expressions of emotions (happiness, anger, fear, disgusted, sadness, and surprise) made in the US to a culturally diverse group of participants. They found that a wider gap between cultures led to decreased emotional recognition accuracy among the participants [16]. Other work focuses on how personalizing robots across cultures promotes acceptance of robots during human-robot interaction [17]. These studies highlight the importance of understanding how users’ cultural backgrounds influence their perception of robot expressions and how these expressions can be personalized across cultures. 3. Proposed Methodology We plan to conduct an online mixed-methods study to identify how multimodal expression, through sound and color, impacts a user’s perception of robot emotion and social cues, and how these perceptions change across cultures. We will follow commonly used frameworks from psychology [18] and focus on both innate primary emotions (joy, sadness, anger, fear, and disgust) and acquired secondary emotions (guilt, regret, pride, and jealousy) [6]. We plan to recruit participants from the US and Mexico to design multimodal robot expressions and social cues to convey these emotions. These locations allow us to explore how cultural elements, such as expressiveness, communication, and attitudes towards technology, impact the design and perception of robot emotions. Furthermore, our research will be primarily conducted in California where there is a relatively high population of people of Mexican heritage. Participants will report the culture with which they self-identify. We will use the CARMEN platform [19, 7], a cognitively assistive robot which supports flexible, expressive modalities that is designed to deliver longitudinal interventions at home. Participants can use CARMEN’s online interface to design their preferred robot expressions to convey emotions with an easy to use block programming system. We will provide participants with a brief tutorial on using the interface to design their own robot emotions in order to better understand how people from different cultural backgrounds perceive these emotions in robots. We will present participants with a predefined neutral robot expression, and they can adjust both the color and sounds in order to isolate the effects of these two modalities. Colored lights will be visible through the robot’s body to enable participants to personalize their design preferences. They can consider features such as frequency, light animation, hue, saturation, and brightness. For sound modalities, verbal and non-verbal sounds will be considered to maximize personalization options for participants where they can adjust features like intonation, pitch, and timbre. After completing the design process, we will ask participants open-ended questions to understand the reasoning behind their choices, including what features they chose and why. We will conduct a thematic analysis of the qualitative data, evaluating how participants weigh each modality and their features, the effect of multimodal expression on emotion and social cue perception, and how this influences users’ perception of the conveyed robot emotion. We will also explore how cultural differences affect users’ perception of the emotions they assign to different robot expressions. In order to understand users’ perception of the robot emotions, we will ask participants to identify the designs from both their own and the other culture with the emotion they perceive, allowing us to compare cross-cultural differences in perception. 4. Future Work This proposed work aims to identify how multimodal expressions of robot emotion and social cues are perceived by users, understand how users’ respective cultures affect their perception of robot expressions, and learn how these modalities can be combined to be more culturally aware. Multimodal expressions have a significantly positive impact on user engagement [20, 21] which may improve robot healthcare interventions deployed longitudinally in the home [10]. In future work, we will also explore these differences among other cultures and how additional modalities, such as facial expressions, can 3 Mathieu DePaul et al. CEUR Workshop Proceedings 1–5 produce more effective modality combinations to improve expression recognition accuracy across cultures [6]. With more modalities to consider, there is the possibility to expand this study to include more emotion and social cues, or even more complex behavior, such as creating more personalized robot personalities. Finally, we want to better understand the impact of these culturally influenced emotions on longitudinal engagement [20, 21], as well as ethical implications such as trust, attachment, and reliance on robots with these abilities [22, 23, 24]. The results of this work may allow researchers to automatically synthesize personalized behaviors based on a user’s cultural background. Our work will enable robots to interact with more culturally diverse populations and ultimately improve equity and accessibility of personalized systems. References [1] K. Terada, A. Yamauchi, A. Ito, Artificial emotion expression for a robot by dynamic color change, in: 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, IEEE, 2012, pp. 314–321. [2] D. A. Sauter, F. Eisner, P. Ekman, S. K. Scott, Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations, Proceedings of the National Academy of Sciences 107 (2010) 2408–2412. [3] V. Lim, M. Rooksby, E. S. Cross, Social robots on a global stage: establishing a role for culture during human–robot interaction, International Journal of Social Robotics 13 (2021) 1307–1333. [4] L. Londoño, A. Röfer, T. Welschehold, A. Valada, Doing right by not doing wrong in human-robot collaboration, arXiv preprint arXiv:2202.02654 (2022). [5] B. J. Zhang, N. T. Fitter, Nonverbal sound in human-robot interaction: a systematic review, ACM Transactions on Human-Robot Interaction 12 (2023) 1–46. [6] D. Löffler, N. Schmidt, R. Tscharn, Multimodal expression of artificial emotion in social robots using color, motion and sound, in: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, 2018, pp. 334–343. [7] A. Bouzida, A. Kubota, D. Cruz-Sandoval, E. W. Twamley, L. D. Riek, Carmen: A cognitively assistive robot for personalized neurorehabilitation at home, in: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, 2024, pp. 55–64. [8] A. Pörtner, L. Schröder, R. Rasch, D. Sprute, M. Hoffmann, M. König, The power of color: A study on the effective use of colored light in human-robot interaction, in: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2018, pp. 3395–3402. [9] H. Su, W. Qi, J. Chen, C. Yang, J. Sandoval, M. A. Laribi, Recent advancements in multimodal human–robot interaction, Frontiers in Neurorobotics 17 (2023) 1084000. [10] Y. Feng, G. Perugia, S. Yu, E. I. Barakova, J. Hu, G. M. Rauterberg, Context-enhanced human-robot interaction: exploring the role of system interactivity and multimodal stimuli on the engagement of people with dementia, International Journal of Social Robotics (2022) 1–20. [11] S. Habibian, A. A. Valdivia, L. H. Blumenschein, D. P. Losey, A review of communicating robot learning during human-robot interaction, arXiv preprint arXiv:2312.00948 (2023). [12] S. Ko, J. Barnes, J. Dong, C. H. Park, A. Howard, M. Jeon, The effects of robot voices and appearances on users’ emotion recognition and subjective perception, International Journal of Humanoid Robotics 20 (2023) 2350001. [13] K. Baraka, M. M. Veloso, Mobile service robot state revealing through expressive lights: formalism, design, and evaluation, International Journal of Social Robotics 10 (2018) 65–92. [14] E.-S. Jee, Y.-J. Jeong, C. H. Kim, H. Kobayashi, Sound design for emotion and intention expression of socially interactive robots, Intelligent Service Robotics 3 (2010) 199–206. [15] B. Orthmann, I. Leite, R. Bresin, I. Torre, Sounding robots: Design and evaluation of auditory dis- plays for unintentional human-robot interaction, ACM Transactions on Human-Robot Interaction 12 (2023) 1–26. 4 Mathieu DePaul et al. CEUR Workshop Proceedings 1–5 [16] P. Laukka, H. A. Elfenbein, Cross-cultural emotion recognition and in-group advantage in vocal expression: A meta-analysis, Emotion Review 13 (2021) 3–11. [17] N. Gasteiger, M. Hellou, H. S. Ahn, Factors for personalization and localization to optimize human–robot interaction: A literature review, International Journal of Social Robotics 15 (2023) 689–701. [18] N. Spatola, O. A. Wudarczyk, Ascribing emotions to robots: Explicit and implicit attribution of emotions and perceived robot anthropomorphism, Computers in Human Behavior 124 (2021) 106934. [19] A. Kubota, R. Pei, E. Sun, D. Cruz-Sandoval, S. Kim, L. D. Riek, Get smart: Collaborative goal setting with cognitively assistive robots, in: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, 2023, pp. 44–53. [20] H. Salam, O. Celiktutan, I. Hupont, H. Gunes, M. Chetouani, Fully automatic analysis of engagement and its relationship to personality in human-robot interactions, IEEE Access 5 (2016) 705–721. [21] F. Del Duchetto, P. Baxter, M. Hanheide, Are you still with me? continuous engagement assessment from a robot’s point of view, Frontiers in Robotics and AI 7 (2020) 116. [22] T. Law, M. Scheutz, Trust: Recent concepts and evaluations in human-robot interaction, Trust in Human-Robot Interaction (2020) 27. [23] T. Sanders, A. Kaplan, R. Koch, M. Schwartz, P. A. Hancock, The relationship between trust and use choice in human-robot interaction, Human factors 61 (2019) 614–626. [24] A. Kubota, M. Pourebadi, S. Banh, S. Kim, L. Riek, Somebody that i used to know: The risks of personalizing robots for dementia care, Proceedings of We Robot (2021). 5