Exploring how users across cultures design and perceive multimodal robot emotion - Abstract

Abstract

As robots enter more human-centered spaces, such as homes, and engage with more diverse populations, they will need to interact with people in a culturally appropriate manner. This interaction plays an important role in maintaining engagement over long periods of time to maximize eficacy for applications, such as delivering health interventions. In our work, we seek to understand how a user's cultural background influences how they design expressions to convey diferent emotions on robots, as well as how they perceive those emotions. We explore how cultural factors impact how people perceive robot emotions composed of diferent modalities, including sounds (verbal and non-verbal expressions) and color. Our proposed work will contribute towards design considerations to make robots more culturally sensitive and inclusive.

Keywords

• eol>Human-robot interaction
• Cross-cultural perception
• Multimodal robot expression

1. Introduction

Perception of robot expressions may vary widely across cultures and contexts [ 1, 2 ], as cultural values influence users’ perception, acceptance, and trust of robots [ 3 ]. However, inappropriate design of these expressions may have the potential to perpetuate cultural biases and stereotypes particularly if designers are not familiar with the culture of the intended end users [ 4 ]. Understanding these factors together may help reduce the perpetuation of cultural stereotypes and biases in robots while promoting social equity [ 3 ].

However, it is unclear how robots can leverage multiple modalities to most efectively convey emotion across cultures. The lack of universality of perceptions of robot emotion and social cues across cultures presents new design considerations for researchers seeking to increase the quality of human-robot interactions [ 5 ]. Furthermore, robot expressions are typically designed by roboticists rather than the intended end users of these systems, possibly leading to misalignments in an intended robot emotion and how users perceive them.

Our work explores synthesizing multimodal robot expressions, focusing on sound and color, which efectively communicate robot emotion [ 6 ]. We aim to identify how combining these modalities afect human perceptions of robot emotion across cultures and how these perceptions may impact design consideration for culturally aware robots, with the long-term goal of supporting autonomous personalization. We propose a mixed-measures study in which participants from various cultural backgrounds will design diferent robot expressions that they perceive to convey specific emotions. We will leverage an online tool for the Cognitively Assistive Robot for Motivation and Neurorehabilitation (CARMEN) [ 7 ] to enable participants to design personalized, multimodal robot expressions on a simulated robot (see Figure 1). We will evaluate how participants perceive expressions designed by other participants from diferent cultures.

We anticipate two main contributions from our proposed work. First, we will provide insight into how various modalities of expression and their combinations afect the perception of emotions and social behaviors across cultures. Second, we will propose design considerations for multimodal expression that researchers can leverage to make socially assistive robots more culturally sensitive and synthesize higher quality interactions between users and robots. Our research also extends to maintaining longitudinal engagement with robot-delivered interventions in the home, where multimodal robot expression may provide more quality and engaging interactions between users and robots across cultures [ 6, 8, 9, 10 ]. Ultimately, our work seeks to create more efective care that promotes inclusiveness across diferent cultures.

2. Related Work

2.1. Robot Emotion and Social Cues

The design of robot expressions can convey complex information to users such as diferent emotions or social cues [ 11, 6 ]. Researchers have found that robot expressions of emotion and social cues have a positive efect on user perception and how accurately they can be recognized [ 6, 1, 8, 5, 12, 13 ]. For instance, utilizing colored lights for expression significantly improves participants’ accuracy when identifying a robot’s internal state and improves trust towards a robot [ 13 ]. Other research on vocal expression revealed that intonation, pitch, and timbre are the primary sound parameters which impact its perception [ 14 ]. Nonverbal sounds may also convey important information to users about robot emotion or social cues [ 5, 15 ]. However, questions remain on how the combinations of these modalities impacts users’ perception of robot expressions across cultures. Thus, we plan to enable participants to design these expressions to better understand how cultural backgrounds afect the perception of robot emotion.

2.2. Cross-Cultural Emotion Perception

Researchers have identified that multimodal robot expression may provide more quality and engaging interactions between users and robots across cultures [ 6, 8, 9, 10 ]. Research on the cross-cultural impact of vocal expression shows that human emotion and social cue expressions manifest as subtle, nuanced patterns for expressing and perceiving such expressions [ 2 ]. A related study [16] provided various human vocal expressions of emotions (happiness, anger, fear, disgusted, sadness, and surprise) made in the US to a culturally diverse group of participants. They found that a wider gap between cultures led to decreased emotional recognition accuracy among the participants [16]. Other work focuses on how personalizing robots across cultures promotes acceptance of robots during human-robot interaction [17]. These studies highlight the importance of understanding how users’ cultural backgrounds influence their perception of robot expressions and how these expressions can be personalized across cultures.

3. Proposed Methodology

We plan to conduct an online mixed-methods study to identify how multimodal expression, through sound and color, impacts a user’s perception of robot emotion and social cues, and how these perceptions change across cultures. We will follow commonly used frameworks from psychology [18] and focus on both innate primary emotions (joy, sadness, anger, fear, and disgust) and acquired secondary emotions (guilt, regret, pride, and jealousy) [ 6 ]. We plan to recruit participants from the US and Mexico to design multimodal robot expressions and social cues to convey these emotions. These locations allow us to explore how cultural elements, such as expressiveness, communication, and attitudes towards technology, impact the design and perception of robot emotions. Furthermore, our research will be primarily conducted in California where there is a relatively high population of people of Mexican heritage. Participants will report the culture with which they self-identify.

We will use the CARMEN platform [ 19, 7 ], a cognitively assistive robot which supports flexible, expressive modalities that is designed to deliver longitudinal interventions at home. Participants can use CARMEN’s online interface to design their preferred robot expressions to convey emotions with an easy to use block programming system. We will provide participants with a brief tutorial on using the interface to design their own robot emotions in order to better understand how people from diferent cultural backgrounds perceive these emotions in robots. We will present participants with a predefined neutral robot expression, and they can adjust both the color and sounds in order to isolate the efects of these two modalities. Colored lights will be visible through the robot’s body to enable participants to personalize their design preferences. They can consider features such as frequency, light animation, hue, saturation, and brightness. For sound modalities, verbal and non-verbal sounds will be considered to maximize personalization options for participants where they can adjust features like intonation, pitch, and timbre.

After completing the design process, we will ask participants open-ended questions to understand the reasoning behind their choices, including what features they chose and why. We will conduct a thematic analysis of the qualitative data, evaluating how participants weigh each modality and their features, the efect of multimodal expression on emotion and social cue perception, and how this influences users’ perception of the conveyed robot emotion. We will also explore how cultural diferences afect users’ perception of the emotions they assign to diferent robot expressions. In order to understand users’ perception of the robot emotions, we will ask participants to identify the designs from both their own and the other culture with the emotion they perceive, allowing us to compare cross-cultural diferences in perception.

4. Future Work

This proposed work aims to identify how multimodal expressions of robot emotion and social cues are perceived by users, understand how users’ respective cultures afect their perception of robot expressions, and learn how these modalities can be combined to be more culturally aware. Multimodal expressions have a significantly positive impact on user engagement [ 20, 21] which may improve robot healthcare interventions deployed longitudinally in the home [ 10 ]. In future work, we will also explore these diferences among other cultures and how additional modalities, such as facial expressions, can produce more efective modality combinations to improve expression recognition accuracy across cultures [ 6 ]. With more modalities to consider, there is the possibility to expand this study to include more emotion and social cues, or even more complex behavior, such as creating more personalized robot personalities. Finally, we want to better understand the impact of these culturally influenced emotions on longitudinal engagement [20, 21], as well as ethical implications such as trust, attachment, and reliance on robots with these abilities [22, 23, 24]. The results of this work may allow researchers to automatically synthesize personalized behaviors based on a user’s cultural background. Our work will enable robots to interact with more culturally diverse populations and ultimately improve equity and accessibility of personalized systems. [16] P. Laukka, H. A. Elfenbein, Cross-cultural emotion recognition and in-group advantage in vocal expression: A meta-analysis, Emotion Review 13 (2021) 3–11. [17] N. Gasteiger, M. Hellou, H. S. Ahn, Factors for personalization and localization to optimize human–robot interaction: A literature review, International Journal of Social Robotics 15 (2023) 689–701. [18] N. Spatola, O. A. Wudarczyk, Ascribing emotions to robots: Explicit and implicit attribution of emotions and perceived robot anthropomorphism, Computers in Human Behavior 124 (2021) 106934. [19] A. Kubota, R. Pei, E. Sun, D. Cruz-Sandoval, S. Kim, L. D. Riek, Get smart: Collaborative goal setting with cognitively assistive robots, in: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, 2023, pp. 44–53. [20] H. Salam, O. Celiktutan, I. Hupont, H. Gunes, M. Chetouani, Fully automatic analysis of engagement and its relationship to personality in human-robot interactions, IEEE Access 5 (2016) 705–721. [21] F. Del Duchetto, P. Baxter, M. Hanheide, Are you still with me? continuous engagement assessment from a robot’s point of view, Frontiers in Robotics and AI 7 (2020) 116. [22] T. Law, M. Scheutz, Trust: Recent concepts and evaluations in human-robot interaction, Trust in

Human-Robot Interaction (2020) 27. [23] T. Sanders, A. Kaplan, R. Koch, M. Schwartz, P. A. Hancock, The relationship between trust and use choice in human-robot interaction, Human factors 61 (2019) 614–626. [24] A. Kubota, M. Pourebadi, S. Banh, S. Kim, L. Riek, Somebody that i used to know: The risks of personalizing robots for dementia care, Proceedings of We Robot (2021).

References

[1] K. Terada , A. Yamauchi , A. Ito , Artificial emotion expression for a robot by dynamic color change , in: 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication , IEEE, 2012 , pp. 314 - 321 .

[2] D. A. Sauter , F. Eisner , P. Ekman , S. K. Scott , Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations , Proceedings of the National Academy of Sciences 107 ( 2010 ) 2408 - 2412 .

[3] V. Lim , M. Rooksby , E. S. Cross , Social robots on a global stage: establishing a role for culture during human-robot interaction , International Journal of Social Robotics 13 ( 2021 ) 1307 - 1333 .

[4] L. Londoño , A. Röfer , T. Welschehold , A. Valada , Doing right by not doing wrong in human-robot collaboration , arXiv preprint arXiv:2202.02654 ( 2022 ).

[5] B. J. Zhang , N. T. Fitter, Nonverbal sound in human-robot interaction: a systematic review , ACM Transactions on Human-Robot Interaction 12 ( 2023 ) 1 - 46 .

[6] D. Löfler , N. Schmidt , R. Tscharn , Multimodal expression of artificial emotion in social robots using color, motion and sound , in: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction , 2018 , pp. 334 - 343 .

[7] A. Bouzida , A. Kubota , D. Cruz-Sandoval , E. W. Twamley , L. D. Riek , Carmen: A cognitively assistive robot for personalized neurorehabilitation at home , in: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , 2024 , pp. 55 - 64 .

[8] A. Pörtner , L. Schröder , R. Rasch , D. Sprute , M. Hofmann , M. König , The power of color: A study on the efective use of colored light in human-robot interaction , in: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , IEEE, 2018 , pp. 3395 - 3402 .

[9] H. Su , W. Qi , J. Chen , C. Yang , J. Sandoval , M. A. Laribi , Recent advancements in multimodal human-robot interaction , Frontiers in Neurorobotics 17 ( 2023 ) 1084000 .

[10] Y. Feng , G. Perugia, S. Yu , E. I. Barakova , J. Hu , G. M. Rauterberg , Context-enhanced human-robot interaction: exploring the role of system interactivity and multimodal stimuli on the engagement of people with dementia , International Journal of Social Robotics ( 2022 ) 1 - 20 .

[11] S. Habibian , A. A. Valdivia , L. H. Blumenschein , D. P. Losey , A review of communicating robot learning during human-robot interaction , arXiv preprint arXiv:2312.00948 ( 2023 ).

[12] S. Ko , J. Barnes , J. Dong , C. H. Park , A. Howard , M. Jeon , The efects of robot voices and appearances on users' emotion recognition and subjective perception , International Journal of Humanoid Robotics 20 ( 2023 ) 2350001 .

[13] K. Baraka , M. M. Veloso , Mobile service robot state revealing through expressive lights: formalism, design, and evaluation , International Journal of Social Robotics 10 ( 2018 ) 65 - 92 .

[14] E.-S. Jee , Y.-J. Jeong , C. H. Kim , H. Kobayashi , Sound design for emotion and intention expression of socially interactive robots , Intelligent Service Robotics 3 ( 2010 ) 199 - 206 .

[15] B. Orthmann , I. Leite , R. Bresin , I. Torre , Sounding robots: Design and evaluation of auditory displays for unintentional human-robot interaction , ACM Transactions on Human-Robot Interaction 12 ( 2023 ) 1 - 26 .