=Paper=
{{Paper
|id=Vol-3906/paper3_WARN
|storemode=property
|title=Exploring how users across cultures design and perceive multimodal robot emotion (short paper)
|pdfUrl=https://ceur-ws.org/Vol-3906/paper3_WARN.pdf
|volume=Vol-3906
|authors=Mathieu DePaul,Dagoberto Cruz-Sandoval,Alyssa Kubota
|dblpUrl=https://dblp.org/rec/conf/ro-man/DePaulCK24
}}
==Exploring how users across cultures design and perceive multimodal robot emotion (short paper)==
https://ceur-ws.org/Vol-3906/paper3_WARN.pdf
Exploring how users across cultures design and perceive
multimodal robot emotion - Abstract
Mathieu DePaul1,* , Dagoberto Cruz-Sandoval2 and Alyssa Kubota1
1
San Francisco State University, School of Engineering, 1600 Holloway Ave, San Francisco, CA 94132, USA
2
University of California San Diego, Computer Science and Engineering, 9500 Gilman Dr, La Jolla, CA 92093, USA
Abstract
As robots enter more human-centered spaces, such as homes, and engage with more diverse populations, they
will need to interact with people in a culturally appropriate manner. This interaction plays an important role in
maintaining engagement over long periods of time to maximize efficacy for applications, such as delivering health
interventions. In our work, we seek to understand how a user’s cultural background influences how they design
expressions to convey different emotions on robots, as well as how they perceive those emotions. We explore how
cultural factors impact how people perceive robot emotions composed of different modalities, including sounds
(verbal and non-verbal expressions) and color. Our proposed work will contribute towards design considerations
to make robots more culturally sensitive and inclusive.
Keywords
Human-robot interaction, Cross-cultural perception, Multimodal robot expression
1. Introduction
Perception of robot expressions may vary widely across cultures and contexts [1, 2], as cultural values
influence users’ perception, acceptance, and trust of robots [3]. However, inappropriate design of
these expressions may have the potential to perpetuate cultural biases and stereotypes particularly if
designers are not familiar with the culture of the intended end users [4]. Understanding these factors
together may help reduce the perpetuation of cultural stereotypes and biases in robots while promoting
social equity [3].
However, it is unclear how robots can leverage multiple modalities to most effectively convey emotion
across cultures. The lack of universality of perceptions of robot emotion and social cues across cultures
presents new design considerations for researchers seeking to increase the quality of human-robot
interactions [5]. Furthermore, robot expressions are typically designed by roboticists rather than the
intended end users of these systems, possibly leading to misalignments in an intended robot emotion
and how users perceive them.
Our work explores synthesizing multimodal robot expressions, focusing on sound and color, which
effectively communicate robot emotion [6]. We aim to identify how combining these modalities affect
human perceptions of robot emotion across cultures and how these perceptions may impact design
consideration for culturally aware robots, with the long-term goal of supporting autonomous personal-
ization. We propose a mixed-measures study in which participants from various cultural backgrounds
will design different robot expressions that they perceive to convey specific emotions. We will leverage
an online tool for the Cognitively Assistive Robot for Motivation and Neurorehabilitation (CARMEN)
[7] to enable participants to design personalized, multimodal robot expressions on a simulated robot
(see Figure 1). We will evaluate how participants perceive expressions designed by other participants
from different cultures.
ALTRUIST, BAILAR, SCRITA, WARN 2024: Workshop on sociAL roboTs for peRsonalized, continUous and adaptIve aSsisTance,
Workshop on Behavior Adaptation and Learning for Assistive Robotics, Workshop on Trust, Acceptance and Social Cues in
Human-Robot Interaction, and Workshop on Weighing the benefits of Autonomous Robot persoNalisation. August 26, 2024,
Pasadena, USA
*
Corresponding author.
$ mdepaul1@sfsu.edu (M. DePaul); dcruzsandoval@ucsd.edu (D. Cruz-Sandoval); akubota@sfsu.edu (A. Kubota)
0000-0002-5936-7240 (D. Cruz-Sandoval)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
1
Mathieu DePaul et al. CEUR Workshop Proceedings 1–5
Figure 1: We will employ CARMEN, a cognitively assistive robot that leverages multiple modalities including
sound, color, and movement, to communicate.
We anticipate two main contributions from our proposed work. First, we will provide insight into how
various modalities of expression and their combinations affect the perception of emotions and social
behaviors across cultures. Second, we will propose design considerations for multimodal expression that
researchers can leverage to make socially assistive robots more culturally sensitive and synthesize higher
quality interactions between users and robots. Our research also extends to maintaining longitudinal
engagement with robot-delivered interventions in the home, where multimodal robot expression may
provide more quality and engaging interactions between users and robots across cultures [6, 8, 9, 10].
Ultimately, our work seeks to create more effective care that promotes inclusiveness across different
cultures.
2. Related Work
2.1. Robot Emotion and Social Cues
The design of robot expressions can convey complex information to users such as different emotions or
social cues [11, 6]. Researchers have found that robot expressions of emotion and social cues have a
positive effect on user perception and how accurately they can be recognized [6, 1, 8, 5, 12, 13]. For
instance, utilizing colored lights for expression significantly improves participants’ accuracy when
identifying a robot’s internal state and improves trust towards a robot [13]. Other research on vocal
expression revealed that intonation, pitch, and timbre are the primary sound parameters which impact
its perception [14]. Nonverbal sounds may also convey important information to users about robot
emotion or social cues [5, 15]. However, questions remain on how the combinations of these modalities
impacts users’ perception of robot expressions across cultures. Thus, we plan to enable participants to
design these expressions to better understand how cultural backgrounds affect the perception of robot
emotion.
2.2. Cross-Cultural Emotion Perception
Researchers have identified that multimodal robot expression may provide more quality and engaging
interactions between users and robots across cultures [6, 8, 9, 10]. Research on the cross-cultural impact
2
Mathieu DePaul et al. CEUR Workshop Proceedings 1–5
of vocal expression shows that human emotion and social cue expressions manifest as subtle, nuanced
patterns for expressing and perceiving such expressions [2]. A related study [16] provided various
human vocal expressions of emotions (happiness, anger, fear, disgusted, sadness, and surprise) made in
the US to a culturally diverse group of participants. They found that a wider gap between cultures led
to decreased emotional recognition accuracy among the participants [16]. Other work focuses on how
personalizing robots across cultures promotes acceptance of robots during human-robot interaction [17].
These studies highlight the importance of understanding how users’ cultural backgrounds influence
their perception of robot expressions and how these expressions can be personalized across cultures.
3. Proposed Methodology
We plan to conduct an online mixed-methods study to identify how multimodal expression, through
sound and color, impacts a user’s perception of robot emotion and social cues, and how these perceptions
change across cultures. We will follow commonly used frameworks from psychology [18] and focus on
both innate primary emotions (joy, sadness, anger, fear, and disgust) and acquired secondary emotions
(guilt, regret, pride, and jealousy) [6]. We plan to recruit participants from the US and Mexico to
design multimodal robot expressions and social cues to convey these emotions. These locations allow
us to explore how cultural elements, such as expressiveness, communication, and attitudes towards
technology, impact the design and perception of robot emotions. Furthermore, our research will be
primarily conducted in California where there is a relatively high population of people of Mexican
heritage. Participants will report the culture with which they self-identify.
We will use the CARMEN platform [19, 7], a cognitively assistive robot which supports flexible,
expressive modalities that is designed to deliver longitudinal interventions at home. Participants can
use CARMEN’s online interface to design their preferred robot expressions to convey emotions with an
easy to use block programming system. We will provide participants with a brief tutorial on using the
interface to design their own robot emotions in order to better understand how people from different
cultural backgrounds perceive these emotions in robots. We will present participants with a predefined
neutral robot expression, and they can adjust both the color and sounds in order to isolate the effects of
these two modalities. Colored lights will be visible through the robot’s body to enable participants to
personalize their design preferences. They can consider features such as frequency, light animation,
hue, saturation, and brightness. For sound modalities, verbal and non-verbal sounds will be considered
to maximize personalization options for participants where they can adjust features like intonation,
pitch, and timbre.
After completing the design process, we will ask participants open-ended questions to understand the
reasoning behind their choices, including what features they chose and why. We will conduct a thematic
analysis of the qualitative data, evaluating how participants weigh each modality and their features, the
effect of multimodal expression on emotion and social cue perception, and how this influences users’
perception of the conveyed robot emotion. We will also explore how cultural differences affect users’
perception of the emotions they assign to different robot expressions. In order to understand users’
perception of the robot emotions, we will ask participants to identify the designs from both their own
and the other culture with the emotion they perceive, allowing us to compare cross-cultural differences
in perception.
4. Future Work
This proposed work aims to identify how multimodal expressions of robot emotion and social cues
are perceived by users, understand how users’ respective cultures affect their perception of robot
expressions, and learn how these modalities can be combined to be more culturally aware. Multimodal
expressions have a significantly positive impact on user engagement [20, 21] which may improve robot
healthcare interventions deployed longitudinally in the home [10]. In future work, we will also explore
these differences among other cultures and how additional modalities, such as facial expressions, can
3
Mathieu DePaul et al. CEUR Workshop Proceedings 1–5
produce more effective modality combinations to improve expression recognition accuracy across
cultures [6]. With more modalities to consider, there is the possibility to expand this study to include
more emotion and social cues, or even more complex behavior, such as creating more personalized
robot personalities. Finally, we want to better understand the impact of these culturally influenced
emotions on longitudinal engagement [20, 21], as well as ethical implications such as trust, attachment,
and reliance on robots with these abilities [22, 23, 24]. The results of this work may allow researchers
to automatically synthesize personalized behaviors based on a user’s cultural background. Our work
will enable robots to interact with more culturally diverse populations and ultimately improve equity
and accessibility of personalized systems.
References
[1] K. Terada, A. Yamauchi, A. Ito, Artificial emotion expression for a robot by dynamic color change,
in: 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive
Communication, IEEE, 2012, pp. 314–321.
[2] D. A. Sauter, F. Eisner, P. Ekman, S. K. Scott, Cross-cultural recognition of basic emotions through
nonverbal emotional vocalizations, Proceedings of the National Academy of Sciences 107 (2010)
2408–2412.
[3] V. Lim, M. Rooksby, E. S. Cross, Social robots on a global stage: establishing a role for culture
during human–robot interaction, International Journal of Social Robotics 13 (2021) 1307–1333.
[4] L. Londoño, A. Röfer, T. Welschehold, A. Valada, Doing right by not doing wrong in human-robot
collaboration, arXiv preprint arXiv:2202.02654 (2022).
[5] B. J. Zhang, N. T. Fitter, Nonverbal sound in human-robot interaction: a systematic review, ACM
Transactions on Human-Robot Interaction 12 (2023) 1–46.
[6] D. Löffler, N. Schmidt, R. Tscharn, Multimodal expression of artificial emotion in social robots
using color, motion and sound, in: Proceedings of the 2018 ACM/IEEE International Conference
on Human-Robot Interaction, 2018, pp. 334–343.
[7] A. Bouzida, A. Kubota, D. Cruz-Sandoval, E. W. Twamley, L. D. Riek, Carmen: A cognitively
assistive robot for personalized neurorehabilitation at home, in: Proceedings of the 2024 ACM/IEEE
International Conference on Human-Robot Interaction, 2024, pp. 55–64.
[8] A. Pörtner, L. Schröder, R. Rasch, D. Sprute, M. Hoffmann, M. König, The power of color: A study
on the effective use of colored light in human-robot interaction, in: 2018 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), IEEE, 2018, pp. 3395–3402.
[9] H. Su, W. Qi, J. Chen, C. Yang, J. Sandoval, M. A. Laribi, Recent advancements in multimodal
human–robot interaction, Frontiers in Neurorobotics 17 (2023) 1084000.
[10] Y. Feng, G. Perugia, S. Yu, E. I. Barakova, J. Hu, G. M. Rauterberg, Context-enhanced human-robot
interaction: exploring the role of system interactivity and multimodal stimuli on the engagement
of people with dementia, International Journal of Social Robotics (2022) 1–20.
[11] S. Habibian, A. A. Valdivia, L. H. Blumenschein, D. P. Losey, A review of communicating robot
learning during human-robot interaction, arXiv preprint arXiv:2312.00948 (2023).
[12] S. Ko, J. Barnes, J. Dong, C. H. Park, A. Howard, M. Jeon, The effects of robot voices and appearances
on users’ emotion recognition and subjective perception, International Journal of Humanoid
Robotics 20 (2023) 2350001.
[13] K. Baraka, M. M. Veloso, Mobile service robot state revealing through expressive lights: formalism,
design, and evaluation, International Journal of Social Robotics 10 (2018) 65–92.
[14] E.-S. Jee, Y.-J. Jeong, C. H. Kim, H. Kobayashi, Sound design for emotion and intention expression
of socially interactive robots, Intelligent Service Robotics 3 (2010) 199–206.
[15] B. Orthmann, I. Leite, R. Bresin, I. Torre, Sounding robots: Design and evaluation of auditory dis-
plays for unintentional human-robot interaction, ACM Transactions on Human-Robot Interaction
12 (2023) 1–26.
4
Mathieu DePaul et al. CEUR Workshop Proceedings 1–5
[16] P. Laukka, H. A. Elfenbein, Cross-cultural emotion recognition and in-group advantage in vocal
expression: A meta-analysis, Emotion Review 13 (2021) 3–11.
[17] N. Gasteiger, M. Hellou, H. S. Ahn, Factors for personalization and localization to optimize
human–robot interaction: A literature review, International Journal of Social Robotics 15 (2023)
689–701.
[18] N. Spatola, O. A. Wudarczyk, Ascribing emotions to robots: Explicit and implicit attribution of
emotions and perceived robot anthropomorphism, Computers in Human Behavior 124 (2021)
106934.
[19] A. Kubota, R. Pei, E. Sun, D. Cruz-Sandoval, S. Kim, L. D. Riek, Get smart: Collaborative goal
setting with cognitively assistive robots, in: Proceedings of the 2023 ACM/IEEE International
Conference on Human-Robot Interaction, 2023, pp. 44–53.
[20] H. Salam, O. Celiktutan, I. Hupont, H. Gunes, M. Chetouani, Fully automatic analysis of engagement
and its relationship to personality in human-robot interactions, IEEE Access 5 (2016) 705–721.
[21] F. Del Duchetto, P. Baxter, M. Hanheide, Are you still with me? continuous engagement assessment
from a robot’s point of view, Frontiers in Robotics and AI 7 (2020) 116.
[22] T. Law, M. Scheutz, Trust: Recent concepts and evaluations in human-robot interaction, Trust in
Human-Robot Interaction (2020) 27.
[23] T. Sanders, A. Kaplan, R. Koch, M. Schwartz, P. A. Hancock, The relationship between trust and
use choice in human-robot interaction, Human factors 61 (2019) 614–626.
[24] A. Kubota, M. Pourebadi, S. Banh, S. Kim, L. Riek, Somebody that i used to know: The risks of
personalizing robots for dementia care, Proceedings of We Robot (2021).
5