Perception of robot expressions may vary widely across cultures and contexts [1,2], as cultural values influence users' perception, acceptance, and trust of robots [3]. However, inappropriate design of these expressions may have the potential to perpetuate cultural biases and stereotypes particularly if designers are not familiar with the culture of the intended end users [4]. Understanding these factors together may help reduce the perpetuation of cultural stereotypes and biases in robots while promoting social equity [3].
However, it is unclear how robots can leverage multiple modalities to most effectively convey emotion across cultures. The lack of universality of perceptions of robot emotion and social cues across cultures presents new design considerations for researchers seeking to increase the quality of human-robot interactions [5]. Furthermore, robot expressions are typically designed by roboticists rather than the intended end users of these systems, possibly leading to misalignments in an intended robot emotion and how users perceive them.
Our work explores synthesizing multimodal robot expressions, focusing on sound and color, which effectively communicate robot emotion [6]. We aim to identify how combining these modalities affect human perceptions of robot emotion across cultures and how these perceptions may impact design consideration for culturally aware robots, with the long-term goal of supporting autonomous personalization. We propose a mixed-measures study in which participants from various cultural backgrounds will design different robot expressions that they perceive to convey specific emotions. We will leverage an online tool for the Cognitively Assistive Robot for Motivation and Neurorehabilitation (CARMEN) [7] to enable participants to design personalized, multimodal robot expressions on a simulated robot (see Figure 1). We will evaluate how participants perceive expressions designed by other participants from different cultures. We anticipate two main contributions from our proposed work. First, we will provide insight into how various modalities of expression and their combinations affect the perception of emotions and social behaviors across cultures. Second, we will propose design considerations for multimodal expression that researchers can leverage to make socially assistive robots more culturally sensitive and synthesize higher quality interactions between users and robots. Our research also extends to maintaining longitudinal engagement with robot-delivered interventions in the home, where multimodal robot expression may provide more quality and engaging interactions between users and robots across cultures [6,8,9,10]. Ultimately, our work seeks to create more effective care that promotes inclusiveness across different cultures.
The design of robot expressions can convey complex information to users such as different emotions or social cues [11,6]. Researchers have found that robot expressions of emotion and social cues have a positive effect on user perception and how accurately they can be recognized [6,1,8,5,12,13]. For instance, utilizing colored lights for expression significantly improves participants' accuracy when identifying a robot's internal state and improves trust towards a robot [13]. Other research on vocal expression revealed that intonation, pitch, and timbre are the primary sound parameters which impact its perception [14]. Nonverbal sounds may also convey important information to users about robot emotion or social cues [5,15]. However, questions remain on how the combinations of these modalities impacts users' perception of robot expressions across cultures. Thus, we plan to enable participants to design these expressions to better understand how cultural backgrounds affect the perception of robot emotion.
Researchers have identified that multimodal robot expression may provide more quality and engaging interactions between users and robots across cultures [6,8,9,10]. Research on the cross-cultural impact of vocal expression shows that human emotion and social cue expressions manifest as subtle, nuanced patterns for expressing and perceiving such expressions [2]. A related study [16] provided various human vocal expressions of emotions (happiness, anger, fear, disgusted, sadness, and surprise) made in the US to a culturally diverse group of participants. They found that a wider gap between cultures led to decreased emotional recognition accuracy among the participants [16]. Other work focuses on how personalizing robots across cultures promotes acceptance of robots during human-robot interaction [17]. These studies highlight the importance of understanding how users' cultural backgrounds influence their perception of robot expressions and how these expressions can be personalized across cultures.
We plan to conduct an online mixed-methods study to identify how multimodal expression, through sound and color, impacts a user's perception of robot emotion and social cues, and how these perceptions change across cultures. We will follow commonly used frameworks from psychology [18] and focus on both innate primary emotions (joy, sadness, anger, fear, and disgust) and acquired secondary emotions (guilt, regret, pride, and jealousy) [6]. We plan to recruit participants from the US and Mexico to design multimodal robot expressions and social cues to convey these emotions. These locations allow us to explore how cultural elements, such as expressiveness, communication, and attitudes towards technology, impact the design and perception of robot emotions. Furthermore, our research will be primarily conducted in California where there is a relatively high population of people of Mexican heritage. Participants will report the culture with which they self-identify.
We will use the CARMEN platform [19,7], a cognitively assistive robot which supports flexible, expressive modalities that is designed to deliver longitudinal interventions at home. Participants can use CARMEN's online interface to design their preferred robot expressions to convey emotions with an easy to use block programming system. We will provide participants with a brief tutorial on using the interface to design their own robot emotions in order to better understand how people from different cultural backgrounds perceive these emotions in robots. We will present participants with a predefined neutral robot expression, and they can adjust both the color and sounds in order to isolate the effects of these two modalities. Colored lights will be visible through the robot's body to enable participants to personalize their design preferences. They can consider features such as frequency, light animation, hue, saturation, and brightness. For sound modalities, verbal and non-verbal sounds will be considered to maximize personalization options for participants where they can adjust features like intonation, pitch, and timbre.
After completing the design process, we will ask participants open-ended questions to understand the reasoning behind their choices, including what features they chose and why. We will conduct a thematic analysis of the qualitative data, evaluating how participants weigh each modality and their features, the effect of multimodal expression on emotion and social cue perception, and how this influences users' perception of the conveyed robot emotion. We will also explore how cultural differences affect users' perception of the emotions they assign to different robot expressions. In order to understand users' perception of the robot emotions, we will ask participants to identify the designs from both their own and the other culture with the emotion they perceive, allowing us to compare cross-cultural differences in perception.
This proposed work aims to identify how multimodal expressions of robot emotion and social cues are perceived by users, understand how users' respective cultures affect their perception of robot expressions, and learn how these modalities can be combined to be more culturally aware. Multimodal expressions have a significantly positive impact on user engagement [20,21] which may improve robot healthcare interventions deployed longitudinally in the home [10]. In future work, we will also explore these differences among other cultures and how additional modalities, such as facial expressions, can produce more effective modality combinations to improve expression recognition accuracy across cultures [6]. With more modalities to consider, there is the possibility to expand this study to include more emotion and social cues, or even more complex behavior, such as creating more personalized robot personalities. Finally, we want to better understand the impact of these culturally influenced emotions on longitudinal engagement [20,21], as well as ethical implications such as trust, attachment, and reliance on robots with these abilities [22,23,24]. The results of this work may allow researchers to automatically synthesize personalized behaviors based on a user's cultural background. Our work will enable robots to interact with more culturally diverse populations and ultimately improve equity and accessibility of personalized systems.