There is a growing interest in employing embodied agents to achieve beneficial outcomes for users, such as improving health, or increasing motivation for learning. The goal of our research is to explore how and to what extent embodied agents can provide social support to victims of cyberbullying. To this end, we implemented a proof of concept virtual buddy that uses verbal and nonverbal behavior to comfort users. This paper presents the results of a study into the ee↵ct of variations in the virtual buddy's emotional expressiveness (no emotion, verbal emotion only, nonverbal emotion only, or verbal & nonverbal emotion) on user experience, the ee↵ctiveness of the support, and perceived social support. The results show that the virtual buddy is successful at conveying support. However, we found no statistically significant die↵rences between conditions.
Introduction Increasingly, embodied agents and robots are being employed to achieve certain e↵ects in users, such as increasing exercise behavior [4], and increasing engagement in a virtual learning system [7]. In order to be able to achieve the beneficial outcomes these companion, coaching and pedagogical agents aim for, they need to behave as social actors. Social actors display and, to some extent, recognize social cues, and show appropriate verbal and nonverbal behavior [12].
The goal of our research is to understand how ECAs can provide social support. Social support refers to communicative attempts to alleviate the emotional distress of another person [5]. We are particularly interested in endowing ECAs with the emotional skills required to comfort users. To this end, we implemented an empathic virtual buddy that uses verbal and nonverbal strategies employed by people to comfort others. In order to be able to provide social support, a context of emotional distress is required. The application domain of the virtual buddy is cyberbullying, that is, bullying through electronic communication devices. Research shows that cyberbullying has a high impact on victims [9], making it a suitable test environment for the virtual buddy. We would like to emphasize that our research is focused on designing supportive interactions between ECAs and users. Our research objective does not include evaluating the buddy’s suitability or e↵ectiveness as a tool against cyberbullying.
The goals of the study presented in this paper are 1) to get more insight into how social support can be conveyed by conversational agents, and 2) to measure the user experience of the virtual buddy proof of concept system. User experience refers to “a person’s perceptions and responses that result from the use or anticipated use of a product, system or service” [1]. Poorly designed user interfaces may cause confusion and frustration [3]. These negative emotions may block the positive emotions the virtual buddy aims to evoke. Therefore, we assume that an acceptable level of user experience is required for a user to experience and be able to benefit from the social support communicated by the virtual buddy.
This paper is organized as follows. The next section describes the virtual buddy proof of concept system. In section 3, we explain the online survey used to conduct the study. The results are presented in section 4. In section 5 the results are discussed. Section 6 reviews related work on embodied agents. Finally, in section 7, we present our conclusions. 2
The Virtual Buddy
The emotion model determines when the virtual buddy expresses sympathy, compliments or encourages the user. It is based on the OCC model of emotions [10]. In OCC, emotions are conceptualized as responses to events, agents, and objects. The OCC model specifies eliciting conditions for all emotion types. The virtual buddy’s emotion model is depicted in figure 3. In the model, response options are interpreted as actions or events. An action or event triggers an OCC emotion type, that is expressed both verbally and nonverbally. In the current implementation, the buddy’s emotional state ranges from sad to happy. Figure 3 shows the facial expressions the virtual buddy displays for each emotional state it is capable of expressing (left to right: sadness, medium sadness, neutral, medium happiness, happiness). If a response option triggers a negative emotion, the buddy displays sadness and provides a sympathetic remark, and if a response option triggers a positive emotion, the buddy displays happiness and either provides a sympathetic remark, encourages, or compliments the user. What supportive strategy is used, depends on the response option selected; for example, if a response option refers to a praiseworthy action performed by the user, the buddy compliments the user.
Not all response options trigger emotions. If a response option does not trigger an emotion, the current emotional state is decayed to neutral (sadness to medium sadness, and medium sadness to neutral). Next, the buddy’s facial expression is updated to reflect the current emotional state. When uttering nonemotional messages, the buddy’s emotional state also decays to neutral.
In addition to expressing sympathy, encouraging, and complimenting the user, the virtual buddy also gives advice and explains how to execute that advice (teaching).
Method The goal of this study is to explore to what extent verbal and/or nonverbal expression of emotions contributes to the perceived e↵ectiveness of the support provided by the virtual buddy and how these variations in emotional expressiveness a↵ect user perceptions of social support. Additionally, since we assume that an acceptable level of user experience is required to be able benefit from interaction with the virtual buddy, a secondary goal of this study was to measure the user experience of the virtual buddy system.
For the experiment, the virtual buddy was embedded in an online survey. It had four modes of behavior, corresponding to four experimental conditions: 1) the buddy did not express emotions (control condition; No-EM), 2) the buddy expressed emotions by changing its facial expression (nonverbal condition; NVEM), 3) the buddy expressed emotions verbally (verbal condition; V-EM), and the buddy expressed emotions both verbally and nonverbally (verbal and nonverbal condition; NV&V-EM). The virtual buddy’s embodiment was displayed in all conditions. The experiment was set up using a between subjects design; participants were randomly assigned to one of the four conditions.
Before involving the virtual buddy’s actual target audience (i.e., children aged 10–14), we decided to perform an experiment with university students. Participants were recruited by e-mail and through social media. The survey was completed by 100 students from di↵erent universities in the Netherlands. There were 25 participants in each condition. Of the 100 participants, 32% were female; the average age was 19.5 (SD=2.0).
Interaction with the virtual buddy was based on a fictitious scenario. The scenario tells the story of Tom, a 14-year-old boy that is verbally abused and threatened by a classmate. In the scenario, the buddy is introduced as a computer program that provides support to cyberbullying victims Tom found online. Participants were asked to take Tom’s perspective during the interaction.
To capture di↵erent aspects of interacting with the virtual buddy and its supportive capacities several measures were included in the survey: – User Experience: User experience was measured by the AttrakDi↵ 2 questionnaire [6]. AttrakDi↵ consists of four scales: Pragmatic Quality (PQ), Hedonic Quality-Identity (HQI), Hedonic Quality-Stimulation (HQS), and Attractiveness (ATT). Each scale consists of 7 semantic di↵erentials on a 7point scale. PQ refers to the utility and usability of products. HQI refers to the identity that is communicated by using certain products. HQS refers to personal development (e.g., development of new skills) triggered by stimulating products. ATT refers to the overall evaluation of the perceived qualities of a product. – E↵ectiveness of the Support : Participants were asked to indicate on a 9-point scale how they think Tom feels (well-being; 1=feeling bad, 9=feeling good) and how severe they think Tom’s problem is (perceived burden of the problem; 1=the problem is not severe, 9=the problem is severe) prior to interacting with the virtual buddy and after the conversation is completed. – Social Support: Users’ perception of social support was measured using a questionnaire containing 7 Likert items on a a 7-point scale (1 = completely disagree and 7 = completely agree). The questionnaire is listed in table 1. – Open Feedback: Participants were asked How can we improve the emotional support provided by Robin? and Do you have other suggestions to improve Robin? We examined whether the buddy’s emotional expressiveness (no emotion, verbal emotion only, nonverbal emotion only, or verbal & nonverbal emotion) a↵ected participants’ user experience, the e↵ectiveness of the support, and/or perceived social support. User experience was measured by the AttrakDi↵ 2 questionnaire that consists of four scales: Pragmatic Quality (PQ), Hedonic Quality-Identity (HQI), Hedonic Quality-Stimulation (HQS), and Attractiveness (ATT). Figure 4 shows the average scores of PQ, HQI, HQS, and ATT for each condition. The average scores of HQI and HQS are close to 4 (the ‘neutral’ score); 4.47 < HQI < 4.61 and 4.15 < HQS < 4.39. PQ and ATT are slightly higher; 5.16 < P Q < 5.33 , and 4.99 < AT T < 5.25. We conclude that the user experience provided by the virtual buddy is acceptable and does not hamper the provision of social support.
Oneway between subjects ANOVA was conducted to compare the e↵ects of
variations in the virtual buddy’s emotional expressiveness on PQ, HQI, HQS,
and ATT. There were no statistically significant di↵erences between the four
conditions; PQ F (
No-EM NV-EM
V-EM
NV&V-EM PQ
HQI HQS ATT
A mixed between-within subjects ANOVA was conducted to assess the impact
of four levels of emotional expressiveness of the virtual buddy on participants’
scores for well-being and perceived burden of the problem before interacting with
the buddy and after interacting with the buddy. The results for well-being and
perceived burden of the problem were similar. There were no significant
interactions between emotional expressiveness and well-being, or between emotional
expressiveness and perceived burden over time; F (
The average perceived social support scores were generally high, especially for for items referring to information support (Advice general, Advice situation, and Persuasion); 5.6 < average scores < 6.4. In contrast, social support ratings for emotional support (Understood emotions, and Compassion) were lowest; 4.2 < average scores < 5.2. These results raise the question to what extent expressing emotions contributes to or is required for users’ perception of social support. Support a ttempt support prob rceivUendderstoUondderstood lem emo tions
Com pa ssion eral
ice gen
Adv Advice 9 8 n7 e rd6 u b5 d e4 v i c3 e r e P2 1 4.4 At the end of the survey, participants were invited to suggest improvements for emotional support and other improvements. In total, 93 of the 100 participants provided one or more remarks. Many participants came up with concrete suggestions on how to improve the experience of emotional support. These suggestions are listed in table 2 together with the number of participants from each condition that made them.
Half of the participants in the no emotion condition that left feedback (12 of 25 participants) suggested to add supportive verbal utterances to the conversation. As formulated by one of the participants in the No-EM condition: In addition to suggesting a practical solution, Robin should show compassion and say nice things that may not directly resolve the situation, but give the impression that Robin is sympathetic and cares about the fact that its conversation partner is being bullied. (P47) Also, 6 participants in the nonverbal emotion only recognized verbal support was missing and suggested to include supportive remarks. Additionally, 3 of 25 participants in the nonverbal and verbal emotion condition suggested to add more supportive verbal expressions. Remarkably, while many participants in the no emotion condition recognized verbal support was missing, this did not lead to significant di↵erences in perceived social support scores between the di↵erent conditions (see figure 6).
Add verbal expressions Add facial expressions Facial expression mismatch Inappropriate verbal expressions Add other support types
Five participants in the verbal emotion only condition suggested to have the virtual buddy change its facial expression during the conversation. Three participants, one in the nonverbal emotions condition and two in the verbal & nonverbal emotions condition, noticed emotion mismatches. For example, one participant thought Robin’s neutral expression was too cheerful: Robin should look less happy; he was smiling when I told my story. That’s rather tactless. (P68; ) In total, nine participants stated that they felt discouraged by some of the messages conveyed by the virtual buddy. The large number of comments that suggest to increase the virtual buddy’s emotional expressiveness indicate that emotional expressiveness is an important factor in the perception of support, even though this is not reflected in the social support scores.
Participants from all conditions suggested other types of support should be added to the conversation, such as explaining why bullies bully, that bullies sometimes randomly select a victim, that Tom is a good person despite what other people say, and that bullying can only be stopped by taking action.
Table 3 lists participants’ feedback on the virtual buddy’s technical limitations. As these limitations were the same in each condition, we only report the total number of participants that made some remark.
Remark Negative about interface design Positive about interface design Negative about appearance of the virtual character More human-like system Typing instead of response options More response options Select multiple response options # participants 8 2 13 13 6 7 11
Eight participants expressed dissatisfaction with the design of the interface, while two participants were positive about the design. Thirteen participants criticized the virtual character’s appearance; they thought it was too robot-like, and/or static. In addition, thirteen participants suggested to make the system (and not just the virtual character) more human-like.
Another recurring topic in the feedback were the response options. Six participants asked for the possibility to type responses instead of selecting them. Seven participants wanted to more response options to choose from. Finally, eleven participants wanted to be able to select multiple response options instead of just one.
Many participants suggested to improve the experience of emotional support by increasing the virtual buddy’s emotional expressiveness. In the verbal and nonverbal emotions condition, the condition in which participants interacted with the most emotionally expressive buddy, there also were participants that suggested to increase the amount of emotional feedback. Additionally, the technical limitations identified by the participants suggest that the system used in the experiment may have been too limited. Even though shortcomings in the virtual buddy’s behavior were recognized by many participants, this did not result in lower social support ratings.
While our study demonstrated that the virtual buddy is able to comfort users, we found no significant di↵erences between the four conditions in user experience, e↵ectiveness of the support, and perceived social support. Additionally, the average perceived social support ratings were relatively high (> 4.24). In this section, we explore explanations for the lack of significant di↵erences between conditions and the high social support ratings.
Nass and Reeves’ media equation states that people apply social rules from human-human interaction to computers (and other media) that provide (simple) social cues [13]. Feedback from participants suggest that the social cues provided by the virtual buddy proof of concept system may have been too simple. However, a pilot study with an earlier version of the virtual buddy system demonstrated that children recognize and accept simple social cues like the ones used in the current study [15]. Nevertheless, repeating the experiment with a more advanced emotion model and/or more natural facial expressions may result in statistically significant di↵erences between conditions.
The lack of significant di↵erences between the conditions might also be (partially) explained by the di↵erences between the virtual buddy’s behavior in the four conditions; these may have been too small. The buddy’s behavior di↵ered in how emotions were expressed. Apart from the control condition in which no emotions were expressed, the amount and valence of the emotions were the same for all conditions (depending on the response options selected by the user). Some participants remarked that the total number of emotions should be increased.
The di↵erences between conditions may also have been too small in the sense that expressing emotions may not be crucial to experience support during the conversation, even though the number of suggestions by participants to increase the virtual buddy’s emotional expressiveness indicates that it is an important factor for the perception of support. The virtual buddy uses a variety of strategies to convey support; in addition to expressing emotions, these strategies include the conversation structure, and providing information (advice and teaching). Also, the fact that many participants suggested other ways in which the virtual buddy could provide support to cyberbullying victims indicates that there are more factors that a↵ect the perception of support than ‘just’ expressing emotions. More research is required to identify these factors, find ways to incorporate them into the conversation, and assess how they a↵ect perceived support.
Even though participants from all conditions were very well able to point out weaknesses in the virtual buddy’s behavior, this critical attitude was not reflected in the perceived social support scores. The average scores were relatively high. These high scores could have been caused by socially desirable behavior triggered by the social relevance of cyberbullying as application domain. 6
The virtual buddy is an example of an application of embodied agents for creating a particular emotional experience, in our case the experience of social support. This section briefly reviews related work on embodied agents that trigger emotional responses.
A related project in the bullying domain is FearNot!. FearNot! is an Intelligent Virtual Environment (IVE), where synthetic characters act out bullying scenarios [11]. The goal of the project was to create virtual agents that elicited empathy by displaying believable social and emotional behavior. User tests confirmed that the agents were able to establish empathic relations with users.
Other virtual agents that try to evoke certain emotional responses are pedagogical agents. A study conducted by Arroyo et al. shows that the interacting with a pedagogical agent that provides emotional and motivational support in an Intelligent Tutoring System for mathematics improved a↵ective learning outcomes; users of the pedagogical agents reported less frustration and increased confidence compared to users that did not interact with with an agent [2].
The emotional experience companion agents strive for is engagement. In particular, the goal of companion agents is to keep user engaged for multiple interactions over longer periods of time. Related work on a robotic chess companion for children shows that keeping users engaged over multiple interactions is challenging; participants of the study lost interest in the companion robot over the course of the five weeks they played against the robot [8]. 7
Conclusion The goals of the study presented in this paper were 1) to determine to what extent verbal and/or nonverbal expression of emotions contribute to the e↵ectiveness of social support by an conversational agent, and 2) to verify the user experience of the virtual buddy proof of concept system does not hamper the provision of social support. The results show that the user experience of the virtual buddy is acceptable; and, therefore, does not impede the virtual buddy’s potential for providing social support. It was also shown that the social support expressed by the virtual buddy is e↵ective. Additionally, perceived social support was generally high.
We found no significant di↵erences between conditions for user experience, e↵ectiveness of the support, and perceived social support. Therefore, we conclude that emotions expressed verbally and/or nonverbally by the virtual buddy proof of concept system do not contribute to the experience of social support in the context of our cyberbullying scenario. However, the large number of participants suggesting to increase the virtual buddy’s emotional expressiveness in order to improve emotional support, indicate that this is an important factor in the perception of support.
The feedback from participants indicated some important limitations of the virtual buddy proof of concept system. We plan to further investigate these limitations and whether social support is conveyed by the virtual buddy in a qualitative evaluation of the system by domain experts and the target audience. Acknowledgements This work is funded by NWO under the Responsible Innovation (RI) program via the project ‘Empowering and Protecting Children and Adolescents Against Cyberbullying’.