Social Robots in Learning Environments: a Case Study of an Empathic Chess Companion

Abstract

We present a scenario where a social robot acts as a chess companion for children, and describe our current e orts towards endowing such robot with empathic capabilities. A multimodal framework for modeling some of the user's a ective states that combines visual and task-related features is presented. Further, we describe how the robot selects adaptive empathic responses considering the model of the user's a ect.

Keywords

• learning companions
• empathy
• a ective user modeling

Introduction

In the last few years, there has been a growing interest in developing animated pedagogical agents for learning environments [ 13 ]. Several studies suggest that pedagogical agents positively a ect the way students perceive the learning experience due to their non-verbal behaviours [ 13 ], physical appearance or voice [ 5 ]. More recently, the ability to recognise and respond to the student's a ective state has also been considered a very important characteristic of pedagogical agents [ 4 ]. In humans, the capacity of understanding and responding appropriately to the a ective states of others is commonly designated as empathy [ 6 ]. Several theorists argue that empathy facilitates the creation and development of social relationships [ 1 ]. A positive student-teacher relationship increases student's trust, cooperation and motivation during the learning process. For these reasons, empathy is often linked with e ective teaching.

Research on arti cial companions has recently started to address the issue of designing systems for the automatic recognition of scenario-dependent, spontaneous a ect-related states. Examples include the system by Kapoor et al. [ 8 ], which can automatically predict frustration, and the work by Nakano and Ishii [ 14 ] to estimate the user's conversational engagement with a conversational agent. In this domain, there has been an increasing attention towards systems utilising contextual information to improve the a ect recognition performance [ 9 ]. In our previous work on the automatic detection of engagement, we showed that a combination of task and visual features allows for the highest recognition rate to be achieved [ 3 ]. While many e orts are being made to detect user's a ective and motivational states, another branch of research addresses the challenge of how a ect-aware agents should react to those states, and in which ways empathic responses improve the interaction. For example, Robison et al. studied the impact of a ective feedback on students interacting with a virtual agent in a narrative-centered learning environment [ 16 ]. In another study, Saerbeck and colleagues [ 17 ] investigated the e ects of a robot's social supportive behaviour on student's learning performance and motivation.

In this paper, we summarise our e orts on the development of a social robot with empathic capabilities that acts as a chess companion for children. By endowing the robot with empathic capabilities, we expect to improve the relationship established between children and the robot, which can ultimately lead them to improve their chess abilities. Thus, to behave empathically, our robot needs to (1) model the child's a ective states and (2) adapt its a ective and prosocial behaviour in response to the a ective states of the child. In the remaining of the paper, we present our approach for modelling empathy in a robotic learning companion. 2

Towards an Empathic Chess Companion

Our application scenario consists of an iCat robot that plays chess with children using an electronic chessboard (see Fig. 1). The iCat provides feedback on the children's moves by employing facial expressions determined by the robot's affective state. Chess can be considered an educational game, as it helps children develop their memory and problem solving skills [ 7 ]. A previous study using this scenario showed that the a ective behaviour expressed by the iCat increased user's perception of the game [ 10 ]. However, in another study, after several interactions children started realising that the robot's behaviour did not take into account their own a ective state [ 11 ]. The results of this study suggested that social presence decreased over time, especially in terms of perceived a ective and behavioural interdependence. These dimensions refer to the extent to which users believe that the behaviour and a ective state of the robot is in uenced by their own behaviour and a ective state. As described earlier, empathy requires the ability of understanding the user's a ective state and responding accordingly. Thus, in the remaining of this section, we describe our current research in these two distinct processes of empathy. O -line analysis of videos recorded during several interactions between children and the iCat showed that children display prototypical emotional expressions only occasionally. Therefore, we aim to endow the robot with the ability to infer scenario-dependent user a ective states, and speci cally a ective states related to the game and the social interaction with the robot: valence of feeling (positive or negative) and engagement with the robot. The valence of the feeling provides information about the overall feeling that the user is experiencing throughout the game, whereas engagement is \the value that a participant in an interaction attributes to the goal of being together with the other participant(s) and continuing the interaction", as de ned by Poggi [ 15 ]. In our previous work, we showed the key role of a subset of user's non-verbal behaviours and contextual features in the discrimination of a ective states [ 2, 3 ]. After modelling the a ective state of the user, the robot should be able to select the empathic responses that are most e ective to keep the user in a positive affective state. Several empathic and pro-social strategies existing in the literature are being considered, such as facial expressions, verbal comments to encourage the player and game-related actions (e.g., allow the user to take back a bad move). Some of these strategies were proven to be successful in a previous study that investigated the in uence of empathic behaviours on people's perceptions of a social robot [ 12 ].

But how should the robot decide, among the set of possible empathic strategies, which one is more appropriate at a certain moment? We are currently implementing an adaptive approach, where the robot learns the best strategies for a particular user by estimating the success of an empathic strategy measuring the user's a ective state right after such strategy is displayed by the iCat. For example, consider a situation where the user is experiencing a negative feeling for loosing an importance piece in the game and the iCat responds with an encouraging verbal comment. If the user's valence changes from negative to positive, then utterances containing encouraging behaviours will become part of the user's preferences in that particular situation. As the same users are expected to interact with the robot for several games, the preferences for a particular user are updated even over di erent interaction sessions. 3

Discussion

In this paper, we described our work towards endowing the robot with empathic capabilities. A multimodal system for predicting and modeling some of the children's a ective states in real time is currently being trained using a corpus with videos previously collected in another experiments using this scenario. With this model of the user, we intend to personalise the learning environment by adapting the robot's empathic responses to the particular needs of the child who is interacting with the robot.

References

1. C. Anderson and D. Keltner . The role of empathy in the formation and maintenance of social bonds . Behavioral and Brain Sciences , 25 ( 01 ): 21 { 22 , 2002 .

2. G. Castellano , I. Leite , A. Pereira , C. Martinho , A. Paiva , and P. McOwan . It's all in the game: Towards an a ect sensitive and context aware game companion . In Proceedings of ACII 2009 , pages 29 { 36 . IEEE, 2009 .

3. G. Castellano , A. Pereira , I. Leite , A. Paiva , and P. W. McOwan . Detecting user engagement with a robot companion using task and social interaction-based features . In Proceedings of ICMI'09 , pages 119 { 126 . ACM Press, 2009 .

4. S. D'Mello , S. Craig , K. Fike , and A. Graesser . Responding to learners' cognitivea ective states with supportive and shakeup dialogues. Human-Computer Interaction . Ambient, Ubiquitous and Intelligent Interaction , pages 595 { 604 , 2009 .

5. S. Domagk . Do pedagogical agents facilitate learner motivation and learning outcomes?: The role of the appeal of agent's appearance and voice . Journal of Media Psychology , 22 ( 2 ): 84 { 97 , 2010 .

6. M. Ho man. Empathy and moral development: Implications for caring and justice . Cambridge Univ Press, 2001 .

7. D. Horgan and D. Morgan . Chess expertise in children. Applied cognitive psychology , 4 ( 2 ): 109 { 128 , 1990 .

8. A. Kapoor , W. Burleson , and R. W. Picard . Automatic prediction of frustration . International Journal of Human-Computer Studies , 65 ( 8 ): 724 { 736 , 2007 .

9. A. Kapoor and R. W. Picard . Multimodal a ect recognition in learning environments . In ACM International Conference on Multimedia , pages 677 { 682 , 2005 .

10. I. Leite, C. Martinho , A. Pereira , and A. Paiva . iCat: an a ective game buddy based on anticipatory mechanisms . In Proceedings of AAMAS'08 , pages 1229 { 1232 . IFAAMAS , 2008 .

11. I. Leite, C. Martinho , A. Pereira , and A. Paiva . As Time goes by: Long-term evaluation of social presence in robotic companions . In Proceedings of RO-MAN 2009 , pages 669 { 674 . IEEE, 2009 .

12. I. Leite, S. Mascarenhas , A. Pereira , C. Martinho , R. Prada , and A. Paiva . \why can't we be friends?" an empathic game companion for long-term interaction . In IVA'10 , volume 6356 of LNCS , pages 315 { 321 . Springer, 2010 .

13. J. C. Lester , S. A. Converse , S. E. Kahler , S. T. Barlow , B. A. Stone , and R. S. Bhogal . The persona e ect: a ective impact of animated pedagogical agents . In Proceedings of CHI '97 , pages 359 { 366 , NY, USA, 1997 . ACM.

14. Y. I. Nakano and R. Ishii . Estimating user's engagement from eye-gaze behaviors in human-agent conversations . In Proceeding of IUI'10 , pages 139 { 148 , New York, NY, USA, 2010 . ACM.

15. I. Poggi. Mind, hands, face and body. A goal and belief view of multimodal communication . Weidler , Berlin, 2007 .

16. J. Robison , S. McQuiggan , and J. Lester . Evaluating the consequences of a ective feedback in intelligent tutoring systems . In Proceedings of ACII'09. IEEE , 2009 .

17. M. Saerbeck , T. Schut , C. Bartneck , and M. Janse . Expressive robots in education: varying the degree of social supportive behavior of a robotic tutor . In Proceedings of CHI'10 , pages 1613 { 1622 . ACM, 2010 .