Empathy and Emotion in Social Robots

Abstract

The topic I plan to address in my doctoral research is based on social robots and their potential impact on interaction. In this paper I outline the goals of the doctoral project, which include creating a cloud-based architecture for modeling user-robot interaction and developing various services such as emotion and face recognition, domain ontology-based services, and user modeling. I also describe what I am focusing on at the moment, empathy in social robots, what I am looking forward to in the nearer future.

Keywords

• Cloud-based robotics services
• Emotion Analysis
• Empathy

1. Introduction

The aim of this PhD research is to define, design, implement and validate models of HumanRobot Interaction (HRI) aimed at improving key aspects of interactions between humans and robots. Robots that can interact with people in a natural manner are called social robots [ 1 ], and they are able to use speech, gestures, facial expressions, and language. These robots are easier to interact with and have many applications in entertainment, services, education, and assistance. This type of robots interacts with people by engaging in social-emotive behaviors, skills, capacities, and rules related to its collaborative role. Thus, social interactive robots need to perceive and understand the richness and complexity of the user’s natural social behavior, to interact with people in a human-like manner. Detecting and recognizing human action and communication provides a good starting point, but more important is the ability to interpret and react to human behavior, and a key mechanism to carry out these actions is user modeling. Thus the social robots should be able to adapt to user’s behavior, their preferences and their emotions, establishing a relationship that is not only social but also emotional. Developments in social robot technology have focused on several key aspects: • Artificial intelligence : AI is a key element in the operation of social robots. Machine learning algorithms and natural language processing have been used to enable robots to understand and respond to human interactions more naturally and intuitively. • Expressiveness: The ability of robots to express emotions and communicate non-verbally is an important aspect of social interaction. Advances in facial animation, hand and body gestures have been developed to make social robots more realistic and understandable to humans. rOcid 0000-0003-0928-8226 (A.

Lillo) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings • Emotion recognition: Social robots have been equipped with human emotion recognition capabilities. Using sensors such as cameras and microphones, robots can evaluate facial expressions, tone of voice and other signals to interpret the emotional state of the human interlocutor. This allows the robot to personalize its responses to the user’s needs. • Personalization: The goal is to create a more personalized interaction experience. Social robots can be programmed to store the user’s preferences, learn from the user’s behaviors, and adapt accordingly. This allows for a closer bond between the robot and the user. • Ethics and trust: As social interaction between humans and robots has increased, the importance of addressing ethics and trust issues has emerged. Developers are working to ensure that social robots are designed to respect the privacy, safety, and dignity of the person involved in the interaction.

This paper has been designed as follow, in Sec. Background I describe what field I come from, in Sec. PhD Goals I describe what the goals of my PhD research are, Sec. Scenario, and Sec. Future work.

2. Background

In my master thesis I have developed a Neural Network for emotion recognition, for more details see [ 2 ]. In particular, I used a Convolutional Neural Network (CNN). The neural network not only recognizes emotions, but it is able to give a prediction based on the user’s body and context. Since the overall expression of emotion may vary across contextual situations, I decided to use the Emotic Dataset[ 3 ], which also uses contextual information in recognizing emotions. The Emotic dataset is a collection of images of people in unconstrained environments annotated according to their apparent emotional states. The dataset contains 23,571 images and 34,320 annotated people. Today, there are huge prejudices in face recognition software and it has shown that even the seemingly impartial world of technology may be subject to racism. Brighter subjects are recognized more accurately than darker ones. Partial or unrepresentative data, such as the faces of black men and women, which are less present in the databases used for face recognition, lead to poor and often incorrect identification. [ 4 ] [ 5 ] [ 6 ] For this reason I decided to use Emotic Dataset, it is made up of images representing people of various ethnicities, sexes and ages. At the end of the project, I obtained a good accuracy for the value of Valence - 0.70991, Arousal - 0.87199, Dominance - 0.90254. After the test, I obtained Mean Average Precision for the prediction of the Emotion Category and Mean Error of the VAD values (Valence, Arousal, Dominance), respectively, of 0.27 and 0.83. It is important to consider that the performance evaluation of an emotional recognition model may vary depending on the dataset used, the complexity of the classification task, and the reference standards. The value I got regarding Mean Average Precision, is not the best, for this reason during my PhD I will try to improve it so as to obtain a better precision.

3. PhD Goals

My PhD goal is to create a cloud-based architecture for modeling the user-robot interaction to re-use the approach with diferent kinds of social robots, in particular assistive robots. The aim of my work is to develop general purpose cloud-based applications, ofering cloud components for managing social, afective and adaptive services for social robots (i.e., Softbank Pepper and NAO etc., the educational robot Wolly [ 7 ], [ 8 ]). The single components listed below can be used individually by external applications that need them (for example, emotion recognition service), but when integrated all together they can enhance the interaction with a social robot. I would like to use a set of services such as: • Face Recognition • Domain Ontology-based services for enriching dialogue’s strategies • Emotion Analysis • User Modeling component • Empathic Interaction

3.1. Face Recognition

Recently, there has been significant attention in the field of computer vision, especially in facial recognition and detecting the localization of facial landmarks. Many meaningful features can be derived directly from the human face, such as age, gender, and emotions. This service has already been developed, and I have utilized the Python Face Recognition library, which is one of the most commonly used libraries for recognizing and manipulating faces. Thanks to this library, the robot can recognize people, allowing it to identify individuals it already knows and respond accordingly.

Furthermore, through this service, I can obtain an estimate of the person’s age and gender.

3.2. Domain Ontology-based services

For the ontology services, I am not dealing with it directly at the moment, but I am collaborating with and supervising some thesis students in making them. Currently we have reached a point where we have a conversational model based on an ontology that has movies, cartoons, and video games as its domain.

3.3. Emotion Analysis

For emotion analysis,as a starting point,I will continue development of CNN cited in session 2. Additionally, I am currently working on recognizing emotions and moods through body language, I am using the BoLD dataset [ 9 ] The BoLD (Body Language Dataset) is a large and growing dataset containing annotated short-video samples of bodily expression of emotions. The dataset contains 9,827 video clips and there are 13,239 instances, i.e., the person of interest in these clips. Each instance is a data sample in this dataset.[ 9 ] I decided to develop another service for emotions, as I am convinced that by getting more data from diferent angles, a more accurate result can be obtained.

3.4. User modeling component

For the user modeling part, my goal is to join all the information received from the three services mentioned above and create a user model to better analyze the user’s personality. A user model may contain the system’s hypotheses about all aspects of the user deemed relevant to adapt the system’s dialogue behavior to the user. This component of an interactive system derives hypotheses about the user based on interaction with the user and/or user’s stereotypes [ 10 ], stores them in an appropriate representational system, infers additional hypotheses from the initial ones, maintains consistency in the current set of hypotheses, and provides other components of the system with hypotheses about the user [ 11 ]. User models can be used by robots for several purposes. First, user models can help the robot understand an individual’s behavior and intentions. Second, they are useful in adapting the robot’s behavior to the user’s diferent abilities, experiences, preferences, and knowledge. Finally, they can determine the form of control and feedback provided to the user (e.g., stimulating interaction).

3.5. Empathic interaction

Concerning empathic interaction, I am currently focusing on empathy in social robots. Empathy can be defined as the ability to understand and share the emotions and perspectives of others, has long been recognized as a fundamental aspect of human social interaction.[ 12 ]. Replicating this capacity for empathy in a computer system represents a complex challenge. In recent years, as advancements in robotics and artificial intelligence have led to the development of social robots, researchers and engineers have been exploring the integration of empathy into these robotic systems. The field of social robotics aims to create robots that can efectively interact and communicate with humans in a socially intelligent and emotionally engaging manner [ 13 ]. The development of this part is still in an initial stage; I believe it is necessary to begin by developing services cited above, that can obtain specific information to proceed with creating a cognitive model for building a computer system or module capable of understanding and expressing empathy towards humans. To develop a formal module of empathy, it is necessary to work on cognitive models, which are internal representations of individuals’ knowledge, beliefs, and intentions. These cognitive models can be constructed using artificial intelligence approaches. The formal empathy module employs these cognitive models to interpret and understand language and human behavior, in order to respond empathetically. This requires the ability to detect and interpret human emotions, analyze the context, and adapt to the needs and perspectives of the individual being interacted with.

4. Scenario

Services mentioned in Sec. PhD Goals 3 will be tested in the context of interaction with autistic children, by improving and continuing the experimentation described in Gena et al [ 14 ]. The study of empathy in respect to this specific social group requires an in-depth understanding of autism and its individual manifestations. In fact, there are traits associated with this neurological disorder that influence the children’s ability to express and understand empathy in a typical way. Milone et al.[ 15 ] as showed dificulties in social communication, mutual understanding, social interaction, and repetitive interests and behaviors. These studies have demonstrated how autistic children may manifest a unique form of empathy. When applying the aforementioned services to people who sufer from autism, it is important to remember that each autistic child is a unique individual with diferent nuances and characteristics, and autism itself is a spectrum. Studies such as Molnar et al. [ 16 ] illustrate as the experiences of empathy vary greatly from one person to another: some may have dificulty understanding the emotions of others or interpreting nonverbal social cues. Working with children with autism also requires to engage with multiple forms of empathy. In fact, both studies [ 17 ] and personal accounts have shown that children with autism can develop a cognitive empathy, based on the ability to understand and recognize the emotions of others through rational thought, rather than through direct experience or intuition. This makes them able to conceptually understand the feelings of the other person in a certain situation, despite being unable to directly experience the same emotions or respond in a typical way. The manifestation of empathy as well may manifest in diferent ways in autistic children than their neurotypical peers, through concrete gestures or actions, rather than through verbalization of emotions. Moreover, as shown in [ 18 ], they tend to demonstrate a strong sense of justice or deep concern for others, even when not communicated in traditional ways. Considering all these premises, it is evident how critical can be to provide an inclusive and supportive environment to encourage the expression of empathy in children with autism. This can include teaching social skills, promoting shared play activities and positive interactions with peers. In addition, educational programs can be a valid instrument to explicitly teach empathy, encouraging understanding of others’ emotions and the use of appropriate communication strategies.

5. Future work

In the future, I will proceed to complete the implementation mentioned in the previous Sec. PhD Goals3 and consequently conduct the experiments. Additionally, I would like to improve the accuracy of the neural network described in Sec. Background 2 . From October to December, I will be in Genoa at the Italian Institute of Technology (IIT), where I will have the opportunity to test and try out my projects on other robots, such as iCub. From January to July, I will be visiting the University of Manchester, specifically with Professor Angelo Cangelosi in his research group, the Cognitive Robotics Lab to continue my research on a computational model of empathy.

References

[1] F. Hegel , C. Muhl , B. Wrede , M. Hielscher-Fastabend , G. Sagerer, Understanding social robots , in: 2009 Second International Conferences on Advances in Computer-Human Interactions, 2009 , pp. 169 - 174 . doi: 10 .1109/ACHI. 2009 . 51 .

[2] C. Gena , A. Lillo , C. Mattutino , E. Mosca, Wolly: an afective and adaptive educational robot , in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization , 2022 , pp. 146 - 150 .

[3] R. Kosti , J. Alvarez , A. Recasens , A. Lapedriza , Context based emotion recognition using emotic dataset , IEEE Transactions on Pattern Analysis and Machine Intelligence ( 2019 ).

[4] La storia della ricercatrice che accusa google di censura , https://www.ilpost.it/2020/12/09/ gebru-google - intelligenza-artificiale/, 2020 . Accessed: 2023 -5-28.

[5] Coded bias , https://www.netflix.com/it/title/81328723, 2020 .

[6] Algorithmic justice league - unmasking AI harms and biases , https://www.ajl.org/, 2023 .

[7] V. Cietto , C. Gena , I. Lombardi, C. Mattutino , C. Vaudano , Co-designing with kids an educational robot , in: 2018 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO) , IEEE, 2018 , pp. 139 - 140 .

[8] C. Gena , C. Mattutino , G. Perosino, M. Trainito , C. Vaudano , D. Cellie , Design and development of a social, educational and afective robot , in: 2020 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS) , IEEE, 2020 , pp. 1 - 8 .

[9] ARBEE : Towards automated recognition of bodily expression of emotion in the wild , ”, International Journal of Computer Vision 128 ( 2020 ) 1 - 25 .

[10] C. Gena , L. Ardissono , On the construction of tv viewer stereotypes starting from lifestyles surveys1 , Workshop on Personalization in Future TV , 2001 UM conference ( 2001 ).

[11] A. Kobsa , User modeling and user-adapted interaction , in: Conference Companion on Human Factors in Computing Systems, CHI '94 , Association for Computing Machinery, New York, NY, USA, 1994 , p. 415 - 416 .

[12] H. Riess , The science of empathy, J. Patient Exp . 4 ( 2017 ) 74 - 77 .

[13] S. Park , M. Whang , Empathy in human-robot interaction: Designing for social robots , Int. J. Environ. Res. Public Health 19 ( 2022 ) 1889 .

[14] C. Gena , C. Mattutino , S. Brighenti , A. Meironeb , F. Petrigliab , L. Mazzottab , F. Lisciob , M. Nazzarioc , V. Riccic , C. Quaratod , et al., Sugar, salt & pepper-humanoid robotics for autism , Proceedings http://ceur-ws. org ISSN 1613 ( 2020 ) 0073 .

[15] A. Milone , L. Cerniglia , C. Cristofani , E. Inguaggiato , V. Levantini , G. Masi, M. Paciello , F. Simone , P. Muratori , Empathy in youths with conduct disorder and callous-unemotional traits , Neural Plast . 2019 ( 2019 ) 9638973 .

[16] I. Molnar-Szakacs , M. J. Wang , E. A. Laugeson , K. Overy , W.-L. Wu , J. Piggot , Autism, emotion recognition and the mirror neuron system: the case of music , Mcgill J. Med . 12 ( 2009 ) 87 .

[17] Y. Song , T. Nie , W. Shi , X. Zhao , Y. Yang , Empathy impairment in individuals with autism spectrum conditions from a multidimensional perspective: A meta-analysis , Front. Psychol . 10 ( 2019 ) 1902 .

[18] Y. Bolourian , A. Losh , N. Hamsho , A. Eisenhower , J. Blacher , General education teachers' perceptions of autism, inclusive practices, and relationship building strategies , J. Autism Dev. Disord . 52 ( 2022 ) 3977 - 3990 .