Towards Empathetic Social Robots: Investigating the Interplay between Facial Expressions and Brain Activity Lorenzo Battisti1 , Sabrina Fagioli2 , Alessio Ferrato1 , Carla Limongelli1 , Stefano Mastandrea2 , Mauro Mezzini2 , Davide Nardo2 and Giuseppe Sansonetti1,∗ 1 Department of Engineering, Roma Tre University, 00146 Rome, Italy 2 Department of Education Sciences, Roma Tre University, 00185 Rome, Italy Abstract The pursuit of creating empathetic social robots that can understand and respond to human emotions is a critical challenge in Robotics and Artificial Intelligence. Social robots, designed to interact with humans in various settings, from healthcare to customer service, require a sophisticated understanding of human emotional states to resonate and effectively assist truly. Our research contributes to this ambitious goal by exploring the relationship between natural facial expressions and brain activity in these human-robot interactions, as captured by electroencephalogram (EEG) signals. This paper presents our initial steps towards this attempt. We want to find which areas in the participant user’s brain are most activated and how these activations correlate with facial expressions. Understanding these correlations is essential for developing social robots that recognize and empathize with various human emotions. Our approach combines neuroscience and computer science, offering a novel perspective in the quest to enhance the emotional intelligence of social robots. We share some preliminary results on a new multimodal dataset that we are developing, providing valuable insights into the potential of our work to improve the personalization and emotional depth of social robot interactions. Keywords Social robot, facial analysis, EEG, dataset 1. Introduction Customizing the interaction experiences with social robots [1] has become increasingly im- portant in recent years [2, 3]. This trend echoes the shift in museum personalization that emerged with the “new museology” movement in 1997 [4], which moved the focus from the institution to the visitor experience. Similar to how museums began to explore personalized visitor experiences, the field of social robotics is now exploring customized interactions based on Joint Proceedings of the ACM IUI Workshops 2024, March 18-21, 2024, Greenville, South Carolina, USA ∗ Corresponding author. Envelope-Open lor.battisti5@stud.uniroma3.it (L. Battisti); sabrina.fagioli@uniroma3.it (S. Fagioli); alessio.ferrato@uniroma3.it (A. Ferrato); limongel@dia.uniroma3.it (C. Limongelli); stefano.mastandrea@uniroma3.it (S. Mastandrea); mauro.mezzini@uniroma3.it (M. Mezzini); davide.nardo@uniroma3.it (D. Nardo); gsansone@dia.uniroma3.it (G. Sansonetti) Orcid 0009-0001-6533-7195 (L. Battisti); 0000-0002-6381-0989 (S. Fagioli); 0000-0001-5405-3018 (A. Ferrato); 0000-0002-0323-7738 (C. Limongelli); 0000-0001-5128-1525 (S. Mastandrea); 0000-0002-5308-0097 (M. Mezzini); 0000-0002-8158-5738 (D. Nardo); 0000-0003-4953-1390 (G. Sansonetti) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings user preferences and reactions [5]. In this paper, we aim to contribute to advancing social-robot interaction by exploring the potential of using natural facial reactions and electroencephalo- gram (EEG) signals to gauge user emotions and preferences. To extend beyond basic expression analysis, in this paper, we explore the possibility of a novel approach that utilizes EEG signals in conjunction with facial reaction analysis to predict user preferences and emotional responses that may help adapt the interactions between humans and social robots. Through this research, we aim to significantly elevate the sophistication of social robot interactions. Our exploration into the intricate relationship between human emotions, facial expressions, and brain activity paves the way for the development of empathetic social robots. These robots, equipped with a deeper understanding of human emotional and cognitive processes, promise to transform the landscape of human-robot interaction, making it more intuitive, responsive, and, importantly, more human-like in its sensitivity and adaptability. 2. Key Concepts This section delves into the key concepts and methodologies that underpin our interdisciplinary project, which aims to advance the field of social robotics by integrating psychological models, neuroscientific techniques, and facial expression analysis. At the heart of our approach is the Circumplex Model of Emotions, proposed by Russell in 1980 [6]. This model is pivotal in psychology for understanding and categorizing emotional responses, organizing them along two primary axes: valence and arousal. Valence measures the positivity or negativity of an emotion, while arousal gauges its intensity. In social robotics, this model is a foundational framework for our research. It is employed to interpret explicit feedback from users interacting with robots, providing a structured way to assess and categorize the emotional responses elicited by these interactions. Electroencephalography (EEG) plays a crucial role in our study. This technology records the brain’s spontaneous electrical activity, offering insights into the cognitive processes triggered during human-robot interactions. Particularly in social robotics, EEG can reveal how users cognitively and emotionally engage with robots [7]. The EEG’s ability to facilitate Event- Related Potential (ERP) analysis is precious [8]. ERP, derived from averaging EEG signals in response to repeated stimuli, sheds light on the cognitive processing steps involved in stimulus recognition and processing, such as attention, memory, and perception. This analysis enables us to understand the brain’s reaction times and the specific regions engaged during interactions, providing a deeper understanding of the user’s emotional and cognitive state. Lastly, the Facial Action Coding System (FACS) [9], developed by Ekman and Friesen in 1978, is integral to our methodology. FACS breaks down facial expressions into individual components known as Action Units (AUs). These AUs are critical for emotion recognition, representing the fundamental movements composing facial expressions. By integrating FACS with EEG data and the Circumplex Model, we aim to meticulously analyze and identify users’ most relevant facial expressions while interacting with social robots. This comprehensive approach allows us to decipher the most significant natural reactions from a neurological and emotional standpoint, thus providing an objective basis for understanding and enhancing user experience in social robotics. 3. First Findings At this stage of the research, we still have a limited number of participants, and consequently, it will only be possible to show some preliminary results on their analysis. To date, we have six participants, four females and two males, between the ages of 20 and 23. All participants finished the trial successfully following the procedure described in [10]. An analysis of the correlation between users’ explicit feedback (Fig. 1) reconfirms the results already found in the literature [11], namely that there is a strong correlation between the following pairs: likability-rewatch, arousal-rewatch, likability-arousal. Next, the face recordings Figure 1: Pearson correlation matrix on explicit feedback. were processed through OpenFace [12]. Features (i.e., average, max, mean, and skewness) were calculated for each video and AU. We then sampled the responses according to stimulus type, and correlations, calculated with phi-k [13] shown in Figure 2 (their significance in Figure 3) were identified. We thus note that for positive images, there is a correlation between the values of two AUs (i.e., Brow Lowerer-AU04 and Lid Tightener-AU07) and arousal. Therefore, we can already hypothesize that given certain types of stimulus, there are portions of the face that are more activated. These correlations reaffirm the intricate relationship between emotional engagement and behavioral intentions, which could have significant implications for social robotics. Finally, we want to show two other interesting results. The first is a particular trend, Figure 2: Correlation matrix with phi-k of max values for each AU for the set of positive stimuli. Figure 3: Significance matrix of the correlation matrix in Figure 2. Figure 4: Trend on the mean value of AU07-Lid Tightener. Figure 5: ERP analysis: the black line is related to negative stimuli, blue to positive, and red to neutral. shown in Figure 4 but also detected for other AUs, in which we can see that the works rated high or low arousal turn out to be less activating than the others. Also, from an initial ERP analysis (see Fig. 5), a strong activation of P300 can be seen for the negative image set. 4. Conclusion and Future works In conclusion, our study has embarked on a fascinating journey to unravel the intricate relation- ships between facial reactions, EEG signals, and user preferences. We have illuminated several critical aspects of this multidisciplinary research through a detailed experimental protocol. Our preliminary findings are particularly promising, offering a new perspective on how users emotionally and cognitively respond to stimuli, with a specific focus on the cultural heritage sector. The potential implications of our findings extend beyond the realm of museums and art appreciation. They hint at intriguing possibilities in the field of social robotics. If a correlation between EEG signals and facial expressions is established, it could revolutionize how social robots interpret and respond to human emotions. The use of advanced camera systems in social robots could enable them to assess deeper feelings and reactions of individuals based solely on facial expressions. This capability would significantly enhance the quality of inter- action between humans and robots, making it more intuitive, empathetic, and personalized. Looking to the future, our research could lay the groundwork for developing social robots adept at understanding complex human emotions. Such robots could be employed in various settings, from education [14] to assistance [15], from telepresence [16] to entertainment [17], offering support and interaction that are deeply attuned to the individual’s emotional state. This advancement would represent a technological leap and a profound step towards more humane and responsive AI systems. While our study currently focuses on the intersection of art, neuroscience, and user experience, its ramifications could be far-reaching, influencing the evolution of social robotics and the way we envision interactions between humans and machines. The ability to discern individuals’ emotions from facial expressions can also bring benefits in other domains. Consider, for instance, recommender systems [18] where inferring implicit levels of appreciation for items remains a notable challenge awaiting conclusive resolution. Within the SOCIALIZE context, the recommendation of points of interest [19] and the creation of itineraries connecting them [20] have the potential to foster social and cultural inclusion for individuals with diverse backgrounds. For example, implementing recommender systems in museums [21] can transform these spaces into invaluable hubs for social and cultural interaction. This transformation can be facilitated by acquiring visitor-related information [22, 23, 24] and integrating multimedia content [25, 26]. The journey toward understanding and leveraging these complex interconnections is just beginning, and the future holds immense potential for enhancing social-robot interactions [27]. References [1] C. Breazeal, Toward sociable robots, Robotics and Autonomous Systems 42 (2003) 167–175. [2] C. Breazeal, K. Dautenhahn, T. Kanda, Social robotics, Springer Handbook of Robotics (2016) 1935–1972. [3] H. Mahdi, S. A. Akgun, S. Saleh, K. Dautenhahn, A survey on the design and evolution of social robots—past, present and future, Robotics and Autonomous Systems 156 (2022). [4] P. Vergo, New museology, Reaktion books, 1997. [5] C. Gena, C. Mattutino, M. Botta, D. Camilleri, F. Di Sario, G. Ignone, F. Cena, et al., Cloud-based user modeling for social robots: a first attempt, volume 2724, CEUR, 2020. [6] J. A. Russell, A circumplex model of affect, Journal of Personality and Social Psychology 39 (1980) 1161–1178. [7] M. Mezzini, Emotion detection using deep learning on spectrogram images of the elec- troencephalogram, in: CEUR Workshop Proceedings, volume 3576, 2023, pp. 1–14. [8] S. J. Luck, An introduction to the event-related potential technique, MIT press, 2014. [9] P. Ekman, W. Friesen, Facial Action Coding System, Consulting Psychologists Press, 1978. [10] L. Battisti, S. Fagioli, A. Ferrato, C. Limongelli, S. Mastandrea, M. Mezzini, D. Nardo, G. Sansonetti, Towards a deeper understanding: Eeg and facial expressions in museums, in: CEUR Workshop Proceedings, volume 3536, 2023, pp. 42–49. [11] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, Exploiting micro facial expressions for more inclusive user interfaces, in: CEUR Workshop Proceedings, volume 2903, 2021. [12] T. Baltrusaitis, A. Zadeh, Y. C. Lim, L.-P. Morency, Openface 2.0: Facial behavior analysis toolkit, in: 13th IEEE International Conference on Automatic Face & Gesture Recognition, IEEE, 2018, pp. 59–66. [13] M. Baak, R. Koopman, H. Snoek, S. Klous, A new correlation coefficient between categorical, ordinal and interval variables with pearson characteristics, Computational Statistics & Data Analysis 152 (2020) 107043. [14] T. Belpaeme, J. Kennedy, A. Ramachandran, B. Scassellati, F. Tanaka, Social robots for education: A review, Science Robotics 3 (2018) 1–9. [15] D. Macis, S. Perilli, C. Gena, Employing socially assistive robots in elderly care, in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, 2022, pp. 130–138. [16] T. B. Tuli, T. O. Terefe, M. M. U. Rashid, Telepresence mobile robots design and control for social interaction, International Journal of Social Robotics 13 (2021) 877–886. [17] S. Forgas-Coll, R. Huertas-Garcia, A. Andriella, G. Alenyà, How do consumers’ gender and rational thinking affect the acceptance of entertainment social robots?, International Journal of Social Robotics 14 (2022) 973–994. [18] F. Gasparetti, G. Sansonetti, A. Micarelli, Community detection in social recommender systems: a survey, Applied Intelligence 51 (2021) 3975–3995. [19] A. De Angelis, F. Gasparetti, A. Micarelli, G. Sansonetti, A social cultural recommender based on linked open data, in: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2017, pp. 329–332. [20] D. D’Agostino, F. Gasparetti, A. Micarelli, G. Sansonetti, A social context-aware recom- mender of itineraries between relevant points of interest, in: HCI International 2016, volume 618, Springer International Publishing, Cham, 2016, pp. 354–359. [21] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, The META4RS Proposal: Museum Emotion and Tracking Analysis For Recommender Systems, in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2022, pp. 406–409. [22] F. Gasparetti, A. Micarelli, G. Sansonetti, Exploiting web browsing activities for user needs identification, in: 2014 International Conference on Computational Science and Computational Intelligence, CSCI 2014, volume 2, 2014, pp. 86–89. [23] M. Mezzini, C. Limongelli, G. Sansonetti, C. De Medio, Tracking museum visitors through convolutional object detectors, in: Adjunct Publication of the 28th Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2020, pp. 352–355. [24] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, Using deep learning for collecting data about museum visitor behavior, Applied Sciences (Switzerland) 12 (2022). [25] A. Micarelli, A. Neri, G. Sansonetti, A case-based approach to image recognition, in: Proceedings of the 5th European Workshop on Advances in Case-Based Reasoning, volume 1898 of EWCBR ’00, Springer-Verlag, Berlin, Heidelberg, 2000, pp. 443–454. [26] G. Sansonetti, F. Gasparetti, A. Micarelli, Cross-domain recommendation for enhancing cultural heritage experience, in: Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2019, pp. 413–415. [27] N. Gasteiger, M. Hellou, H. S. Ahn, Factors for personalization and localization to optimize human–robot interaction: A literature review, Int. J. of Social Robotics 15 (2023) 689–701.