The pursuit of creating empathetic social robots that can understand and respond to human emotions is a critical challenge in Robotics and Artificial Intelligence. Social robots, designed to interact with humans in various settings, from healthcare to customer service, require a sophisticated understanding of human emotional states to resonate and efectively assist truly. Our research contributes to this ambitious goal by exploring the relationship between natural facial expressions and brain activity in these human-robot interactions, as captured by electroencephalogram (EEG) signals. This paper presents our initial steps towards this attempt. We want to find which areas in the participant user's brain are most activated and how these activations correlate with facial expressions. Understanding these correlations is essential for developing social robots that recognize and empathize with various human emotions. Our approach combines neuroscience and computer science, ofering a novel perspective in the quest to enhance the emotional intelligence of social robots. We share some preliminary results on a new multimodal dataset that we are developing, providing valuable insights into the potential of our work to improve the personalization and emotional depth of social robot interactions.
CEUR ceur-ws.org
(G. Sansonetti)
Customizing the interaction experiences with social robots [
This section delves into the key concepts and methodologies that underpin our interdisciplinary project, which aims to advance the field of social robotics by integrating psychological models, neuroscientific techniques, and facial expression analysis.
At the heart of our approach is the Circumplex Model of Emotions, proposed by Russell in 1980 [6]. This model is pivotal in psychology for understanding and categorizing emotional responses, organizing them along two primary axes: valence and arousal. Valence measures the positivity or negativity of an emotion, while arousal gauges its intensity. In social robotics, this model is a foundational framework for our research. It is employed to interpret explicit feedback from users interacting with robots, providing a structured way to assess and categorize the emotional responses elicited by these interactions.
Electroencephalography (EEG) plays a crucial role in our study. This technology records the brain’s spontaneous electrical activity, ofering insights into the cognitive processes triggered during human-robot interactions. Particularly in social robotics, EEG can reveal how users cognitively and emotionally engage with robots [7]. The EEG’s ability to facilitate EventRelated Potential (ERP) analysis is precious [8]. ERP, derived from averaging EEG signals in response to repeated stimuli, sheds light on the cognitive processing steps involved in stimulus recognition and processing, such as attention, memory, and perception. This analysis enables us to understand the brain’s reaction times and the specific regions engaged during interactions, providing a deeper understanding of the user’s emotional and cognitive state.
Lastly, the Facial Action Coding System (FACS) [9], developed by Ekman and Friesen in 1978, is integral to our methodology. FACS breaks down facial expressions into individual components known as Action Units (AUs). These AUs are critical for emotion recognition, representing the fundamental movements composing facial expressions. By integrating FACS with EEG data and the Circumplex Model, we aim to meticulously analyze and identify users’ most relevant facial expressions while interacting with social robots. This comprehensive approach allows us to decipher the most significant natural reactions from a neurological and emotional standpoint, thus providing an objective basis for understanding and enhancing user experience in social robotics.
At this stage of the research, we still have a limited number of participants, and consequently, it will only be possible to show some preliminary results on their analysis. To date, we have six participants, four females and two males, between the ages of 20 and 23. All participants ifnished the trial successfully following the procedure described in [ 10].
An analysis of the correlation between users’ explicit feedback (Fig. 1) reconfirms the results already found in the literature [11], namely that there is a strong correlation between the following pairs: likability-rewatch, arousal-rewatch, likability-arousal. Next, the face recordings were processed through OpenFace [12]. Features (i.e., average, max, mean, and skewness) were calculated for each video and AU. We then sampled the responses according to stimulus type, and correlations, calculated with phi-k [13] shown in Figure 2 (their significance in Figure 3) were identified. We thus note that for positive images, there is a correlation between the values of two AUs (i.e., Brow Lowerer-AU04 and Lid Tightener-AU07) and arousal. Therefore, we can already hypothesize that given certain types of stimulus, there are portions of the face that are more activated. These correlations reafirm the intricate relationship between emotional engagement and behavioral intentions, which could have significant implications for social robotics. Finally, we want to show two other interesting results. The first is a particular trend, shown in Figure 4 but also detected for other AUs, in which we can see that the works rated high or low arousal turn out to be less activating than the others. Also, from an initial ERP analysis (see Fig. 5), a strong activation of P300 can be seen for the negative image set.
In conclusion, our study has embarked on a fascinating journey to unravel the intricate relationships between facial reactions, EEG signals, and user preferences. We have illuminated several critical aspects of this multidisciplinary research through a detailed experimental protocol. Our preliminary findings are particularly promising, ofering a new perspective on how users emotionally and cognitively respond to stimuli, with a specific focus on the cultural heritage sector. The potential implications of our findings extend beyond the realm of museums and art appreciation. They hint at intriguing possibilities in the field of social robotics. If a correlation between EEG signals and facial expressions is established, it could revolutionize how social robots interpret and respond to human emotions. The use of advanced camera systems in social robots could enable them to assess deeper feelings and reactions of individuals based solely on facial expressions. This capability would significantly enhance the quality of interaction between humans and robots, making it more intuitive, empathetic, and personalized. Looking to the future, our research could lay the groundwork for developing social robots adept at understanding complex human emotions. Such robots could be employed in various settings, from education [14] to assistance [15], from telepresence [16] to entertainment [17], ofering support and interaction that are deeply attuned to the individual’s emotional state. This advancement would represent a technological leap and a profound step towards more humane and responsive AI systems. While our study currently focuses on the intersection of art, neuroscience, and user experience, its ramifications could be far-reaching, influencing the evolution of social robotics and the way we envision interactions between humans and machines. The ability to discern individuals’ emotions from facial expressions can also bring benefits in other domains. Consider, for instance, recommender systems [18] where inferring implicit levels of appreciation for items remains a notable challenge awaiting conclusive resolution. Within the SOCIALIZE context, the recommendation of points of interest [19] and the creation of itineraries connecting them [20] have the potential to foster social and cultural inclusion for individuals with diverse backgrounds. For example, implementing recommender systems in museums [21] can transform these spaces into invaluable hubs for social and cultural interaction. This transformation can be facilitated by acquiring visitor-related information [22, 23, 24] and integrating multimedia content [25, 26]. The journey toward understanding and leveraging these complex interconnections is just beginning, and the future holds immense potential for enhancing social-robot interactions [27]. [5] C. Gena, C. Mattutino, M. Botta, D. Camilleri, F. Di Sario, G. Ignone, F. Cena, et al.,
Cloud-based user modeling for social robots: a first attempt, volume 2724, CEUR, 2020. [6] J. A. Russell, A circumplex model of afect, Journal of Personality and Social Psychology 39 (1980) 1161–1178. [7] M. Mezzini, Emotion detection using deep learning on spectrogram images of the electroencephalogram, in: CEUR Workshop Proceedings, volume 3576, 2023, pp. 1–14. [8] S. J. Luck, An introduction to the event-related potential technique, MIT press, 2014. [9] P. Ekman, W. Friesen, Facial Action Coding System, Consulting Psychologists Press, 1978. [10] L. Battisti, S. Fagioli, A. Ferrato, C. Limongelli, S. Mastandrea, M. Mezzini, D. Nardo, G. Sansonetti, Towards a deeper understanding: Eeg and facial expressions in museums, in: CEUR Workshop Proceedings, volume 3536, 2023, pp. 42–49. [11] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, Exploiting micro facial expressions for more inclusive user interfaces, in: CEUR Workshop Proceedings, volume 2903, 2021. [12] T. Baltrusaitis, A. Zadeh, Y. C. Lim, L.-P. Morency, Openface 2.0: Facial behavior analysis toolkit, in: 13th IEEE International Conference on Automatic Face & Gesture Recognition, IEEE, 2018, pp. 59–66. [13] M. Baak, R. Koopman, H. Snoek, S. Klous, A new correlation coeficient between categorical, ordinal and interval variables with pearson characteristics, Computational Statistics & Data Analysis 152 (2020) 107043. [14] T. Belpaeme, J. Kennedy, A. Ramachandran, B. Scassellati, F. Tanaka, Social robots for education: A review, Science Robotics 3 (2018) 1–9. [15] D. Macis, S. Perilli, C. Gena, Employing socially assistive robots in elderly care, in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, 2022, pp. 130–138. [16] T. B. Tuli, T. O. Terefe, M. M. U. Rashid, Telepresence mobile robots design and control for social interaction, International Journal of Social Robotics 13 (2021) 877–886. [17] S. Forgas-Coll, R. Huertas-Garcia, A. Andriella, G. Alenyà, How do consumers’ gender and rational thinking afect the acceptance of entertainment social robots?, International Journal of Social Robotics 14 (2022) 973–994. [18] F. Gasparetti, G. Sansonetti, A. Micarelli, Community detection in social recommender systems: a survey, Applied Intelligence 51 (2021) 3975–3995. [19] A. De Angelis, F. Gasparetti, A. Micarelli, G. Sansonetti, A social cultural recommender based on linked open data, in: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2017, pp. 329–332. [20] D. D’Agostino, F. Gasparetti, A. Micarelli, G. Sansonetti, A social context-aware recommender of itineraries between relevant points of interest, in: HCI International 2016, volume 618, Springer International Publishing, Cham, 2016, pp. 354–359. [21] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, The META4RS Proposal: Museum Emotion and Tracking Analysis For Recommender Systems, in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2022, pp. 406–409. [22] F. Gasparetti, A. Micarelli, G. Sansonetti, Exploiting web browsing activities for user needs identification, in: 2014 International Conference on Computational Science and Computational Intelligence, CSCI 2014, volume 2, 2014, pp. 86–89. [23] M. Mezzini, C. Limongelli, G. Sansonetti, C. De Medio, Tracking museum visitors through convolutional object detectors, in: Adjunct Publication of the 28th Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2020, pp. 352–355. [24] A. Ferrato, C. Limongelli, M. Mezzini, G. Sansonetti, Using deep learning for collecting data about museum visitor behavior, Applied Sciences (Switzerland) 12 (2022). [25] A. Micarelli, A. Neri, G. Sansonetti, A case-based approach to image recognition, in: Proceedings of the 5th European Workshop on Advances in Case-Based Reasoning, volume 1898 of EWCBR ’00, Springer-Verlag, Berlin, Heidelberg, 2000, pp. 443–454. [26] G. Sansonetti, F. Gasparetti, A. Micarelli, Cross-domain recommendation for enhancing cultural heritage experience, in: Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2019, pp. 413–415. [27] N. Gasteiger, M. Hellou, H. S. Ahn, Factors for personalization and localization to optimize human–robot interaction: A literature review, Int. J. of Social Robotics 15 (2023) 689–701.