To improve the acceptability of robots especially in assistance scenarios, where higher legibility and conformity to cognitive and emotional aspects of the social interaction are required, new human-robot interactive paradigms have to be modeled, in which the robot is humanized thanks to empowered AI-based social and cognitive abilities to recognize human users' emotions. Highlights on some works on AI approaches to emotion recognition in Assistive Robotics domains are presented. 2 Transferring time-consuming human tasks to machines (improving QoL for Clinicians) 3 Enabling patients to self-service their care needs when possible (increasing QoL for patients/caregivers)
The development of social robots interacting with
humans is becoming an important focus of robotics research
leading the EU commission to invest in social and
cognitive robotics by boosting the view of trustworthy
humancentric and Artificial Intelligence (AI) based technologies
that can lead to high benefits for the entire society. In
this regard, last years updates of the coordinated plan on
AI put strategy in place concerning the adoption of a new
generation of AI-Powered Robotics. Current advances
in AI, in fact, enabled computers to have social
capacities. AI-powered social robots, for example, provide
psychological, social, and emotional support. Persons
with cognitive disabilities may benefit from social robot
help in daily duties since these systems allow them to
preserve their freedom at home for long time [
robots are expected to enter our daily lives as companions, instructors, coworkers, caregivers, or tutors for children. Most of the research projects in the Social and Assistive Robotics (SAR) are oriented towards the development of assistive technologies for the support, and improvement of the quality of life (QoL) of elderly and disabled people. Despite the promise of enormous improvement of QoL, robots are still far to be accepted, probably due to a lack of understanding and trustworthiness.
Most robotic applications are based on user static models and possible interaction contexts. This makes such systems incapable of adapting autonomously and proactively to changes in users’ needs and preferences. More generally, assistive technology products do not take cognitive and personality characteristics into account (including an individual’s specific deficits, emotional and behavioral problems, attitudes toward technology, and their physical and social environment) that can afect their acceptance, use and efectiveness. A valid robotic system with a high degree of acceptability must necessarily be based not only on the knowledge of potential users, through the integration of detailed information from clinicians and psychologists, but also on the knowledge of the users that the robot is able to reconstruct through the observation and interpretation of human users’ overt and internal behavior (cognitive/emotional state and mental models), which provides essential elements for adaptive planning and customization of the interaction.
This contribution aims at summarizing the main achievement of our Inter-University Research Group in this field.
reduction of attentional focus, increasing of positive emotions, etc..
In the following subsections, we will summarize our research activities dealing with the use of diverse AI techniques to allow emotion recognition in HRI, especially in Assistive robotics contextes.
states of humans facilitates human-robot mutual afective
understanding and, in turn, allows an empathetic inter- 3.1. EEG-based Stress Assessment in HRI
action enhancing legibility and acceptability of robots. While on the one hand, a robot can be used to help
disThus, for a constructive and intelligent social human- abled people by providing cognitive rehabilitation
exerrobot interaction (HRI), robots should be endowed with cises and assistance, on the other hand, its presence and
the ability to detect and interpret humans’ basic afective actions could sometimes provoke negative emotions like
responses to adapt their behavior accordingly. This is es- stress or discomfort. This may cause serious dificulties
pecially desirable in the field of assistive robotics, where by negatively impacting the users’ health and, in turn,
the interaction often takes place with disabled or vulner- achieving a counter-productive result. In this context,
able people (with diseases compromised the emotional robots could use the afect-sensing capability to adapt
response). Additionally, equipping robots with this kind their behavior to be more comfortable for the person. In
of cognitive power may increase interlocutor engage- this direction we published a work [
Specifically, the robot’s emotional system must be de- aim, we implemented a Multi-Layer Perceptron (MLP)
signed in terms of afective elicitation as well as sens- to identify common patterns in users’ brain power
specing [
Recent studies showed that bio-signals analysis provides a valuable technique for afective state assessment since it guarantees repeatability and objectivity.
EEG devices have been widely used during last years in (a) (b)
the robotics field particularly for robot automatic control
[
Yet, there are a few studies that use BCI-based
techniques and technologies in combination with robotic HRI experiments were performed involving 10
particiequipment to categorize human interior states. Addi- pants, and EEG signals were recorded and analysed while
tionally, it is worth noticing the potential of EEG signals these subjects underwent cognitive tests. Results showed
analysis to detect time-varying changes in certain spec- the feasibility to assess afective states by using machine
tral bands in the brain activity that can be correlated learning algorithms applied to EEG signals. Interestingly,
to afective/cognitive states, such as increased anxiety, results on cognitive tests scores suggested that an
authoritarian interaction style can improve the performance,
(a)
(b)
(c)
although not significant diferences have been observed classical Support Vector Machine (SVM), Random Forest
between the friendly and authoritarian robots. (RF) and Decision Tree (DT) but also new emerging
techniques based on Deep Neural Networks (DNN) [
Internal State Monitoring with AI ionfgthteheamlgocroiuthldmssi.gAnsificaanretlsyulitm,ipnrothvies tphaepepre,rwfoermpraonpcoese
This work aims to demonstrate the efectiveness of EEG a Global Optimization Ensemble (GOE) Model based on
measurements in determining the emotional state in ex- a combination of feature selection and hyper-parameter
perimental subjects during the interaction with a robot. optimization techniques that can improve the overall
In particular, we investigate the correlation between the accuracy of various machine learning and deep
learnparameters of valence and arousal and the emotional ing models tested. In this work, we are interested in
state of users in correspondence with a particular
personality profile of the robot, namely positive and negative,
designed to induce greater positive responses (positive
engagament) and lower negative responces (such as stress)
during the interaction (see Fig.2-a and -Fig.2-b. Our
hypothesis is that a positive personality aiming at creating
a positive relationship with the users helps increasing
the user engagement and in turn the success of the
interaction, which, especially in the context of therapeutic
and cognitive interventions, is one of the main objectives
[
The results show a correlation between the extracted (and subsequently selected) features and the emotional state of the subject in the two experimental tests (See the Cconfusion matrix in Fig.2-c. Figure 3: Global Optimization Workflow. 3.3. Global Optimization Model for evaluating whether a diferent personality (positive or Classifying Emotions in HRI negative) of the robot could imply a diferent level of mental state of users who interact with it, which we plan In general, studies on the use of EEG for emotion recog- to monitor through the analysis of human brain waves nition in robotics applications are demonstrating the fea- acquired through the use of EEG-helmet. The intent is sibility and efectiveness of this technology. However, to validate neuroscience theories that demonstrate that there are still technical challenges to overcome, includ- by observing particular relationships between alpha and ing the bias-variance trade-of, dimensionality, and noise beta waves in specific areas of the brain it is possible to in the input data space. The latter, in particular, may detect a person’s emotional state in terms of valence and significantly impair the predictive quality of the tested arousal. models, and it is thus seen how an accurate phase of The tests are carried out with the aid of a humanoid noise removal may improve overall performance. There robot which ofers the dual advantage of being able to are several neural network models or classifiers such as inspire its cognitive abilities to neuroscience (for example brain-inspired algorithms) and on the other hand to isolation and economic loss for all their family. In [9] we provide a means for assessing the human brain. conducted a systematic litterature review aiming at understanding and summarizing the current research work 3.4. An alternative way to trait on the use of wearables for supporting detection and monitoring people with ASD and analyze the limitations
and open challenges to address future work. Results reIn this work (written with colleagues of this inter- vealed that the topic is very relevant, but there are many university group and still under review), a digital version limitations in the considered studies, such as reduced of Furhat was used, configured to simulate a bartender number of participants, absence of datasets and experiwith or without identity: a) the robot with identity is mentation in real contexts, need for considering privacy endowed with social intelligence abilities such as irony issues, and the adoption of appropriate validation apand empathetic behavior b) the robot without identity proaches. The issues highlighted in this analysis may be shows an apathetic/not expressive behavior (See Fig.4). useful for improving machine learning techniques and highlighting areas of interest in which experimenting with the use of diferent noninvasive sensors. Sensors may also be adopted for detecting challenging behavior of people with ASD [10].
uals is dramatically increasing. For them, it is dificult In [13] we presented a user-centered design approach to to get an early diagnosis or to intervene for preventing develop an emotion detection system by using: i) nonchallenging behaviors, which may be the cause of social invasive, wearable, low-cost sensors, ii) artificial intelli(a) Ironic Furath.
(b) Apathetic Furath.
Epoc+ headset equipped with 16 sensors, including 14 EEG channels.The goal of this task is to use the EEG signal obtained from an interaction between a person and a robot to classify a positive or negative emotion. Unlike traditional approaches, this work doesn’t apply classification to raw nor elaborated EEG-signals, but it transforms EEG into topomaps and adopts a CNN for the classification task (Fig. 5).
and treatment of people with ASD
Social stories have been introduced by Carol Gray and the use of this educational intervention methodology has been widely adopted, both by schools and at home, to help children with ASD to adapt to changes and face novel situations. Caregivers and therapists should be able to create customized digital social stories that take into account the ASD person’s level of severity and his/her peculiarity. Gray provides guidelines for the construction of efective social stories , but this may not be an easy task for all caregivers who, in the majority of cases, are often relatives of the individual, and may not be experts in the use of digital devices and/or the creation of social stories. Thus, automatic or semi-automatic support may be helpful. In [11] we introduced an idea for the design of an intelligent editor of social stories. In [12], we desired to empower caregivers of people with ASD by providing an application with a multimodal interface that supports and facilitates the creation of social stories. Caregivers know the target user better than anyone else. Therefore they are the best proposers of an efective social story for the peculiarity of the person they care of. Thus, we proposed a multimodal conversational interface of a mobile application for creating social stories integrated with an intelligent editor and supported the application with innovative existing usability guidelines suitable for a multimodal conversational interface. Seven caregivers of people with ASD have participated in the evaluation.
Usability results were encouraging. 3.7. Emotion recognition gence models for classifying emotions and iii) afective pants should be aware of the realabilty of this kind of analysis trough the Self-Assessment Manikin (SAM) ques- tool, otherwise they may decide to use it for "making tionnaire for assessing the impact of the system on the homework, programming and any tasks". Often this very user afective state. We conducted an emotional analysis positive opinion and intention to use survive also when by using diferent ML and DL models. Emotion detection an incorrect content is provided. based on patient’s speech and video analysis has been also adopted in [14] for supporting the clinician in the depression screening. In [15] we presented a patient- 4. Work in progress centered interaction project involving the Pepper humanoid robot for therapy applications for children with attention deficit. This new therapeutic methodology was created to support and make therapeutic work more attractive. The robot integrated with AI analyzes the child’s expressions and recognizes his state of mind. This helps the therapist to build the right therapeutic path and the right behavioral measures to take.
We are currently investigating the possibility to use
AIempowered Robot to support clinicians in assessment
and training activities. During last years, the use of social
robots has been in fact explored to provide assistance to
clinicians to the administration of
pshycological/phsycometric assessment of patients with diverse disorders
[
AI interaction, particularly in scenarios where AI systems are designed to assist or augment human decision- Supporting schizophrenia PatiEnts’ Care wiTh making. AI-based systems are increasingly being em- Robotics and Artificial Intelligence. At the present, ployed in various fields such as healthcare, finance, and we are investigating how robots and AI may be adopted transportation, where accurate and reliable decisions are to support the diagnosis of people sufering from critical. In such cases, trust is necessary to ensure that schizophrenia who generally are characterized by speech humans accept the recommendations and decisions of disorders such as logorrea, dissociation of thought, or the AI system. Furthermore, reliability is crucial to en- perseveration [17]. To subministrate diagnostic tests is sure that the AI system consistently provides accurate a time consuming activity conducted by clinicians. We results that meet the users’ expectations. Without trust are developing an approach involving robots for suband reliability, the efectiveness of AI systems is signifi- ministrating the test and able to identify many speech cantly reduced, leading to diminished user satisfaction disorders by exploiting NLP models and signal processing and possibly negative outcomes. We investigated in [16] techniques. the impact of trust and reliability on the user experience and satisfaction towards one of the most recent and powerful AI tool, ChatGPT, a general-purpose language model, in providing accurate responses to a variety of questions. The study involved fifteen participants, all of whom had prior experience with ChatGPT, and were asked to provide a list of questions to be posed to the language model and highlighted that in many cases the tool produced fake info. These questions covered a range of topics, such as problem-solving, creativity, and search activities. The participants were asked to rate their satisfaction with ChatGPT both before and after the study, using a seven-point Likert scale. The results showed that Figure 6: Main RoboTald modules. while the participants’ satisfaction decreased after using the language model, it was not as significant as expected.
The median score passed from 6 before the experience We propose an approach named RoboTald in which a to 5 after it. Examples of ChatGPT "hallucinations" were robot is used to formulating questions according to the included in the paper, including a mathematical problem Thought and Language Disorder (TALD) scale to patients and a logic test, where ChatGPT provided incorrect so- and to record their answers. The audio files are then aulutions. Participants appreciated ChatGPT’s ability to tomatically transcripted and inputted to a deeplearninggenerate realistic and polite responses, even if they were based system aiming at assessing the speech disorders not always accurate. The study revealed that partici- basing on the TALD classification. Faced with the need to improve and speed up the classification and quantifi- rangam, S. Mesay, Stress classification using brain cation of disorders, various Artificial Intelligence (AI) signals based on lstm network, Computational Inmodels have been used. In this paper, we propose an telligence and Neuroscience, Article ID 7607592, 13 approach in which Natural Language Processing (NLP) pages (2022). doi:10.1155/2022/7607592. techniques are adopted to support the Clinicians in the [9] R. Francese, X. Yang, Supporting autism spectrum classification of the SZ patients degree of speech dis- disorder screening and intervention with machine orders according to the TALD scale by examining the learning and wearables: a systematic literature redefinition of each element of the scale and by providing view, Complex & Intelligent Systems 8 (2022) 3659– an algorithm for scoring the single items. Additionally, 3674. to reduce the time spent by the Doctors to administer the [10] R. Francese, M. Risi, G. Tortora, F. D. Salle, Thea: interviews, a humanized social robot is used to formulate empowering the therapeutic alliance of children questions according to the TALD scale to patients. A with ASD by multimedia interaction, Multim. Tools human-in-the-loop approach will be used to permit the Appl. 80 (2021) 34875–34907. specialist to provide a direct feedback to the robot to cor- [11] R. Francese, A. Guercio, V. Rossano, An intelligent rect sentences or suggest additional ones more suitable system to support social storytelling for people with for the particular patient. ASD, in: J. Kim, M. Singh, J. Khan, U. S. Tiwary, M. Sur, D. Singh (Eds.), Intelligent Human Computer Interaction - 13th Int. Conf., IHCI 2021, Kent, References OH, USA, December 20-22, 2021, Revised Selected Papers, volume 13184 of Lecture Notes in Computer
Science, Springer, 2021, pp. 372–378. [12] R. Francese, A. Guercio, V. Rossano, D. Bhati, A multimodal conversational interface to support the creation of customized social stories for people with ASD, in: P. Bottoni, E. Panizzi (Eds.), AVI 2022: International Conference on Advanced Visual Interfaces, Frascati, Rome, Italy, June 6 - 10, 2022, ACM, 2022, pp. 19:1–19:5. doi:10.1145/3531073.
3531118. [13] R. Francese, M. Risi, G. Tortora, A user-centered approach for detecting emotions with low-cost sensors, Multim. Tools Appl. 79 (2020) 35885–35907.
doi:10.1007/s11042-020-09576-0. [14] R. Francese, P. Attanasio, Emotion detection for supporting depression screening, Multim.
Tools Appl. 82 (2023) 12771–12795. doi:10.1007/ s11042-022-14290-0. [15] F. Amato, M. D. Gregorio, C. Monaco, M. Sebillo,
G. Tortora, G. Vitiello, Socially assistive robotics combined with artificial intelligence for ADHD, in: 18th IEEE Annual Consumer Communications & Networking Conference, CCNC 2021, Las Vegas,
NV, USA, January 9-12, 2021, IEEE, 2021, pp. 1–6. [16] I. Amaro, A. Della Greca, R. Francese, G. Tortora,
C. Tucci, Ai unreliable answers: A case study on chatgpt, in: HCI International 2023: 25th International Conference on human-Computer Interaction, HCI 2023, Danemark, June 23–28, 2023, Proceedings, To Appear, ???? [17] I. Amaro, R. D’errico L., Francese, M. Stafa, C. Tucci,
Supporting schizophrenia language disturb assessment through robots interaction and deep learning, Submitted to an International Conference (2023).