=Paper=
{{Paper
|id=Vol-2474/shortpaper9
|storemode=property
|title=Facial expression recognition from nao robot within a memory training program for individuals with mild cognitive impairment
|pdfUrl=https://ceur-ws.org/Vol-2474/shortpaper9.pdf
|volume=Vol-2474
|authors=Berardina Nadja De Carolis,Giuseppe Palestra,Olimpia Pino
|dblpUrl=https://dblp.org/rec/conf/smc/CarolisPP19
}}
==Facial expression recognition from nao robot within a memory training program for individuals with mild cognitive impairment==
https://ceur-ws.org/Vol-2474/shortpaper9.pdf
Facial expression recognition from nao robot
within a memory training program for individuals
with mild cognitive impairment
1st Berardina De Carolis 2nd Giuseppe Palestra 3rd Olimpia Pino
Department of Computer Science Research Department Department of Psychology
University of Bari “A. Moro” HERO srl University of Parma
Bari, Italy Apulia, Italy Parma, Italy
berardina.decarolis@uniba.it giuseppepalestra@gmail.com olimpia.pino@unipr.it
Abstract—Mild Cognitive Impairment refers to a borderline state provide helpful feedback, to analyze patients’ performances
between healthy aging and dementia. Memory-training program over the time [4]. The psychologists’ primary objective for MCI
plays a crucial role in the reduction of the possible conversion in
dementia and a robot mediated memory training is useful to patients is to keep their cognitive ability when functional
overcome limits of traditional programs. The present study addresses capabilities and independence are not compromised [2]. The
the effectiveness of a system in automatically recognize facial worldwide incidence of MCI expects to increase in the next few
expression from video recorded sessions of a robot mediated memory
training lasted 2 months involving 21 patients. The system is able to
years [3]; however, space and personnel shortages are already
recognize facial expressions from group sessions handling partially becoming a problem owing to an unprecedented rise in life
occluded faces. Findings showed that in all participants the system is expectancy [3]. In very recent years, new tools and
able to recognize facial expressions. technologies based on machine learning and robotics are
Index Terms—facial expression recognition, social robot, memory
training, Mild Cognitive Impairment
successfully applied to the field of psychology and could be
used to in memory training program for people with MCI.
Humanoid robots are able to improve mood, emotional
I. INTRODUCTION expressiveness and social relationships among patients with
dementia [5]– [7] also executing many assistive functionalities
Mild Cognitive Impairment (MCI) concerns a stage between
[8]–[10] and providing life assistance demonstrating that the
normal aging and early dementia marked by cognitive deficit
information support provided by the robot also has the
characterized by scores below the norm on psychometric
potential to improve the daily life of persons with a mild level
tests, preserved functional abilities and high levels of quality
of dementia [5]. Most recent advances in information and
of life [1]. The prevalence of MCI in individuals >65 years of age
communication technologies have enabled the development
is between 10 and 20%. MCI is highly likely to convert in
of telepresence robots to connect a family member and a
dementia at a rate of about 13% per year and in the rest of the
person with dementia as a means of enhancing
patients the impairment persists stable or even return to
communication between these two parties [11]. The
normal over time [2]. Dementia was estimated to have been
humanoids skills are progressively enhanced: they are able to
detected around the globe at the rate of one new patient
recognize faces, call people by their name, shape their
about every 7 seconds. Therefore, MCI has become a relevant
behavior considering the mood of people interacting with
research topic because it could play a critical role in
them [12], [13]. Some robots can also reproduce emotions
distinguishing developmental changes in lifespan memory
[12], [14], making their human mate feel welcomed, and
from those that are real signs of the disorder. Delaying the
simulating empathy [15], [16]. Kinetics technology can help
onset of dementia by as little as one year could decrease the
them reproduce movements [17], while speech recognition
global burden of Alzheimer’s by 9 million of patients in 2050
software allows them to respond to what people say, even in
[3]. In order to maintaining cognitive functions non-
many different languages. During human-robot interaction,
pharmacological programs are developed. These programs
the mirror-circuit, responsible for social interaction, is verified
involve qualified psychologist and therapists in order to
to be active [18] suggesting that humans can consider robots
conduct new tasks and new exercises, to monitor the
as real companions with their own intentions. Many studies
performance of the patients, to
have employed the robot NAO. If appropriately programmed,
it is able to decode human emotions, simulate emotions
SAT19: 1st Workshop on Socio-Affective Technologies: an interdisciplinary
approach, October 7, 2019, Bari, Italy through the color of his eyes or the position of the body,
recognize faces and model physical exercise to a group of
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
seniors [19], and equipped to measure of health and interfere with study participation, such Parkinson’s disease,
environmental parameters [20]. Robotics could partially fill in HIV/AIDS, or other contraindications. Informed consent was
some of the identified gaps in current health care and home obtained from all the patients or from their legal
care/self-care provisions for promising applications in these representatives when appropriate. 21 individuals (10 females
fields that we expect to play relevant roles in the near future. and 11 males) participated in the experiment with a mean age
With emerging research suggesting that mobile robot systems of 73.45 years (SD = 7.71). The mean education level was of
can improve elderly care [6], [7], [10], also through the 9.90 years (SD = 4.58) with a minimum value of 5 years
development of a coding system aimed at measuring corresponding to the conclusion of the elementary school and
engagement-related behavior across activities in people with a maximum value of 18 years which corresponds to a bachelor
dementia [21], [22]. With the growing incidence of pathologies degree.
and cognitive impairment associated with aging, there will be
B. Robot Mediated Memory Training Program
an increasing demand for maintaining care systems and
services for elderly with the imperative of economic cost- The memory training exercises were implemented on NAO
effectiveness of care provision. In our previous work [23], NAO on the basis of exercises described in literature [24] and aimed
has been evaluated as mediator in a memorytraining program to train: i) focused attention; ii) divided and alternate
for people with MCI in a center for cognitive disorders therapy. attention; and iii) categorization and association as learning
The focus of the present paper was evaluate the feasibility strategies. Five tasks were implemented in NAO, considering
and usefulness of NAO platform in cognitive stimulation. In the characteristics of the robot:
human interaction emotional and social signals expressing 1) Reading stories;
additional information are essential. For this aim, particular 2) Questions about the story;
attention was paid to quantifying the effectiveness of robot on 3) Associated/not associated words;
well being of the training recipients and this was realized 4) Associated/not associated word recall;
measuring facial expressions at the same time gaze when 5) Song-singer match;
human participants interact with the synthetic agent. This
study aims to exploit video clips recorded during the 2-month C. Video corpus
experiment of the previous study analyzing facial expressions In this study, we used a corpus of 48 memory program
of the participants in the experiment. The system presented is session video clips of one hour from 24 therapeutic sessions.
able: i) to recognize group facial expression thanks a multiface Each video clips recorded three or four participants. These
detector; ii) to recognize facial expression in partially occluded videos were recorder during two months by two cameras
faces. This paper is organized as follows. Section 2 presents placed in the therapeutic room and they have been used to
Materials and Methods used. Section 3 reports experimental record all participants. Overall, at the end of the experiment
results. Finally, conclusions are drawn. for each participant a total of eight hours of video have been
collected.
II. MATERIALS AND METHODS
A. Participants D. Group Facial Expression Recognition System
The participants were selected from the population of The group facial expression recognition system detect faces
outpatients attending the Center for Cognitive Disorders and in the corpus and then recognize 6 basic emotions (anger,
Dementia of AUSL Parma (Italy). All the participants were disgust, fear, happiness, sadness and surprise) plus neutral
firstly evaluated by memory-disorders specialists. The expression. The video analysis system is based on a previous
diagnosis of MCI was based on a detailed medical history, study that aimed to recognize six basic emotions through facial
relevant physical and neurological examinations, negative expression [25] but the system has been improved in order: i)
laboratory findings, and neuroimaging studies. Subjects are to work in groups using a multi-face detector; ii) handle partial
enrolled according to the following inclusion criteria: a) occlusions of the face. In group facial expression recognition
diagnosis of MCI obtained through Petersen guidelines, and handle occlusions is very important in order to ensure a high
full marks in the two tests measuring daily living activities (ADL accuracy rate. Indeed, sometimes the face of the participants
and IADL); b) both genders; c) chronological age comprised is occluded by a hand as well as by an arm of the other
between 45 and 85 years; and d) without pharmacological participants as shown in Figure 1.
treatment. Exclusion criteria were a diagnosis of major
neurocognitive disorder (defined using DSM 5 criteria), history
of symptomatic stroke (although silent brain infarction was not
an exclusion), history of other central nervous system
diseases, serious medical or psychiatric illness that would
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
expression and an average accuracy of 94,24% on 6 basic facial
expression plus neutral.
Overall, the system extracts 32 geometric features that have
been used in total or in part (to handle occlusions) in order to
train a model. To recognize facial expression, the system uses
a classification module that, through a Random Forest
classifier, analyzes the geometric characteristic vectors to
determine the facial expression.
III. RESULTS
Fig. 1. Examples of occlusion in the video corpus. In order to test the system, the video clips of the corpus have
been down sampled at 1 frame per second. For each
After identifying a face, the system extracts, for each face, participant 28,800 frames have been analyzed (1
facial landmarks locating 77 key points. Once the 77 points are frame/second x 8 hours of recording). To evaluate the facial
identified, the software tracks linear, polygonal, elliptical and expression recognition during the memory training program
mediated by the robot: i) the number of detected face (nFD);
ii) and the number of each facial expression recognized (nFE)
for each frame for each participants in the video corpus have
been used as metrics. Overall, the system was able to analyze
all the
Fig. 2. Facial expression analyzing software flow diagram.
Fig. 3. Facial expression recognized in the video corpus in percentage.
angular characteristics, i.e. the distance between two points to
find the following: three lines describing the left eyebrow; two
defining the left eye; one for the cheeks; one for nose; eight video corpus and all the 21 faces of the participants have been
for the mouth. The system then determines polygonal detected. With respect to nFD, it has been observed that, in
features, calculating the area delimited by irregular polygons percentage, a face has been detected in total of the corpus
created using three or more key reference points, specifically: with a success rate of 56%. Moreover, respect to the nFE, it has
one for the left eye; one forming a triangle between the been observed that in each frame where a face had been
corners of the left eye and the left corner of the mouth; one detected (also if partially occluded) the system was able to
for the mouth. Thus, the system traces the elliptic recognize a facial expression. In percentage, the three most
characteristics, calculated by the ratio between the major axis common facial expressions are neutral that has been founded
and the minor axis of the ellipse, in particular seven ellipses in the 41% of the frames, happiness in the 17% and sadness in
are chosen between the reference points: one for the left the 15% as shown in Figure 3.
eyebrow; three for the eye, left upper and lower eyelid; three In Table I are reported the number of frames where a face
for the mouth, lower and upper lips. The pipeline of the system had been detected and Table II reports the number of facial
is depicted in Figure 2. The system has been tested on the expression recognized for each participant divided into facial
Extended Cohn-Kanade (CK+) data set, a well known facial expressions.
expression image database of 123 individuals of different
gender, ethnicity and age. The system reach an average facial TABLE I
expression recognition accuracy of 95,46% on six basic facial NUMBER OF FACE DETECTED (NFD) FOR EACH PARTICIPANT (#).
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Participant (#) nFD track of the mood of the individuals involved during the
1 23,567 training sessions. In future work, other analysis on the video
2 22,678 corpus will be done to understand the engagement, how the
3 16,709 participants have been involved during the memory training
4 18,541
program and how to better involve the individuals with MCI in
5 20,341
new memory training program mediated by a social robot.
6 16,354
7 15,980
8 12,434 REFERENCES
9 11,345
10 16,709 [1] O. Pino, “Memory impairments and rehabilitation: Evidence-based
11 18,541 effects of approaches and training programs,” The Open Rehabilitation
12 13,678 Journal, vol. 8, no. 1, 2015.
[2] S. Gauthier, B. Reisberg, M. Zaudig, R. C. Petersen, K. Ritchie, K. Broich,
13 12,454
S. Belleville, H. Brodaty, D. Bennett, H. Chertkow et al., “Mild cognitive
14 13,544 impairment,” The Lancet, vol. 367, no. 9518, pp. 1262– 1270, 2006.
15 25,123 [3] T. G. Ton, T. DeLeire, S. G. May, N. Hou, M. G. Tebeka, E. Chen, and J.
16 16,341 Chodosh, “The financial burden and health care utilization patterns
17 11,490 associated with amnestic mild cognitive impairment,” Alzheimer’s &
18 10,431 Dementia, vol. 13, no. 3, pp. 217–224, 2017.
19 14,235 [4] O. Pino, Ricucire i ricordi: la memoria, i suoi disturbi, le evidenze di
efficacia dei trattamenti riabilitativi. Mondadori universita, 2017.`
20 12,357
[5] S. Sabanovic, C. C. Bennett, W.-L. Chang, and L. Huber, “Paro robot
21 18,235 affects diverse interaction modalities in group sensory therapy for older
IV. CONCLUSIONS adults with dementia,” in Rehabilitation Robotics (ICORR), 2013 IEEE
International Conference on. IEEE, 2013, pp. 1–6.
In this paper a system for group facial expression recognition [6] H.-M. Gross, C. Schroeter, S. Mueller, M. Volkhardt, E. Einhorn, A. Bley,
with handling of partial occlusions has been presented T. Langner, C. Martin, and M. Merten, “I’ll keep an eye on you: Home
TABLE II robot companion for elderly people with cognitive impairment,” in
NUMBER OF FACIAL EXPRESSION RECOGNIZED (NFE) FOR EACH Systems, Man, and Cybernetics (SMC), 2011 IEEE International
PARTICIPANT (#) DIVIDED INTO FACIAL EXPRESSIONS (AN=ANGER, Conference on. IEEE, 2011, pp. 2481–2488.
DI=DISGUST, FE=FEAR, HA=HAPPINESS, SA=SADNESS, SU=SURPRISE, NE=NEUTRAL). [7] F. Mart´ın, C. Aguero, J. M. Ca¨ nas, G. Abella, R. Ben˜ ´ıtez, S. Rivero, M.
Valenti, and P. Mart´ınez-Mart´ın, “Robots in therapy for dementia
# nFE AN DI FE HA SA SU NE
patients,” Journal of Physical Agents, vol. 7, no. 1, pp. 48–55, 2013.
1 23567 987 345 2312 4054 2345 870 12654 [8] C. D. Kidd and C. Breazeal, “Robots at home: Understanding longterm
2 22678 1022 1233 1982 3281 1757 765 12638 human-robot interaction,” in Intelligent Robots and Systems, 2008. IROS
3 16709 998 374 1534 2918 3982 551 6352 2008. IEEE/RSJ International Conference on. IEEE, 2008, pp. 3230–3235.
4 18541 1543 235 2234 3002 4526 549 6452 [9] B. Graf, C. Parlitz, and M. Hagele, “Robotic home assistant care-o-bot®¨
5 20341 738 397 2391 3384 1902 847 10682 3 product vision and innovation platform,” in International Conference
6 16354 543 235 1534 3002 1138 549 9353 on Human-Computer Interaction. Springer, 2009, pp. 312–320.
[10] H.-M. Gross, C. Schroeter, S. Mueller, M. Volkhardt, E. Einhorn, A. Bley,
7 15980 637 332 932 3363 1332 321 9063
T. Langner, M. Merten, C. Huijnen, H. van den Heuvel et al., “Further
8 12434 1101 356 2291 2308 2347 394 3637
progress towards a home robot companion for people with mild
9 11345 1438 234 1982 1221 1395 254 4821 cognitive impairment,” in Systems, Man, and Cybernetics (SMC), 2012
10 16709 1412 316 1535 2237 4223 332 6654 IEEE International Conference on. IEEE, 2012, pp. 637–644.
11 18541 873 339 1982 3922 2023 748 8654 [11] W. Moyle, C. Jones, M. Cooke, S. ODwyer, B. Sung, and S. Drummond,
12 13678 667 254 1625 3323 2145 782 4882 “Connecting the person with dementia and family: a feasibility study of
13 12454 823 443 1912 3231 1198 256 4591 a telepresence robot,” BMC geriatrics, vol. 14, no. 1, p. 7, 2014.
[12] B. De Carolis, S. Ferilli, and G. Palestra, “Simulating empathic behavior in
14 13544 2383 487 1112 1005 1345 870 6342
a social assistive robot,” Multimedia Tools and Applications, vol. 76, no.
15 25123 1587 429 2542 4124 2891 794 12756
4, pp. 5073–5094, 2017.
16 16341 673 345 1612 2905 945 651 9210 [13] B. De Carolis, N. Macchiarulo, and G. Palestra, “Soft biometrics for social
17 11490 2012 234 1234 1123 3487 870 2530 adaptive robots,” in International Conference on Industrial, Engineering
18 10431 970 221 873 3130 1881 703 2653 and Other Applications of Applied Intelligent Systems. Springer, 2019,
19 14235 2189 884 3823 1290 3321 218 2510 pp. 687–699.
20 12357 1123 543 2912 2931 2283 214 2351 [14] B. N. De Carolis, S. Ferilli, G. Palestra, and V. Carofiglio, “Towards an
21 18235 987 345 1982 2052 2345 870 9654 empathic social robot for ambient assisted living.” in ESSEM@ AAMAS,
pp. 19–34. [Online]. Available:
as well as its application to the field of psychology. A particular https://dl.acm.org/citation.cfm?id=3054107
application such as a memory training program for MCI in [15] V. Gallese, C. Keysers, and G. Rizzolatti, “A unifying view of the basis of
adults can benefit from computer vision and machine learning social cognition,” Trends in cognitive sciences, vol. 8, no. 9, pp. 396– 403,
2004.
technologies to understand better how patients react to robot
[16] B. De Carolis, S. Ferilli, and G. Palestra, “Improving speech-based human
mediated memory training program. Moreover, from robot interaction with emotion recognition,” in International
automatic facial expression recognition psychologists can keep
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Symposium on Methodologies for Intelligent Systems. Springer, 2015,
pp. 273–279.
[17] B. S. Ertugrul, C. Gurpinar, H. Kivrak, and H. Kose, “Gesture recogni-˘ tion
for humanoid assisted interactive sign language tutoring,” in Signal
Processing and Communications Applications Conference (SIU), 2013
21st. IEEE, 2013, pp. 1–4.
[18] V. Gazzola, G. Rizzolatti, B. Wicker, and C. Keysers, “The
anthropomorphic brain: the mirror neuron system responds to human
and robotic actions,” Neuroimage, vol. 35, no. 4, pp. 1674–1684, 2007.
[19] I. Back, K. Makela, and J. Kallio, “Robot-guided exercise program for¨ the
rehabilitation of older nursing home residents,” Annals of Long Term
Care, vol. 21, no. 6, 2013.
[20] J. P. Vital, M. S. Couceiro, N. M. Rodrigues, C. M. Figueiredo, and N. M.
Ferreira, “Fostering the nao platform as an elderly care robot,” in 2013
IEEE 2nd International Conference on Serious Games and Applications for
Health (SeGAH). IEEE, 2013, pp. 1–5.
[21] G. Perugia, R. van Berkel, M. D´ıaz-Boladeras, A. Catala-Mallofr` e,´ M.
Rauterberg, and E. Barakova, “Understanding engagement in dementia
through behavior. the ethographic and laban-inspired coding system of
engagement (elicse) and the evidence-based model of
engagementrelated behavior (emodeb),” Frontiers in psychology, vol. 9,
2018.
[22] P. Boissy, H. Corriveau, F. Michaud, D. Labonte, and M.-P. Royer,´ “A
qualitative study of in-home robotic telepresence for home care of
community-living elderly subjects,” Journal of telemedicine and telecare,
vol. 13, no. 2, pp. 79–84, 2007.
[23] O. Pino, G. Palestra, R. Trevino, and B. De Carolis, “The humanoid robot
nao as trainer in a memory program for elderly people with mild
cognitive impairment,” International Journal of Social Robotics, pp. 1–
13, 2019.
[24] D. Gollin, A. Ferrari, and A. Peruzzi, Una palestra per la mente.
Stimolazione cognitiva per l’invecchiamento cerebrale e le demenze.
Edizioni Erickson, 2007.
[25] G. Palestra, A. Pettinicchio, M. Del Coco, P. Carcagni, M. Leo, and C.
Distante, “Improved performance in facial expression recognition using
32 geometric features,” in International Conference on Image Analysis
and Processing. Springer, 2015, pp. 518–528.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).