=Paper=
{{Paper
|id=Vol-3172/short2
|storemode=property
|title=An Approach to Exploiting Personal Memories in Humanoid Robots Serious Games for Cognitive Stimulation of Older Adults
|pdfUrl=https://ceur-ws.org/Vol-3172/short2.pdf
|volume=Vol-3172
|authors=Benedetta Catricalà,Davide Coffaro,Marco Manca,Andrea Mattioli,Fabio Paternò,Carmen Santoro
|dblpUrl=https://dblp.org/rec/conf/avi/CatricalaCMMPS22
}}
==An Approach to Exploiting Personal Memories in Humanoid Robots Serious Games for Cognitive Stimulation of Older Adults==
https://ceur-ws.org/Vol-3172/short2.pdf
An Approach to Exploiting Personal Memories in Humanoid
Robots Serious Games for Cognitive Stimulation of Older
Adults
Benedetta Catricalà1, Davide Coffaro1, Marco Manca1, Andrea Mattioli1, Fabio Paternò1*,
Carmen Santoro1
CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1, 56124 Pisa, Italy
Abstract
One of the goals of Ambient Assisted Living (AAL) solutions is to be able to stimulate the
cognitive resources of older adults. An innovative way to address such stimulation is the
use of serious games delivered through humanoid robots. Such devices are based on Internet
of Things technologies since they are collections of sensors and actuators in a human shape.
Serious games delivered through humanoid robots can provide an engaging way to perform
exercises useful for training human memory, attention, processing, and planning activities.
This paper presents an approach to supporting cognitive stimulation based on personal
memories. The humanoid robot can exhibit different behaviours through different
modalities, and propose the games in a way personalized to specific individuals’
requirements, preferences, abilities, and motivations, which can vary among older adults,
and even dynamically evolve over time for the same person depending on changing user
needs and health conditions.
Keywords 1
Humanoid robot, Personalization, Serious Games, Ambient Assisted Living
1 Introduction
By 2050, the number of individuals over the age of 85 is projected to be three times more than
today [26]. In this scenario, most older adults will need physical, social, and cognitive assistance.
Indeed, aging has a considerable impact on the health of older adults in terms of cognitive and
EMPATHY: 3rd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems.
Workshop co-located with AVI 2022, June 06, 2022, Frascati, Rome, Italy.
* Corresponding author
EMAIL: b.catricala@studenti.unipi.it (B. Catricalà); d.coffaro@studenti.unipi.it (D. Coffaro);
marco.manca@isti.cnr.it (M. Manca); andrea.mattioli@isti.cnr.it (A. Mattioli); fabio.paterno@isti.cnr.it (F.
Paternò); carmen.santoro@isti.cnr.it (C. Santoro)
ORCID: 0000-0003-1029-9934 (M. Manca); 0000-0001-6766-7916 (A. Mattioli); 0000-0001-8355-6909 (F.
Paternò); 0000-0002-0556-7538 (C. Santoro)
© 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
physical impairments, which influence the abilities to complete and perform basic activities of daily
living, such as cooking, shopping, managing the home, bathing, and dressing. Nowadays, a large
proportion of cognitive assistance is provided by informal caregivers, usually family members.
These caregivers often experience a negative impact on their psychological, emotional, and physical
well-being due to the high workload [3]. Given the high health care expenditure at older ages, and
their effects on family caregivers, new technologies to assist older adults with cognitive impairments
are urgently needed.
Non-pharmacological interventions, such as physical training, cognitive training, and social
stimulation activities have been used to mitigate the cognitive decline by maintaining or improving
cognitive abilities, social well-being, and quality of life of older adults [3, 5]. However, traditional
interventions require experienced instructors who may be unavailable. Assistive technologies can
provide useful support to address this problem. They are technologies that aim to assist different
types of users during their rehabilitation. They can help older adults maintain their independence
during daily routines and can also be an important instrument during their rehabilitation [15].
In recent years, humanoid robots have increased their similarity to human behaviour starting from
gestures and facial expressions to the ability to understand questions and provide answers. Thanks
to such humanlike characteristics, the interaction between people and robots is becoming more
natural. The behaviour of such robots can also be personalized through end-user development
approaches, such as the use of trigger-action rules and associated support [10]. A recent literature
review [16] indicates that the humanoid robot is an interactive technology still not sufficiently
investigated for supporting the cognitive stimulation of older adults.
In this paper, we present a novel approach based on a Pepper humanoid robot, with the purpose
to obtain games for cognitive stimulation of older adults. We consider this kind of technology in this
context because it can potentially promote seniors’ cognitive, physical, and emotional well-being
and also reduce the workload of the healthcare system [25]. A humanoid robot is a system that can
employ different interaction strategies, such as verbal and non-verbal communication, the use of
facial expressions, communicative gestures, and sensors. These capabilities are essential to creating
social and emotional interaction with users to increase their acceptability and users engagement,
which may increase the possibility to reach the goal of assistance in less time and with better results
[3].
Using robots to support and assist patients can be a valuable tool to help them during their
cognitive training. In such context, digital cognitive training through serious games may potentially
benefit those with cognitive impairments more than traditional training due to enhanced motivation
and engagement. In the literature, different studies show how digital games can obtain positive
results in stimulating older adults and helping them improve their cognitive abilities compared to
traditional training [24].
Combining a humanoid robot and a serious game can be an exciting solution to obtain measurable
progress in cognitive functions and stimulate the user to continue the training program. We aim to
offer novel digital training through serious games designed using personally relevant material from
older adults‘ life. They will be based on elements associated with their biography, thus making
interactions personalised, relevant, and more engaging.
2
2 The Remind Approach
The psychological well-being of older adults may be affected by some age-related conditions,
such as approaching death, loss of family members, and reduced autonomy. A meta-analysis [2]
indicates that the practice of life review, even more than reminiscence (reminiscence involves
describing a memory itself, while life review is based on discussing what a memory means), is a
good instrument for improving the psychological well-being of older adults: in fact, its effect sizes
are comparable to those of cognitive-behavioural therapy. Serrano et al. [17] found that the practice
of autobiographical memory improved the mood of the elderly by improving their life satisfaction.
Furthermore, Damianakis et al. [6] report that interventions that contextualize history, personality,
and life experiences can contribute to improving both communication and social interactions
between family members and between family members and formal caregivers.
Based on previous experiences [12], we have started the development of a new prototype in which
the serious games installed on the humanoid robot will motivate older adults by engaging them in
playful situations that draw on their personal memories, with which they can interact. Indeed, such
serious games are designed using personally relevant material from older adults’ life. Specifically,
the games are based on elements associated with the biography of the older adults, thus making
interactions more relevant and more likely to keep them engaged while enhancing their well-being
as well. In the following, we describe the approach proposed for a platform able to support cognitive
stimulation through humanoid robots, the multimodal app for collecting memories, and the first set
of games able to exploit such memories through the humanoid robot.
2.1 The REMIND Platform
Biographical information is exploited in a group of games that aim to support tasks related to
Activities of Daily Living (ADL), which seem to have some potential in terms of both utility and
cognitive stimulation according to a recent systematic review [16] that has analyzed various game
types employed in this domain. An example game is one whose object is carrying out cooking
activities, such as replication of recipes: an activity that can stimulate multiple cognitive domains
simultaneously (memory, attention, planning), but that can also be adapted and calibrated to
stimulate in a more targeted way each of the cognitive domains mentioned above. In addition, the
evocation and detection of a user’s emotional state are becoming a crucial element in the aim of
developing more effective interfaces between humans and computers, especially in applications such
as games and e-learning tools [14]. This is typically done by analyzing emotions while playing,
which can be performed by using sensors in a wristband to detect relevant information. For this
purpose, we aim to infer emotional states from physiological measures. Galvanic skin response, skin
temperature, heart rate, and Inter-beat (RR) intervals are all valuable in inferring emotional states.
We consider some of these measures detected also through an Empatica E4 wristband in order to
assess how engaged participants feel while playing the games. The humanoid robots will serve as
personal trainers, proposing exercises and communicating through various modalities, and
challenging users in cognitive games relevant to their daily life (e.g. by memorizing scheduled events
or names of family members and friends).
3
The solution aims to allow doctors and caregivers to configure the exercises and choose the most
suitable games to stimulate the cognitive abilities of end users and enhance their experience. The
REMIND platform is based on a modular architecture allowing the deployment of the multimodal
serious cognitive games on a humanoid robot, which can stimulate interest and engagement from
seniors that would not be possible with other types of smaller and more limited robots thanks to its
human-like appearance and behaviour.
On the one hand the older adults, by interacting with the biographical app, provide relevant
biographical data that are mainly used to customize the games, which thereby will be highly
personalised for them. On the other hand, seniors will also interact with the games in order to
stimulate their cognitive abilities; the data produced during the interactive sessions will be exploited
by the Game Logic (to improve the adaptation of the game according to the data gathered in previous
game sessions) as well as by the Analytics services, which will improve its analysis by also including
such data. The caregivers will be able to interact with a Game personalisation tool, which allows
them to provide further relevant personalisations for the interaction with the games. In addition, the
caregiver will also interact with an Analytics tool, to have both overview and detailed information
regarding user performance and state. The goal is to develop end-user configurable games. Game
developers implement the features, functionalities, and graphics of the core game. The caregivers
will be able to configure the game, according to the skill and learning level of the end users. For this
purpose, previous programming experience is not required, as the game's configuration will be
performed through intuitive user interfaces. In addition, the games include a custom tracking system,
which tracks the data about user performance (such as time, date, pass/fail, score) and other
important metrics necessary for the analytics of serious games (comments, number of errors,
completion level, skill level achieved etc.).
2.2 The Biography App
A responsive Web application has been developed to collect older adults’ memories. It can
receive them both through graphical and vocal interaction. At the beginning, users are asked whether
they want to enter a new memory or review those previously entered. A first set of possible categories
of memories has been identified, and shown at the user access (see Figure 1, left). They can refer to
music, specific events, games that the user liked to play, locations, food, and hobbies.
Figure 1: (left) User interface for selecting memory category; (right) user interface for adding a
hobby-related memory
4
When a memory category is selected, then the user can provide the information associated with
the specific memory. For example, Figure 1 (right) shows the user interface associated with entering
a memory related to a hobby. The user has to indicate a name characterizing the hobby, then provide
a description (it can be done either vocally or by keyboard), the user can also indicate the age at the
time of the hobby, and then provide a list of activities required by the hobby. Such list can then be
used by the Pepper game for specific exercises.
2.3 The Games with the Humanoid Robot
The prototype aims to stimulate cognitive functions through play sessions, which should last 15-
20 minutes. The games present various exercises such as completion of a memory with the main
element missing, choosing from a multiple choice of elements, and in case of a correct answer the
memory is reread to the older adult. The exercises should be useful for making the participants think
and reason to provide the correct answer. An example is a memory: “In 1950 I was in Paris with my
daughter Eleonora and I ate at the Angel restaurant”, and Pepper will introduce it with some missing
elements, asking for example “Do you remember the missing element of your memory in 1950 when
you were in Paris with your daughter Eleonora and ate in the …. restaurant"?
Another exercise is to reorder a list of activities to perform a task (hobby, job or home procedures),
starting with an unordered list (this stimulates executive functions and procedural memory). For
example, considering the list of home activities when users wake up, Pepper proposes the following
list: make the bed, rinse your face in the bathroom, do morning exercises, have breakfast, go to the
newsstand; and the older adult should reorganise it.
One exercise is about the association of a memory to a main connected element. For example,
Pepper can present three memories and three locations and ask the user to indicate for each memory
the corresponding location. A further exercise considers the year of a memory, and asks the user to
select which important event in the world happened that year from a set of events that are listed.
Another exercise asks questions about an element drawn from the memories already saved for a
specific older adult. Suppose the memory is about something that happened in the city of Pisa. Then
there can be questions such as: What are the main squares or places of interest in Pisa? What is the
name of the river that crosses Pisa? What are the names of the inhabitants of Pisa? In the case of
memories about a song, then the questions can be about the year of the song or the singer.
We also plan to use memories uploaded by relatives or people close to a participant, tell the
memory to the older adult, and pose questions or ask them to complete sentences regarding the
relative's memory. For example, Pepper says “I have a memory from your daughter Eleonora, do
you want me to tell you about it? The memory is when she visited Florence, she saw the most famous
monuments and entered the Uffizi to see the works inside; her favourite was Birth of Venus ". Then,
Pepper asks questions such as “What was Eleonora's favorite work in the Uffizi?”
5
Figure 2: (left) Session start with the robot; (centre) robot with the list of available game topics;
(right) a question from the memory game “missing element completion”.
3 Conclusions
In this paper, we introduce a novel approach to personalizing serious games for cognitive
stimulation of older adults delivered through a humanoid robot. It is based on a multimodal Web app
to collect memories of the older adults, and then such content is exploited in a set of games
implemented for a Pepper robot.
In future work, we plan to validate the approach with a user test involving a group of people aged
65+, participating in a Train the Brain programme in which they regularly attend cognitive
stimulation exercises in a care centre. We also plan to develop a customization and analytics
environment to further extend the possible personalizations of games.
Acknowledgements
This work has been partly supported by the PRIN 2017 “EMPATHY: Empowering People in
Dealing with Internet of Things Ecosystems”, www.empathy-project.eu/ and the CNR project
SERENI https://hiis.isti.cnr.it/sereni/index.html.
REFERENCES
[1] Abdollahi, H., Mollahosseini, A., Lane, J., & Mahoor, M. (2017). A pilot study on using an
intelligent life-like robot as a companion for elderly individuals with dementia and depression.
IEEE-RAS 17th International Conference on Humanoid Robotics, (pp. 541-546).
6
[2] Bohlmeijer, E., Roemer, M., Cuijpers, P., & Smit, F. (2007). The effects of reminiscence on
psychological well-being in older adults: A meta-analysis. Aging and Mental Health, 11(3),
291-300.
[3] Carros, F., Meurer, J., Loffer, D., & Unbehaun, D. (2020). Exploring human-robot interaction
with the elderly: results from a ten-week case study in a care home. Proceedings of the 2020
CHI Conference on Human Factors in Computing Systems, (pp. 1-12).
[4] J. Costa, T. Paul, of personality theories: Theoretical contexts for the five-factor model, The
five-factor model of personality: Theoretical perspectives 51 (1996).
[5] Cruz-Sandoval, D., Morales-Tellez, A., Sandoval, E. B., & Favela, J. (2020). A social robot as
therapy facilitator in interventions to deal with dementia-related behavioral symptoms.
Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction
(pp. 161-169). ACM/IEEE.
[6] Damianakis, T., Crete-Nishihata, M., Smith, K. L., Baecker, R. M., & Marziali, E. (2010). The
psychosocial impacts of multimedia biographies on persons with cognitive impairments. The
Gerontologist, 50(1), 23-35.
[7] Hall, Edward T. ”A system for the notation of proxemic behavior.” American anthropologist
65.5 (1963): 1003-1026.
[8] John, O. P., & Srivastava, S. (1999). The Big Five Trait taxonomy: History, measurement, and
theoretical perspectives. Handbook of personality:Theory and research, 2nd ed.
[9] Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M. T. (2014). Socially assistive robots in
elderly care: a mixed-method systematic literature review. International Journal of Human-
Computer Interaction, 30(5), 369-393.
[10] N.Leonardi, M.Manca, F.Paternò, C.Santoro, Trigger-Action Programming for Personalising
Humanoid Robot, the 2019 ACM Conference on Human Factors in Computing Systems
(CHI'19), Paper 445
[11] Lippa, R. A., & Dietz, J. K. (2000). The relation of gender, personality, and intelligence to
judges’ accuracy in judging strangers’ personality from brief video segments. Journal of
Nonverbal Behavior, 24, 25–43.
[12] Manca, M., Paternò, F., Santoro, C., Zedda, E., Braschi, C., R., F., & Sale, A. (2021). The
impact of serious games with humanoid robots on mild cognitive impairment older adults.
International Journal of Human-Computer Studies, 145, 102509.
[13] Manera, V., Petit D.P., Derreumaux, A., Orvieto, I., Romagnoli, A., Lyttle, G., . Robert, ..
(2015). Kitchen and cooking, a serious game for mild cognitive impairment and Alzheimer’s
disease: a pilot study. Frontiers in aging neuroscience.
[14] Nakasone, A., Prendinger, H., & Ishizuka, M. (2005, September). Emotion recognition from
electromyography and skin conductance. In Proc. of the 5th international workshop on biosignal
interpretation (pp. 219-222).
[15] Nishiura, Y., Nihei, M., Nakamura-Thomas, H., & Inoue, T. (2021). Effectiveness of using
assistive technology for time orientation and memory, in older adults with or without dementia.
Disability and Rehabilitation: Assistive Technology 16(5), 472-478.
[16] Palumbo, V., & Paternò, F. (2020). Serious games to cognitively stimulate older adults: a
systematic literature review. PETRA (pp. 27:1-27:10). ACM Press.
7
[17] Serrano, J. P., Latorre, J. M., Gatz, M., & Montanes, J. (2004). Life review therapy using
autobiographical retrieval practice for older adults with depressive symptomatology.
Psychology and aging, 19(2), 272.
[18] Smith, E., Nolen-Hoeksema, S., Fredrickson, B., & Loftus, G. (2002). Atkinson and Hilgard’s
Introduction to Psychology. Wadsworth.
[19] Softbank Robotics. (2021, 07 10). Pepper SDK-Tools. Retrieved from QiSDK:
https://qisdk.softbankrobotics.com/sdk/doc/pepper-sdk/ch3_tools/tools.html
[20] Tanaka, M., Ishii, A., Yamano, E., Ogikubo, H., Okazaki, M., & Kamimura, K. (2012). Effect
of a human-type communication robot on cognitive function in elderly women living alone.
Medical science monitor: international medical journal of experimental and clinical research,
18(9).
[21] Tapus, A., & Vieru, A. M. (2013). Robot cognitive stimulation for the elderly. International
Work-Conference on the Interplay Between Natural and Artificial Computation (pp. pp. 94-
102). Springer.
[22] A.Tapus, C. Tapus, C., & Matarić, M. .. (2008). User—robot personality matching and assistive
robot behavior adaptation for post-stroke rehabilitation therapy. Intelligent Service Robotics,
169-183.
[23] Tay, B., Jung, Y., & Park, T. (2014). When stereotypes meet robots: the double-edge sword of
robot gender and personality in human–robot interaction. . Computers in Human Behavior, 38,
75-84.
[24] Tong, T., & Chignell, M. (2013). Designing game-based cognitive assessments for elderly
adults. Proceedings of the First International Conference on Gameful design, research, and
applications, (pp. 127-130).
[25] Vänni, K., & Salin, S. (2019). Attitudes of Professionals Toward the Need for Assistive and
Social Robots in the Healthcare Sector. Korn O. (eds) Social Robots: Technological, Societal
and Ethical Aspects of Human-Robot Interaction. Human–Computer Interaction Series.
Springer, Cham.
[26] World Health Organization. (2021, 07 12). Ageing and health. Retrieved from World Health
Organization: https://www.who.int/news-room/fact-sheets/detail/ageing-and-health.
8