=Paper=
{{Paper
|id=Vol-3794/paper08
|storemode=property
|title=Train the Brain with SERENI
|pdfUrl=https://ceur-ws.org/Vol-3794/paper8.pdf
|volume=Vol-3794
|authors=Benedetta Catricalà,Marco Manca,Fabio Paternò,Carmen Santoro,Eleonora Zedda
|dblpUrl=https://dblp.org/rec/conf/rfh/Catricala0PSZ24
}}
==Train the Brain with SERENI==
https://ceur-ws.org/Vol-3794/paper8.pdf
Train the Brain with SERENI
Benedetta Catricalà1,† , Marco Manca1,† , Fabio Paternò1,∗,† , Carmen Santoro1,† and Eleonora Zedda1,†
1
CNR-ISTI, HIIS Laboratory, Pisa, Italy
Abstract
One way to address cognitive decline in ageing is the use of serious games delivered through humanoid robots, to provide engaging
ways to perform exercises to train memory, attention, processing, and planning activities. We present an approach in which a humanoid
robot, by using various modalities, proposes the games in a way personalised to specific individuals’ experiences using their personal
memories associated with facts and events that occurred in older adults’ lives. We discuss how such an approach has been deployed in a
trial and how we plan to extend it in a new trial.
Keywords
Humanoid robot, Personalisation, Serious Games, Cognitive training,
1. Introduction non-verbal communication, facial expressions, and commu-
nicative gestures, and can detect the surrounding context
The increasing number of older adults implies an increas- by using various sensors (tactile sensors, cameras, micro-
ing need for their physical, social, and cognitive assistance. phones). These capabilities are essential to creating social
Indeed, ageing has a considerable impact on the health of and emotional interaction with users to increase their ac-
older adults in terms of cognitive and physical impairments, ceptability and user engagement, which may increase the
which influence the abilities to complete and perform basic possibility of reaching the goal of assisting in less time and
activities of daily living, such as cooking, shopping, man- with better results [1].
aging the home, bathing, and dressing. Nowadays, infor- Using robots to support and assist patients can be a valu-
mal caregivers, usually family members, provide a large able tool to help them during their cognitive training. In
proportion of cognitive assistance. These caregivers often such a context, digital cognitive training through serious
experience a negative impact on their psychological, emo- games may potentially benefit those with cognitive impair-
tional, and physical well-being due to the high workload ments more than traditional training due to enhanced moti-
[1]. Given the high healthcare expenditure at older ages vation and engagement. In literature, different studies show
and their effects on family caregivers, new technologies how digital games can obtain positive results in helping
to assist older adults with cognitive impairments are ur- seniors improve their cognitive abilities compared to tradi-
gently needed. Non-pharmacological interventions, such as tional training [5]. Since older adults are varied in terms of
physical training, cognitive training, and social stimulation preferences, interests, and abilities, it is important to pro-
activities, have been used to mitigate cognitive decline by pose serious games for cognitive training that are able to
maintaining or improving cognitive abilities, social well- personalise, and thus be more relevant for them. Combining
being, and quality of life of older adults [1, 2]. However, a humanoid robot and a set of personalised serious games
traditional interventions require experienced instructors can be a solution to obtain measurable progress in cognitive
who may be unavailable. functions and stimulate the user to continue the training [6].
In recent years, humanoid robots have increased their sim- Personalised serious games for cognitive intervention have
ilarity to human behaviour, from gestures and facial expres- been explored with mobile apps [7] but have not been inves-
sions to understanding questions and providing answers. tigated with humanoid robots. We aim to offer novel digital
Thanks to such humanlike characteristics, the interaction training through serious games designed using personally
between people and robots is becoming more natural. The relevant material from older adults‘ lives. They will be based
behaviour of such robots can also be personalised through on elements associated with their biography, thus making
end-user development approaches, such as trigger-action interactions personalised, relevant, and more engaging.
rules and associated support [3]. A recent literature re-
view [4] indicates that the humanoid robot is an interactive
technology still not sufficiently investigated for supporting 2. The SERENI Approach
the cognitive stimulation of older adults. In this paper, we
present a novel approach based on a Pepper humanoid robot, The psychological well-being of older adults may be affected
which exploits serious games for the cognitive stimulation by some age-related conditions, such as approaching death,
of older adults. A humanoid robot is a system that can loss of family members, and reduced autonomy. A meta-
employ different interaction strategies, such as verbal and analysis [8] indicates that the practice of life review (dis-
cussing what a set of personal memories means for an indi-
Workshop Robots for Humans 2024, Advanced Visual Interfaces, Arenzano, vidual), even more than reminiscence (recalling and describ-
June 3rd, 2024 ing individual memories), is a good instrument for improv-
∗
Corresponding author. ing the psychological well-being of older adults and that its
†
These authors contributed equally. effect sizes are comparable to those of cognitive-behavioural
Envelope-Open benedetta.catricala@isti.cnr.it (B. Catricalà); therapy. Serrano et al. [9] found that the practice of auto-
marco.manca@isti.cnr.it (M. Manca); fabio.paterno@isti.cnr.it
(F. Paternò); carmen.santoro@isti.cnr.it (C. Santoro);
biographical memory improved the mood of the elderly by
eleonora.zedda@isti.cnr.it (E. Zedda) improving their life satisfaction. Furthermore, Damianakis
Orcid 0009-0004-9042-978X (B. Catricalà); 0000-0003-1029-9934 et al. [10] report that interventions that contextualise his-
(M. Manca); 0000-0001-8355-6909 (F. Paternò); 0000-0002-0556-7538 tory, personality, and life experiences can improve commu-
(C. Santoro); 0000-0002-6541-5667 (E. Zedda) nication and social interactions between family members
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
and between family members and formal caregivers. Pre- goal, the games include a custom tracking system, which
vious experiences [11]indicated that the humanoid robot tracks the data about user performance and other game
stimulates more engagement in cognitive training of older analytics data (such as time, number of errors, pass/fail,
adults with respect to a device such as a tablet, which is score, completion level, etc.).
often used for this purpose [4]. Thus, we have started the To facilitate entering the memories through the respon-
development of a new prototype in which the serious games sive Web application (Remind) developed to collect older
installed on the humanoid robot will motivate older adults adults’ memories, we thought it was useful to categorise the
by engaging them in playful situations that draw on their biographical aspect, also because different types of mem-
personal memories, with which they can interact. Indeed, ories need different types of questions for being entered.
such serious games are designed to use personally relevant Based on the results of a survey filled out by 50 people (23
material and events from older adults’ lives. Specifically, the males and 27 females) aged between 65 and 84 years (Mean:
games are based on elements associated with the biography 72, SD: 5,09), the categories selected were Beloved, Events,
of the users (mainly taken from their youth), thus making Games, Hobbies, Places, Music.
interactions more relevant and more likely to keep them At the beginning of the interaction with the Web Remind
engaged while enhancing their well-being. application, users are asked whether they want to enter
According to such motivations, we have designed the a new memory or review those previously entered. After
SERENI (SERious gamEs with humanoid robots in cogNI- selecting a memory category, the user can provide the as-
tive training) platform to deliver serious games using per- sociated information associated with the specific memory.
sonally relevant material from older adults’ lives through For example, for entering a memory related to a particular
a humanoid robot. It aims to stimulate cognitive functions event in life the user indicates a name for the event and
through play sessions, which should last 15-20 minutes. The provides a description, which can be entered either vocally
exercises should be useful for making the participants think or by keyboard. The users can also indicate their age when
and reason before providing the correct answer. The plat- such an event occurred, and optionally provide an image
form can be a solution for day-care centres where older associated with it. In the case of memory in the Hobby cat-
adults with mild cognitive impairments can go to perform egory, the user can also provide a list of activities required
relevant exercises. On the one hand the older adults, by by the hobby. All such information can then be used by the
interacting with the biographical app, provide relevant bio- games provided by the Pepper robot for specific exercises.
graphical data that are mainly used to customise the games, In general, it is not necessary that the older adults directly
which thereby will be highly personalised for them. On enter the memories, to facilitate the process they can tell
the other hand, seniors will also interact with the games to them to some formal or informal caregiver, who can also
stimulate their cognitive abilities. The data produced dur- help them in specifying relevant memories. The Pepper
ing the interactive sessions will be exploited to improve the application presents various exercises useful for making the
adaptation of the game itself (according to the data gathered participants think and reason to provide the correct answer.
in previous game sessions) and also to feed the associated An initial set of five games were identified:
analytics services.
The SERENI platform is based on a modular architecture • Memory completion. Pepper presents a memory
allowing the deployment of multimodal serious cognitive with a missing detail, which the user should select
games on a humanoid robot. Thanks to its human-like from some elements. For example: “I liked playing
appearance and behaviour, it can stimulate interest and with my grandchildren in the kitchen, we used:”
engagement from seniors that would be more difficult with and the robot shows three possible options: Flours,
other types of smaller and more limited robots can stimulate Eggs, Fake recipes) or “I used to listen to that singer
interest and engagement from seniors that would be more when I travelled by car with my father” with possible
difficult with other types of smaller and more limited robots, answers: Modugno, Morandi, Celentano, Guccini;
thanks to its human-like appearance and behaviour. The • Activities ordering. It is only applied to the Hobby
platform is based on various components. The first one is the category: a set of activities presented in an un-
Remind App, a responsive multimodal Web application to ordered list should be put in the right temporal se-
collect memories from older adults and their relatives. The quential order by the user (this can stimulate execu-
memories can be entered both through graphical and vocal tive functions and procedural memory);
interaction. Biographical information is exploited in a group • Memory association. In this game, three memories
of games that aim to stimulate and train various cognitive are briefly listed as well as some details: users have
resources in older adults (memory, attention, planning). The to connect each memory with the corresponding
platform (see Figure 1) is also able to store data regarding detail, for example associating events to the corre-
user performance (i.e. when and for how long the user sponding places (to stimulate attention and mem-
played with a given game, the number of errors in a session, ory);
type of games played). In the resulting environment, the • Memory-related event question. The user has to
humanoid robots will serve as personal trainers, proposing guess an event that happened in the same year of
exercises and communicating through various modalities, the memory: the robot asks the user to select that
and challenging users in cognitive games relevant to their event from a list of possible events. For example:
daily life (e.g. by remembering past events or names of what happened in the same year you got married
family members and friends). The solution aims to allow (1945)? Possible answers: “the end of the Second
caregivers to configure the exercises and choose the most World War”, “the first man on the moon”, “women
suitable games to stimulate the cognitive skills of users gain the right to vote in Italy“? (useful to stimulate
and enhance their experience. Caregivers can also interact long-term memory).
with an Analytics tool, to have both overview and detailed • Music game, the robot plays the initial part of a song
information regarding user performance and state. For this popular at the time of the memories and the user has
to guess its singer or title. In general, music has a
positive effect on the users’ engagement, and in this
case, music related to their memories is proposed.
In a session at the beginning, the robot asks for the name
of the user, and then through such information, it retrieves
the memories that the user entered, which are available
from the biography application backend through a restful
service and transmitted in JSON format. The memories
arrive in the robot with the indication of the corresponding
category, which is useful to determine how to exploit them
in the various exercises. In the case of a missing detail in the Figure 1: The SERENI Approach
Memory completion exercise, the robot proposes a memory
and a list of possible missing details derived from that user’s
memories. For the memory-related event exercises, the
list of options in terms of real events is taken by external session, users had to fill out a paper questionnaire through
services. The activities ordering exercise refers only to the which they expressed their emotional state associated with
Hobby category because only in that case users are asked the gaming experience. At the end of the 12 weeks, each
to enter the steps required to perform the hobby. Thus, user filled out a questionnaire to evaluate the user expe-
users can first select the type of game they want to play, and rience (UEQ: User Experience Questionnaire), the gaming
then they have the opportunity to perform the associated experience and the interaction with the robot.
exercises, with personalised content. The user interactions were logged in order to facilitate
their analysis. Various pieces of information have been
stored for analysis. Overall, the personalised version seems
3. First Trial to stimulate slightly better performance (reaction time, num-
ber of errors), even if the difference in the quantitative data
In order to validate the approach, a trial has been carried collected was not statistically significant. A detailed descrip-
out. It involved 15 older adults 67+ who have been recruited tion of the data collected and their discussion can be found
in a Train the Brain programme managed by the CNR Neu- in [12]. Users praised memory-based games for evoking
roscience Institute in Pisa. They were diagnosed MCI by the emotions and triggering memories. The games that they
local university medical centre. In the trial to better under- liked most were Music, Memory association, and Memory
stand the role of the personal memories two versions of the completion (those that more explicitly refer to memories
games were proposed: a version based on their memories stimulated more curiosity and interest).
and therefore with a personalized gaming experience and a
standard version with general culture questions that does
not change according to the users and their memories. The 4. The Design of the Second Trial
application maintains the same structure but the contents Planning
change, which are linked to users’ memories only in one ver-
sion. The test was of the within-subjects type in which users In the new trial, we are going to change some aspects of
tested both conditions of the same application alternately: the training exercise. First of all, the older adults will be
in one week they played with the customized version, and free to interact with vocal or touch-based interaction during
in the following week they played with the non-customized the exercises. Next, we will add an exercise (memory) in
version. The purpose of this type of experimentation was which, at the beginning, the robot shows a set of pictures
to better understand the impact of the personalised version, and then hides their content. The user has to remember the
from the point of view of the user experience, the user’s position of the couple of pictures identical. The pictures
game performance and cognitive training. For the creation that will be shown will be provided by the participants.
of the contents of the non-personalised version, the interests Then, the users will participate with a Pepper exhibiting
and ages of the target user were taken into consideration. two personalities (introvert and extravert) [13] in order to
The trial was organized in twelve sessions per user held analyse how they impact the user experience in a cognitive
in the period March-May 2023 with 15 patients of the Train training programme.
The Brain program. Preceded by 2 interviews per user held
in January 2023 to collect the memories of each user to be
included in the web application. Testing meetings were held 5. Conclusions and Future Work
twice a week for 12 weeks, each week choosing a category
of memories for users to play with. In each session, users In this paper, we introduce a novel approach to personalising
had to play at least once with each game available exploiting serious games for the cognitive stimulation of older adults
the memories associated with the category initially selected. delivered through a humanoid Pepper robot. It is based on a
The two groups of users played with the same category multimodal Web app to collect memories of older adults, and
but while one group played with the customized version then such content is exploited in a set of games aiming to
of the application, the other group played with the non- stimulate several cognitive resources of seniors. We carried
customized version. In the following week, the category out a trial in which they were asked to interact with both
changed and the two groups switched sides playing the ver- the version of the games exploiting personal memories and
sion of the application they had not played in the previous another version with standard content in a within-subjects
week. Each user played 2 times with each of the 6 memory study.
categories. The sessions were conducted individually and We also present the design of a new trial that is currently
lasted approximately 15 minutes each. At the end of each being organised. We describe the main changes introduced
for such a new trial, which started in May 2024. adults, International Journal of Human-Computer
For future work, we plan to introduce a game able to Studies 145 (2021) 102509.
exploit ChatGPT, with an exercise able to consider the per- [12] B. Catricalà, M. Manca, F. Paternò, C. Santoro, E. Zedda,
sonal memories provided as a prompt in the conversation C. Braschi, F. Ricciardi, A. Sale, Exploiting personal
with the robot. memories in humanoid robot serious games for mild
cognitive impaired older adults, 2024. Submitted paper.
[13] E. Zedda, M. Manca, F. Paternò, et al., Older
6. Acknowledgments adults’ user experience with introvert and extravert
humanoid robot personalities, Universal Access
This work is partly supported by the CNR project SERENI
in the Information Society (2023). doi:10.1007/
https://hiis.isti.cnr.it/sereni/index.html document.
s10209- 023- 01054- 2 .
References
[1] F. Carros, J. Meurer, D. Löffer, D. Unbehaun, Exploring
human-robot interaction with the elderly: results from
a ten-week case study in a care home, in: Proceed-
ings of the 2020 CHI Conference on Human Factors
in Computing Systems, ACM, 2020, pp. 1–12.
[2] D. Cruz-Sandoval, A. Morales-Tellez, E. B. Sandoval,
J. Favela, A social robot as therapy facilitator in in-
terventions to deal with dementia-related behavioral
symptoms, in: Proceedings of the 2020 ACM/IEEE In-
ternational Conference on Human-Robot Interaction,
ACM/IEEE, 2020, pp. 161–169.
[3] N. Leonardi, M. Manca, F. Paternò, C. Santoro, Trigger-
action programming for personalising humanoid
robot, in: Proceedings of the 2019 ACM Conference
on Human Factors in Computing Systems (CHI’19),
ACM, 2019, p. Paper 445.
[4] V. Palumbo, F. Paternò, Serious games to cognitively
stimulate older adults: a systematic literature review,
in: Proceedings of the 2020 ACM International Confer-
ence on Pervasive Technologies Related to Assistive
Environments (PETRA), ACM, 2020, pp. 27:1–27:10.
[5] V. Manera, P.-D. Petit, A. Derreumaux, I. Orvieto,
A. Romagnoli, D. R. Lyttle, P. H. Robert, Kitchen and
cooking, a serious game for mild cognitive impair-
ment and alzheimer’s disease: a pilot study, Frontiers
in Aging Neuroscience (2015).
[6] A. Tapus, A. M. Vieru, Robot cognitive stimulation for
the elderly, in: International Work-Conference on the
Interplay Between Natural and Artificial Computation,
Springer, 2013, pp. 94–102.
[7] Q. Xu, V. Subbaraju, C. H. Cheong, A. Wang, K. Kang,
M. Bashir, Y. Dong, L. Li, J.-H. Lim, Personalized se-
rious games for cognitive intervention with lifelog
visual analytics, in: Proceedings of the 26th ACM In-
ternational Conference on Multimedia (MM ’18), ACM,
2018.
[8] E. Bohlmeijer, M. Roemer, P. Cuijpers, F. Smit, The
effects of reminiscence on psychological well-being
in older adults: A meta-analysis, Aging and Mental
Health 11 (2007) 291–300.
[9] J. P. Serrano, J. M. Latorre, M. Gatz, J. Montanes, Life
review therapy using autobiographical retrieval prac-
tice for older adults with depressive symptomatology,
Psychology and Aging 19 (2004) 272.
[10] T. Damianakis, M. Crete-Nishihata, K. L. Smith, R. M.
Baecker, E. Marziali, The psychosocial impacts of
multimedia biographies on persons with cognitive im-
pairments, The Gerontologist 50 (2010) 23–35.
[11] M. Manca, F. Paternò, C. Santoro, E. Zedda, C. Braschi,
F. Ricciardi, A. Sale, The impact of serious games with
humanoid robots on mild cognitive impairment older