One way to address cognitive decline in ageing is the use of serious games delivered through humanoid robots, to provide engaging ways to perform exercises to train memory, attention, processing, and planning activities. We present an approach in which a humanoid robot, by using various modalities, proposes the games in a way personalised to specific individuals' experiences using their personal memories associated with facts and events that occurred in older adults' lives. We discuss how such an approach has been deployed in a trial and how we plan to extend it in a new trial.
The increasing number of older adults implies an
increasing need for their physical, social, and cognitive assistance.
Indeed, ageing has a considerable impact on the health of
older adults in terms of cognitive and physical impairments,
which influence the abilities to complete and perform basic
activities of daily living, such as cooking, shopping,
managing the home, bathing, and dressing. Nowadays,
informal caregivers, usually family members, provide a large
proportion of cognitive assistance. These caregivers often
experience a negative impact on their psychological,
emotional, and physical well-being due to the high workload
[
In recent years, humanoid robots have increased their
similarity to human behaviour, from gestures and facial
expressions to understanding questions and providing answers.
Thanks to such humanlike characteristics, the interaction
between people and robots is becoming more natural. The
behaviour of such robots can also be personalised through
end-user development approaches, such as trigger-action
rules and associated support [
Using robots to support and assist patients can be a
valuable tool to help them during their cognitive training. In
such a context, digital cognitive training through serious
games may potentially benefit those with cognitive
impairments more than traditional training due to enhanced
motivation and engagement. In literature, diferent studies show
how digital games can obtain positive results in helping
seniors improve their cognitive abilities compared to
traditional training [
ceur-ws.org
and between family members and formal caregivers.
Previous experiences [
According to such motivations, we have designed the SERENI (SERious gamEs with humanoid robots in cogNItive training) platform to deliver serious games using personally relevant material from older adults’ lives through a humanoid robot. It aims to stimulate cognitive functions through play sessions, which should last 15-20 minutes. The exercises should be useful for making the participants think and reason before providing the correct answer. The platform can be a solution for day-care centres where older adults with mild cognitive impairments can go to perform relevant exercises. On the one hand the older adults, by interacting with the biographical app, provide relevant biographical data that are mainly used to customise the games, which thereby will be highly personalised for them. On the other hand, seniors will also interact with the games to stimulate their cognitive abilities. The data produced during the interactive sessions will be exploited to improve the adaptation of the game itself (according to the data gathered in previous game sessions) and also to feed the associated analytics services.
The SERENI platform is based on a modular architecture allowing the deployment of multimodal serious cognitive games on a humanoid robot. Thanks to its human-like appearance and behaviour, it can stimulate interest and engagement from seniors that would be more dificult with other types of smaller and more limited robots can stimulate interest and engagement from seniors that would be more dificult with other types of smaller and more limited robots, thanks to its human-like appearance and behaviour. The platform is based on various components. The first one is the Remind App, a responsive multimodal Web application to collect memories from older adults and their relatives. The memories can be entered both through graphical and vocal interaction. Biographical information is exploited in a group of games that aim to stimulate and train various cognitive resources in older adults (memory, attention, planning). The platform (see Figure 1) is also able to store data regarding user performance (i.e. when and for how long the user played with a given game, the number of errors in a session, type of games played). In the resulting environment, the humanoid robots will serve as personal trainers, proposing exercises and communicating through various modalities, and challenging users in cognitive games relevant to their daily life (e.g. by remembering past events or names of family members and friends). The solution aims to allow caregivers to configure the exercises and choose the most suitable games to stimulate the cognitive skills of users and enhance their experience. Caregivers can also interact with an Analytics tool, to have both overview and detailed information regarding user performance and state. For this goal, the games include a custom tracking system, which tracks the data about user performance and other game analytics data (such as time, number of errors, pass/fail, score, completion level, etc.).
To facilitate entering the memories through the responsive Web application (Remind) developed to collect older adults’ memories, we thought it was useful to categorise the biographical aspect, also because diferent types of memories need diferent types of questions for being entered. Based on the results of a survey filled out by 50 people (23 males and 27 females) aged between 65 and 84 years (Mean: 72, SD: 5,09), the categories selected were Beloved, Events, Games, Hobbies, Places, Music.
At the beginning of the interaction with the Web Remind application, users are asked whether they want to enter a new memory or review those previously entered. After selecting a memory category, the user can provide the associated information associated with the specific memory. For example, for entering a memory related to a particular event in life the user indicates a name for the event and provides a description, which can be entered either vocally or by keyboard. The users can also indicate their age when such an event occurred, and optionally provide an image associated with it. In the case of memory in the Hobby category, the user can also provide a list of activities required by the hobby. All such information can then be used by the games provided by the Pepper robot for specific exercises. In general, it is not necessary that the older adults directly enter the memories, to facilitate the process they can tell them to some formal or informal caregiver, who can also help them in specifying relevant memories. The Pepper application presents various exercises useful for making the participants think and reason to provide the correct answer. An initial set of five games were identified: • Memory completion. Pepper presents a memory with a missing detail, which the user should select from some elements. For example: “I liked playing with my grandchildren in the kitchen, we used:” and the robot shows three possible options: Flours, Eggs, Fake recipes) or “I used to listen to that singer when I travelled by car with my father” with possible answers: Modugno, Morandi, Celentano, Guccini; • Activities ordering. It is only applied to the Hobby category: a set of activities presented in an unordered list should be put in the right temporal sequential order by the user (this can stimulate executive functions and procedural memory); • Memory association. In this game, three memories are briefly listed as well as some details: users have to connect each memory with the corresponding detail, for example associating events to the corresponding places (to stimulate attention and memory); • Memory-related event question. The user has to guess an event that happened in the same year of the memory: the robot asks the user to select that event from a list of possible events. For example: what happened in the same year you got married (1945)? Possible answers: “the end of the Second World War”, “the first man on the moon”, “women gain the right to vote in Italy“? (useful to stimulate long-term memory). • Music game, the robot plays the initial part of a song popular at the time of the memories and the user has to guess its singer or title. In general, music has a positive efect on the users’ engagement, and in this case, music related to their memories is proposed.
In a session at the beginning, the robot asks for the name of the user, and then through such information, it retrieves the memories that the user entered, which are available from the biography application backend through a restful service and transmitted in JSON format. The memories arrive in the robot with the indication of the corresponding category, which is useful to determine how to exploit them in the various exercises. In the case of a missing detail in the Memory completion exercise, the robot proposes a memory and a list of possible missing details derived from that user’s memories. For the memory-related event exercises, the list of options in terms of real events is taken by external services. The activities ordering exercise refers only to the Hobby category because only in that case users are asked to enter the steps required to perform the hobby. Thus, users can first select the type of game they want to play, and then they have the opportunity to perform the associated exercises, with personalised content.
In order to validate the approach, a trial has been carried out. It involved 15 older adults 67+ who have been recruited in a Train the Brain programme managed by the CNR Neuroscience Institute in Pisa. They were diagnosed MCI by the local university medical centre. In the trial to better understand the role of the personal memories two versions of the games were proposed: a version based on their memories and therefore with a personalized gaming experience and a standard version with general culture questions that does not change according to the users and their memories. The application maintains the same structure but the contents change, which are linked to users’ memories only in one version. The test was of the within-subjects type in which users tested both conditions of the same application alternately: in one week they played with the customized version, and in the following week they played with the non-customized version. The purpose of this type of experimentation was to better understand the impact of the personalised version, from the point of view of the user experience, the user’s game performance and cognitive training. For the creation of the contents of the non-personalised version, the interests and ages of the target user were taken into consideration.
The trial was organized in twelve sessions per user held in the period March-May 2023 with 15 patients of the Train The Brain program. Preceded by 2 interviews per user held in January 2023 to collect the memories of each user to be included in the web application. Testing meetings were held twice a week for 12 weeks, each week choosing a category of memories for users to play with. In each session, users had to play at least once with each game available exploiting the memories associated with the category initially selected. The two groups of users played with the same category but while one group played with the customized version of the application, the other group played with the noncustomized version. In the following week, the category changed and the two groups switched sides playing the version of the application they had not played in the previous week. Each user played 2 times with each of the 6 memory categories. The sessions were conducted individually and lasted approximately 15 minutes each. At the end of each session, users had to fill out a paper questionnaire through which they expressed their emotional state associated with the gaming experience. At the end of the 12 weeks, each user filled out a questionnaire to evaluate the user experience (UEQ: User Experience Questionnaire), the gaming experience and the interaction with the robot.
The user interactions were logged in order to facilitate
their analysis. Various pieces of information have been
stored for analysis. Overall, the personalised version seems
to stimulate slightly better performance (reaction time,
number of errors), even if the diference in the quantitative data
collected was not statistically significant. A detailed
description of the data collected and their discussion can be found
in [
In the new trial, we are going to change some aspects of
the training exercise. First of all, the older adults will be
free to interact with vocal or touch-based interaction during
the exercises. Next, we will add an exercise (memory) in
which, at the beginning, the robot shows a set of pictures
and then hides their content. The user has to remember the
position of the couple of pictures identical. The pictures
that will be shown will be provided by the participants.
Then, the users will participate with a Pepper exhibiting
two personalities (introvert and extravert) [
In this paper, we introduce a novel approach to personalising serious games for the cognitive stimulation of older adults delivered through a humanoid Pepper robot. It is based on a multimodal Web app to collect memories of older adults, and then such content is exploited in a set of games aiming to stimulate several cognitive resources of seniors. We carried out a trial in which they were asked to interact with both the version of the games exploiting personal memories and another version with standard content in a within-subjects study.
We also present the design of a new trial that is currently being organised. We describe the main changes introduced for such a new trial, which started in May 2024.
For future work, we plan to introduce a game able to exploit ChatGPT, with an exercise able to consider the personal memories provided as a prompt in the conversation with the robot.
This work is partly supported by the CNR project SERENI https://hiis.isti.cnr.it/sereni/index.html document.