One of the goals of Ambient Assisted Living (AAL) solutions is to be able to stimulate the cognitive resources of older adults. An innovative way to address such stimulation is the use of serious games delivered through humanoid robots, as they can provide an engaging way to perform exercises useful for training human memory, attention, processing, and planning activities. This paper presents an approach to supporting cognitive stimulation based on personal memories. The humanoid robot can exhibit different behaviours using various modalities, and propose the games in a way personalised to specific individuals' requirements, preferences, abilities, and motivations, which can vary among older adults, and even dynamically evolve over time for the same person depending on changing user needs and health conditions. Using personal memories associated with facts and events that occurred in older adults life in the serious games can increase their engagement, and thus potentially reduce the cognitive training drop-out.
According to the World Health Organization, by 2050 the number of individuals over the age of
85 is projected to be three times more than today. In this scenario, most older adults will need
physical, social, and cognitive assistance. Indeed, aging has a considerable impact on the health of
older adults in terms of cognitive and physical impairments, which influence the abilities to complete
and perform basic activities of daily living, such as cooking, shopping, managing the home, bathing,
and dressing. Nowadays, a large proportion of cognitive assistance is provided by informal
caregivers, usually family members. These caregivers often experience a negative impact on their
psychological, emotional, and physical well-being due to the high workload [
Non-pharmacological interventions, such as physical training, cognitive training, and social
stimulation activities have been used to mitigate the cognitive decline by maintaining or improving
cognitive abilities, social well-being, and quality of life of older adults [
In recent years, humanoid robots have increased their similarity to human behaviour starting from
gestures and facial expressions to the ability to understand questions and provide answers. Thanks
to such humanlike characteristics, the interaction between people and robots is becoming more
natural. The behaviour of such robots can also be personalised through end-user development
approaches, such as the use of trigger-action rules and associated support [
In this paper, we present a novel approach based on a Pepper humanoid robot, which exploits
serious games for cognitive stimulation of older adults. A humanoid robot is a system that can
employ different interaction strategies, such as verbal and non-verbal communication, facial
expressions, communicative gestures, and can detect the surrounding context by using various
sensors (tactile sensors, camera, microphones). These capabilities are essential to creating social and
emotional interaction with users to increase their acceptability and users engagement, which may
increase the possibility of reaching the goal of assistance in less time and with better results [
Using robots to support and assist patients can be a valuable tool to help them during their cognitive training. In such context, digital cognitive training through serious games may potentially benefit those with cognitive impairments more than traditional training due to enhanced motivation and engagement. In the literature, different studies show how digital games can obtain positive results in stimulating older adults and helping them improve their cognitive abilities compared to traditional training [24]. Since older adults are a category of users very varied in terms of their preferences, interests, and abilities, it is important to propose serious games for cognitive training that are able to personalise, and thus be more relevant for each of them. Combining a humanoid robot and a set of personalised serious games can be an exciting solution to obtain measurable progress in cognitive functions and stimulate the user to continue the training program [16]. We aim to offer novel digital training through serious games designed using personally relevant material from older adults‘ life. They will be based on elements associated with their biography, thus making interactions personalised, relevant, and more engaging.
In the paper we present our approach to personalised serious games for cognitive training, and the platform that we have designed for supporting. Next, we describe the multimodal Web app for collecting older adults memories, and how we have considered feedback received by target audience for categorising such memories. We then illustrate the first set of games that have been designed and implemented for providing exercises that exploit personal memories. Lastly, we draw some conclusions and provide indications for future work.
The psychological well-being of older adults may be affected by some age-related conditions,
such as approaching death, loss of family members, and reduced autonomy. A meta-analysis [
Based on previous experiences [
According to the motivations discussed in the previous section, we have designed the SERENI platform, to deliver serious games designed using personally relevant material from older adults’ life through a humanoid robot. It aims to stimulate cognitive functions through play sessions, which should last 15-20 minutes. The exercises should be useful for making the participants think and reason before providing the correct answer. The platform can be a solution for daycare centres where older adults with mild cognitive impairments can go to perform relevant exercises. On the one hand the older adults, by interacting with the biographical app, provide relevant biographical data that are mainly used to customise the games, which thereby will be highly personalised for them. On the other hand, seniors will also interact with the games to stimulate their cognitive abilities. The data produced during the interactive sessions will be exploited to improve the adaptation of the game itself (according to the data gathered in previous game sessions) and also to feed the associated analytics services.
The SERENI platform is based on a modular architecture allowing the deployment of the multimodal serious cognitive games on a humanoid robot, which can stimulate interest and engagement from seniors that would be more difficult with other types of smaller and more limited robots, thanks to its human-like appearance and behaviour.
The platform is based on various components (see Figure 1). The first one is the Remind App, which is a responsive multimodal Web application to collect memories from older adults and their relatives. The memories can be entered both through graphical and vocal interaction.
Biographical information is exploited in a group of games that aim to stimulate and train varoious
cognitive resources in older adults (memory, attention, planning), …. The platform is also able to
store data regarding user performance, such as when and for how long the user played with a given
game, the number of errors in a session, the type of games that have been played. In addition, the
evocation and detection of a user’s emotional state are becoming a crucial element in the aim of
developing more effective interfaces between humans and computers, especially in applications such
as games and e-learning tools [
In the resulting environment, the humanoid robots will serve as personal trainers, proposing exercises and communicating through various modalities, and challenging users in cognitive games relevant to their daily life (e.g. by smemorising scheduled events or names of family members and friends).
The solution aims to allow doctors and caregivers to configure the exercises and choose the most suitable games to stimulate the cognitive abilities of end users and enhance their experience. We also plan to include a Game personalisation tool, which allows them to provide further relevant personalisations for the elderly’s interaction with the games. In addition, the caregiver will also interact with an Analytics tool, to have both overview and detailed information regarding user performance and state. For this purpose, the games include a custom tracking system, which tracks the data about user performance and other important information necessary for the analytics of serious games (such as time, number of errors, pass/fail, score, completion level, skill level achieved etc.). 2.2
A responsive Web application (Remind) has been developed to collect older adults’ memories. It can receive them both through graphical and vocal interaction. To facilitate entering the memories, we thought it was useful to categorise the biographical aspect, also because different types of memories need different types of questions for being entered. Based on our previous experiences in projects in the Ambient Assisted Living area and informal discussions with relevant stakeholders, the first version of the app we identified a first set of possible categories of memories: music, specific events, games that the user liked to play, locations, food, and hobbies.
We then decided to carry out an empirical validation of such classification with the target audience (older adults). We thus proposed a questionnaire to people aged 65+, in Italian language. The questionnaire was composed of three parts. The initial part was dedicated to demographic information, in the second part they were asked to freely indicate at least four categories that they deemed particularly relevant to claissify their personal memories, and to select which categories they find relevant in a list comprising (Food, Events, Family, Travels, Music, Hobby, Work, Love/ friendship, Studying, Health). Then, they had to rate on a scale from 1 to 5 the relevance of the categories used in the initial version of the games (Locations, Games, Hobby, Food, Music, and Personal Events), with also the possibility of indicating what category they would add or remove from such classification.
The questionnaire was completed in paper form by 50 participants identified thanks to leafleting, word of mouth and home visits to friends and acquaintances. The sample consisted of 50 people (23 males and 27 females) aged between 65 and 84 years (mean 72, SD 5,09). 40% of them have a higher education while only 38% have a degree. 86% said they were very familiar with electronic devices such as smartphones, tablets and PCs; the remaining 14% only use smartphones mainly out of necessity.
The following considerations emerged from the review: - 80% (40 out of 50 participants) indicated “Family” as the most representative category for their memories; the examples proposed concern the birth of children and grandchildren, the memory of parents and grandparents and the childhood home.
- 40% (20 people out of 50) indicated “Work”, in particular the first experiences and satisfactions during their career.
- 50% (29 out of 50 participants) cited “Affections”, and provided examples such as meeting their first love, childhood friendships, and events such as engagement and marriage.
Participants were also asked to indicate, among the initially proposed categories, those most relevant to them. Participants rated each category on a scale of 1 to 5, with 1 being Not Relevant and 5 being Relevant. In particular, in the participants’ ratings: • • • • • • 54% (27/50) rated the Hobbies category as Not Relevant (scores below 3 on a 1-5 scale) 40% (20/50) rated the Food category as Not Relevant (scores below 3 on a 1-5 scale) 74% (37/50) rated the Music category as Relevant (scores above 3 on a 1-5 scale) 86% (43/50) rated the Places category as Relevant (scores above 3 on a 1-5 scale) 98% (49/50) rated the Events category as Relevant (scores above 3 on a 1-5 scale) 68% (34/50) rated the Games category as Relevant (scores above 3 on a 1-5 scale) The most relevant category among those proposed by us was the one linked to Events with an average score of 4.7 out of a maximum of 5: the participants appreciated the category as it was considered very versatile, as it allows the inclusion of different types of memories.
The least relevant categories turned out to be Hobbies and Food. Hobbies received an average score of about 2.6 out of a maximum of 5, The main criticism concerned the name of the category because, according to the participants, it was not very pertinent compared to the other proposals. As possible replacements, terms such as “leisure” or “entertainment” have been suggested. Participants also showed very low interest in the Food category as this type of memories did not have a significant impact in their life experiences for most of them.
In conclusion, the answers of the candidates showed that the most significant memories concern the dearest affections and the most important events in life such as graduation, marriage or the birth of children. Of the six categories proposed, those related to Music and Personal Events aroused the greatest interest. Everyone expressed their appreciation for the project, declaring that they are inclined to use the web application to share their memories. Thus, in the new version of the Remind app we introduced the Affects category (see Figure 2 left).
At the beginning of the interaction with the Remind application, users are asked whether they want to enter a new memory or review those previously entered. When a memory category is selected, then the user can provide the information associated with the specific memory. For example, Figure 2 (right) shows the user interface for entering a memory related to a particular event in life. The user has to indicate a name for the event, then provide a description, which can be done either vocally or by keyboard. The user can also indicate the age they had when they had when such event occurred, and optionally provide an image associated with it. In the case of a memory in the Hobby category, the user can also provide a list of activities required by the hobby. All such information can then be used by the games provided by Pepper for specific exercises.
In general it is not necessary that the older adults directly enter the memories, to facilitate the process they can just tell them to some formal or informal caregiver, who can also help them in indicating relevant memories. 2.3
The Pepper application prototype aims to stimulate cognitive functions through play sessions, which should last 15-20 minutes. The games present various exercises, which should be useful for making the participants think and reason to provide the correct answer. An initial set of four types of games have been identified: • Memory completion, where the robot presents a memory with a missing detail, which should be indicated by the user by choosing from a multiple choice of elements, and in case of a correct answer, the memory is reread to the older adult. For example: “when I was 12 years old I used to spend summer time in…”(and the robot shows three possible options: Marina di Pisa, Tirrenia, San Vincenzo or Castiglioncello) or “I used to listen to that singer when I traveled by car with my father” with possible answers: Modugno, Morandi, Celentano, Guccini; • Activities ordering, which can be applied to hobby, in which sequence of tasks presented in an unordered list should be put in the right order by the user (this can stimulate executive functions and procedural memory); • Memory association, where 3-4 memories are briefly described along with a set of details, and the user has to connect each memory with the corresponding detail (for example associate song titles to the corresponding singers); • Memory-related event question, where the user has to guess the event that happened in the same year of the presented memory, thus it asks the user to select which important event in the world happened that year from a set of events that are listed (for example: what happened in the same year you got married (1945)? Possible answers: “the end of second world war”, “the first man on the moon”, “women gain the right to vote in Italy“?).
In a session at the beginning the robot asks the name of the user (Figure 3, left), then through such information it retrieves the memories that the user entered, which are available from the biography application backend through a restful service and transmitted in JSON format. The memories arrive in the robot with the indication of the corresponding category, which is useful to determine how to exploit them in the various exercises. In the case of a missing detail exercise the robot proposes a memory and a list of possible missing details, which are derived from the memories of that user. In case there are not sufficient memories to identify the proposed options, the details are taken by the memories of another user who has a similar age. For the memory-related event exercises, the list of possible options in terms of real events to choose from are taken by external services. The activities ordering exercise refers only to the Hobby category of memories because only in that case the users are asked to enter the steps required by the hobby, thus the robot can retrieve sufficient information to propose it. Then, users can first select the type of game they want to play (Figure 3, center), and then they have the opportunity to perform the associated exercise, with personalised content (Figure 3, right).
In this paper, we introduce a novel approach to personalising serious games for the cognitive stimulation of older adults delivered through a humanoid robot. It is based on a multimodal Web app to collect memories of older adults, and then such content is exploited in a set of games aiming to stimulate several cognitive resources in older adults implemented for a Pepper robot.
In future work, we plan to validate the approach with trials involving a group of older adults with mild cognitive impairments participating in a Train the Brain programme in which they regularly attend cognitive stimulation exercises in a care centre managed by CNR in Pisa. We also plan to develop a customisation and analytics environment to further extend the possible personalisations of games and their analysis.
https://hiis.isti.cnr.it/sereni/index.html. supported by the
CNR project
[16] Palumbo, V., & Paternò, F. (2020). Serious games to cognitively stimulate older adults: a systematic literature review. PETRA (pp. 27:1-27:10). ACM Press. [17] Serrano, J. P., Latorre, J. M., Gatz, M., & Montanes, J. (2004). Life review therapy using autobiographical retrieval practice for older adults with depressive symptomatology.
Psychology and aging, 19(2), 272. [18] Smith, E., Nolen-Hoeksema, S., Fredrickson, B., & Loftus, G. (2002). Atkinson and Hilgard’s
Introduction to Psychology. Wadsworth. [19] Softbank Robotics. (2021, 07 10). Pepper SDK-Tools. Retrieved from QiSDK: https://qisdk.softbankrobotics.com/sdk/doc/pepper-sdk/ch3_tools/tools.html [20] Tanaka, M., Ishii, A., Yamano, E., Ogikubo, H., Okazaki, M., & Kamimura, K. (2012). Effect of a human-type communication robot on cognitive function in elderly women living alone. Medical science monitor: international medical journal of experimental and clinical research, 18(9). [21] Tapus, A., & Vieru, A. M. (2013). Robot cognitive stimulation for the elderly. International Work-Conference on the Interplay Between Natural and Artificial Computation (pp. pp. 94102). Springer. [22] A.Tapus, C. Tapus, C., & Matarić, M. .. (2008). User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intelligent Service Robotics, 169-183. [23] Tay, B., Jung, Y., & Park, T. (2014). When stereotypes meet robots: the double-edge sword of robot gender and personality in human–robot interaction. . Computers in Human Behavior, 38, 75-84. [24] Tong, T., & Chignell, M. (2013). Designing game-based cognitive assessments for elderly adults. Proceedings of the First International Conference on Gameful design, research, and applications, (pp. 127-130).