An Environment to Collect Personal Memories of Older Adults and Use them to Personalise Serious Games with Humanoid Robots Benedetta Catricalà1, Miriam Ledda1, Marco Manca1, Fabio Paternò*1, Carmen Santoro1, Eleonora Zedda1 CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1, 56124 Pisa, Italy Abstract One of the goals of Ambient Assisted Living (AAL) solutions is to be able to stimulate the cognitive resources of older adults. An innovative way to address such stimulation is the use of serious games delivered through humanoid robots, as they can provide an engaging way to perform exercises useful for training human memory, attention, processing, and planning activities. This paper presents an approach to supporting cognitive stimulation based on personal memories. The humanoid robot can exhibit different behaviours using various modalities, and propose the games in a way personalised to specific individuals’ requirements, preferences, abilities, and motivations, which can vary among older adults, and even dynamically evolve over time for the same person depending on changing user needs and health conditions. Using personal memories associated with facts and events that occurred in older adults life in the serious games can increase their engagement, and thus potentially reduce the cognitive training drop-out. Keywords 1 Humanoid robot, Personalisation, Serious Games, Ambient Assisted Living 1 Introduction According to the World Health Organization, by 2050 the number of individuals over the age of 85 is projected to be three times more than today. In this scenario, most older adults will need physical, social, and cognitive assistance. Indeed, aging has a considerable impact on the health of 2nd Workshop on sociAL roboTs for peRsonalized, continUous and adaptIve aSsistTance (ALTRUIST), December 16, 2022, Florence, Italy. * Corresponding author EMAIL: b.catricala@studenti.unipi.it (B. Catricalà); miriam.ledda@isti.cnr.it (M. Ledda); marco.manca@isti.cnr.it (M. Manca); fabio.paterno@isti.cnr.it (F. Paternò); carmen.santoro@isti.cnr.it (C. Santoro); eleonora.zedda@isti.cnr.it (E.Zedda) ORCID: 0000-0003-1029-9934 (M. Manca); 0000-0001-8355-6909 (F. Paternò); 0000-0002-0556-7538 (C. Santoro); 0000-0002-6541-5667 (E. Zedda); ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) older adults in terms of cognitive and physical impairments, which influence the abilities to complete and perform basic activities of daily living, such as cooking, shopping, managing the home, bathing, and dressing. Nowadays, a large proportion of cognitive assistance is provided by informal caregivers, usually family members. These caregivers often experience a negative impact on their psychological, emotional, and physical well-being due to the high workload [3]. Given the high health care expenditure at older ages, and their effects on family caregivers, new technologies to assist older adults with cognitive impairments are urgently needed. Non-pharmacological interventions, such as physical training, cognitive training, and social stimulation activities have been used to mitigate the cognitive decline by maintaining or improving cognitive abilities, social well-being, and quality of life of older adults [3, 5]. However, traditional interventions require experienced instructors who may be unavailable. Assistive technologies can provide useful support to address this problem. They are technologies that aim to assist different types of users during their rehabilitation. They can help older adults maintain their independence during daily routines and can also be an important instrument during their rehabilitation [15]. In recent years, humanoid robots have increased their similarity to human behaviour starting from gestures and facial expressions to the ability to understand questions and provide answers. Thanks to such humanlike characteristics, the interaction between people and robots is becoming more natural. The behaviour of such robots can also be personalised through end-user development approaches, such as the use of trigger-action rules and associated support [10]. A recent literature review [16] indicates that the humanoid robot is an interactive technology still not sufficiently investigated for supporting the cognitive stimulation of older adults. In this paper, we present a novel approach based on a Pepper humanoid robot, which exploits serious games for cognitive stimulation of older adults. A humanoid robot is a system that can employ different interaction strategies, such as verbal and non-verbal communication, facial expressions, communicative gestures, and can detect the surrounding context by using various sensors (tactile sensors, camera, microphones). These capabilities are essential to creating social and emotional interaction with users to increase their acceptability and users engagement, which may increase the possibility of reaching the goal of assistance in less time and with better results [3]. Using robots to support and assist patients can be a valuable tool to help them during their cognitive training. In such context, digital cognitive training through serious games may potentially benefit those with cognitive impairments more than traditional training due to enhanced motivation and engagement. In the literature, different studies show how digital games can obtain positive results in stimulating older adults and helping them improve their cognitive abilities compared to traditional training [24]. Since older adults are a category of users very varied in terms of their preferences, interests, and abilities, it is important to propose serious games for cognitive training that are able to personalise, and thus be more relevant for each of them. Combining a humanoid robot and a set of personalised serious games can be an exciting solution to obtain measurable progress in cognitive functions and stimulate the user to continue the training program [16]. We aim to offer novel digital training through serious games designed using personally relevant material from older adults‘ life. They will be based on elements associated with their biography, thus making interactions personalised, relevant, and more engaging. In the paper we present our approach to personalised serious games for cognitive training, and the platform that we have designed for supporting. Next, we describe the multimodal Web app for collecting older adults memories, and how we have considered feedback received by target audience 2 for categorising such memories. We then illustrate the first set of games that have been designed and implemented for providing exercises that exploit personal memories. Lastly, we draw some conclusions and provide indications for future work. 2 The SERENI Approach The psychological well-being of older adults may be affected by some age-related conditions, such as approaching death, loss of family members, and reduced autonomy. A meta-analysis [2] indicates that the practice of life review, even more than reminiscence (reminiscence involves describing a memory itself, while life review is based on discussing what a memory means), is a good instrument for improving the psychological well-being of older adults and that its effect sizes are comparable to those of cognitive-behavioural therapy. Serrano et al. [17] found that the practice of autobiographical memory improved the mood of the elderly by improving their life satisfaction. Furthermore, Damianakis et al. [6] report that interventions that contextualise history, personality, and life experiences can contribute to improving both communication and social interactions between family members and between family members and formal caregivers. Based on previous experiences [12], we have started the development of a new prototype in which the serious games installed on the humanoid robot will motivate older adults by engaging them in playful situations that draw on their personal memories, with which they can interact. Indeed, such serious games are designed to use personally relevant material and events from older adults’ life. Specifically, the games are based on elements associated with the biography of the older adults, thus making interactions more relevant and more likely to keep them engaged while enhancing their well- being as well. In the following, we describe the approach proposed for a platform able to support cognitive stimulation through humanoid robots, a multimodal app for collecting memories, and a first set of games able to exploit such memories through the humanoid robot. 2.1 The SERENI Platform According to the motivations discussed in the previous section, we have designed the SERENI platform, to deliver serious games designed using personally relevant material from older adults’ life through a humanoid robot. It aims to stimulate cognitive functions through play sessions, which should last 15-20 minutes. The exercises should be useful for making the participants think and reason before providing the correct answer. The platform can be a solution for daycare centres where older adults with mild cognitive impairments can go to perform relevant exercises. On the one hand the older adults, by interacting with the biographical app, provide relevant biographical data that are mainly used to customise the games, which thereby will be highly personalised for them. On the other hand, seniors will also interact with the games to stimulate their cognitive abilities. The data produced during the interactive sessions will be exploited to improve the adaptation of the game itself (according to the data gathered in previous game sessions) and also to feed the associated analytics services. The SERENI platform is based on a modular architecture allowing the deployment of the multimodal serious cognitive games on a humanoid robot, which can stimulate interest and 3 engagement from seniors that would be more difficult with other types of smaller and more limited robots, thanks to its human-like appearance and behaviour. The platform is based on various components (see Figure 1). The first one is the Remind App, which is a responsive multimodal Web application to collect memories from older adults and their relatives. The memories can be entered both through graphical and vocal interaction. Biographical information is exploited in a group of games that aim to stimulate and train varoious cognitive resources in older adults (memory, attention, planning), …. The platform is also able to store data regarding user performance, such as when and for how long the user played with a given game, the number of errors in a session, the type of games that have been played. In addition, the evocation and detection of a user’s emotional state are becoming a crucial element in the aim of developing more effective interfaces between humans and computers, especially in applications such as games and e-learning tools [14]. This is typically done by sanalysing emotions while playing, which can be performed by using sensors in a wristband to detect relevant information. For this purpose, we have started the integration of a E4 Empatica wristband in the platform. It is a device equipped with various sensors to measure Blood Volume Pulse (BVP), from which heart rate variability can be derived, and the constantly fluctuating changes in specific electrical properties of the skin, capture motion-based activity through a 3-axis accelerometer, read peripheral skin temperature, and tag events and link them to physiological signals. In this way, it can be possible to perform unobtrusive monitoring of anxiety and mood-related information of older adults. In the resulting environment, the humanoid robots will serve as personal trainers, proposing exercises and communicating through various modalities, and challenging users in cognitive games relevant to their daily life (e.g. by smemorising scheduled events or names of family members and friends). Figure 1: The SERENI Platform The solution aims to allow doctors and caregivers to configure the exercises and choose the most suitable games to stimulate the cognitive abilities of end users and enhance their experience. We 4 also plan to include a Game personalisation tool, which allows them to provide further relevant personalisations for the elderly’s interaction with the games. In addition, the caregiver will also interact with an Analytics tool, to have both overview and detailed information regarding user performance and state. For this purpose, the games include a custom tracking system, which tracks the data about user performance and other important information necessary for the analytics of serious games (such as time, number of errors, pass/fail, score, completion level, skill level achieved etc.). 2.2 The Biography App A responsive Web application (Remind) has been developed to collect older adults’ memories. It can receive them both through graphical and vocal interaction. To facilitate entering the memories, we thought it was useful to categorise the biographical aspect, also because different types of memories need different types of questions for being entered. Based on our previous experiences in projects in the Ambient Assisted Living area and informal discussions with relevant stakeholders, the first version of the app we identified a first set of possible categories of memories: music, specific events, games that the user liked to play, locations, food, and hobbies. We then decided to carry out an empirical validation of such classification with the target audience (older adults). We thus proposed a questionnaire to people aged 65+, in Italian language. The questionnaire was composed of three parts. The initial part was dedicated to demographic information, in the second part they were asked to freely indicate at least four categories that they deemed particularly relevant to claissify their personal memories, and to select which categories they find relevant in a list comprising (Food, Events, Family, Travels, Music, Hobby, Work, Love/ friendship, Studying, Health). Then, they had to rate on a scale from 1 to 5 the relevance of the categories used in the initial version of the games (Locations, Games, Hobby, Food, Music, and Personal Events), with also the possibility of indicating what category they would add or remove from such classification. The questionnaire was completed in paper form by 50 participants identified thanks to leafleting, word of mouth and home visits to friends and acquaintances. The sample consisted of 50 people (23 males and 27 females) aged between 65 and 84 years (mean 72, SD 5,09). 40% of them have a higher education while only 38% have a degree. 86% said they were very familiar with electronic devices such as smartphones, tablets and PCs; the remaining 14% only use smartphones mainly out of necessity. The following considerations emerged from the review: - 80% (40 out of 50 participants) indicated “Family” as the most representative category for their memories; the examples proposed concern the birth of children and grandchildren, the memory of parents and grandparents and the childhood home. - 40% (20 people out of 50) indicated “Work”, in particular the first experiences and satisfactions during their career. - 50% (29 out of 50 participants) cited “Affections”, and provided examples such as meeting their first love, childhood friendships, and events such as engagement and marriage. 5 Participants were also asked to indicate, among the initially proposed categories, those most relevant to them. Participants rated each category on a scale of 1 to 5, with 1 being Not Relevant and 5 being Relevant. In particular, in the participants’ ratings: • 54% (27/50) rated the Hobbies category as Not Relevant (scores below 3 on a 1-5 scale) • 40% (20/50) rated the Food category as Not Relevant (scores below 3 on a 1-5 scale) • 74% (37/50) rated the Music category as Relevant (scores above 3 on a 1-5 scale) • 86% (43/50) rated the Places category as Relevant (scores above 3 on a 1-5 scale) • 98% (49/50) rated the Events category as Relevant (scores above 3 on a 1-5 scale) • 68% (34/50) rated the Games category as Relevant (scores above 3 on a 1-5 scale) The most relevant category among those proposed by us was the one linked to Events with an average score of 4.7 out of a maximum of 5: the participants appreciated the category as it was considered very versatile, as it allows the inclusion of different types of memories. The least relevant categories turned out to be Hobbies and Food. Hobbies received an average score of about 2.6 out of a maximum of 5, The main criticism concerned the name of the category because, according to the participants, it was not very pertinent compared to the other proposals. As possible replacements, terms such as “leisure” or “entertainment” have been suggested. Participants also showed very low interest in the Food category as this type of memories did not have a significant impact in their life experiences for most of them. In conclusion, the answers of the candidates showed that the most significant memories concern the dearest affections and the most important events in life such as graduation, marriage or the birth of children. Of the six categories proposed, those related to Music and Personal Events aroused the greatest interest. Everyone expressed their appreciation for the project, declaring that they are inclined to use the web application to share their memories. Thus, in the new version of the Remind app we introduced the Affects category (see Figure 2 left). Figure 2: (left) The user interface for selecting a category; (right) the user interface for adding an event-related memory 6 At the beginning of the interaction with the Remind application, users are asked whether they want to enter a new memory or review those previously entered. When a memory category is selected, then the user can provide the information associated with the specific memory. For example, Figure 2 (right) shows the user interface for entering a memory related to a particular event in life. The user has to indicate a name for the event, then provide a description, which can be done either vocally or by keyboard. The user can also indicate the age they had when they had when such event occurred, and optionally provide an image associated with it. In the case of a memory in the Hobby category, the user can also provide a list of activities required by the hobby. All such information can then be used by the games provided by Pepper for specific exercises. In general it is not necessary that the older adults directly enter the memories, to facilitate the process they can just tell them to some formal or informal caregiver, who can also help them in indicating relevant memories. 2.3 The Games with the Humanoid Robot The Pepper application prototype aims to stimulate cognitive functions through play sessions, which should last 15-20 minutes. The games present various exercises, which should be useful for making the participants think and reason to provide the correct answer. An initial set of four types of games have been identified: • Memory completion, where the robot presents a memory with a missing detail, which should be indicated by the user by choosing from a multiple choice of elements, and in case of a correct answer, the memory is reread to the older adult. For example: “when I was 12 years old I used to spend summer time in…”(and the robot shows three possible options: Marina di Pisa, Tirrenia, San Vincenzo or Castiglioncello) or “I used to listen to that singer when I traveled by car with my father” with possible answers: Modugno, Morandi, Celentano, Guccini; • Activities ordering, which can be applied to hobby, in which sequence of tasks presented in an unordered list should be put in the right order by the user (this can stimulate executive functions and procedural memory); • Memory association, where 3-4 memories are briefly described along with a set of details, and the user has to connect each memory with the corresponding detail (for example associate song titles to the corresponding singers); • Memory-related event question, where the user has to guess the event that happened in the same year of the presented memory, thus it asks the user to select which important event in the world happened that year from a set of events that are listed (for example: what happened in the same year you got married (1945)? Possible answers: “the end of second world war”, “the first man on the moon”, “women gain the right to vote in Italy“?). In a session at the beginning the robot asks the name of the user (Figure 3, left), then through such information it retrieves the memories that the user entered, which are available from the biography 7 application backend through a restful service and transmitted in JSON format. The memories arrive in the robot with the indication of the corresponding category, which is useful to determine how to exploit them in the various exercises. In the case of a missing detail exercise the robot proposes a memory and a list of possible missing details, which are derived from the memories of that user. In case there are not sufficient memories to identify the proposed options, the details are taken by the memories of another user who has a similar age. For the memory-related event exercises, the list of possible options in terms of real events to choose from are taken by external services. The activities ordering exercise refers only to the Hobby category of memories because only in that case the users are asked to enter the steps required by the hobby, thus the robot can retrieve sufficient information to propose it. Then, users can first select the type of game they want to play (Figure 3, center), and then they have the opportunity to perform the associated exercise, with personalised content (Figure 3, right). Figure 3: (left) Session start with the robot; (centre) robot with the list of available game topics; (right) a question from an instance of the memory game “missing element completion”. 3 Conclusions In this paper, we introduce a novel approach to personalising serious games for the cognitive stimulation of older adults delivered through a humanoid robot. It is based on a multimodal Web app to collect memories of older adults, and then such content is exploited in a set of games aiming to stimulate several cognitive resources in older adults implemented for a Pepper robot. In future work, we plan to validate the approach with trials involving a group of older adults with mild cognitive impairments participating in a Train the Brain programme in which they regularly attend cognitive stimulation exercises in a care centre managed by CNR in Pisa. We also plan to develop a customisation and analytics environment to further extend the possible personalisations of games and their analysis. Acknowledgements This work is partly supported by the CNR project SERENI https://hiis.isti.cnr.it/sereni/index.html. 8 REFERENCES [1] Abdollahi, H., Mollahosseini, A., Lane, J., & Mahoor, M. (2017). A pilot study on using an intelligent life-like robot as a companion for elderly individuals with dementia and depression. IEEE-RAS 17th International Conference on Humanoid Robotics, (pp. 541-546). [2] Bohlmeijer, E., Roemer, M., Cuijpers, P., & Smit, F. (2007). The effects of reminiscence on psychological well-being in older adults: A meta-analysis. Aging and Mental Health, 11(3), 291-300. [3] Carros, F., Meurer, J., Loffer, D., & Unbehaun, D. (2020). Exploring human-robot interaction with the elderly: results from a ten-week case study in a care home. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, (pp. 1-12). [4] McCrae, R., Costa, P., The five-factor model of personality: Theoretical perspectives 51 (1996). [5] Cruz-Sandoval, D., Morales-Tellez, A., Sandoval, E. B., & Favela, J. (2020). A social robot as therapy facilitator in interventions to deal with dementia-related behavioral symptoms. Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 161-169). ACM/IEEE. [6] Damianakis, T., Crete-Nishihata, M., Smith, K. L., Baecker, R. M., & Marziali, E. (2010). The psychosocial impacts of multimedia biographies on persons with cognitive impairments. The Gerontologist, 50(1), 23-35. [7] Hall, Edward T. ”A system for the notation of proxemic behavior.” American anthropologist 65.5 (1963): 1003-1026. [8] John, O. P., & Srivastava, S. (1999). The Big Five Trait taxonomy: History, measurement, and theoretical perspectives. Handbook of personality:Theory and research, 2nd ed. [9] Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M. T. (2014). Socially assistive robots in elderly care: a mixed-method systematic literature review. International Journal of Human- Computer Interaction, 30(5), 369-393. [10] Leonardi, N., Manca, M., Paternò, F., Santoro, C., Trigger-Action Programming for Personalising Humanoid Robot, the 2019 ACM Conference on Human Factors in Computing Systems (CHI'19), Paper 445 [11] Lippa, R. A., & Dietz, J. K. (2000). The relation of gender, personality, and intelligence to judges’ accuracy in judging strangers’ personality from brief video segments. Journal of Nonverbal Behavior, 24, 25–43. [12] Manca, M., Paternò, F., Santoro, C., Zedda, E., Braschi, C., R., F., & Sale, A. (2021). The impact of serious games with humanoid robots on mild cognitive impairment older adults. International Journal of Human-Computer Studies, 145, 102509. [13] Manera, V., Petit D.P., Derreumaux, A., Orvieto, I., Romagnoli, A., Lyttle, G., . Robert, .. (2015). Kitchen and cooking, a serious game for mild cognitive impairment and Alzheimer’s disease: a pilot study. Frontiers in aging neuroscience. [14] Nakasone, A., Prendinger, H., & Ishizuka, M. (2005, September). Emotion recognition from electromyography and skin conductance. In Proc. of the 5th international workshop on biosignal interpretation (pp. 219-222). [15] Nishiura, Y., Nihei, M., Nakamura-Thomas, H., & Inoue, T. (2021). Effectiveness of using assistive technology for time orientation and memory, in older adults with or without dementia. Disability and Rehabilitation: Assistive Technology 16(5), 472-478. 9 [16] Palumbo, V., & Paternò, F. (2020). Serious games to cognitively stimulate older adults: a systematic literature review. PETRA (pp. 27:1-27:10). ACM Press. [17] Serrano, J. P., Latorre, J. M., Gatz, M., & Montanes, J. (2004). Life review therapy using autobiographical retrieval practice for older adults with depressive symptomatology. Psychology and aging, 19(2), 272. [18] Smith, E., Nolen-Hoeksema, S., Fredrickson, B., & Loftus, G. (2002). Atkinson and Hilgard’s Introduction to Psychology. Wadsworth. [19] Softbank Robotics. (2021, 07 10). Pepper SDK-Tools. Retrieved from QiSDK: https://qisdk.softbankrobotics.com/sdk/doc/pepper-sdk/ch3_tools/tools.html [20] Tanaka, M., Ishii, A., Yamano, E., Ogikubo, H., Okazaki, M., & Kamimura, K. (2012). Effect of a human-type communication robot on cognitive function in elderly women living alone. Medical science monitor: international medical journal of experimental and clinical research, 18(9). [21] Tapus, A., & Vieru, A. M. (2013). Robot cognitive stimulation for the elderly. International Work-Conference on the Interplay Between Natural and Artificial Computation (pp. pp. 94- 102). Springer. [22] A.Tapus, C. Tapus, C., & Matarić, M. .. (2008). User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intelligent Service Robotics, 169-183. [23] Tay, B., Jung, Y., & Park, T. (2014). When stereotypes meet robots: the double-edge sword of robot gender and personality in human–robot interaction. . Computers in Human Behavior, 38, 75-84. [24] Tong, T., & Chignell, M. (2013). Designing game-based cognitive assessments for elderly adults. Proceedings of the First International Conference on Gameful design, research, and applications, (pp. 127-130). 10