Immersive applications are widely used in educational and training settings across various contexts, such as museums, classrooms, or industries. They ofer interactive and engaging experiences that help users focus their attention and enhance skills like critical thinking. This paper explores the potential of designing immersive experiences to complement traditional therapies for individuals with dementia. This target population presents specific needs due to their medical conditions and technological profile. For this reason, to develop practical applications, it is essential to collaborate with experts and patients to understand how to design tailored solutions. We propose an action research methodology to create immersive experiences to target specific cognitive skills, such as memory, recognition, and association, especially for people with dementia. We collaborated with psychiatry specialists and individuals diagnosed with mild cognitive impairment to identify a research opportunity to create two AR applications: one for tablets and another for smart glasses. The applications are developed as part of an iterative process to identify significant insights into how patients interact with the devices and interfaces within clinical settings. Finally, these prototypes underwent testing in a clinical study involving ten patients diagnosed with mild cognitive impairment, conducted in the presence of their therapists.
Neurodegenerative disorders currently afect 15% of the total population of the world, with
a significant increase during the last 30 years and an expectation to at least double this
percentage during the next two decades [
In this direction, finding innovative ways to stimulate their brains and practice cognitive skills is crucial. Various studies have explored the advantages of advanced technologies to support medical treatments, such as tablets, wearables, and Augmented and Virtual Reality. A significant consideration when working with individuals afected by neurodegenerative disorders is their potential dificulty in learning new technologies and interfaces that depend on a great variety of factors, including age, previous experience with technology, and cultural and economic situation. Technology should consider these factors and others that could appear later when the symptoms evolve and there are personal diferences among patients.
This paper explores the potential of immersive experiences for supporting people with dementia (PwD) in practicing specific cognitive skills, such as memory, recognition, and association. In particular, we are interested in augmented reality (AR) as an immersive technology that allows users to connect with the physical objects around them and avoid disorientation and dizziness [4, 5]. Other solutions have delved into the benefits of AR for aiding patients in tasks like object recognition or identifying hazardous situations [6, 7, 8], furnishing therapists with additional patient data and health status updates [9], and enhancing the interactivity of cognitive stimulation exercises [10, 11, 12].
We propose an action research methodology for creating AR applications involving therapists and patients and collecting insights based on their personal experiences concerning dementia. The action research approach is based on observing a real scenario to identify insights for driving the research work and defining an iterative process where the involvement of the stakeholders is crucial to gain insights into their needs [13]. The proposed methodology consists of four cycles of data collection, design, evaluation, and lessons learned, involving two therapists and twelve patients.
The therapists helped us detail the practical exercises to propose as part of the immersive experience and select the patients who could collaborate in testing the prototypes. In the initial cycle of the methodology, we interviewed the therapists to identify the cognitive skills to practice: object recognition, memory recall, and creating connections among object attributes. After, we ran three other cycles involving both therapists and patients to design an early-stage prototype of four AR exercises to practice the identified cognitive skills: Recognize for object recognition, Remember for memory recall of individual objects, Play Around for establishing a connection between attributes like shapes, colors, and sounds, and Simon for memory recall of sequence of sounds. Based on the results of these sessions, we proposed a final version of the AR applications.
The number of patients diagnosed with dementia will continue to increase in the coming years [14] due to population growth, life expectancy improvement, and the lack of efective cures for most symptoms [15]. Thus, there will be an increasing need to find solutions for caring for PwD and guaranteeing a good quality of life as long as possible [16]. The evolution of the symptoms damages the brain’s nerve cells of the patients. Cognitive training is among the treatments ofered to slow down the speedup of these efects [ 17], improve happiness and rest (i.e., de-stress), and reduce the usage of medical drugs for stress and depression issues [18], with a consequent reduction of the caregivers’ workload [8].
As a contribution to the impact of dementia from both a social and an economic point of view, diferent technologies have started to be explored as support for patients, caregivers, relatives, and health professionals to ofer innovative solutions to facilitate day-to-day care, improve the quality of life and mitigate the symptoms [19]. Most solutions have been designed for smartphones, tablets, Head Mounted Displays (HMDs) for virtual and augmented reality, and even physical computing [20]. This research focuses on designing immersive experiences, considering that virtual and augmented reality applications can ofer exciting benefits to deal with dementia but with some relevant diferences. Virtual reality aims to create a synthetic world, isolating the user from reality. It has been used mainly by therapists, caregivers, and relatives for empathy training [21] and by patients as diagnostic tools and memory training in spatial and navigation-controlled environments [22]. Its usage seems limited in PwD due to possible drawbacks such as cyber-sickness and eye fatigue [23]. AR technology deals with these issues by keeping a view of reality and mixing digital information with the natural world [5].
An example is Dementia Eyes, a tool to simulate vision degeneration as one of the symptoms of dementia syndrome [24] and make other people realize the dificulties that PwD faces every day. My Daily Routine allows the caregivers to set a list of reminders that the patients will receive using an HMD in the form of a great variety of multimedia content, including 3D models and videos [7]. In [25], AR has been used to add labels to a lifelogging system to stimulate reminiscences. Boletsis and McCallum designed six AR minigames to interact with ten cubes associated with a marker [12]. The patient uses a tablet to interact with the cubes and solve the games. Each game focuses on diferent cognitive abilities, such as perception, attention, language processing, and executive functions, such as problem-solving, decision-making, and lfexibility. Another relevant work to our scope is MemHolo, a tool with three activities for practicing short-term and spatial memory designed in an iterative process that involves experts in dementia and evaluated with several older adults not diagnosed with dementia syndrome [11].
We can observe several open issues from the literature that need further consideration. One is the lack of an actual evaluation with patients in a controlled environment to check their reactions to the applications and devices used to interact with the digital elements in the scene. Another is the involvement of therapists and patients to design solutions that fulfill their expectations. In this paper, we are interested in both directions to explore the benefits of AR for supporting PwD in practicing diferent cognitive abilities.
The experience of both therapists and patients is crucial for designing solutions that efectively support cognitive therapies. Therapists should be able to define the skills to practice depending on the stage and symptoms of the disorder. Patients need to interact with technologies easily and enjoyably, avoiding frustrations and misunderstandings. In both cases, it is crucial to profile the specific needs and requirements of each stakeholder interested in the application.
In this paper, we propose a methodology inspired by the action research approach and its four characteristics [13]: (i) practical nature to address real-world problems; (ii) change to adapt the solution to the issues pointed out by the problem to solve; (iii) cyclical process to implement the solution iterating over design and evaluation; (iv) participation of practitioners directly interested by the problems to solve. The methodology consists of cycling over four phases (see Figure 1): data collection, design or re-design, participants, and lessons learned.
As detailed in Figure 1, the activities, the profile, and the number of participants change depending on the cycle. The first cycle defines the problem to solve and provides all the details needed to design the solution. To this extent, we interviewed two therapists specializing in neurodegenerative diseases about the current treatment practices and the cognitive skills trained during the therapies. They suggested focusing on the memory to recall and recognize objects and create associations among diferent attributes of an object. Moreover, they brainstormed examples of typical exercises for brain training.
This cycle results in the design of early-stage prototypes of four AR applications [26]. The first is a tablet-based AR application called Recognize for object recognition. The exercise consists of showing aleatory objects on the tablet along with three buttons, each bearing a distinct label, where only one corresponds to the correct answer. Users are seated in front of the tablet; the tablet points to a marker to position the object on the table, and the users have to tap on one of the buttons. The application provides visual and auditory feedback depending on whether the answer is correct or incorrect, as well as a final screen with a summary of all the answers. The second is Remember for memory recall of individual objects. It works like the previous one but with a slight diference in the logic of the activity. In this case, the application shows two objects for a limited time and then hides them. The user has to recognize and remember them by tapping on the buttons with the correct labels.
The other two applications are glasses-based (i.e., Hololens): Play Around for establishing a connection between attributes like shapes, colors, and sounds, and Simon for memory recall of sequence of sounds. Both show four cubes in front of the user. In Play Around, the user can touch any cube and discover which sound is associated with it. In Simon, the user has to reproduce a sequence of sounds, remembering which sound is associated with each cube and touching the corresponding cube. In this case, the users can interact by approaching the cubes with their hands, but they do not have any restriction on the gesture to reproduce.
From the second cycle, we focus on designing the AR applications as iterative prototypes and showing them to the involved participants for their opinions and suggestions. At this stage, we limited the prototypes to the two tablet-based applications and the glasses-based Play Around. Before continuing with the development, we wanted feedback about the opportunities of a device like the Hololens for the patients. As in the previous cycle, we interviewed the two therapists again to evaluate the interface usability and the device used for the interaction (e.g., tablet and see-through glasses). As a result, they emphasized the importance of avoiding frustration and misunderstanding because these feelings could lead the patients to stop being receptive. One of the suggestions was to review the logic behind the proposed activities and adapt them to the patient’s cognitive abilities.
After improving the prototypes, in the third cycle, we involved two patients diagnosed with early dementia and asked them to interact using the table and the glasses. The fewer participants allowed us to spend more time interviewing them and letting them use the applications freely. We proposed two interaction modes for the two tablet applications: touching digital elements on the screen and telling the object’s name. As part of the lessons learned during this cycle, we found out that the voice generates several issues related mainly to the dificulties they could have pronouncing certain words and the fact that they wore a mask covering their mouths. For these reasons, we finally decided to keep joust touch and gestures as interaction modes. Another interesting result from this first contact with the patients was their enthusiasm towards the Hololens. At this stage, we wanted to check whether the glasses could be comfortable for them before continuing to develop the Simon application, receiving enthusiastic comments. One of the aspects that the patients highlighted was the possibility of trying on something new that could represent an incentive for practicing and doing exercises.
In the last cycle, we tested the improved version of the prototypes and, in particular, the Simon application. In collaboration with the therapists, we could involve ten patients, seven women and three men, aged 70-83 years old, attending a routine appointment in a health center in Madrid [26]. Based on their symptoms, the patients were diagnosed with diferent stages of dementia: four in a mild stage, four in a moderate stage, and two in a severe stage. We interviewed each of them in an individual session accompanied by one of the therapists.
The collected feedback from the evaluation suggests that, in general, the patients enjoyed the four activities. Still, they preferred to interact with the tablet, considering that it is a device that most of them are used to. They found the Remember application the most challenging, but after a few attempts, they gained fluency and improved the number of correct hits.
About the Hololens, nobody objected to putting them on or experienced discomfort while using the device. The patients enjoyed interacting with the virtual cubes and sounds of the Play around application and even lost track of time. In most cases, we had to stop them as more than ten minutes passed. However, the Simon game proved too challenging for some patients. They found diferent dificulties, such as recognizing the sound, diferentiating it from another, remembering the association cube to a sound, and the sequence of sounds. Sometimes, they couldn’t finish the game, leading to frustration. Therapists suggested adding more cues to make tasks easier and give them more time or attempts. Despite the challenges, participants enjoyed the experience and found it intuitive, appreciating being still connected to reality thanks to the see-through glasses.
Analyzing the feedback collected during the methodology, we focused on redesigning the glasses-based application Simon to ofer a more enjoyable and less frustrating experience. The redesign considered two main aspects: personalization and game logic. Personalization refers to the possibility of adapting the experience to the patient’s specific needs, thinking they could have diferent ways to perceive and react while interacting with the application. We implemented a control panel for caregivers and therapists for starting the game, positioning the digital elements in the space, adjusting the dificulty as the number of sounds in the sequence or the time interval, switching to the Play around application, displaying the game progress and the scene seen through the Hololens by the patient. The control panel can be used in a desktop and mobile web browser. In Figure 2, part a shows the options for personalizing the game, and part b shows the scene that the user sees through the glasses.
Regarding the game logic, we changed the cube for instruments to help the users recognize the sounds by looking at the instruments that produce them. We added animations around the instruments that are playing as movements and musical notes (see part c in Figure 2). The upper part of the scene displays the sequence of the sounds already played and recognized by the user in case they forget them.
In this paper, we have explored the challenges of AR technologies for practicing cognitive capabilities like memory, recognition, and association. As already seen in the literature, AR ofers several benefits that could be exploited for building solutions to deal with the day-to-day care of patients diagnosed with early dementia [5]. Since dementia is an umbrella term for a wide range of syndromes, it is crucial to involve experts and patients to understand how to design a valuable contribution for them. For this reason, we propose an action research methodology that could be derived from collaboration with therapists and patients’ insights about the design of AR experiences. We ran four cycles of the methodology, and in each one, we (a) (b)
(c) collected data needed for making decisions about how to improve the design of the prototypes and how to move forward in the following steps.
The result is the development of four AR applications where the interaction occurs using a tablet and a pair of Hololens. The introduction of the Hololens was one of the most interesting results of this work. The patients were enthusiastic about using an innovative, eficient device that was easy to interact with. Most of them got frustrated with the logic behind the proposed games, which led us to a deep redesign considering the suggestions from the therapists and the patients. In the future, we plan to run an additional iteration of the methodology to test the new version of the game.
This work is supported by the project sense2MakeSense, funded by the Spanish State Agency of Research (PID2019-109388GB-I00). [4] M. S. Farooq, Z. Zahid, U. Omer, R. Tehseen, A. Alvi, U. Farooq, Z. Atal, Applications of Augmented Reality in Neurology: Architectural Model and Guidelines, IEEE Access 10 (2022) 102804–102830. doi:10.1109/ACCESS.2022.3206600, conference Name: IEEE Access. [5] J. Hayhurst, How Augmented Reality and Virtual Reality is Being Used to Support People Living with Dementia—Design Challenges and Future Directions, in: T. Jung, M. C. tom Dieck (Eds.), Augmented Reality and Virtual Reality: Empowering Human, Place and Business, Progress in IS, Springer International Publishing, Cham, 2018, pp. 295–305. URL: https://doi.org/10.1007/978-3-319-64027-3_20. doi:10.1007/978-3-319-64027-3_20. [6] D. Wolf, D. Besserer, K. Sejunaite, M. Riepe, E. Rukzio, cARe: An Augmented Reality Support System for Dementia Patients, in: Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 2018, pp. 42–44. URL: https://dl.acm.org/doi/10.1145/3266037.3266095. doi:10.1145/3266037. 3266095. [7] M. A. Hamilton, A. P. Beug, H. J. Hamilton, W. J. Norton, Augmented Reality Technology for People Living with Dementia and their Care Partners, in: 2021 the 5th International Conference on Virtual and Augmented Reality Simulations, ACM, Melbourne VIC Australia, 2021, pp. 21–30. URL: https://dl.acm.org/doi/10.1145/3463914.3463918. doi:10.1145/3463914.3463918. [8] N. Rohrbach, P. Gulde, A. R. Armstrong, L. Hartig, A. Abdelrazeq, S. Schröder, J. Neuse, T. Grimmer, J. Diehl-Schmid, J. Hermsdörfer, An augmented reality approach for ADL support in Alzheimer’s disease: a crossover trial, Journal of NeuroEngineering and Rehabilitation 16 (2019) 66. URL: https://doi.org/10.1186/s12984-019-0530-z. doi:10.1186/ s12984-019-0530-z. [9] A. Vovk, A. Patel, D. Chan, Augmented Reality for Early Alzheimer’s Disease Diagnosis, in: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, Association for Computing Machinery, New York, NY, USA, 2019, pp. 1–6.
URL: https://doi.org/10.1145/3290607.3313007. doi:10.1145/3290607.3313007. [10] A. Niknam, An Augmented Reality Mobile Game Design to Enhance Spatial Memory in
Elderly with Dementia (2021). [11] B. Aruanno, F. Garzotto, MemHolo: mixed reality experiences for subjects with Alzheimer’s disease, Multimedia Tools and Applications 78 (2019) 13517–13537. URL: https://doi.org/ 10.1007/s11042-018-7089-8. doi:10.1007/s11042-018-7089-8. [12] C. Boletsis, S. McCallum, Augmented Reality Cubes for Cognitive Gaming: Preliminary Usability and Game Experience Testing, International Journal of Serious Games 3 (2016). URL: https://journal.seriousgamessociety.org/index.php/IJSG/article/view/106. doi:10.17083/ijsg.v3i1.106, number: 1. [13] M. Denscombe, The Good Research Guide: Research Methods for Small-Scale Social
Research Projects, McGraw-Hill Education (UK), 2021. [14] M. Prince, A. Comas-Herrera, M. Knapp, M. Guerchet, M. Karagiannidou, World Alzheimer report 2016: improving healthcare for people living with dementia: coverage, quality and costs now and in the future (2016). Publisher: Alzheimer’s Disease International (ADI). [15] S. Kumari, K. Bagri, R. Deshmukh, Dementia: A journey from cause to cure, in:
Nanomedicine-Based Approaches for the Treatment of Dementia, Elsevier, 2023, pp. 37–56. [16] G. Wong, M. Knapp, Should we move dementia research funding from a cure to its care?, 2020. Issue: 4 Pages: 303–305 Publication Title: Expert review of neurotherapeutics Volume: 20. [17] K. Chen, V. W. Q. Lou, S. S. C. Lo, Exploring the acceptance of tablets usage for cognitive training among older people with cognitive impairments: A mixed-methods study, Applied Ergonomics 93 (2021) 103381. Publisher: Elsevier. [18] W. Moyle, C. Jones, T. Dwan, T. Petrovich, Efectiveness of a virtual reality forest on people with dementia: A mixed methods pilot study, The Gerontologist 58 (2018) 478–487.
Publisher: Oxford University Press US. [19] F. Meiland, A. Innes, G. Mountain, L. Robinson, H. van der Roest, J. A. García-Casal, D. Gove, J. R. Thyrian, S. Evans, R.-M. Dröes, Technologies to support community-dwelling persons with dementia: a position paper on issues regarding development, usability, efectiveness and cost-efectiveness, deployment, and ethics, JMIR rehabilitation and assistive technologies 4 (2017) e6376. Publisher: JMIR Publications Inc., Toronto, Canada. [20] S. Enshaeifar, P. Barnaghi, S. Skillman, A. Markides, T. Elsaleh, S. T. Acton, R. Nilforooshan, H. Rostill, The internet of things for dementia care, IEEE Internet Computing 22 (2018) 8–17. Publisher: IEEE. [21] J. Hirt, T. Beer, Use and impact of virtual reality simulation in dementia care education: A scoping review, Nurse education today 84 (2020) 104207. Publisher: Elsevier. [22] J. Strong, Immersive virtual reality and persons with dementia: a literature review, Journal of gerontological social work 63 (2020) 209–226. Publisher: Taylor & Francis. [23] S. Davis, K. Nesbitt, E. Nalivaiko, A systematic review of cybersickness, in: Proceedings of the 2014 conference on interactive entertainment, 2014, pp. 1–9. [24] X. Shen, Y. S. Pai, D. Kiuchi, K. Oishi, K. Bao, T. Aoki, K. Minamizawa, Dementia Eyes:
Perceiving Dementia with Augmented Reality, in: SIGGRAPH Asia 2021 XR, 2021, pp. 1–2. [25] M. N. Sakib, M. M. Rahman, H. Mahmud, M. K. Hasan, Augmented reality-based lifelogging system for reminiscence, in: Proceedings of International Conference on Fourth Industrial Revolution and Beyond 2021, Springer, 2022, pp. 493–504. [26] T. Onorati, P. Díaz, A. Montero, I. Aedo, Designing AR Applications for People Living with Dementia, in: J. Abdelnour Nocera, M. Kristín Lárusdóttir, H. Petrie, A. Piccinno, M. Winckler (Eds.), Human-Computer Interaction – INTERACT 2023, Springer Nature Switzerland, Cham, 2023, pp. 449–453. doi:10.1007/978-3-031-42293-5_52.