=Paper=
{{Paper
|id=Vol-3715/paper9
|storemode=property
|title=An Action Research Methodology for Designing AR Experiences for People with Dementia
|pdfUrl=https://ceur-ws.org/Vol-3715/paper9.pdf
|volume=Vol-3715
|authors=Teresa Onorati,Paloma Díaz,Ignacio Aedo,Álvaro Montero
|dblpUrl=https://dblp.org/rec/conf/ini-dh/Onorati0AM24
}}
==An Action Research Methodology for Designing AR Experiences for People with Dementia==
https://ceur-ws.org/Vol-3715/paper9.pdf
An Action Research Methodology for Designing AR
Experiences for People with Dementia
Teresa Onorati1,*,† , Paloma Díaz1,† , Ignacio Aedo1,† and Álvaro Montero1,†
1
Computer Science Department, Universidad Carlos III de Madrid, Madrid, Spain
Abstract
Immersive applications are widely used in educational and training settings across various contexts,
such as museums, classrooms, or industries. They offer interactive and engaging experiences that
help users focus their attention and enhance skills like critical thinking. This paper explores the
potential of designing immersive experiences to complement traditional therapies for individuals with
dementia. This target population presents specific needs due to their medical conditions and technological
profile. For this reason, to develop practical applications, it is essential to collaborate with experts and
patients to understand how to design tailored solutions. We propose an action research methodology
to create immersive experiences to target specific cognitive skills, such as memory, recognition, and
association, especially for people with dementia. We collaborated with psychiatry specialists and
individuals diagnosed with mild cognitive impairment to identify a research opportunity to create two
AR applications: one for tablets and another for smart glasses. The applications are developed as part
of an iterative process to identify significant insights into how patients interact with the devices and
interfaces within clinical settings. Finally, these prototypes underwent testing in a clinical study involving
ten patients diagnosed with mild cognitive impairment, conducted in the presence of their therapists.
Keywords
Action Research, Augmented Reality, People with Dementia
1. Introduction
Neurodegenerative disorders currently affect 15% of the total population of the world, with
a significant increase during the last 30 years and an expectation to at least double this per-
centage during the next two decades [1]. Among the neurodegenerative disorders, dementia
is an umbrella term that includes a wide range of symptoms, like memory loss, difficulties in
concentration, planning, organization, and disorientation regarding time or place [2]. It is not
directly related to aging but to the impact that a disease has on the nerve cells in the brain,
causing one or more symptoms. Consequently, the damages and the evolution of the disease
vary depending on each individual and their prior health condition, identifying three main
stages as early (or mild), middle (or moderate), and late (or severe) [3]. Considering that there
INI-DH 2024: Workshop on Innovative Interfaces in Digital Healthcare, in conjunction with International Conference on
Advanced Visual Interfaces 2024 (AVI 2024), June 3–7, 2024, Arenzano, Genoa, Italy (2024)
*
Corresponding author.
†
These authors contributed equally.
$ tonorati@inf.uc3m.es (T. Onorati); pdp@inf.uc3m.es (P. Díaz); aedo@ia.uc3m.es (I. Aedo);
ammontes@inf.uc3m.es ( Montero)
0000-0002-3154-249X (T. Onorati); 0000-0002-9493-7739 (P. Díaz); 0000-0001-5819-0511 (I. Aedo);
0000-0002-2511-9986 ( Montero)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
is no cure for dementia, but different treatments depending on the symptoms and the health
status of the patients. The treatments include administering medications and psychological
therapies and tailoring the care to the specific condition of each individual with the common
objective of guaranteeing a good quality of life as long as possible [2].
In this direction, finding innovative ways to stimulate their brains and practice cognitive skills
is crucial. Various studies have explored the advantages of advanced technologies to support
medical treatments, such as tablets, wearables, and Augmented and Virtual Reality. A significant
consideration when working with individuals affected by neurodegenerative disorders is their
potential difficulty in learning new technologies and interfaces that depend on a great variety
of factors, including age, previous experience with technology, and cultural and economic
situation. Technology should consider these factors and others that could appear later when
the symptoms evolve and there are personal differences among patients.
This paper explores the potential of immersive experiences for supporting people with demen-
tia (PwD) in practicing specific cognitive skills, such as memory, recognition, and association.
In particular, we are interested in augmented reality (AR) as an immersive technology that
allows users to connect with the physical objects around them and avoid disorientation and
dizziness [4, 5]. Other solutions have delved into the benefits of AR for aiding patients in
tasks like object recognition or identifying hazardous situations [6, 7, 8], furnishing therapists
with additional patient data and health status updates [9], and enhancing the interactivity of
cognitive stimulation exercises [10, 11, 12].
We propose an action research methodology for creating AR applications involving therapists
and patients and collecting insights based on their personal experiences concerning dementia.
The action research approach is based on observing a real scenario to identify insights for driving
the research work and defining an iterative process where the involvement of the stakeholders
is crucial to gain insights into their needs [13]. The proposed methodology consists of four
cycles of data collection, design, evaluation, and lessons learned, involving two therapists and
twelve patients.
The therapists helped us detail the practical exercises to propose as part of the immersive
experience and select the patients who could collaborate in testing the prototypes. In the
initial cycle of the methodology, we interviewed the therapists to identify the cognitive skills to
practice: object recognition, memory recall, and creating connections among object attributes.
After, we ran three other cycles involving both therapists and patients to design an early-stage
prototype of four AR exercises to practice the identified cognitive skills: Recognize for object
recognition, Remember for memory recall of individual objects, Play Around for establishing a
connection between attributes like shapes, colors, and sounds, and Simon for memory recall of
sequence of sounds. Based on the results of these sessions, we proposed a final version of the
AR applications.
2. Augmented Reality for Dementia
The number of patients diagnosed with dementia will continue to increase in the coming years
[14] due to population growth, life expectancy improvement, and the lack of effective cures
for most symptoms [15]. Thus, there will be an increasing need to find solutions for caring
for PwD and guaranteeing a good quality of life as long as possible [16]. The evolution of the
symptoms damages the brain’s nerve cells of the patients. Cognitive training is among the
treatments offered to slow down the speedup of these effects [17], improve happiness and rest
(i.e., de-stress), and reduce the usage of medical drugs for stress and depression issues [18], with
a consequent reduction of the caregivers’ workload [8].
As a contribution to the impact of dementia from both a social and an economic point of
view, different technologies have started to be explored as support for patients, caregivers,
relatives, and health professionals to offer innovative solutions to facilitate day-to-day care,
improve the quality of life and mitigate the symptoms [19]. Most solutions have been designed
for smartphones, tablets, Head Mounted Displays (HMDs) for virtual and augmented reality,
and even physical computing [20]. This research focuses on designing immersive experiences,
considering that virtual and augmented reality applications can offer exciting benefits to deal
with dementia but with some relevant differences. Virtual reality aims to create a synthetic
world, isolating the user from reality. It has been used mainly by therapists, caregivers, and
relatives for empathy training [21] and by patients as diagnostic tools and memory training in
spatial and navigation-controlled environments [22]. Its usage seems limited in PwD due to
possible drawbacks such as cyber-sickness and eye fatigue [23]. AR technology deals with these
issues by keeping a view of reality and mixing digital information with the natural world [5].
An example is Dementia Eyes, a tool to simulate vision degeneration as one of the symptoms
of dementia syndrome [24] and make other people realize the difficulties that PwD faces every
day. My Daily Routine allows the caregivers to set a list of reminders that the patients will
receive using an HMD in the form of a great variety of multimedia content, including 3D models
and videos [7]. In [25], AR has been used to add labels to a lifelogging system to stimulate
reminiscences. Boletsis and McCallum designed six AR minigames to interact with ten cubes
associated with a marker [12]. The patient uses a tablet to interact with the cubes and solve
the games. Each game focuses on different cognitive abilities, such as perception, attention,
language processing, and executive functions, such as problem-solving, decision-making, and
flexibility. Another relevant work to our scope is MemHolo, a tool with three activities for
practicing short-term and spatial memory designed in an iterative process that involves experts
in dementia and evaluated with several older adults not diagnosed with dementia syndrome
[11].
We can observe several open issues from the literature that need further consideration. One is
the lack of an actual evaluation with patients in a controlled environment to check their reactions
to the applications and devices used to interact with the digital elements in the scene. Another
is the involvement of therapists and patients to design solutions that fulfill their expectations.
In this paper, we are interested in both directions to explore the benefits of AR for supporting
PwD in practicing different cognitive abilities.
3. Action Research Methodology
The experience of both therapists and patients is crucial for designing solutions that effectively
support cognitive therapies. Therapists should be able to define the skills to practice depending
on the stage and symptoms of the disorder. Patients need to interact with technologies easily
Figure 1: Action Research Methodology in four cycles and four stages for each cycle: data collection,
solution design, participants, and lessons learned
and enjoyably, avoiding frustrations and misunderstandings. In both cases, it is crucial to profile
the specific needs and requirements of each stakeholder interested in the application.
In this paper, we propose a methodology inspired by the action research approach and its
four characteristics [13]: (i) practical nature to address real-world problems; (ii) change to adapt
the solution to the issues pointed out by the problem to solve; (iii) cyclical process to implement
the solution iterating over design and evaluation; (iv) participation of practitioners directly
interested by the problems to solve. The methodology consists of cycling over four phases (see
Figure 1): data collection, design or re-design, participants, and lessons learned.
3.1. First Cycle
As detailed in Figure 1, the activities, the profile, and the number of participants change
depending on the cycle. The first cycle defines the problem to solve and provides all the details
needed to design the solution. To this extent, we interviewed two therapists specializing in
neurodegenerative diseases about the current treatment practices and the cognitive skills trained
during the therapies. They suggested focusing on the memory to recall and recognize objects
and create associations among different attributes of an object. Moreover, they brainstormed
examples of typical exercises for brain training.
This cycle results in the design of early-stage prototypes of four AR applications [26]. The first
is a tablet-based AR application called Recognize for object recognition. The exercise consists of
showing aleatory objects on the tablet along with three buttons, each bearing a distinct label,
where only one corresponds to the correct answer. Users are seated in front of the tablet; the
tablet points to a marker to position the object on the table, and the users have to tap on one
of the buttons. The application provides visual and auditory feedback depending on whether
the answer is correct or incorrect, as well as a final screen with a summary of all the answers.
The second is Remember for memory recall of individual objects. It works like the previous one
but with a slight difference in the logic of the activity. In this case, the application shows two
objects for a limited time and then hides them. The user has to recognize and remember them
by tapping on the buttons with the correct labels.
The other two applications are glasses-based (i.e., Hololens): Play Around for establishing a
connection between attributes like shapes, colors, and sounds, and Simon for memory recall
of sequence of sounds. Both show four cubes in front of the user. In Play Around, the user
can touch any cube and discover which sound is associated with it. In Simon, the user has to
reproduce a sequence of sounds, remembering which sound is associated with each cube and
touching the corresponding cube. In this case, the users can interact by approaching the cubes
with their hands, but they do not have any restriction on the gesture to reproduce.
3.2. Second Cycle
From the second cycle, we focus on designing the AR applications as iterative prototypes and
showing them to the involved participants for their opinions and suggestions. At this stage, we
limited the prototypes to the two tablet-based applications and the glasses-based Play Around.
Before continuing with the development, we wanted feedback about the opportunities of a
device like the Hololens for the patients. As in the previous cycle, we interviewed the two
therapists again to evaluate the interface usability and the device used for the interaction
(e.g., tablet and see-through glasses). As a result, they emphasized the importance of avoiding
frustration and misunderstanding because these feelings could lead the patients to stop being
receptive. One of the suggestions was to review the logic behind the proposed activities and
adapt them to the patient’s cognitive abilities.
3.3. Third Cycle
After improving the prototypes, in the third cycle, we involved two patients diagnosed with
early dementia and asked them to interact using the table and the glasses. The fewer participants
allowed us to spend more time interviewing them and letting them use the applications freely.
We proposed two interaction modes for the two tablet applications: touching digital elements
on the screen and telling the object’s name. As part of the lessons learned during this cycle, we
found out that the voice generates several issues related mainly to the difficulties they could
have pronouncing certain words and the fact that they wore a mask covering their mouths.
For these reasons, we finally decided to keep joust touch and gestures as interaction modes.
Another interesting result from this first contact with the patients was their enthusiasm towards
the Hololens. At this stage, we wanted to check whether the glasses could be comfortable for
them before continuing to develop the Simon application, receiving enthusiastic comments.
One of the aspects that the patients highlighted was the possibility of trying on something new
that could represent an incentive for practicing and doing exercises.
3.4. Fourth Cycle
In the last cycle, we tested the improved version of the prototypes and, in particular, the Simon
application. In collaboration with the therapists, we could involve ten patients, seven women
and three men, aged 70-83 years old, attending a routine appointment in a health center in
Madrid [26]. Based on their symptoms, the patients were diagnosed with different stages
of dementia: four in a mild stage, four in a moderate stage, and two in a severe stage. We
interviewed each of them in an individual session accompanied by one of the therapists.
The collected feedback from the evaluation suggests that, in general, the patients enjoyed the
four activities. Still, they preferred to interact with the tablet, considering that it is a device that
most of them are used to. They found the Remember application the most challenging, but after
a few attempts, they gained fluency and improved the number of correct hits.
About the Hololens, nobody objected to putting them on or experienced discomfort while
using the device. The patients enjoyed interacting with the virtual cubes and sounds of the Play
around application and even lost track of time. In most cases, we had to stop them as more
than ten minutes passed. However, the Simon game proved too challenging for some patients.
They found different difficulties, such as recognizing the sound, differentiating it from another,
remembering the association cube to a sound, and the sequence of sounds. Sometimes, they
couldn’t finish the game, leading to frustration. Therapists suggested adding more cues to make
tasks easier and give them more time or attempts. Despite the challenges, participants enjoyed
the experience and found it intuitive, appreciating being still connected to reality thanks to the
see-through glasses.
4. Redesign of AR Experiences for PwD
Analyzing the feedback collected during the methodology, we focused on redesigning the
glasses-based application Simon to offer a more enjoyable and less frustrating experience. The
redesign considered two main aspects: personalization and game logic. Personalization refers
to the possibility of adapting the experience to the patient’s specific needs, thinking they
could have different ways to perceive and react while interacting with the application. We
implemented a control panel for caregivers and therapists for starting the game, positioning the
digital elements in the space, adjusting the difficulty as the number of sounds in the sequence
or the time interval, switching to the Play around application, displaying the game progress and
the scene seen through the Hololens by the patient. The control panel can be used in a desktop
and mobile web browser. In Figure 2, part a shows the options for personalizing the game, and
part b shows the scene that the user sees through the glasses.
Regarding the game logic, we changed the cube for instruments to help the users recognize
the sounds by looking at the instruments that produce them. We added animations around the
instruments that are playing as movements and musical notes (see part c in Figure 2). The upper
part of the scene displays the sequence of the sounds already played and recognized by the user
in case they forget them.
5. Conclusions and Future Works
In this paper, we have explored the challenges of AR technologies for practicing cognitive
capabilities like memory, recognition, and association. As already seen in the literature, AR
offers several benefits that could be exploited for building solutions to deal with the day-to-day
care of patients diagnosed with early dementia [5]. Since dementia is an umbrella term for
a wide range of syndromes, it is crucial to involve experts and patients to understand how
to design a valuable contribution for them. For this reason, we propose an action research
methodology that could be derived from collaboration with therapists and patients’ insights
about the design of AR experiences. We ran four cycles of the methodology, and in each one, we
(a) (b) (c)
Figure 2: (a) Main screen of the control panel to personalize the game (b) User point of view of the
scene visible through the glasses (c) Scene of the Simon game with the animated guitar and cello.
collected data needed for making decisions about how to improve the design of the prototypes
and how to move forward in the following steps.
The result is the development of four AR applications where the interaction occurs using a
tablet and a pair of Hololens. The introduction of the Hololens was one of the most interesting
results of this work. The patients were enthusiastic about using an innovative, efficient device
that was easy to interact with. Most of them got frustrated with the logic behind the proposed
games, which led us to a deep redesign considering the suggestions from the therapists and the
patients. In the future, we plan to run an additional iteration of the methodology to test the
new version of the game.
Acknowledgments
This work is supported by the project sense2MakeSense, funded by the Spanish State Agency of
Research (PID2019-109388GB-I00).
References
[1] J. Van Schependom, M. D’haeseleer, Advances in Neurodegenerative Diseases, Journal
of Clinical Medicine 12 (2023) 1709. URL: https://www.ncbi.nlm.nih.gov/pmc/articles/
PMC10002914/. doi:10.3390/jcm12051709.
[2] What is dementia? | Alzheimer’s Society, 2024. URL: https://www.alzheimers.org.uk/
about-dementia/types-dementia/what-is-dementia. doi:Lastaccess2024-03-27.
[3] L. Clare, R. T. Woods, Cognitive training and cognitive rehabilitation for people with early-
stage Alzheimer’s disease: A review, Neuropsychological Rehabilitation 14 (2004) 385–401.
URL: https://doi.org/10.1080/09602010443000074. doi:10.1080/09602010443000074,
publisher: Routledge _eprint: https://doi.org/10.1080/09602010443000074.
[4] M. S. Farooq, Z. Zahid, U. Omer, R. Tehseen, A. Alvi, U. Farooq, Z. Atal, Applications of
Augmented Reality in Neurology: Architectural Model and Guidelines, IEEE Access 10
(2022) 102804–102830. doi:10.1109/ACCESS.2022.3206600, conference Name: IEEE
Access.
[5] J. Hayhurst, How Augmented Reality and Virtual Reality is Being Used to Support People
Living with Dementia—Design Challenges and Future Directions, in: T. Jung, M. C. tom
Dieck (Eds.), Augmented Reality and Virtual Reality: Empowering Human, Place and
Business, Progress in IS, Springer International Publishing, Cham, 2018, pp. 295–305. URL:
https://doi.org/10.1007/978-3-319-64027-3_20. doi:10.1007/978-3-319-64027-3_20.
[6] D. Wolf, D. Besserer, K. Sejunaite, M. Riepe, E. Rukzio, cARe: An Augmented Reality
Support System for Dementia Patients, in: Adjunct Proceedings of the 31st Annual ACM
Symposium on User Interface Software and Technology, ACM, Berlin Germany, 2018,
pp. 42–44. URL: https://dl.acm.org/doi/10.1145/3266037.3266095. doi:10.1145/3266037.
3266095.
[7] M. A. Hamilton, A. P. Beug, H. J. Hamilton, W. J. Norton, Augmented Reality Technol-
ogy for People Living with Dementia and their Care Partners, in: 2021 the 5th Inter-
national Conference on Virtual and Augmented Reality Simulations, ACM, Melbourne
VIC Australia, 2021, pp. 21–30. URL: https://dl.acm.org/doi/10.1145/3463914.3463918.
doi:10.1145/3463914.3463918.
[8] N. Rohrbach, P. Gulde, A. R. Armstrong, L. Hartig, A. Abdelrazeq, S. Schröder, J. Neuse,
T. Grimmer, J. Diehl-Schmid, J. Hermsdörfer, An augmented reality approach for ADL
support in Alzheimer’s disease: a crossover trial, Journal of NeuroEngineering and
Rehabilitation 16 (2019) 66. URL: https://doi.org/10.1186/s12984-019-0530-z. doi:10.1186/
s12984-019-0530-z.
[9] A. Vovk, A. Patel, D. Chan, Augmented Reality for Early Alzheimer’s Disease Diagnosis, in:
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems,
CHI EA ’19, Association for Computing Machinery, New York, NY, USA, 2019, pp. 1–6.
URL: https://doi.org/10.1145/3290607.3313007. doi:10.1145/3290607.3313007.
[10] A. Niknam, An Augmented Reality Mobile Game Design to Enhance Spatial Memory in
Elderly with Dementia (2021).
[11] B. Aruanno, F. Garzotto, MemHolo: mixed reality experiences for subjects with Alzheimer’s
disease, Multimedia Tools and Applications 78 (2019) 13517–13537. URL: https://doi.org/
10.1007/s11042-018-7089-8. doi:10.1007/s11042-018-7089-8.
[12] C. Boletsis, S. McCallum, Augmented Reality Cubes for Cognitive Gaming: Prelimi-
nary Usability and Game Experience Testing, International Journal of Serious Games
3 (2016). URL: https://journal.seriousgamessociety.org/index.php/IJSG/article/view/106.
doi:10.17083/ijsg.v3i1.106, number: 1.
[13] M. Denscombe, The Good Research Guide: Research Methods for Small-Scale Social
Research Projects, McGraw-Hill Education (UK), 2021.
[14] M. Prince, A. Comas-Herrera, M. Knapp, M. Guerchet, M. Karagiannidou, World Alzheimer
report 2016: improving healthcare for people living with dementia: coverage, quality and
costs now and in the future (2016). Publisher: Alzheimer’s Disease International (ADI).
[15] S. Kumari, K. Bagri, R. Deshmukh, Dementia: A journey from cause to cure, in:
Nanomedicine-Based Approaches for the Treatment of Dementia, Elsevier, 2023, pp. 37–56.
[16] G. Wong, M. Knapp, Should we move dementia research funding from a cure to its care?,
2020. Issue: 4 Pages: 303–305 Publication Title: Expert review of neurotherapeutics Volume:
20.
[17] K. Chen, V. W. Q. Lou, S. S. C. Lo, Exploring the acceptance of tablets usage for cognitive
training among older people with cognitive impairments: A mixed-methods study, Applied
Ergonomics 93 (2021) 103381. Publisher: Elsevier.
[18] W. Moyle, C. Jones, T. Dwan, T. Petrovich, Effectiveness of a virtual reality forest on
people with dementia: A mixed methods pilot study, The Gerontologist 58 (2018) 478–487.
Publisher: Oxford University Press US.
[19] F. Meiland, A. Innes, G. Mountain, L. Robinson, H. van der Roest, J. A. García-Casal,
D. Gove, J. R. Thyrian, S. Evans, R.-M. Dröes, Technologies to support community-dwelling
persons with dementia: a position paper on issues regarding development, usability,
effectiveness and cost-effectiveness, deployment, and ethics, JMIR rehabilitation and
assistive technologies 4 (2017) e6376. Publisher: JMIR Publications Inc., Toronto, Canada.
[20] S. Enshaeifar, P. Barnaghi, S. Skillman, A. Markides, T. Elsaleh, S. T. Acton, R. Nilforooshan,
H. Rostill, The internet of things for dementia care, IEEE Internet Computing 22 (2018)
8–17. Publisher: IEEE.
[21] J. Hirt, T. Beer, Use and impact of virtual reality simulation in dementia care education: A
scoping review, Nurse education today 84 (2020) 104207. Publisher: Elsevier.
[22] J. Strong, Immersive virtual reality and persons with dementia: a literature review, Journal
of gerontological social work 63 (2020) 209–226. Publisher: Taylor & Francis.
[23] S. Davis, K. Nesbitt, E. Nalivaiko, A systematic review of cybersickness, in: Proceedings
of the 2014 conference on interactive entertainment, 2014, pp. 1–9.
[24] X. Shen, Y. S. Pai, D. Kiuchi, K. Oishi, K. Bao, T. Aoki, K. Minamizawa, Dementia Eyes:
Perceiving Dementia with Augmented Reality, in: SIGGRAPH Asia 2021 XR, 2021, pp. 1–2.
[25] M. N. Sakib, M. M. Rahman, H. Mahmud, M. K. Hasan, Augmented reality-based lifelogging
system for reminiscence, in: Proceedings of International Conference on Fourth Industrial
Revolution and Beyond 2021, Springer, 2022, pp. 493–504.
[26] T. Onorati, P. Díaz, A. Montero, I. Aedo, Designing AR Applications for People Living
with Dementia, in: J. Abdelnour Nocera, M. Kristín Lárusdóttir, H. Petrie, A. Piccinno,
M. Winckler (Eds.), Human-Computer Interaction – INTERACT 2023, Springer Nature
Switzerland, Cham, 2023, pp. 449–453. doi:10.1007/978-3-031-42293-5_52.