=Paper=
{{Paper
|id=Vol-3172/short3
|storemode=property
|title=An IoT-based Immersive Smart Home System for Seniors with Neurocognitive Disorders
|pdfUrl=https://ceur-ws.org/Vol-3172/short3.pdf
|volume=Vol-3172
|authors=Lorans Alabood,Frank Maurer
|dblpUrl=https://dblp.org/rec/conf/avi/AlaboodM22
}}
==An IoT-based Immersive Smart Home System for Seniors with Neurocognitive Disorders==
https://ceur-ws.org/Vol-3172/short3.pdf
An IoT-based Immersive Smart Home System for
Seniors with Neurocognitive Disorders
Lorans Alabood1,* , Frank Maurer2
1
The Computational Media Design program, University of Calgary. 528, ICT Building, 856 Campus Pl NW, Calgary, AB
T2N 4V8, Canada
2
The Department of Computer Science, University of Calgary. 550, ICT Building, 856 Campus Pl NW, Calgary, AB T2N
4V8, Canada
Abstract
Commercial Internet of Things (IoT) devices allow for creating affordable and customized smart home
systems for seniors with neurocognitive disorders (SwNCDs), such as dementia and Alzheimer’s. An
IoT-based system can support SwNCDs in completing daily living activities by sending reminders and
notifications via smartphone applications. However, this approach comes with two main challenges
that negatively affect the overall system usability. Firstly, customizing and interacting with IoT devices
requires a certain level of technology literacy, which many SwNCDs and caregivers don’t have. Secondly,
relying solely on smartphone notifications is not suitable for homecare purposes. In response to these
challenges, we present the design, development and architecture of our immersive IoT-based smart home
system prototype. Our proposed system allows senior users to interact with the smart home system
effortlessly and hands-free via a head-mounted mixed reality device. In addition, the current prototype
supports two use cases: medication reminders and cooking safety.
Keywords
Mixed reality, IoT, Smart home systems, Immersive UX, Dementia homecare,
1. Introduction
The Internet of Things (IoT) provides a basis for customized and expandable supportive smart
home systems for senior homecare [1]. The design of a supportive system for Seniors with
Neurocognitive Disorders (SwNCDs) should account for end-users’ special requirements, such
as short-term memory loss, neuro-delay, learning difficulties, and high levels of frustration [2].
Using smartphone applications to interact with the smart home system can be challenging for
seniors, and their caregivers due to technology literacy and learning difficulties [3]. Furthermore,
carrying a smartphone at home all the time to receive notifications or interact with the home
system is not practical for homecare purposes. For instance, missing a notification regarding
taking medication could be crucial for the health and well-being of the individual. From a User
Experience (UX) perspective, a single type of notification is insufficient because SwNCDs tend
to become confused and miss phone notifications [4]. Ultimately, commercial IoT services don’t
EMPATHY: 3rd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems. Workshop
co-located with AVI 2022, June 06, 2022, Frascati, Rome, Italy.
*
Corresponding author.
$ lorans.alabood@ucalgary.ca (L. Alabood); fmaurer@ucalgary.ca (F. Maurer)
0000-0003-2155-6449 (L. Alabood); 0000-0002-0240-715X (F. Maurer)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
account for SwNCDs special requirements as they are not intended for homecare purposes [5].
Therefore, a better user-system interaction method tailored for this user category is required.
In a conventional home care setting, the term ’memory prompts’ describes the various
techniques that caregivers use to prompt the memory of seniors to complete daily tasks. Some
of the most common memory prompts are verbal reminders, alarms, sticky notes, calendar
notes, to-do lists, labels, and tags [6]. Notably, these prompts use audio and visual cues and are
integrated with the senior’s living space. For instance, caregivers utilize sticky notes and labels
in the kitchen to remind seniors of different items. IoT-based systems can provide senior users
with similar supportive environments if integrated with a better immersive user experience.
Mixed Reality (MR) is one type of immersive technology that blends the physical and digital
worlds to unlock natural and intuitive human-holographic interactions. In this model, designing
seamless UXs tailored for SwNCDs is more feasible [7]. A major benefit of using a head-mounted
MR device is that it allows users to interact with the home system instantly and effortlessly [8].
Additionally, it offers hands-free interactions without isolating the users from their environment
[9]. The possibility to display holograms everywhere around the user enables free movement
while ensuring users still receive prompts. Moreover, designing a User Interface (UI) for an MR
application is not bound to traditional elements, such as buttons, windows or menus, allowing
for more design liberty [10].
Integrating IoT devices and MR unlocks new possibilities for homecare. For instance, an IoT
device could trigger an MR application to display an interactive hologram designed specifically
for SwNCDs homecare. Additionally, it is possible to combine holographic prompts with
wearable IoT devices to receive vibration signals or haptic feedback. This approach improves
user responsiveness by engaging more than one sense. Immersive smart homes, with careful
and empathetic UX and UI design, can improve overall system usability and reduce homecare
challenges [11].
This paper presents our immersive IoT-based smart home system to support SwNCDs. The
system prototype is designed based on the User-Centered Design (UCD) method following
five phases: investigate, ideate, prototype, evaluate and implement. We started the process by
conducting a requirements elicitation study with a sample of seniors and caregivers to better
understand user requirements and elicit use cases. Then, we selected two use cases: medication
reminders and cooking safety. After that, we introduced two video prototypes demoing our
initial system design. We used these video prototypes in an online Design Critique evaluation
study with 24 North American participants during the Covid-19 pandemic. This evaluation
included SwNCDs, formal and informal caregivers, domain experts and MR developers. All
Design Critique sessions were video recorded, scripted and later imported into NVivo software
for qualitative data analysis. We used the Thematic Analysis method to elicit feedback and
design recommendations. Based on the new results, we reiterated our design and developed a
high-fidelity system prototype described in the following sections. The current system utilizes
a HoloLens2 application, a set of IoT devices, a pill dispenser, an Apple Watch and a Home
Assistant local server.
Table 1
Detailed descriptions for all provided memory prompts
Memory prompt Description
Holographic message Simple hologram displaying short sentences backed with visual elements that could convey the message if the user didn’t read
Visual attractions Using Unity particle system to create a floating ring around IoT pill dispenser to attract the user’s attention
Indoor navigation A dynamic arrow to guide the user to smart home items (stove, pill dispenser)
Holographic labels Using MRTK tool-tip to label IoT devices and physical objects. These labels are self-oriented towards the user
Pre-recorded video message Video messages show the user how to complete a task such as using the pill dispenser. Videos are recorded by a familiar person
Flashing LED LED strip attached to the bottom of the pill dispenser to attract the user’s attention
Pre-recorded audio message Pre-recorded audio messages with a familiar voice to remind the user about certain tasks
Spatial music cues Spatial music to guide the user towards a device
2. High Fidelity System Prototype
In this section, we describe our high-fidelity system prototype; see Figure 1 for UI screenshots.
We used the Unity game engine and the Microsoft Mixed Reality Toolkit (MRTK) to develop the
MR application. The current version of the application supports two user modes: senior user
and caregiver user, which we describe in the following subsections.
2.1. Caregiver Mode
The purpose of the caregiver mode is to set up the system or customize preferences. When the
application runs for the first time, caregivers will be asked to introduce real-world IoT devices
to the application. Each IoT device is represented by one virtual object (game object). In order
to assist the user, the system displays holographic animated hand gestures illustrating how
to interact with these virtual objects (MRTK Hand Coach prefab). Afterward, caregivers are
asked to grab a virtual object representing the pill dispenser and align it with the real-world pill
dispenser. The same step is required to locate the stove. This approach allows us to identify the
position of both devices in space and measure the distance between them and the HoloLens2
using Unity ’Vetcotr3’ struct.
Upon introducing the IoT devices to the application, caregivers can customize the system
preferences by exploring various pre-set memory prompts that we designed specifically for
SwNCDs homecare based on our Design Critique study. Using simple holographic ’toggle
switches’ hovering near their left hand, caregivers can select appropriate memory prompts
for their senior user. The current prototype supports the following memory prompts: pre-
recorded messages, spatial music cues, indoor navigation, holographic labels, visual attractions,
pre-recorded video messages, and a flashing LED light strip– see Table 1 for more details. In
addition, the caregiver can access the setup mode at any time using a keyword phrase or a
hidden button to reconfigure the system. This step prevents senior users from modifying the
system preferences accidentally.
2.2. Senior User Mode
The senior user mode is designed in a simple and minimal fashion to deliver memory prompts
tailored to SwNCDs. Initially, the system sends a vibration signal to the Apple Watch to attract
user attention. Then, a holographic message is displayed above the wrist when the user looks at
the watch. The reason behind this design decision is to engage more than one sense, increasing
Figure 1: Screenshots presenting three different memory prompts as seen by senior user
user responsiveness. In addition, a short break between the vibration signal and the memory
prompts gives the senior time to process information, as SwNCDs often suffer from neuro-delay.
Failure to complete any task after receiving the initial memory prompt will trigger the system
to proceed to the following memory prompt. The sequence of events after the initial prompt is
entirely configured by the caregiver based on the senior’s needs.
3. Use Case Scenarios
3.1. Medication Reminder
When it is time to take a medication, an automation is triggered to send a vibration signal to
the users’ Apple Watch to attract their attention. When the user looks at their watch, they
see a two-word message (medication time) and a 3D model of two pills floating above the
message. The text in this message is chosen specifically for SwNCDs where phrases such as
’remember to’ or ’don’t forget to’ are avoided. These phrases are known to cause frustration and
anger among SwNCDs. The colours are selected carefully to stand out from the surrounding
environment; however, they can be configured by the caregiver if needed. If the user ignores
the first prompt, the second one will start after five minutes showing the same hologram and a
pre-recorded audio message. It is important to use a familiar voice, such as a family member or
a friend, to better prompt the user’s memory. Passed the second prompt, the system response
will depend on the caregiver’s pre-set preferences. The entire sequence gets terminated when
the user removes the pill dispenser and returns it to its place within five minutes. When the
pill dispenser is removed from its place, the magnetic sensor status changes to ’open’ and the
system will register the change and assume the user has taken their medication. After that, the
system will wait for the user to return the dispenser to its place. Repetitive failure at taking
medication or returning the dispenser will trigger the system to notify the caregiver.
3.2. Cooking Safety
The main goal of this system feature is to keep the user safe by preventing the stove from
running for a long time unattended. We used the previous prompting technique (vibration
signal first, then memory prompt) for this use case. Consistent UX design improves the overall
user responsiveness and could establish formulating a new user habit [12]. In addition to the
memory prompts, we added a safety feature where a maximum cooking time could be pre-set.
Another safety measurement is incorporated where if the user is away from the kitchen area for
five minutes, the system will prompt the user to return to the stove. If the prompts are ignored,
the system will notify the caregiver and turn off the IoT electricity plug to shut off the stove.
4. System Architecture
We used an open-source smart home operating system called Home Assistant (Hass.io) running
on a local server using a RaspberryPI microprocessor. Hass.io comes with 1952 built-in ’Integra-
tions’ to connect with almost any commercially available IoT device. A primary benefit of this
system architecture is to have a single platform for managing and monitoring the entire smart
home system. To address our two use cases, our system prototype utilized one Wyze magnetic
sensor attached to the pill dispenser to monitor usage and an LED strip. In addition, we used
a TP-Link smart electricity plug with current measuring capabilities to monitor and manage
stove usage. Using Hass.io, we created automation recipes following Trigger-Action logic. In
some cases, we wrote YAML scripts to create more complex automation related to monitoring
IoT devices status, firing different system responses and sending messages to the caregiver in
the case of incomplete tasks. The Hass.io comes with a customized smartphone and an Apple
Watch application. Accounting for various caregivers’ technology literacy backgrounds, we
customized Hass.io mobile app homepage to display a house map with icons representing the
stove and the pill dispenser. Taping on these icons would display more options. The local server
exchanges data with the HoloLens application via a single REST API.
5. Discussion and Conclusion
In this paper, we presented our immersive smart home system prototype for SwNCDs homecare.
Thanks to the MR application running on a head-mount device, senior users can receive various
types of memory prompts in an immersive fashion seamlessly and effortlessly. Our system
design unlocks possibilities for using IoT devices to create supportive, smart home systems and
facilitate seniors’ homecare. Although the UX and UI of our system are designed based on our
previous Design Critique study, the final system prototype is yet to be tested in person with
potential end-users to investigate its usability further. With the ease of Covid-19 restrictions,
we hope to conduct in-person testing in the future and add more use cases. Finally, it is essential
to acknowledge that current MR headsets are unsuitable for day-long use. However, the near
future promises lighter and more user-friendly devices [13]. In the meantime, we continue to
use the HoloLens for prototyping purposes.
6. Acknowledgement
This paper is part of a PhD research that was fully funded by the Computational Media Design
(CMD) departmental grant at the University of Calgary. In addition, partial funding was received
from the Agile Surface Engineering (ASE) group.
References
[1] H. K. Ngankam, H. Pigot, M. Parenteau, M. Lussier, A. Aboujaoudé, C. Laliberté, M. Cou-
ture, N. Bier, S. Giroux, An iot architecture of microservices for ambient assisted living
environments to promote aging in smart cities, in: International Conference on Smart
Homes and Health Telematics, Springer, Cham, 2019, pp. 154–167.
[2] M. Amiribesheli, H. Bouchachia, A tailored smart home for dementia care, Journal of
Ambient Intelligence and Humanized Computing 9 (2018) 1755–1782.
[3] M. Mohammed, S. Desyansah, S. Al-Zubaidi, E. Yusuf, An internet of things-based smart
homes and healthcare monitoring and management system, in: Journal of Physics: Con-
ference Series, volume 1450, 2020, p. 012079.
[4] H. M. Mohadisdudis, N. M. Ali, A study of smartphone usage and barriers among the
elderly, in: 2014 3rd International Conference on User Science and Engineering (i-USEr),
IEEE, 2014, pp. 109–114.
[5] N. Dimitrioglou, D. Kardaras, S. Barbounaki, Multicriteria evaluation of the internet of
things potential in health care: The case of dementia care, in: 2017 IEEE 19th Conference
on Business Informatics (CBI), volume 1, IEEE, 2017, pp. 454–462.
[6] D. Bruno, The Preservation of Memory, Psychology Press, 2015.
[7] A. C. M. Bauer, G. Andringa, The potential of immersive virtual reality for cognitive
training in elderly, Gerontology 66 (2020) 614–623.
[8] I. Buchem, S. Vorwerg, O. Stamm, K. Hildebrand, Y. Bialek, Gamification in mixed-reality
exergames for older adult patients in a mobile immersive diagnostic center: A pilot study
in the beware project, in: 2021 7th International Conference of the Immersive Learning
Research Network (iLRN), IEEE, 2021, pp. 1–8.
[9] M. Rossi, G. D’Avenio, S. Morelli, M. Grigioni, Cogar: an augmented reality app to improve
quality of life of the people with cognitive impairment, in: 2020 IEEE 20th Mediterranean
Electrotechnical Conference (MELECON), IEEE, 2020, pp. 339–343.
[10] R. I. Garcia-Betances, V. Jiménez-Mixco, M. T. Arredondo, M. F. Cabrera-Umpiérrez, Using
virtual reality for cognitive training of the elderly, American Journal of Alzheimer’s
Disease & Other Dementias® 30 (2015) 49–54.
[11] F. Garzotto, E. Torelli, F. Vona, B. Aruanno, Hololearn: Learning through mixed reality
for people with cognitive disability, in: 2018 IEEE International Conference on Artificial
Intelligence and Virtual Reality (AIVR), IEEE, 2018, pp. 189–190.
[12] K. Langvardt, Regulating habit-forming technology, Fordham L. Rev. 88 (2019) 129.
[13] M. C. Howard, M. M. Davis, A meta-analysis and systematic literature review of mixed
reality rehabilitation programs: Investigating design characteristics of augmented reality
and augmented virtuality, Computers in Human Behavior (2022) 107197.