Commercial Internet of Things (IoT) devices allow for creating afordable and customized smart home systems for seniors with neurocognitive disorders (SwNCDs), such as dementia and Alzheimer's. An IoT-based system can support SwNCDs in completing daily living activities by sending reminders and notifications via smartphone applications. However, this approach comes with two main challenges that negatively afect the overall system usability. Firstly, customizing and interacting with IoT devices requires a certain level of technology literacy, which many SwNCDs and caregivers don't have. Secondly, relying solely on smartphone notifications is not suitable for homecare purposes. In response to these challenges, we present the design, development and architecture of our immersive IoT-based smart home system prototype. Our proposed system allows senior users to interact with the smart home system efortlessly and hands-free via a head-mounted mixed reality device. In addition, the current prototype supports two use cases: medication reminders and cooking safety.
The Internet of Things (IoT) provides a basis for customized and expandable supportive smart
home systems for senior homecare [
In a conventional home care setting, the term ’memory prompts’ describes the various
techniques that caregivers use to prompt the memory of seniors to complete daily tasks. Some
of the most common memory prompts are verbal reminders, alarms, sticky notes, calendar
notes, to-do lists, labels, and tags [
Mixed Reality (MR) is one type of immersive technology that blends the physical and digital
worlds to unlock natural and intuitive human-holographic interactions. In this model, designing
seamless UXs tailored for SwNCDs is more feasible [
Integrating IoT devices and MR unlocks new possibilities for homecare. For instance, an IoT
device could trigger an MR application to display an interactive hologram designed specifically
for SwNCDs homecare. Additionally, it is possible to combine holographic prompts with
wearable IoT devices to receive vibration signals or haptic feedback. This approach improves
user responsiveness by engaging more than one sense. Immersive smart homes, with careful
and empathetic UX and UI design, can improve overall system usability and reduce homecare
challenges [
This paper presents our immersive IoT-based smart home system to support SwNCDs. The system prototype is designed based on the User-Centered Design (UCD) method following ifve phases: investigate, ideate, prototype, evaluate and implement. We started the process by conducting a requirements elicitation study with a sample of seniors and caregivers to better understand user requirements and elicit use cases. Then, we selected two use cases: medication reminders and cooking safety. After that, we introduced two video prototypes demoing our initial system design. We used these video prototypes in an online Design Critique evaluation study with 24 North American participants during the Covid-19 pandemic. This evaluation included SwNCDs, formal and informal caregivers, domain experts and MR developers. All Design Critique sessions were video recorded, scripted and later imported into NVivo software for qualitative data analysis. We used the Thematic Analysis method to elicit feedback and design recommendations. Based on the new results, we reiterated our design and developed a high-fidelity system prototype described in the following sections. The current system utilizes a HoloLens2 application, a set of IoT devices, a pill dispenser, an Apple Watch and a Home Assistant local server.
In this section, we describe our high-fidelity system prototype; see Figure 1 for UI screenshots. We used the Unity game engine and the Microsoft Mixed Reality Toolkit (MRTK) to develop the MR application. The current version of the application supports two user modes: senior user and caregiver user, which we describe in the following subsections.
The purpose of the caregiver mode is to set up the system or customize preferences. When the application runs for the first time, caregivers will be asked to introduce real-world IoT devices to the application. Each IoT device is represented by one virtual object (game object). In order to assist the user, the system displays holographic animated hand gestures illustrating how to interact with these virtual objects (MRTK Hand Coach prefab). Afterward, caregivers are asked to grab a virtual object representing the pill dispenser and align it with the real-world pill dispenser. The same step is required to locate the stove. This approach allows us to identify the position of both devices in space and measure the distance between them and the HoloLens2 using Unity ’Vetcotr3’ struct.
Upon introducing the IoT devices to the application, caregivers can customize the system preferences by exploring various pre-set memory prompts that we designed specifically for SwNCDs homecare based on our Design Critique study. Using simple holographic ’toggle switches’ hovering near their left hand, caregivers can select appropriate memory prompts for their senior user. The current prototype supports the following memory prompts: prerecorded messages, spatial music cues, indoor navigation, holographic labels, visual attractions, pre-recorded video messages, and a flashing LED light strip– see Table 1 for more details. In addition, the caregiver can access the setup mode at any time using a keyword phrase or a hidden button to reconfigure the system. This step prevents senior users from modifying the system preferences accidentally.
The senior user mode is designed in a simple and minimal fashion to deliver memory prompts tailored to SwNCDs. Initially, the system sends a vibration signal to the Apple Watch to attract user attention. Then, a holographic message is displayed above the wrist when the user looks at the watch. The reason behind this design decision is to engage more than one sense, increasing user responsiveness. In addition, a short break between the vibration signal and the memory prompts gives the senior time to process information, as SwNCDs often sufer from neuro-delay. Failure to complete any task after receiving the initial memory prompt will trigger the system to proceed to the following memory prompt. The sequence of events after the initial prompt is entirely configured by the caregiver based on the senior’s needs.
When it is time to take a medication, an automation is triggered to send a vibration signal to the users’ Apple Watch to attract their attention. When the user looks at their watch, they see a two-word message (medication time) and a 3D model of two pills floating above the message. The text in this message is chosen specifically for SwNCDs where phrases such as ’remember to’ or ’don’t forget to’ are avoided. These phrases are known to cause frustration and anger among SwNCDs. The colours are selected carefully to stand out from the surrounding environment; however, they can be configured by the caregiver if needed. If the user ignores the first prompt, the second one will start after five minutes showing the same hologram and a pre-recorded audio message. It is important to use a familiar voice, such as a family member or a friend, to better prompt the user’s memory. Passed the second prompt, the system response will depend on the caregiver’s pre-set preferences. The entire sequence gets terminated when the user removes the pill dispenser and returns it to its place within five minutes. When the pill dispenser is removed from its place, the magnetic sensor status changes to ’open’ and the system will register the change and assume the user has taken their medication. After that, the system will wait for the user to return the dispenser to its place. Repetitive failure at taking medication or returning the dispenser will trigger the system to notify the caregiver.
The main goal of this system feature is to keep the user safe by preventing the stove from
running for a long time unattended. We used the previous prompting technique (vibration
signal first, then memory prompt) for this use case. Consistent UX design improves the overall
user responsiveness and could establish formulating a new user habit [
We used an open-source smart home operating system called Home Assistant (Hass.io) running on a local server using a RaspberryPI microprocessor. Hass.io comes with 1952 built-in ’Integrations’ to connect with almost any commercially available IoT device. A primary benefit of this system architecture is to have a single platform for managing and monitoring the entire smart home system. To address our two use cases, our system prototype utilized one Wyze magnetic sensor attached to the pill dispenser to monitor usage and an LED strip. In addition, we used a TP-Link smart electricity plug with current measuring capabilities to monitor and manage stove usage. Using Hass.io, we created automation recipes following Trigger-Action logic. In some cases, we wrote YAML scripts to create more complex automation related to monitoring IoT devices status, firing diferent system responses and sending messages to the caregiver in the case of incomplete tasks. The Hass.io comes with a customized smartphone and an Apple Watch application. Accounting for various caregivers’ technology literacy backgrounds, we customized Hass.io mobile app homepage to display a house map with icons representing the stove and the pill dispenser. Taping on these icons would display more options. The local server exchanges data with the HoloLens application via a single REST API.
In this paper, we presented our immersive smart home system prototype for SwNCDs homecare.
Thanks to the MR application running on a head-mount device, senior users can receive various
types of memory prompts in an immersive fashion seamlessly and efortlessly. Our system
design unlocks possibilities for using IoT devices to create supportive, smart home systems and
facilitate seniors’ homecare. Although the UX and UI of our system are designed based on our
previous Design Critique study, the final system prototype is yet to be tested in person with
potential end-users to investigate its usability further. With the ease of Covid-19 restrictions,
we hope to conduct in-person testing in the future and add more use cases. Finally, it is essential
to acknowledge that current MR headsets are unsuitable for day-long use. However, the near
future promises lighter and more user-friendly devices [
This paper is part of a PhD research that was fully funded by the Computational Media Design (CMD) departmental grant at the University of Calgary. In addition, partial funding was received from the Agile Surface Engineering (ASE) group.