An IoT-based Immersive Smart Home System for Seniors with Neurocognitive Disorders Lorans Alabood1,* , Frank Maurer2 1 The Computational Media Design program, University of Calgary. 528, ICT Building, 856 Campus Pl NW, Calgary, AB T2N 4V8, Canada 2 The Department of Computer Science, University of Calgary. 550, ICT Building, 856 Campus Pl NW, Calgary, AB T2N 4V8, Canada Abstract Commercial Internet of Things (IoT) devices allow for creating affordable and customized smart home systems for seniors with neurocognitive disorders (SwNCDs), such as dementia and Alzheimer’s. An IoT-based system can support SwNCDs in completing daily living activities by sending reminders and notifications via smartphone applications. However, this approach comes with two main challenges that negatively affect the overall system usability. Firstly, customizing and interacting with IoT devices requires a certain level of technology literacy, which many SwNCDs and caregivers don’t have. Secondly, relying solely on smartphone notifications is not suitable for homecare purposes. In response to these challenges, we present the design, development and architecture of our immersive IoT-based smart home system prototype. Our proposed system allows senior users to interact with the smart home system effortlessly and hands-free via a head-mounted mixed reality device. In addition, the current prototype supports two use cases: medication reminders and cooking safety. Keywords Mixed reality, IoT, Smart home systems, Immersive UX, Dementia homecare, 1. Introduction The Internet of Things (IoT) provides a basis for customized and expandable supportive smart home systems for senior homecare [1]. The design of a supportive system for Seniors with Neurocognitive Disorders (SwNCDs) should account for end-users’ special requirements, such as short-term memory loss, neuro-delay, learning difficulties, and high levels of frustration [2]. Using smartphone applications to interact with the smart home system can be challenging for seniors, and their caregivers due to technology literacy and learning difficulties [3]. Furthermore, carrying a smartphone at home all the time to receive notifications or interact with the home system is not practical for homecare purposes. For instance, missing a notification regarding taking medication could be crucial for the health and well-being of the individual. From a User Experience (UX) perspective, a single type of notification is insufficient because SwNCDs tend to become confused and miss phone notifications [4]. Ultimately, commercial IoT services don’t EMPATHY: 3rd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems. Workshop co-located with AVI 2022, June 06, 2022, Frascati, Rome, Italy. * Corresponding author. $ lorans.alabood@ucalgary.ca (L. Alabood); fmaurer@ucalgary.ca (F. Maurer)  0000-0003-2155-6449 (L. Alabood); 0000-0002-0240-715X (F. Maurer) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) account for SwNCDs special requirements as they are not intended for homecare purposes [5]. Therefore, a better user-system interaction method tailored for this user category is required. In a conventional home care setting, the term ’memory prompts’ describes the various techniques that caregivers use to prompt the memory of seniors to complete daily tasks. Some of the most common memory prompts are verbal reminders, alarms, sticky notes, calendar notes, to-do lists, labels, and tags [6]. Notably, these prompts use audio and visual cues and are integrated with the senior’s living space. For instance, caregivers utilize sticky notes and labels in the kitchen to remind seniors of different items. IoT-based systems can provide senior users with similar supportive environments if integrated with a better immersive user experience. Mixed Reality (MR) is one type of immersive technology that blends the physical and digital worlds to unlock natural and intuitive human-holographic interactions. In this model, designing seamless UXs tailored for SwNCDs is more feasible [7]. A major benefit of using a head-mounted MR device is that it allows users to interact with the home system instantly and effortlessly [8]. Additionally, it offers hands-free interactions without isolating the users from their environment [9]. The possibility to display holograms everywhere around the user enables free movement while ensuring users still receive prompts. Moreover, designing a User Interface (UI) for an MR application is not bound to traditional elements, such as buttons, windows or menus, allowing for more design liberty [10]. Integrating IoT devices and MR unlocks new possibilities for homecare. For instance, an IoT device could trigger an MR application to display an interactive hologram designed specifically for SwNCDs homecare. Additionally, it is possible to combine holographic prompts with wearable IoT devices to receive vibration signals or haptic feedback. This approach improves user responsiveness by engaging more than one sense. Immersive smart homes, with careful and empathetic UX and UI design, can improve overall system usability and reduce homecare challenges [11]. This paper presents our immersive IoT-based smart home system to support SwNCDs. The system prototype is designed based on the User-Centered Design (UCD) method following five phases: investigate, ideate, prototype, evaluate and implement. We started the process by conducting a requirements elicitation study with a sample of seniors and caregivers to better understand user requirements and elicit use cases. Then, we selected two use cases: medication reminders and cooking safety. After that, we introduced two video prototypes demoing our initial system design. We used these video prototypes in an online Design Critique evaluation study with 24 North American participants during the Covid-19 pandemic. This evaluation included SwNCDs, formal and informal caregivers, domain experts and MR developers. All Design Critique sessions were video recorded, scripted and later imported into NVivo software for qualitative data analysis. We used the Thematic Analysis method to elicit feedback and design recommendations. Based on the new results, we reiterated our design and developed a high-fidelity system prototype described in the following sections. The current system utilizes a HoloLens2 application, a set of IoT devices, a pill dispenser, an Apple Watch and a Home Assistant local server. Table 1 Detailed descriptions for all provided memory prompts Memory prompt Description Holographic message Simple hologram displaying short sentences backed with visual elements that could convey the message if the user didn’t read Visual attractions Using Unity particle system to create a floating ring around IoT pill dispenser to attract the user’s attention Indoor navigation A dynamic arrow to guide the user to smart home items (stove, pill dispenser) Holographic labels Using MRTK tool-tip to label IoT devices and physical objects. These labels are self-oriented towards the user Pre-recorded video message Video messages show the user how to complete a task such as using the pill dispenser. Videos are recorded by a familiar person Flashing LED LED strip attached to the bottom of the pill dispenser to attract the user’s attention Pre-recorded audio message Pre-recorded audio messages with a familiar voice to remind the user about certain tasks Spatial music cues Spatial music to guide the user towards a device 2. High Fidelity System Prototype In this section, we describe our high-fidelity system prototype; see Figure 1 for UI screenshots. We used the Unity game engine and the Microsoft Mixed Reality Toolkit (MRTK) to develop the MR application. The current version of the application supports two user modes: senior user and caregiver user, which we describe in the following subsections. 2.1. Caregiver Mode The purpose of the caregiver mode is to set up the system or customize preferences. When the application runs for the first time, caregivers will be asked to introduce real-world IoT devices to the application. Each IoT device is represented by one virtual object (game object). In order to assist the user, the system displays holographic animated hand gestures illustrating how to interact with these virtual objects (MRTK Hand Coach prefab). Afterward, caregivers are asked to grab a virtual object representing the pill dispenser and align it with the real-world pill dispenser. The same step is required to locate the stove. This approach allows us to identify the position of both devices in space and measure the distance between them and the HoloLens2 using Unity ’Vetcotr3’ struct. Upon introducing the IoT devices to the application, caregivers can customize the system preferences by exploring various pre-set memory prompts that we designed specifically for SwNCDs homecare based on our Design Critique study. Using simple holographic ’toggle switches’ hovering near their left hand, caregivers can select appropriate memory prompts for their senior user. The current prototype supports the following memory prompts: pre- recorded messages, spatial music cues, indoor navigation, holographic labels, visual attractions, pre-recorded video messages, and a flashing LED light strip– see Table 1 for more details. In addition, the caregiver can access the setup mode at any time using a keyword phrase or a hidden button to reconfigure the system. This step prevents senior users from modifying the system preferences accidentally. 2.2. Senior User Mode The senior user mode is designed in a simple and minimal fashion to deliver memory prompts tailored to SwNCDs. Initially, the system sends a vibration signal to the Apple Watch to attract user attention. Then, a holographic message is displayed above the wrist when the user looks at the watch. The reason behind this design decision is to engage more than one sense, increasing Figure 1: Screenshots presenting three different memory prompts as seen by senior user user responsiveness. In addition, a short break between the vibration signal and the memory prompts gives the senior time to process information, as SwNCDs often suffer from neuro-delay. Failure to complete any task after receiving the initial memory prompt will trigger the system to proceed to the following memory prompt. The sequence of events after the initial prompt is entirely configured by the caregiver based on the senior’s needs. 3. Use Case Scenarios 3.1. Medication Reminder When it is time to take a medication, an automation is triggered to send a vibration signal to the users’ Apple Watch to attract their attention. When the user looks at their watch, they see a two-word message (medication time) and a 3D model of two pills floating above the message. The text in this message is chosen specifically for SwNCDs where phrases such as ’remember to’ or ’don’t forget to’ are avoided. These phrases are known to cause frustration and anger among SwNCDs. The colours are selected carefully to stand out from the surrounding environment; however, they can be configured by the caregiver if needed. If the user ignores the first prompt, the second one will start after five minutes showing the same hologram and a pre-recorded audio message. It is important to use a familiar voice, such as a family member or a friend, to better prompt the user’s memory. Passed the second prompt, the system response will depend on the caregiver’s pre-set preferences. The entire sequence gets terminated when the user removes the pill dispenser and returns it to its place within five minutes. When the pill dispenser is removed from its place, the magnetic sensor status changes to ’open’ and the system will register the change and assume the user has taken their medication. After that, the system will wait for the user to return the dispenser to its place. Repetitive failure at taking medication or returning the dispenser will trigger the system to notify the caregiver. 3.2. Cooking Safety The main goal of this system feature is to keep the user safe by preventing the stove from running for a long time unattended. We used the previous prompting technique (vibration signal first, then memory prompt) for this use case. Consistent UX design improves the overall user responsiveness and could establish formulating a new user habit [12]. In addition to the memory prompts, we added a safety feature where a maximum cooking time could be pre-set. Another safety measurement is incorporated where if the user is away from the kitchen area for five minutes, the system will prompt the user to return to the stove. If the prompts are ignored, the system will notify the caregiver and turn off the IoT electricity plug to shut off the stove. 4. System Architecture We used an open-source smart home operating system called Home Assistant (Hass.io) running on a local server using a RaspberryPI microprocessor. Hass.io comes with 1952 built-in ’Integra- tions’ to connect with almost any commercially available IoT device. A primary benefit of this system architecture is to have a single platform for managing and monitoring the entire smart home system. To address our two use cases, our system prototype utilized one Wyze magnetic sensor attached to the pill dispenser to monitor usage and an LED strip. In addition, we used a TP-Link smart electricity plug with current measuring capabilities to monitor and manage stove usage. Using Hass.io, we created automation recipes following Trigger-Action logic. In some cases, we wrote YAML scripts to create more complex automation related to monitoring IoT devices status, firing different system responses and sending messages to the caregiver in the case of incomplete tasks. The Hass.io comes with a customized smartphone and an Apple Watch application. Accounting for various caregivers’ technology literacy backgrounds, we customized Hass.io mobile app homepage to display a house map with icons representing the stove and the pill dispenser. Taping on these icons would display more options. The local server exchanges data with the HoloLens application via a single REST API. 5. Discussion and Conclusion In this paper, we presented our immersive smart home system prototype for SwNCDs homecare. Thanks to the MR application running on a head-mount device, senior users can receive various types of memory prompts in an immersive fashion seamlessly and effortlessly. Our system design unlocks possibilities for using IoT devices to create supportive, smart home systems and facilitate seniors’ homecare. Although the UX and UI of our system are designed based on our previous Design Critique study, the final system prototype is yet to be tested in person with potential end-users to investigate its usability further. With the ease of Covid-19 restrictions, we hope to conduct in-person testing in the future and add more use cases. Finally, it is essential to acknowledge that current MR headsets are unsuitable for day-long use. However, the near future promises lighter and more user-friendly devices [13]. In the meantime, we continue to use the HoloLens for prototyping purposes. 6. Acknowledgement This paper is part of a PhD research that was fully funded by the Computational Media Design (CMD) departmental grant at the University of Calgary. In addition, partial funding was received from the Agile Surface Engineering (ASE) group. References [1] H. K. Ngankam, H. Pigot, M. Parenteau, M. Lussier, A. Aboujaoudé, C. Laliberté, M. Cou- ture, N. Bier, S. Giroux, An iot architecture of microservices for ambient assisted living environments to promote aging in smart cities, in: International Conference on Smart Homes and Health Telematics, Springer, Cham, 2019, pp. 154–167. [2] M. Amiribesheli, H. Bouchachia, A tailored smart home for dementia care, Journal of Ambient Intelligence and Humanized Computing 9 (2018) 1755–1782. [3] M. Mohammed, S. Desyansah, S. Al-Zubaidi, E. Yusuf, An internet of things-based smart homes and healthcare monitoring and management system, in: Journal of Physics: Con- ference Series, volume 1450, 2020, p. 012079. [4] H. M. Mohadisdudis, N. M. Ali, A study of smartphone usage and barriers among the elderly, in: 2014 3rd International Conference on User Science and Engineering (i-USEr), IEEE, 2014, pp. 109–114. [5] N. Dimitrioglou, D. Kardaras, S. Barbounaki, Multicriteria evaluation of the internet of things potential in health care: The case of dementia care, in: 2017 IEEE 19th Conference on Business Informatics (CBI), volume 1, IEEE, 2017, pp. 454–462. [6] D. Bruno, The Preservation of Memory, Psychology Press, 2015. [7] A. C. M. Bauer, G. Andringa, The potential of immersive virtual reality for cognitive training in elderly, Gerontology 66 (2020) 614–623. [8] I. Buchem, S. Vorwerg, O. Stamm, K. Hildebrand, Y. Bialek, Gamification in mixed-reality exergames for older adult patients in a mobile immersive diagnostic center: A pilot study in the beware project, in: 2021 7th International Conference of the Immersive Learning Research Network (iLRN), IEEE, 2021, pp. 1–8. [9] M. Rossi, G. D’Avenio, S. Morelli, M. Grigioni, Cogar: an augmented reality app to improve quality of life of the people with cognitive impairment, in: 2020 IEEE 20th Mediterranean Electrotechnical Conference (MELECON), IEEE, 2020, pp. 339–343. [10] R. I. Garcia-Betances, V. Jiménez-Mixco, M. T. Arredondo, M. F. Cabrera-Umpiérrez, Using virtual reality for cognitive training of the elderly, American Journal of Alzheimer’s Disease & Other Dementias® 30 (2015) 49–54. [11] F. Garzotto, E. Torelli, F. Vona, B. Aruanno, Hololearn: Learning through mixed reality for people with cognitive disability, in: 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), IEEE, 2018, pp. 189–190. [12] K. Langvardt, Regulating habit-forming technology, Fordham L. Rev. 88 (2019) 129. [13] M. C. Howard, M. M. Davis, A meta-analysis and systematic literature review of mixed reality rehabilitation programs: Investigating design characteristics of augmented reality and augmented virtuality, Computers in Human Behavior (2022) 107197.