Digital Ecosystem Model for the Production of Mixed Reality Environments to Assist Senile Dementia Patients Erwin B. Montes-Chaparro1, Jaime Muñoz-Arteaga1, Angel E. Muñoz-Zavala1 and Miguel A. Ortiz-Esparza2* 1 Universidad Autónoma de Aguascalientes, Av. Universidad # 940, Ciudad Universitaria, C.P. 20100, Aguascalientes, Ags. México. 2 Center for Research in Mathematics, Quantum Knowledge City, 98160, Zacatecas, Mexico Abstract Senile dementia is one of the most common sufferings in the elderly population as brain functions start to deteriorate, making their day-to-day routine increasingly dependent on care staff. We need to build rapport between the Elderly and technology tools to allow aged people have an easier way to complete daily tasks without losing their independence and allowing them to work out to maintain their health and mobility with as less intervention with rehab staff as possible via remote monitoring using Internet of Things to detect anomalies in the elderly daily routine to prevent other age-related diseases in an early stage. This article proposes an elderly assistance ecosystem with focus on senile dementia using IoT and applying Software Engineering for the architecture design side allowing to make a more detailed and suited implementation. Keywords Mixed Reality, Internet of Things, Senile Dementia, UX, Digital Ecosystem 1 1. Introduction Despite the Technological advances being adopted by more people every time [1], often, the Elderly population, (Baby boomers and older generations) never had the opportunity of living the Technological evolution in an active way. Most modern technology corporations do not design products for the elder users daily living, not to mention ease of use of Graphic User Interfaces (GUI) for them, making it harder to adopt modern gadgets and devices that could make their life easier [2]. Elderly people need to be trained when they are not familiar with modern tech devices, and the way to let elder people learn by themselves is designing interfaces as simple as possible but with the generational breach in mind so they can use new technology with a minimal assistance, as they are not known for being technologically enthusiastic [1,3]. Elder people could develop a low self-esteem after retirement because of the contrast with their new daily routine, diminishing their physical and social activity to a large degree. All persons in retirement should maintain their dignity and the right to be useful; the users should be able to discover his own limitations and capacities. The Quality of life concept is nearly linked to social, mental and physical wellbeing [4]. Senile dementia is known for being in the top 3 causes of death in the elderly, just after cancer and hearth diseases [5], but it is feasible to lower the repercussions in mental health using the benefits of aerobic exercises [6]; this, in combination with virtual environments specially designed for the elderly physical activity, has demonstrated an increasing interest in workout CISETC 2023: International Congress on Education and Technology in Sciences 2023, December 04–06, 2023, Zacatecas, Mexico al14965@edu.uaa.mx (E. B. Montes-Chaparro); jaime.munoz@edu.uaa.mx (J. Muñoz-Arteaga); eduardo.munoz@edu.uaa.mx (A. Muñoz-Zavala); miguel.ortiz@cimat.mx (M. A. Ortiz-Esparza) XXXX-XXXX-XXXX-XXXX (E. B. Montes-Chaparro); 0000-0002-3635-7592 (J. Muñoz-Arteaga); 0000-0002-7484- 2097 (A. Muñoz-Zavala); 0000-0001-8762-5780 (M. Ortiz-Esparza) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ht tp: // ceur -ws .or g Works hop I SSN1613- 0073 Pr oceedi ngs CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings routines because of their null or low social interaction with other persons, increasing the cognitive stimulation [3]. As technology has a big presence in our daily routine, and it has been an important part of our entire lives, in a direct or indirect way. The younger generations are able to adapt quickly to technology created to be friendly and intuitive for almost any user. User Experience (UX) is an almost new knowledge area in the modern devices and applications development; there are notable changes among device generations and their interactions with users. There is an evident good UX design in the field of Smart Assistants (Google Assistant, Amazon Alexa, etc.) allowing recognition of multiple accents in a language, the fulfillment for users is prominent [7]. This work has been formed in seven sections: Second section, named Background, contains basic definitions of the technologies related, Third section contains some work with VR devices for treating senile dementia patients, Fourth section contains the problem outline, with other works using the integrated technologies of this approach, Fifth section describes a Digital Ecosystem Model for the development of Mixed Reality Environments, showing a general architecture for possible scenarios and organizing the model by interaction layers in order to assist the patients with senile dementia; A case study is presented in section six, applying several technologies in a real scenario. Finally, the conclusions are described in section seven. 2. Background VR, also known as virtual environments, consists in an ambience surrounding a person with sound and images that can that transmit the feel of being present physically in that virtual environment [8]. Thanks to the versatility of current Game engines, Virtual Environments development can be very straightforward compared to some Game engine generations ago, making them reliable tools to enhance UX in combination with the daily routine [3,8,9]. The use of Virtual Environments in Social Interaction has a promising future supporting education, decision making simulation, training, health and clinical practices [10], allowing innovative techniques to communicate ideas. Augmented Reality (AR) refers to a set of technologies that permit the users seeing most of the real environment surrounding them through a device capable of adding extra graphical information over real objects, adding virtual layers to the real environment view. AR can make some activities easier for people applied to the daily life, by highlighting objects with colors or showing notifications. Mixed Reality (MR), as shown in Figure 1, consists in a combination of real objects and the interaction with virtual ones, giving the ability of using corporeal objects with 3D objects rendered on a MR headset [11]. The use of physical peripherals can be more adequate for elder users, making them feel comfortable with the feedback found in the haptic vibration. The current electronic miniaturization techniques allow the production of reliable and powerful mobile microcomputers as the Raspberry Pi, being the perfect choice for reliable devices to be the brain unit for custom and open-source home automation systems [12] to control different electronic devices as sensors, electric motors and actuators with Arduino as low-level hardware interface. Figure 1: Example figure caption The concept of Digital Ecosystem can be defined as loose networks of interacting organization that are digitally connected and enabled by modularity, and that affect and are affected by each other’s offering [14], in this way, a digital ecosystem can be integrated by the interaction among Internet of Things, Virtual Environments, AR/VR headsets, Wearable devices, using Smartphones, Tablets and Smart Assistants as intermediaries with the elder patient. 3. Related Work The use of AR/VR applied to Senile Dementia is an almost new area in the treatment of Mental Diseases. Due the cost reduction in Virtual reality headsets and the increase in power for mobile devices, VR applications have become affordable for more people like ever before. The creation of virtual environments experience have had an exponential growth combining modern engines engines like Unity [3,5,9] (free of charge for non-commercial applications) and Mobile phones with AR/VR accessories as cardboard (Figure 2), letting users not aware of virtual environments get a taste of Mixed reality technology and developers to create experimental AR/VR oriented apps relatively free of charge. Since 2012 a growth in the VR apps for dementia had been noticed [15] without the necessity of taking the patients out of the nursing home allows to explore a great variety of scenarios and study the reaction of the elders to virtual nature environments as a forest [8]. A proposal of an adaptive VR environment can be found, using Real-Time feedback from the user to change what the user can see and interact with and monitor brain activity [9]. Figure 2: Cardboard design by Google, originally created to encourage the development of Virtual Reality Applications (Photo from Aliexpress). The combination of Virtual environments and work out allow the elderly to avoid routine and “visit” (in a virtual way) other places outside the asylum they live in [3], giving positive feedback by users, expressing they want to work out for longer times; this work focuses on a virtual environment using Unity and part of hardware design adapting an arm and leg exerciser machine. Diseases as Alzheimer’s have promising results when combining Virtual Reality [16] showing the benefits of a tight integration with MR and theoretical models [17] using reminiscence therapy. Project CogARC [18] integrates real objects with AR add-ons and a visual detection software installed on a tablet for the elder patient to play small games; even when they obtained positive results, in the part of negative feedback shows that the interactive aspects for some games have some flaws. HalleyAssist [19] uses a complex ecosystem trying to prevent the majority of elder patient’s health related ailments combining sensor sets to achieve a better detection of anomalies in movement according to patterns stored in the knowledge base; on the other side, this work leaves the software and user interface design to make easier for the elder to adopt this kind of devices and applications for their day-to-day living [2,3,5]. Microsoft HoloLens, as a MR headset [4] has demonstrated being a great optimal device with memory loss patients including a wide range of media content available. Panoramic video experiences in 360° show better attention of the patients, but interface design is needed especially for elderly users, not to mention that bad device calibration can cause dizziness (digital point-of-view strain and motion sickness). DCPAR project [2] uses a combination of a pico-projector hanging around the neck and a tablet app for the user. The users had problems with the weight and bulkiness of the pico-projector around the neck despite having a good tablet app design and the good response of the elder users. Eldergames Project has a MR environment with Tabletop games in it, this to train cognitive functions [11]; this work deepens in the emotional aspects of the player interactions with each other, having positive results and proving that a good design in UX for elder patients is feasible. 4. Problem Outline Several areas of opportunity can be found when talking about MR applied to Elderly patients: • Mobility issues: one of the most frequent ailing in the elder population caused by muscular, cardiovascular, metabolism and brain deterioration related to age [6]; some brain diseases symptoms are known for its delay with age with the integration of exercise routines. This work is in search of elders doing constant physical activity to maintain brain health, with the use of virtual environments and lightweight devices to promote user engagement. • Device ergonomics: Systems Design for occupational therapy [20] has shown good results in motor rehabilitation using virtual environments and peripherals. Here is where it lays the balance among ergonomics, intuitive interfaces, and pleasant environments, to increase the elder user engagement, making therapy sessions enjoyable. • Software Engineering approach: Many of the research articles related to AR for the Elderly have a practical approach, device functionality and experiment design depending on the case scenario [5,11,21], with the software design part put aside. 5. Digital Ecosystem Model The reason to define a digital ecosystem arquitecture is because of the presence of different technologies interacting with each other using multiple communication protocols that could produce confusion among all the kinds of devices connected [14]. The use of multiple distinct environments will be required for the different patient scenarios, in this work the proposal consist in a digital ecosystem architecture combining AR, VR, MR, and tools we can find in software design like the Unified Modeling Language (UML) [22] and System Modeling Language (SysML) [23], using the Human-Computer Interface [24] to adapt User Interfaces (UI) and User Experience (UX). We are looking for a way to easily adapt to the most possible kinds of mental diseases starting from a generic architecture and follow the evolution of the participants sufferings. Figure 3: Digital Ecosystem Model for the Production of Mixed Reality Environments to Assist Senile Dementia Patient 5.1. Direct Interaction We define Direct Interaction as all the devices the patient can interact with in a physical way (tablets, smartphones, AR/VR headsets, body-attached wearable devices), including biometric sensors to monitor the heart rate, step counter with the help of a smartphone for fall detection [25]. Some kinds of dementia can cause the patient to forget that is wearing a VR headset or any other kind of wearable device, so the need of a non-intrusive gadget is a must to make the experience comfortable for the elderly. Tablets and smartphones allow the elder users them to be aware of their own medical situation and a medium to receive calls and notifications from relatives and care staff to maintain a constant interpersonal contact, meanwhile the sensor set found in smartphones can register accidental falls, movement patterns and daily activities for pattern analysis. AR/VR Headsets running routines in virtual environments will permit the patient to have multiple therapies for physical workout, alternating the virtual scenario to make the experience less monotonous. 5.2. Indirect Interaction The area of Indirect interaction will be integrated by devices the patient can interact with and connected to a server in the local network to process data locally. Usually, wearables have an incredibly low processing, battery, and storage capacity, so a more powerful device is needed to collect all the local information. Microcomputers like Raspberry Pi [12] have shown their powerful processors that can be used for data readings from wearables and other monitoring local network devices as: cameras for pattern recognition and live remote monitoring; smart assistants as Google Home or Alexa [7] for direct user feedback and requests using voice commands; room ambience monitoring to record data and decide for making simple actions (air cooling systems, ceiling fans, etc.) or notifications to the user in different temperature situations adjusting related devices as room lights. After processing the basic patient situation, the Raspberry will connect to the Internet to give live information to the care staff and the deploy of alerts in anomalous situations. In this layer of interaction, the patient can trigger simple commands in the voice Assistant to change lights or even adjust the room temperature; in case these parameters are not dependent of the patient, other near relatives can make the respective adjustments to the room control. 5.3. Remote Interaction The remote interaction is made through the Internet connection, this is the way caring staff, using audio and video, and distant relatives, using social network notifications, in a and can maintain contact and monitor the elder patient, permitting them to know the status of their physical situation and to make direct calls using the smart Assistant speaker or mobile devices. The medical staff can receive alerts when an anomaly is detected with the patient that is being monitored and make audio calls directly to the mobile phone, tablet or smart assistant speaker to give the patient feedback of their situation to immediately take actions. 5.4. Data Analysis An encrypted connection is needed to protect all the data collected from patients when is stored in the applications servers, where all the big data processing will be done. This is where anomalies in the patient’s behavior can be detected following differences in the movement or voice patterns, allowing to detect other diseases related to the Elderly in an early stage. Server infrastructure will be needed to host more users from different places, including hospitals, nursing homes or the patient’s home and classify by type of disease and sufferings; this will allow to detect anomalous patterns and early symptoms in patients that share the same suffering. All the ecosystem server infrastructure needs to be ready to adopt methods for big data processing and algorithms that will classify the data. 6. Case Study The target audiences are the elderly patients in public nursing homes that belong to the “Integral Development for the Family” institute (DIF) in Aguascalientes, México. An interview with the caring staff is necessary to study the elders work out routines, recreational activities, sleeping schedule, main interests, other aging ailments not causally related to the Senile Dementia, general user preferences, openness to using technological gadgets and wearable devices, etc. The Mixed reality infrastructure scenario is showed in Figure 4 where we can show in more detail the process of the information management and all the ways the care staff can receive the information and monitor the patient. Figure 4: Digital Ecosystem for the Assistant of Patient with Senile Dementia from DIF Institute. The Senile Dementia disease we are going to focus on is the Alzheimer’s disease [26], adapting virtual environments with therapy techniques and follow the evolution of the patients participating in the project. Adaptations to devices will be needed in case some patients have physical or mental limitations, but the idea is to have the devices ready for the most possible scenarios. 6.1. Direct Interaction Xiaomi mi Band (Figure 5) is an especially useful wearable to monitor the steps per day and the heart rate of the patient at a low cost with an acceptable reading quality for 15 to 40 USD and with a great battery duration that is from one to three weeks depending on the generation of the wearable and the features that are being used. A smartphone is needed for every single Mi Band device to store the data we can retrieve from the patients via Bluetooth Connection. The Mi Band can collect data from one to two weeks until the battery is completely depleted. The readings will give us an idea of how many steps the patient walks a day and adjust their exercise routine when it is necessary. A simple heart rate monitor is integrated in the smart band and could help the caring staff to have more frequent heart rate readings without visiting the patient. Figure 5: Elder user wearing the Xiaomi Mi Band 3, a great wearable with simple and compact design. For the first stage of the project the selected VR headset is the Oculus Rift (Figure. 6) as part of the interface between the patient and the virtual environment. For further stages, the objective is using the Cardboard (Figure 2) headset with a smartphone, as a low-cost VR headset and easy to replicate in case the headset needs spare parts. Figure 6: Elder user with the first retail version of Oculus Rift, that will be used in the first stage of the project. 6.2. Indirect Interaction Amazon Alexa and Google Home (Figure 7) are reliable voice-controlled Smart Assistants with great cloud voice processing algorithms [7] to give the user an almost immediate response to commands using natural language. This device can give information for the patient about weather, the clock, or their therapy schedule; also, this device can give the patients a virtual interaction with the Smart assistance including asking for any kind of information, telling jokes, playing voice games and solving riddles to make a more enjoyable routine for the patient. Figure 7: General Smart Assistant Architecture view, showing how Alexa and Google Home work with their respective online services. 6.3. Remote Interaction Using APIs of the most popular Social Networks like Facebook, Instagram or X (aka Twitter) or even some Instant Messaging applications like Telegram can send notifications to the interested relatives and friends concerning the patient’s condition; in a similar way, medical staff will receive special notifications but with biometric information instead of social network interactions to know the status of the patient in a more real-time oriented communication. 6.4. Data Analysis The first step for data analysis is to build a knowledge base with all the data collected from the devices around the patient; with the data retrieved from all the patients’ routines for data mining will be developed to detect patterns in the information using statistics and heuristics. In a further stage there will be a possibility to detect anomalies in movement and some sufferings in an early phase, giving the medical staff the opportunity to treat diseases opportunely. 7. Conclusion The digital ecosystem modeling is a necessary part of the interconection of devices and the type of interplay with the actors involved. Each device role is essential in the environment to maintain a consistent connectivity between layers. Comprehend the specific case of each user will permit us to have a wide view of the needs for the different kind of dementia present in the elderly. Interactions between elder users and virtual environments though mixed reality can offer an inmense expansion of the elderly’s daily social interaction without leaving their homes, allowing elders to get out of the monotony and follow their daily therapy sessions in an easier way; the AR part could help the elders to have a better perception of their home environment when being left alone. The UX design will allow an increasing number of elder patients to have a more bearable experience in rehabilitation and get out of the routine they live in the nursing homes. Virtual environments are great tools with AR/VR Headsets altogether to achieve the users engagement with the ecosystem. We need to present our elders the technologies they have been missing for years, or even decades, to make their daily living a more enjoyable experience at a relative low- cost and keeping their independence as much as possible. References [1] Hwangbo, H., Yoon, S.H., Jin, B.S., Han, Y.S., Ji, Y.G.: A Study of Pointing Performance of Elderly Users on Smartphones. International Journal of Human-Computer Interaction. 29, 604–618 (2013). https://doi.org/10.1080/10447318.2012.729996. [2] Saracchini, R., Catalina-Ortega, C., Bordoni, L.: A mobile augmented reality assistive technology for the elderly. Comunicar: Revista Científica de Comunicación y Educación. 23, 65–74 (2015). https://doi.org/10.3916/C45-2015-07. [3] Bruun-Pedersen, J.R., Pedersen, K.S., Serafin, S., Kofoed, L.B.: Augmented exercise biking with virtual environments for elderly users: A preliminary study for retirement home physical therapy. In: 2014 2nd Workshop on Virtual and Augmented Assistive Technology (VAAT). pp. 23–27. IEEE, Minneapolis, MN (2014). https://doi.org/10.1109/VAAT.2014.6799464. [4] Adela Herrera, P., Andrés Guzmán, H.: Reflexiones sobre calidad de vida, dignidad y envejecimiento. Revista Médica Clínica Las Condes. 23, 65–76 (2012). https://doi.org/10.1016/S0716-8640(12)70275-4. [5] Ro, H., Park, Y.J., Han, T.-D.: A Projection-based Augmented Reality for Elderly People with Dementia. 4 (2019). https://arxiv.org/abs/1908.06046v1 [6] Felipe Salech, M., Rafael Jara, L., Luis Michea, A.: Cambios fisiológicos asociados al envejecimiento. Revista Médica Clínica Las Condes. 23, 19–29 (2012). https://doi.org/10.1016/S0716-8640(12)70269-9. [7] Pal, D., Arpnikanondt, C., Funilkul, S., Varadarajan, V.: User Experience with Smart Voice Assistants: The Accent Perspective. In: 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT). pp. 1–6. IEEE, Kanpur, India (2019). https://doi.org/10.1109/ICCCNT45670.2019.8944754. [8] Moyle, W., Jones, C., Dwan, T., Petrovich, T.: Effectiveness of a Virtual Reality Forest on People With Dementia: A Mixed Methods Pilot Study. The Gerontologist. 58, 478–487 (2018). https://doi.org/10.1093/geront/gnw270. [9] Kritikos, J., Alevizopoulos, G., Koutsouris, D.: Personalized Virtual Reality Human-Computer Interaction for Psychiatric and Neurological Illnesses: A Dynamically Adaptive Virtual Reality Environment That Changes According to Real-Time Feedback From Electrophysiological Signal Responses. Front. Hum. Neurosci. 15, 596980 (2021). https://doi.org/10.3389/fnhum.2021.596980. [10] Persky, S., McBride, C.M.: Immersive Virtual Environment Technology: A Promising Tool for Future Social and Behavioral Genomics Research and Practice. Health Communication. 24, 677–682 (2009). https://doi.org/10.1080/10410230903263982. [11] Gamberini, L., Martino, F., Seraglia, B., Spagnolli, A., Fabregat, M., Ibanez, F., Alcaniz, M., Andres, J.M.: Eldergames project: An innovative mixed reality table-top solution to preserve cognitive functions in elderly people. In: 2009 2nd Conference on Human System Interactions. pp. 164–169. IEEE, Catania, Italy (2009). https://doi.org/10.1109/HSI.2009.5090973. [12] Jamil, M.M.A., Ahmad, M.S.: A pilot study: Development of home automation system via raspberry Pi. In: 2015 2nd International Conference on Biomedical Engineering (ICoBE). pp. 1–4. IEEE, Penang, Malaysia (2015). https://doi.org/10.1109/ICoBE.2015.7235916. [13] Kishino, F., Milgram, P.: A Taxonomy of Mixed Reality Visual Displays. IEICE TRANSACTIONS on Information and Systems. E77-D, 1321–1329 (1994). https://ci.nii.ac.jp/naid/110003209335/ [14] Valdez-De-Leon, O.: How to Develop a Digital Ecosystem – a Practical Framework. TIM Review. 9, 43–54 (2019). https://doi.org/10.22215/timreview/1260. [15] Garcia, L., Kartolo, A., Methot-Curtis, E.: A Discussion of the Use of Virtual Reality in Dementia. In: Eichenberg, C. (ed.) Virtual Reality in Psychological, Medical and Pedagogical Applications. InTech (2012). https://doi.org/10.5772/46412. [16] Clay, F., Howett, D., FitzGerald, J., Fletcher, P., Chan, D., Price, A.: Use of Immersive Virtual Reality in the Assessment and Treatment of Alzheimer’s Disease: A Systematic Review. JAD. 75, 23–43 (2020). https://doi.org/10.3233/JAD-191218. [17] Tsao, Shu, Lan: Development of a Reminiscence Therapy System for the Elderly Using the Integration of Virtual Reality and Augmented Reality. Sustainability. 11, 4792 (2019). https://doi.org/10.3390/su11174792. [18] Boletsis, C., McCallum, S.: Augmented Reality Cubes for Cognitive Gaming: Preliminary Usability and Game Experience Testing. IJSG. 3, (2016). https://doi.org/10.17083/ijsg.v3i1.106. [19] Forkan, A.R.M., Branch, P., Jayaraman, P.P., Ferretto, A.: HalleyAssist: A Personalised Internet of Things Technology to Assist the Elderly in Daily Living. 10 (2019). https://doi.org/10.24251/HICSS.2019.507 [20] 18 19Cardona Reyes, H., Muñoz Arteaga, J.: Multidisciplinary production of interactive environments to support occupational therapies. Journal of Biomedical Informatics. 63, 90– 99 (2016). https://doi.org/10.1016/j.jbi.2016.08.002. [21] Schall, M.C., Rusch, M.L., Lee, J.D., Dawson, J.D., Thomas, G., Aksan, N., Rizzo, M.: Augmented Reality Cues and Elderly Driver Hazard Perception. Hum Factors. 55, 643–658 (2013). https://doi.org/10.1177/0018720812462029. [22] Gomaa, H.: Designing concurrent, distributed, and real-time applications with UML. In: Proceeding of the 28th international conference on Software engineering - ICSE ’06. p. 1059. ACM Press, Shanghai, China (2006). https://doi.org/10.1145/1134285.1134504. [23] Huang, E., Ramamurthy, R., McGinnis, L.F.: System and simulation modeling using SYSML. In: 2007 Winter Simulation Conference. pp. 796–803. IEEE, Washington, DC, USA (2007). https://doi.org/10.1109/WSC.2007.4419675. [24] Dillon, A., Watson, C.: User analysis in HCI — the historical lessons from individual differences research. International Journal of Human-Computer Studies. 45, 619–637 (1996). https://doi.org/10.1006/ijhc.1996.0071. [25] Tsinganos, P., Skodras, A.: A smartphone-based fall detection system for the elderly. In: Proceedings of the 10th International Symposium on Image and Signal Processing and Analysis. pp. 53–58. IEEE, Ljubljana, Slovenia (2017). https://doi.org/10.1109/ISPA.2017.8073568. [26] Bucciarelli, A.: Alzheimer’s disease. Mercury Learning and Information, Dulles, Virginia; Boston, Massachusetts; New Delhi (2016).