This paper describes the design approach for promoting health and well-being of elderly patients with hypertension with the help of wearable and augmented reality technologies, which have been applied in the bewARe project, focusing on a sensor-supported movement training for senior citizens in an intelligent Augmented Reality system. The bewARe system enables movement, coordination and reaction training and is based on research in the field of geriatric medicine. The use of wearable sensors and AR glasses makes it possible to augment and record the environment of the user into and during training, to derive information from it and to return the information to the user in a supportive and motivating manner. The use of the trainer avatar and gamification elements aims to increase motivation, change behaviour and to strengthen adherence and persistence. The bewARe system supports a non-drug, athome therapy for elderly patients with hypertension in conjunction with medical diagnosis and guided rehabilitation programs. In this way, bewARe contributes to the UN's Sustainable Development Goals by focusing on good health and well-being for all at all ages. The paper present the current state of design in the bewARe project and concludes with recommendations for future research and development with focus on ethical evaluation of socio-technological arrangements which utilise wearable and AR technologies.
Hypertension is a major risk factor for a number of cardiovascular diseases and a chronic disease
with the highest prevalence in Germany, affecting 85% of people over 70 years of age. Hypertension
management may be supported by pharmacological and non-pharmacological interventions. Non drug
interventions encompass lifestyle modifications which aim to improve overall health, reduce
medication and delay the progression of hypertension [
According to motivational theories and research, intrinsic motivation is critical for adherence in
physical exercising and sports. While extrinsic motives may be as effective and intrinsic motives for
initiation of participation in physical activities, the enjoyment of the physical activity itself leads to
increased persistence and contributes to positive psychological feelings [
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
for the design of physical activity programs are to focus on enjoyment as a means to sustained
physical activity, regardless of the initial motives [
The question is then how to facilitate user enjoyment in technology-enhanced settings. This
question can be answered from the perspective of research in the field of technology acceptance.
Research in acceptance of software systems has shown that systems may be used in a utilitarian and/
or recreational manner. While utilitarian systems are used for reasons of productivity, recreational
(pleasure-oriented) systems, such as games, focus on prolonged and hedonic rather than productive or
utilitarian use [
The bewARe project presented in this paper aims to develop a technology-enhanced movement
and mobility training for seniors with hypertension using emerging technologies including augmented
reality and wearable sensors. BewARe is a three-year (08/2018 - 08/2021) research and development
project founded by the German Federal Ministry of Education and Research (BMBF) as part of the
program Interactive systems in virtual and real spaces – innovative technologies for a healthy life. The
bewARe system leverages a combination of different sensors like rgb- and depth-cameras,
acceleration sensors, eye tracking devices and wristbands to measures blood pressure and heart
frequency [
A deeper understanding of characteristics, constraints and abilities of users is essential for creating an engaging and effective interaction design. This section gives an overview of the methods applied in the bewARe project to gather requirements of the target group of senior patients with hypertension and developing personas with varying conditions based on the results from user research. 2.1.
Research in technology acceptance among elderly users has been informed by the Senior
Technology Acceptance Model (STAM) model, which takes into consideration the unique
characteristics of the elderly [
Despite common changes related to ageing, elderly users are not a homogeneous group [
The focus group methodology comprised a series of tasks related to the use of AR/VR glasses (e.
g. putting on and taking off AR glasses, recognising the user interface, differentiating between real vs.
virtual elements, gesture control). Exergame prototypes were used to create and compare
understanding of AR and VR [
Based on the results from user interviews and the focus group, three personas with varying conditions were designed to condense relevant characteristics of the target group. The description of each persona includes information about age, gender, marital status, diseases, blood pressure, maximum training pulse, fitness goals, experience goals, a typical day timeline, motivations, frustrations, physical activities, attitude towards tracking of health-related data and devices used. Each description includes the about section to make personas more vivid. The about sections of the three personas are: 1. Persona “Klaus”. Klaus has to take care of his disabled wife. He doesn’t have so much time for himself. Nevertheless, he tries to exercise by biking to wherever he needs to go, such as to the grocery store. He has been going to the same weekly group training since five years. He enjoys seeing the familiar faces there and making laughters during the training session. He values that the trainer knows his medical history and is someone who can shake up his inner couch potato once in a while. 2. Persona “Victoria”. Oliver, a 5 years old golden retriever, has been Victoria’s companion since her husband passed away. She keeps herself busy with attending seminars at the senior university and different training sessions during the week. She enjoys being outdoor such as gardening and walking her dog. She is happy to have walked more than 10000 steps a day, based on the Fitbit device, a gift from her daughter at last Christmas. In the evenings, she looks forward to cooking for her family and playing with her grandchildren. 3. Persona “Sven”. Since his heart surgery, Sven has been exercising daily to maintain his fitness level, with every training session reaching his target heart rate zone. His heart condition limits his physical activity intensity. But with persistence, he keeps his health in check so that he can still participate in the annual hikes with his buddies.
The design of the bewARe system aims at accommodating all potential users taking accessibility as one of the critical factors in interaction design for senior patients with different conditions such as hypertension into account. This section lays out some of the key approaches to the interaction design in the bewARe project including multimodal, avatar and gamification approaches. The subsequent section also address limitations of fully immersive VR systems which require wearing a Head Mounted Displays which block the view of the real world and enhance the imbalance problem. 3.1.
The methodology of the multimodal interaction design in the bewARe project has followed the principles of the Design Thinking methodology. In the initial stage the focus has been on the design user scenarios for each of the three personas. The design challenges were: How do users get in contact with the bewARe system and calibrate the system accordingly their needs under medical supervision? How can the bewARe system guide the user in a way that feels intuitive so that the user can easily understand and perform the exercise correctly? How to design the trainer’s avatar and the avatar’s interactions with the user so that it feels personal, motivating and engaging?
In order to explore specific usability aspects including avatar vs. non-avatar based visual guidance, specified vs. free-form movement experience, the integration of sound effect, user tests were conducted in form of the guided experience ideation workshop with 5 seniors. The workshop included three phases, i. e. pre-training (introduction, measuring user's pulse, putting on goggles and wristbands), training (3 exercises: virtual geometry, spatial marker and partner mirroring) and posttraining (measuring user's pulse, post-training result and message from a previous user). During the workshop a test script and non-digital materials (such as cardboard mockups of goggles and wristbands) were used as props to simulate an immersive AR experience. The experience was guided by researchers who engaged with users through role-plays. The aim was to apply the embodied practice to act out the features of the bewARe system and the physical movements performed during a set of physical exercises. The workshop was video recorded to support the evaluation. The analysis was based on on-site observation, recorded video materials and presented in form of quantitative scores (completion rates for exercises) and errors (e. g. instruction was unclear) for each participant.
The key insights from the guided experience ideation workshop included preferences of senior users for personalisation (e. g. addressing users by their names), performance review (e. g. evaluation of the pulse over time), visual instructions and demonstrations at the beginning to improve understanding of the exercise instruction as well as use of music which was reported to be calming and positive for concentration. Some users expressed a preference for group training, while others voiced their preference for exercising alone at home. Based on the insights from the guided experience ideation workshop, a set of primary and secondary scenarios, user journeys (with 5 phases: before training, start training, during training, end training and after training) and user stories (in 3 categories: movement, posture and performance) were developed for each persona to further guide the design process. Building on these results, the bewARe MVP was created in Unity. 3.2.
The design of the first prototype of the trainer avatar for the AR settings was based on the insights from the guided experience ideation workshop and from the follow-up trainer avatar ideation workshop. The objective of the workshop was to compile a list of actions to be recorded during motion capture session and to define the role of the trainer avatar in providing an engaging user experience. In the first step, trainer avatar maps were created to describe some of the desired characteristics such as the looks and the size of the avatar, the type of voice and movement, as well as the type of interaction and feedback provided by the trainer avatar.
Based on the insights from the trainer avatar ideation workshop, a full body trainer avatar
prototype was designed implemented in the current MVP of the AR training environment. The trainer
avatar was developed with Adobe Fuse CC, edited in Cinema4D and further developed in Unity. The
animations of the avatar were recorded with the motion capture system Qualisys and then edited in
Unity. The trainer avatar (Anna) is represented by a realistic, whole-body, female trainer figure
(silhouette). The whole-body, human-like avatar was chosen to enhance positive experience and
enjoyment. Previous research, e. g. in context of AR [
The gamification design as part of the multimodal interaction design of the bewARe system
combines digital storytelling [
As a follow-up design activity, groups of students at the partner university developed gamification concepts and mockups for each persona as part of the Designing Thinking process. Both Bartle’s Player types and game preferences of senior users (interview results) were taking into consideration to design each scenario. The first step was to design gamification user stories for each persona and match them with each phase in the user journey. The gamification designs included: travelling the world to explore and learn different dances, time travelling and adventure scenarios with missions, pathways with checkpoints for exercising and different avatar designs.
Based on the insights from this initial phase, a first gamification prototype was designed and
implemented for the training unit Hustle Dance. Huste Dance is one of the nine training units in the
bewARe system, in which pre-set sequences of dance movements are demonstrated by the trainer
avatar and repeated by the user to the background music. The training unit was gamified using digital
storytelling and meaningful gamification approaches. The gamification concept begins with a
challenge which is to dance the hustle dance like John Travolta in the Saturday Night Fever. The
training sequence is embedded in the Saturday Night Fever scenery, which changes the physical
environment of the user to the dance floor from the 1960s. The training sequence is structured
following the RECIPE model [
This section presents the three-component bewARe system architecture, wearable sensors and AR glasses applied to provide a hybrid system for training of senior patients with hypertension. 4.1.
The architecture of the bewARe system consists of several distributed components and is designed
to be extendable and configurable beyond project lifetime. One goal is to adapt the system to the
userspecific needs and therefore, enable a highly customised experience. Even though the system can
technically be divided into a local and an online set of components, it is also possible to only execute
it locally. The option to run the whole system locally especially benefits field study evaluations in a
mobile scenario [
Input sensors are a core component of the bewARe system. The sensors supply the analysis software with desired information for efficient motion and heart rate analysis with focus on healthy movements while improving general well-being. All selected sensor devices have been chosen to reduce small footprint and increase usability and accessibility. For example, wearing a watch feels natural to most of the people and the use of a smart watch makes it straightforward to implement heart rate tracking. The heart rate data can be used to derive the patients current state of health while adjusting the training for an optimal training stimulus. Modern motion capturing cameras like Microsoft’s Azure Kinect2 or Qualisys’ Miqus3 can be combined with machine learning models [19, 20, 21] to determine skeleton-based 3-D joint position information. The resulting information is used to compute medical joint angles which are necessary to evaluate correct movements. Moreover, motion capturing data can be applied for various additional tasks like visualising a user avatar to mirror his actions or for similarity search to identify what movement is currently performed. Smart insoles from Motion Science4 may be applied as an element of the bewARe system to compute pressure information of foot movements which are essential for inverse kinematic and dynamic calculations. This pressure information gives a detailed overview of the patient's center of gravity and can be derived to identify wrong execution of movements. Currently, specific movement designs with the use of smart insoles, such as squad exercises, are being designed for AR training settings. 4.3.
One of the important wearable components of the BewARe system are AR glasses which are part of the training set-up. A number of AR and VR glasses was tested with senior users in the project,
including AR glasses Microsoft’s HoloLens 1 and Leap Motion’s Project North Star, as well as VR glasses HTC Vive Pro and HTC Vive Focus. Currently, new tests with HoloLens 2 are planned.
Whereas in the initial phase of the project the use of VR devices was explored as mentioned in the sections above, AR was established as the focus of the project in later stages. During the project's initial phase in 2018, all available AR technologies lacked the capacity to properly display crucial interaction elements, such as the trainer avatar and the projected visuals, due to a limited field of view. In particular, displaying life-size avatars in close environments in AR was a challenge. Therefore, the second phase of the project has addressed potentials and challenges of designing interaction for a AR head-mounted device for senior users. Specifically elderly people face the problem of balance, where normal fields of view with augmented information come in handy. The results from user tests with VR glasses showed that the risks of imbalance and motion sickness may not only impede user experience but also create critical safety issues in case of senior patients with hypertension. Being able to see the known and not being confronted with a complete new environment as it is the case with AR technologies might be less overwhelming and more adequate for end users of the bewARe system.
Since the bewARe project focuses on AR use and the benefits of AR for the target group (including as safety), a workaround with VR technology was established to design and display an "ideal" application of augmentation in real space under the hypothesis that the development of AR goggles will eventually catch up and provide a wide(r) field of view, enabling a more elaborated interaction. With this workaround, we have been able to test a variety of interaction elements and gained valuable insight for the further development of the training. With new AR technologies now becoming available, the project plans to use Microsoft’s Hololens 2 to evaluate possible modes of interaction for seniors with the technology in AR settings.
Emerging technologies such as wearable sensors and AR glasses have been considered beneficial for improving the quality of life and the promotion of health and well-being of senior users [22]. The design of complex systems, such as the bewARe system, includes a number of components which have to be brought together to deliver an engaging and enjoyable experience. The complexity of such systems as well as the limitations of available technologies bear some challenges and risks for end users, among others the need to use specialist equipment, difficulties in distinguishing between real and virtual elements, the need for one-to-one assistance when using wearable and AR technologies, as well as concerns related using new technologies including safety and security [22]. In order to identify possible ethically problematic effects and to develop ways to solve them, an ethical analysis based on the MEESTAR model (Model for the ethical evaluation of socio-technological arrangements) [23] was conduced in the bewARe project. MEESTAR as an analytical toolkit for the ethical evaluation and problem-solving of assistance systems for elderly users can be used in a wide range of technical applications, especially e-health. Following the MEESTAR methodology, seven ethical dimensions (i. e. care, self-determination, security, justice, privacy, participation, self-understanding) were explored from the perspective of risks and adequate interventions to minimise these risks in the initial stage of the project. Such ethical evaluation allows to anticipate risks and reflect on the role of (senior) users in the broader context of e-health, especially in relation to sovereignty, self-empowerment and adherence to therapy. The critical fields from the MEESTAR analysis were priorities using the MoSCoW method. Some of the key risks identified encompassed Exclusion of people with disabilities, motion sickness, enhancement of addictive behaviour, dependence on technical systems, paternalism through software and transfer of data to third parties. The MEESTAR analysis allowed to systematically addresses these and other risks in the early stages of the project by working out an effective solution to each risk. The bewARe project has considered this ethical analysis as an essential component in the development and delivery of the movement and mobility training systems for senior patients with hypertension. The recommendation from the bewARe project is to include an ethical analysis already in the early stages of design to explore relevant ethical concerns and evaluate potential risks of applied technologies on end users as well as to derive effective solutions early on. It is advisable to take he key challenges of specific technologies into consideration, e. g. persuasiveness and safety of AR applications, privacy and data control issues related to the use of wearable sensors [24]. This recommendation is especially relevant in case of systems designed for vulnerable groups which include the elderly, ill, disabled and children.
This publication was produced as part of the bewARe project with the name: Sensor-supported movement training for senior citizens in an intelligent Augmented Reality System. The bewARe project is a three-year (08/2018 - 08/2021) research and development project founded by the German Federal Ministry of Education and Research (BMBF) as part of the program Interactive systems in virtual and real spaces – innovative technologies for a healthy life. For more information please visit the project website: https://www.beware-projekt.de/en/index.html. This paper is made available under a Creative Commons License Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). 7. References 18. D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, A. Zisserman, Counting Out Time: Class Agnostic Video Repetition Counting in the Wild, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020): n. pag. 19. W. Yang, W. Ouyang, X. Wang, J. Ren, H. Li, X. Wang, 3D Human Pose Estimation in the Wild by Adversarial Learning, IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, (2018): 5255-5264. 20. C. Zhe, T. Šimon, S. E. Wei, Y. Sheikh, Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017): 1302-1310. 21. M. Wang, X. Chen, W. Liu, C. Qian, L. Lin, L. Ma, DRPose3D: Depth Ranking in 3D Human
Pose Estimation, ArXiv, abs/1805.08973, (2018): n. pag. 22. L. N. Lee, M. J. Kim, W. J. Hwang, Potential of augmented reality and virtual reality technologies to promote wellbeing in older adults, Applied Sciences (Switzerland), 9(17), (2019). 23. A. Manzeschke, MEESTAR: Ein Modell angewandter Ethik im Bereich assistiver Technologien.
In: K. Weber, D. Frommeld, A. Manzeschke, H. Fangerau H. (Eds.) Technisierung des Alltags.
Beitrag für eingutesLeben? Steiner, Stuttgart, (2015): 263–283. 24. S. G. Pase, Ethical Considerations in Augmented Reality Applications, in: Proceedings of the International Conference on e-Learning, e-Business, Enterprise Information Systems, and eGovernment (EEE): The Steering Committee of The World Congress in Computer Science, Computer (Las Vegas, NV), 1, (2012).