Creation of Physiatric Exercises for Remote Use in Rehabilitation Exergames Dario Maggiorini Laura Anna Ripamonti Davide Gadia University of Milano University of Milano University of Milano Milano, Italy Milano, Italy Milano, Italy dario@di.unimi.it ripamonti@di.unimi.it gadia@di.unimi.it ABSTRACT working age – that is to say 15–64 years old [2, 9, 16]. In The current demographic ageing in Europe is the result of a the following decades, the so-called ‘baby-boomers’ (i.e., relevant economic, social, and medical development. the huge generation born in the ‘50s–‘60s) will start to Nevertheless, at the same time, it is also leading to a retire, further exacerbating the situation. The demographic significant increase in the demand for long-term care ageing in Europe is the result of a relevant economic, (LTC), especially for seniors. One viable way to offer social, and medical development, that provide us with qualified cares at home, while at the same time containing longer and better lives compared to those of past costs, is to exploit digital technologies as enablers of a generations. Nevertheless, this progressive increase in constant interaction between seniors and assisting population lifespan, besides creating new opportunities, personnel. In particular, (video) games have already been impacts deeply on a number of areas, such as: healthcare, identified as a viable way to foster motivation and retirement, housing, community care, welfare, etc. Among engagement in the long term. While technical solutions to these areas, one of the most afflicted is the long-term socio- provide at home LTC has already been proposed, the medical assistance. This increase in the demand for Long scientific community is still working on general Term Care (LTC) is supported by declining (or, at best, methodologies to streamline the process on the caretaker’s growing at a slower pace) public financial resources. side. In this paper, we focus on a software application to Consequences of this emerging situation call for a design and propose physiatric exercises from a remote reorganization of the assistance supply through the location. These exercises can be extremely tailored on the development of innovative management approaches, the requirements of each patient and can be monitored in an enrichment of socio-medical services, the integration automated way exploiting off-the-shelf gaming between hospitals and local communities, and the adoption technologies such as Microsoft Kinect. The proposed of multidisciplinary perspectives. As of today, the most solution aims to shorten the feedback loop between patient promising solutions rely on exploiting digital technologies and caretaker in order to increase the quality of the therapy in order to provide services supporting LTC at home. As a and improve the recovery time. matter of fact, domiciliary cares can be an effective Author Keywords alternative to long-term hospitalization: the psychological HCI; visual interfaces; healthcare; long-term care; LTC. and affective benefit for the patient would be huge, and shorter hospitalizations would mean shorter queues to ACM Classification Keywords access public health services and shrinkage in costs. Among H.5.2 User Interfaces; J.3 Life and medical Science: Health; all possible digital technologies, (video) games have D.2.m Software Engineering: Miscellaneous. already been identified as a viable teaching/training media INTRODUCTION in 2002 with the foundation of the Serious Games Initiative It has been estimated that, by 2025, people over 60 [15] by the Woodrow Wilson Center for International worldwide will be 1.2 billion, and by 2050 they will reach 2 Scholar in Washington, D.C. In particular, serious games billion (in 2000 they used to be ‘only’ 600 million). Also, in can provide a way to increase engagement in a 2050, in Europe, the number of over 60 will equal the 40% rehabilitation therapy: exercises disguised as sessions of of the total population, and the 60% of the population in serious gaming are more likely to be performed on a regular basis and will not require the constant presence of a therapist. Moreover, the scoring system may also offer a quick and easy way to assess the rehabilitation path and/or to raise real-time alarms. Designing a serious game for rehabilitation [5, 11] as well as defining a distributed infrastructure to support non- GHItaly18: 2nd Workshop on Games-Human Interaction, May 29th, 2018, invasive patient monitoring and remote assistance [12] are Castiglione della Pescaia, Grosseto (Italy) not impossible challenges. Nevertheless, an important issue Copyright © 2018 for the individual papers by the papers' authors. Copying still stands about how to easily define the right exercise for permitted for private and academic purposes. This volume is published and copyrighted by its editors. each patient and deploy it on a device located in the households. Exercises should not be described using code because that would be impractical for medical staff but, at the same time, they must be easy to integrate in an existing gaming environment. With the above goal in mind, this paper is focused on proposing a visual interface which will be viable for a medical operator and where an exercise can be easily defined while taking into account patient’s monitoring data. This exercise can then be deployed remotely as a physiatric serious game and be part of a long-term rehabilitation therapy. RELATED WORK The idea of using games to foster medical therapies has already been accepted long before computers and electronic devices became a way to convey entertainment [1]. In Figure 1: General Structure of Care@Home. particular, in recent times, we have been witnessing a fair number of projects and experimentations on this topic [3, 4, house. The created exercises may not be specific to a 6, 7, 8, 10, 11, 12, 13, 14]. medical condition and should be suitable for a number of exergames. Our contribution in this paper is going to be an In [13], authors design an exergame for post-surgery knee extension to Care@Home and, for this reason, we will now rehabilitation. The provided solution uses Kinect to monitor present its framework in detail. the patient and allows remote assistance, but the final application is limited to assist on knee conditions. Other The Care@Home Project contributions are focusing on post-stroke patients; in The Care@Home project has been developed to allow particular, [8] proposes an exergame for upper-limbs remote interaction, on a daily basis, with elders (especially rehabilitation. As in the previous case, authors are focused those living alone) requiring LTC. Care@Home integrates a on a specific game rather than a more comprehensive number of sensor devices in the patient environment and solution. Authors of [3] are also targeting after-stroke allows to assign exercises, verify progresses in mobility, upper-limb rehabilitation, but a more comprehensive study and monitor health/environmental parameters. In particular, is offered about the usefulness of Augmented Reality (AR) the Kinect gaming device has been exploited as a in designing and deploying rehabilitation exergames. Other movement and body measurement sensor as well as for its contributions [6, 7] are proposing exergames designed for easy integration in a gaming platform. A set-top-box is wrist rehabilitation. In this case, while still limited to a installed in the elder home to manage the Kinect together specific condition, authors try to exploit mobile phones to will all other environmental sensors and to interface the increase accessibility and user experience. household with a caretaker’s control center (see Fig. 1). Other contributions try to be more general and proposes full frameworks [14] or general methodologies [4, 10] to improve patient motivation and prove the all-around effectiveness of games in rehabilitation therapies. This group, anyway, seems to be more focused on monitoring the patient and evaluating the effectiveness of a therapy rather than actually hosting and dispensing exercises. All the solutions referenced so far, are either implemented around an existing game or focus on providing design guidelines for exergames; none of them are proposing an integrated environment to manage multiple exercises/games in the context of a rehabilitation process taking place over a long time. A more comprehensive solution to host and manage exergames, and to monitor patients’ performance from a remote location is represented by Care@Home [11, 12]. Care@Home is a complete framework where a caretaker Figure 2: Care@Home hi-level architecture. can interact with each patient using a remote application. This remote application is capable to define exercises to be proposed as games via a set-top-box located in the patient’s example. Each exercise may implement its own checking policy and focus on monitoring specific postures, which may be critical for a given patient. CREATING EXERCISES As already mentioned, an important feature required from a remote rehabilitation service is a convenient way to define new exercises tailored on each patient’s needs and performances. When designing such a tool and its interface, we must also take into account that the creation of exercises will be performed by a medical operator; this implies that any description methodology requiring to write code is not viable. Moreover, the physiotherapist must be able to Figure 3: Tracking patient while performing an exercise. perform all operations of storage, retrieval, and deployment Care@Home has been implemented as a distributed of exercises without asking the assistant of technical staff. architecture whose purpose is to provide an efficient, The proposed solution reliable, and scalable real-time communication between In order to create an efficient yet very intuitive interface, sensing equipment located in the home of each patient and a we resolved to use a Natural User Interface (NUI) also on central control system. A schema of the hi-level the caretaker side. As a consequence, Microsoft Kinect is architecture is reported in Fig. 2. In the picture, we can used to let the caretaker define the tracking positions of observe that this architecture is composed by many each exercise. The caretaker can track her own body to components; some of them are located inside the define an exercise and then edit the result. This approach caretaker’s organization datacenter (shaded area), while proved to be very well accepted by medical staff because others are dislocated where convenient: either in the physiotherapists are already used to show to their patients caretaker’s office or in the patient’s household. In how to perform each exercise. The only learning curve particular, on the patient side, a DeviceProxy is in charge to required from the caretaker is about the interface for editing manage the connection and collect data from the local and deployment, which is used in a second stage. sensors. Moreover, the DeviceProxy is storing all the information related to the exercises to be “dispensed” to the Once an exercise is decorated with all required metadata patient. (e.g., number of repetitions, reference patient, and description), it can be stored in an online database located When an exercise is performed, the Kinect comes into play in the caretaker’s datacenter. From there, the PhysioServ and starts tracking the patient’s position. A sequence of application server will take care to push the exercise toward positions is presented on the screen as dots overlapped to the correct DeviceProxy. the avatar reporting the current body position (see Fig. 3). Each exercise requires from the patient to reach a sequence The caretaker’s database can also be used to retrieve of positions with her body. If, while performing an exercise, existing exercises and tailor them for different patients due to new, hopefully improved, conditions. the tracking is reported too off of the required asset, the dots are drawn in red, to provide immediate visual feedback Editor Architecture and an alarm is raised on the screen; see Fig. 4 for an The exercise editor has been implemented using Windows Presentation Foundation (WPF): a Kinect-compatible framework for visual applications distributed by Microsoft. In WPF, the interface is defined by means of an XML file (taking xaml as extension). A specific class, usually implemented in C#, manages the interface as described in the xaml file and bridges the GUI to external elements and the Kinect hardware. The hi-level architecture of the exercise editor is reported in Fig. 5. In this architecture, the Main Window class (with its xaml file) is responsible to manage the main application panel. From within this panel, we can initialize all data structures, perform data management tasks, and drive Kinect sensors. The Main Window interface exposes standard data management Figure 4: Wrong posture reported: right hand should be functionalities for exercises (new, open, save, and so forth) repositioned (avatar is represented as if in a mirror). as well as players control (start, stop, record). Figure 5: General architecture of the exercise editor. Figure 6: Metadata Editor, panel to add additional information to the exercise (some data have been The Metadata Editor class is responsible to manage all obfuscated for privacy reasons). additional data for the exercise. These data include both contextual information, such as how many repetitions are required and when the exercise should be performed, as well as generic information such as medical notes from the caretaker. Screenshots of the Metadata Editor windows are proposed in Fig. 6 and Fig. 7. In the Metadata Editor, it is also possible to select single joints (Fig. 7) in order to provide additional directions for the exercise software. These directions may be about a joint which is not supposed to be moving or a limb bending which should not exceed a given angle, as in the alarm raised in Fig. 4. Both Metadata Editor and Main Window use two service classes designed to store data about the whole exercise and each single joint, respectively. Exercise and Joints will be serialized and stored in a backend database for deployment and/or later retrieval. The last class in our architecture is Player. The Player class takes care of all functionalities related to capturing the caretaker’s movement using Kinect and replaying the edited Figure 7: Metadata Editor, joint selection exercise before saving it in the backend database. for detailed editing. Moreover, the Player can also retrieve an existing exercise from the database and let the caretaker create a customized version. Figure 8 shows the Player window during the recording of an exercise. In the figure, it is possible to see a left panel with a real-time tracking of the caretaker body while performing the exercise (the red border means that recording is in progress) and a right-hand panel with a hierarchical representation of the exercises database in the backend. The exercise database is available from the Player window in order to ease the process of browsing and playing out existing exercises. The database is represented in a hierarchical way with a tree branching based on the interested limbs or body part for each exercise. This data organization proved to allow for quick retrieval from medical staff without requiring to fill an explicit search form. The last element of the Payer window, in its lower section, is a selector to pick a subset of the joints which are important for monitoring. This way, the recording will be Figure 8: Player window while capturing an exercise. limited to the selected subset and the rest of the body will which is feasible for non-tech-savvy caretakers. This not be used to evaluate the patient’s performance. Once application exploits Kinect on the medical personnel’s side finished the recording, the Player window can be closed to to define an exercise via body tracking and let the user perform further processing and deployment from the Main decorate it with metadata useful for archiving and Window. deployment. Creating and Managing Exercises The proposed solution is currently under testing and The workflow, while creating an exercise, can be divided preliminary unstructured feedback from medical staff seems into a sequence of steps. During the first step, the caretaker promising. will record the required movement, select the relevant joints, and add all important metadata to the exercise. A As a future extension, we are planning to implement an serialized version of the Exercise instance can then be adaptation layer to let each exercise to be instantiated inside saved locally in a binary format. This local save will be a generic game, as in the early prototype shown in Fig. 9. In then pushed to the backend database and indexed using the this prototype, the player/patient is required to raise metadata provided by the caretaker. alternatively the right and left leg for a given number of times. Our intention is to insert several kinds of leg-related Once in the backend database, the information will be exercises into this game without bounding movements or managed by the PhysioServ application server: basing on requiring the modification of game rules and mechanics. the indexing metadata, each exercise will be pushed to one ACKNOWLEDGEMENTS (or more) set-top-boxes in patients’ households when We would like to thank our student Ana Maria Alexandru connection will take place. We cannot assume the client for her contribution in defining the first interface and side of the architecture is always connected and sync is guidelines for this application. performed based on opportunity. When the exercise is deployed on a set-top-box, the patient will see the Exercise REFERENCES de-serialized and instantiated inside a rehabilitation 1. C. Abt. 1970. Serious Games. The Viking Press, New exergame. York. CONCLUSION AND FUTURE WORK 2. H. Afsarmanesh, S.S. Msanjila. 2010. ePAL Vision In this paper we discussed about exergames to support 2020 for active ageing of senior professionals. In rehabilitation of patient on LTC. While it is already very Proceedings of IFIP Advances in Information and clear the importance of (video) games for rehabilitation, we Communication Technology (PRO-VE 2010), Vol. are still missing a complete and streamlined solution to 336, pp. 60–72. DOI:10.1007/978-3-642-15961-9_6 dispense gaming activity as part of a therapy. In the context 3. J.W. Burke, M.D.J. McNeill, D.K. Charles, P.J. of Care@Home, an infrastructure for remote administration Morrow, J.H. Crosbie, S.M. McDonough. 2010. of LTC at home, we addressed here the problem of defining Augmented reality games for upper-limb stroke exercises with a functional and easy-to-use application rehabilitation. In proceedings of 2nd International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES 2010). pp.75–78. DOI:10.1109/VS-GAMES.2010.21. 4. J.W. Burke, M.D.J. McNeill, D.K. Charles, P.J. Morrow, J.H. Crosbie, S.M. McDonough. 2009. Serious games for upper limb rehabilitation following stroke. In proceedings of the 2009 Conference in Games and Virtual Worlds for Serious Applications (VS-GAMES 2009). pp.103–110. DOI: 10.1109/VS- GAMES.2009.17 5. N. D’Aquaro, D. Maggiorini, G. Mancuso, L. A. Ripamonti, 2011. Videogames and Elders: a New Path in LTC?. In proceedings International Conference on Human-Computer Interaction (HCII 2011). DOI:10.1007/978-3-642-21799-9_28 6. D. Deponti, D. Maggiorini, C.E. Palazzi. 2009. DroidGlove: an android-based application for wrist rehabilitation. In proceedings of International Figure 9: Early prototype game for legs exercise (this Conference on Ultra Modern Telecommunications application is using standard assets from Unity3D). (UMSA 2009). DOI:10.1109/ICUMT.2009.5345442. 7. D. Deponti, D. Maggiorini, C.E. Palazzi. 2011. 12. D. Maggiorini, L.A. Ripamonti, E. Zanon. 2012. Smartphone’s physiatrist serious game. In proceedings Supporting Seniors Rehabilitation Through Videogame of 1st International Conference on Serious Games and Technology: a Distributed Approach. In proceedings Applications for Health (SeGAH 2011). DOI: International Workshop on Games and Software 10.1109/SeGAH.2011.6165434. Engineering (GAS 2012). 8. I. Di Loreto, G. Abdelkader, N. Hocine. 2011. Mixed DOI:10.1109/GAS.2012.6225920. reality serious games for post-stroke rehabilitation. in 13. B. Maurer, F. Bergner, P. Kober, R. Baumgartner. proceedings of 5th International Conference on 2012. Improving rehabilitation process after total knee Pervasive Computing Technologies for Healthcare, replacement surgery through visual feedback and pp.530–537. enhanced communication in a serious game. In DOI:10.4108/icst.pervasivehealth.2011.246102. proceedings of the 30th ACM International Conference 9. C. Jeavans. 2004. Will we still be working at 70?. on Design of Communication. pp.355–356. http://news.bbc.co.uk/2/hi/uk_news/4016969.stm DOI:10.1145/2379057.2379124. (accessed April 3, 2018). 14. S. Saini, D.R.A. Rambli, S. Sulaiman, M.N. Zakaria, 10. E. lores, G. Tobon, E. Cavallaro, F.I. Cavallaro, J.C. S.R. Mohd Shukri. 2012. A low-cost game framework Perry, T. Keller. 2008. Improving patient motivation in for a home-based stroke rehabilitation system. In game development for motor deficit rehabilitation. In proceedings of 2012 International Conference on proceedings of the 2008 International Conference on Computer and Information Science (ICCIS 2012). Advances in Computer Entertainment Technology pp.55–60. DOI:10.1109/ICCISci.2012.6297212. (ACE 2008). pp.381–384. DOI: 15. Serious Games Initiative. 10.1145/1501750.1501839 http://www.seriousgames.org/ (accessed April 3, 2018) 11. D. Maggiorini, L.A. Ripamonti, A. Scambia. 2012. 16. M. Stranges. 2008. Immigration as a remedy for Videogame Technology to Support Seniors. In population decline? An overview of the European proceedings International ICST Conference on countries. In European Papers on the New Welfare, Simulation Tools and Techniques (SIMUTOOLS Special Issue on the Counter Ageing Society, Paper 2012). DOI:10.4108/icst.simutools.2012.247782 no. 8.