=Paper=
{{Paper
|id=Vol-2246/GHItaly18_paper_07
|storemode=property
|title=Creation of Physiatric Exercises for Remote Use in Rehabilitation Exergames
|pdfUrl=https://ceur-ws.org/Vol-2246/GHItaly18_paper_07.pdf
|volume=Vol-2246
|authors=Dario Maggiorini,Laura Anna Ripamonti,Davide Gadia
|dblpUrl=https://dblp.org/rec/conf/avi/MaggioriniRG18
}}
==Creation of Physiatric Exercises for Remote Use in Rehabilitation Exergames==
https://ceur-ws.org/Vol-2246/GHItaly18_paper_07.pdf
Creation of Physiatric Exercises
for Remote Use in Rehabilitation Exergames
Dario Maggiorini Laura Anna Ripamonti Davide Gadia
University of Milano University of Milano University of Milano
Milano, Italy Milano, Italy Milano, Italy
dario@di.unimi.it ripamonti@di.unimi.it gadia@di.unimi.it
ABSTRACT working age – that is to say 15–64 years old [2, 9, 16]. In
The current demographic ageing in Europe is the result of a the following decades, the so-called ‘baby-boomers’ (i.e.,
relevant economic, social, and medical development. the huge generation born in the ‘50s–‘60s) will start to
Nevertheless, at the same time, it is also leading to a retire, further exacerbating the situation. The demographic
significant increase in the demand for long-term care ageing in Europe is the result of a relevant economic,
(LTC), especially for seniors. One viable way to offer social, and medical development, that provide us with
qualified cares at home, while at the same time containing longer and better lives compared to those of past
costs, is to exploit digital technologies as enablers of a generations. Nevertheless, this progressive increase in
constant interaction between seniors and assisting population lifespan, besides creating new opportunities,
personnel. In particular, (video) games have already been impacts deeply on a number of areas, such as: healthcare,
identified as a viable way to foster motivation and retirement, housing, community care, welfare, etc. Among
engagement in the long term. While technical solutions to these areas, one of the most afflicted is the long-term socio-
provide at home LTC has already been proposed, the medical assistance. This increase in the demand for Long
scientific community is still working on general Term Care (LTC) is supported by declining (or, at best,
methodologies to streamline the process on the caretaker’s growing at a slower pace) public financial resources.
side. In this paper, we focus on a software application to Consequences of this emerging situation call for a
design and propose physiatric exercises from a remote reorganization of the assistance supply through the
location. These exercises can be extremely tailored on the development of innovative management approaches, the
requirements of each patient and can be monitored in an enrichment of socio-medical services, the integration
automated way exploiting off-the-shelf gaming between hospitals and local communities, and the adoption
technologies such as Microsoft Kinect. The proposed of multidisciplinary perspectives. As of today, the most
solution aims to shorten the feedback loop between patient promising solutions rely on exploiting digital technologies
and caretaker in order to increase the quality of the therapy in order to provide services supporting LTC at home. As a
and improve the recovery time. matter of fact, domiciliary cares can be an effective
Author Keywords alternative to long-term hospitalization: the psychological
HCI; visual interfaces; healthcare; long-term care; LTC. and affective benefit for the patient would be huge, and
shorter hospitalizations would mean shorter queues to
ACM Classification Keywords access public health services and shrinkage in costs. Among
H.5.2 User Interfaces; J.3 Life and medical Science: Health; all possible digital technologies, (video) games have
D.2.m Software Engineering: Miscellaneous. already been identified as a viable teaching/training media
INTRODUCTION in 2002 with the foundation of the Serious Games Initiative
It has been estimated that, by 2025, people over 60 [15] by the Woodrow Wilson Center for International
worldwide will be 1.2 billion, and by 2050 they will reach 2 Scholar in Washington, D.C. In particular, serious games
billion (in 2000 they used to be ‘only’ 600 million). Also, in can provide a way to increase engagement in a
2050, in Europe, the number of over 60 will equal the 40% rehabilitation therapy: exercises disguised as sessions of
of the total population, and the 60% of the population in serious gaming are more likely to be performed on a regular
basis and will not require the constant presence of a
therapist. Moreover, the scoring system may also offer a
quick and easy way to assess the rehabilitation path and/or
to raise real-time alarms.
Designing a serious game for rehabilitation [5, 11] as well
as defining a distributed infrastructure to support non-
GHItaly18: 2nd Workshop on Games-Human Interaction, May 29th, 2018,
invasive patient monitoring and remote assistance [12] are
Castiglione della Pescaia, Grosseto (Italy) not impossible challenges. Nevertheless, an important issue
Copyright © 2018 for the individual papers by the papers' authors. Copying still stands about how to easily define the right exercise for
permitted for private and academic purposes. This volume is published and
copyrighted by its editors. each patient and deploy it on a device located in the
households. Exercises should not be described using code
because that would be impractical for medical staff but, at
the same time, they must be easy to integrate in an existing
gaming environment.
With the above goal in mind, this paper is focused on
proposing a visual interface which will be viable for a
medical operator and where an exercise can be easily
defined while taking into account patient’s monitoring data.
This exercise can then be deployed remotely as a physiatric
serious game and be part of a long-term rehabilitation
therapy.
RELATED WORK
The idea of using games to foster medical therapies has
already been accepted long before computers and electronic
devices became a way to convey entertainment [1]. In
Figure 1: General Structure of Care@Home.
particular, in recent times, we have been witnessing a fair
number of projects and experimentations on this topic [3, 4, house. The created exercises may not be specific to a
6, 7, 8, 10, 11, 12, 13, 14]. medical condition and should be suitable for a number of
exergames. Our contribution in this paper is going to be an
In [13], authors design an exergame for post-surgery knee
extension to Care@Home and, for this reason, we will now
rehabilitation. The provided solution uses Kinect to monitor
present its framework in detail.
the patient and allows remote assistance, but the final
application is limited to assist on knee conditions. Other The Care@Home Project
contributions are focusing on post-stroke patients; in The Care@Home project has been developed to allow
particular, [8] proposes an exergame for upper-limbs remote interaction, on a daily basis, with elders (especially
rehabilitation. As in the previous case, authors are focused those living alone) requiring LTC. Care@Home integrates a
on a specific game rather than a more comprehensive number of sensor devices in the patient environment and
solution. Authors of [3] are also targeting after-stroke allows to assign exercises, verify progresses in mobility,
upper-limb rehabilitation, but a more comprehensive study and monitor health/environmental parameters. In particular,
is offered about the usefulness of Augmented Reality (AR) the Kinect gaming device has been exploited as a
in designing and deploying rehabilitation exergames. Other movement and body measurement sensor as well as for its
contributions [6, 7] are proposing exergames designed for easy integration in a gaming platform. A set-top-box is
wrist rehabilitation. In this case, while still limited to a installed in the elder home to manage the Kinect together
specific condition, authors try to exploit mobile phones to will all other environmental sensors and to interface the
increase accessibility and user experience. household with a caretaker’s control center (see Fig. 1).
Other contributions try to be more general and proposes full
frameworks [14] or general methodologies [4, 10] to
improve patient motivation and prove the all-around
effectiveness of games in rehabilitation therapies. This
group, anyway, seems to be more focused on monitoring
the patient and evaluating the effectiveness of a therapy
rather than actually hosting and dispensing exercises.
All the solutions referenced so far, are either implemented
around an existing game or focus on providing design
guidelines for exergames; none of them are proposing an
integrated environment to manage multiple exercises/games
in the context of a rehabilitation process taking place over a
long time.
A more comprehensive solution to host and manage
exergames, and to monitor patients’ performance from a
remote location is represented by Care@Home [11, 12].
Care@Home is a complete framework where a caretaker Figure 2: Care@Home hi-level architecture.
can interact with each patient using a remote application.
This remote application is capable to define exercises to be
proposed as games via a set-top-box located in the patient’s
example. Each exercise may implement its own checking
policy and focus on monitoring specific postures, which
may be critical for a given patient.
CREATING EXERCISES
As already mentioned, an important feature required from a
remote rehabilitation service is a convenient way to define
new exercises tailored on each patient’s needs and
performances. When designing such a tool and its interface,
we must also take into account that the creation of exercises
will be performed by a medical operator; this implies that
any description methodology requiring to write code is not
viable. Moreover, the physiotherapist must be able to
Figure 3: Tracking patient while performing an exercise. perform all operations of storage, retrieval, and deployment
Care@Home has been implemented as a distributed of exercises without asking the assistant of technical staff.
architecture whose purpose is to provide an efficient, The proposed solution
reliable, and scalable real-time communication between In order to create an efficient yet very intuitive interface,
sensing equipment located in the home of each patient and a we resolved to use a Natural User Interface (NUI) also on
central control system. A schema of the hi-level the caretaker side. As a consequence, Microsoft Kinect is
architecture is reported in Fig. 2. In the picture, we can used to let the caretaker define the tracking positions of
observe that this architecture is composed by many each exercise. The caretaker can track her own body to
components; some of them are located inside the define an exercise and then edit the result. This approach
caretaker’s organization datacenter (shaded area), while proved to be very well accepted by medical staff because
others are dislocated where convenient: either in the physiotherapists are already used to show to their patients
caretaker’s office or in the patient’s household. In how to perform each exercise. The only learning curve
particular, on the patient side, a DeviceProxy is in charge to required from the caretaker is about the interface for editing
manage the connection and collect data from the local and deployment, which is used in a second stage.
sensors. Moreover, the DeviceProxy is storing all the
information related to the exercises to be “dispensed” to the Once an exercise is decorated with all required metadata
patient. (e.g., number of repetitions, reference patient, and
description), it can be stored in an online database located
When an exercise is performed, the Kinect comes into play in the caretaker’s datacenter. From there, the PhysioServ
and starts tracking the patient’s position. A sequence of application server will take care to push the exercise toward
positions is presented on the screen as dots overlapped to the correct DeviceProxy.
the avatar reporting the current body position (see Fig. 3).
Each exercise requires from the patient to reach a sequence The caretaker’s database can also be used to retrieve
of positions with her body. If, while performing an exercise, existing exercises and tailor them for different patients due
to new, hopefully improved, conditions.
the tracking is reported too off of the required asset, the
dots are drawn in red, to provide immediate visual feedback Editor Architecture
and an alarm is raised on the screen; see Fig. 4 for an The exercise editor has been implemented using Windows
Presentation Foundation (WPF): a Kinect-compatible
framework for visual applications distributed by Microsoft.
In WPF, the interface is defined by means of an XML file
(taking xaml as extension). A specific class, usually
implemented in C#, manages the interface as described in
the xaml file and bridges the GUI to external elements and
the Kinect hardware. The hi-level architecture of the
exercise editor is reported in Fig. 5. In this architecture, the
Main Window class (with its xaml file) is responsible to
manage the main application panel. From within this panel,
we can initialize all data structures, perform data
management tasks, and drive Kinect sensors. The Main
Window interface exposes standard data management
Figure 4: Wrong posture reported: right hand should be functionalities for exercises (new, open, save, and so forth)
repositioned (avatar is represented as if in a mirror). as well as players control (start, stop, record).
Figure 5: General architecture of the exercise editor. Figure 6: Metadata Editor, panel to add additional
information to the exercise (some data have been
The Metadata Editor class is responsible to manage all obfuscated for privacy reasons).
additional data for the exercise. These data include both
contextual information, such as how many repetitions are
required and when the exercise should be performed, as
well as generic information such as medical notes from the
caretaker. Screenshots of the Metadata Editor windows are
proposed in Fig. 6 and Fig. 7. In the Metadata Editor, it is
also possible to select single joints (Fig. 7) in order to
provide additional directions for the exercise software.
These directions may be about a joint which is not supposed
to be moving or a limb bending which should not exceed a
given angle, as in the alarm raised in Fig. 4.
Both Metadata Editor and Main Window use two service
classes designed to store data about the whole exercise and
each single joint, respectively. Exercise and Joints will be
serialized and stored in a backend database for deployment
and/or later retrieval.
The last class in our architecture is Player. The Player class
takes care of all functionalities related to capturing the
caretaker’s movement using Kinect and replaying the edited Figure 7: Metadata Editor, joint selection
exercise before saving it in the backend database. for detailed editing.
Moreover, the Player can also retrieve an existing exercise
from the database and let the caretaker create a customized
version. Figure 8 shows the Player window during the
recording of an exercise. In the figure, it is possible to see a
left panel with a real-time tracking of the caretaker body
while performing the exercise (the red border means that
recording is in progress) and a right-hand panel with a
hierarchical representation of the exercises database in the
backend. The exercise database is available from the Player
window in order to ease the process of browsing and
playing out existing exercises. The database is represented
in a hierarchical way with a tree branching based on the
interested limbs or body part for each exercise. This data
organization proved to allow for quick retrieval from
medical staff without requiring to fill an explicit search
form. The last element of the Payer window, in its lower
section, is a selector to pick a subset of the joints which are
important for monitoring. This way, the recording will be Figure 8: Player window while capturing an exercise.
limited to the selected subset and the rest of the body will which is feasible for non-tech-savvy caretakers. This
not be used to evaluate the patient’s performance. Once application exploits Kinect on the medical personnel’s side
finished the recording, the Player window can be closed to to define an exercise via body tracking and let the user
perform further processing and deployment from the Main decorate it with metadata useful for archiving and
Window. deployment.
Creating and Managing Exercises The proposed solution is currently under testing and
The workflow, while creating an exercise, can be divided preliminary unstructured feedback from medical staff seems
into a sequence of steps. During the first step, the caretaker promising.
will record the required movement, select the relevant
joints, and add all important metadata to the exercise. A As a future extension, we are planning to implement an
serialized version of the Exercise instance can then be adaptation layer to let each exercise to be instantiated inside
saved locally in a binary format. This local save will be a generic game, as in the early prototype shown in Fig. 9. In
then pushed to the backend database and indexed using the this prototype, the player/patient is required to raise
metadata provided by the caretaker. alternatively the right and left leg for a given number of
times. Our intention is to insert several kinds of leg-related
Once in the backend database, the information will be exercises into this game without bounding movements or
managed by the PhysioServ application server: basing on requiring the modification of game rules and mechanics.
the indexing metadata, each exercise will be pushed to one
ACKNOWLEDGEMENTS
(or more) set-top-boxes in patients’ households when
We would like to thank our student Ana Maria Alexandru
connection will take place. We cannot assume the client
for her contribution in defining the first interface and
side of the architecture is always connected and sync is
guidelines for this application.
performed based on opportunity. When the exercise is
deployed on a set-top-box, the patient will see the Exercise REFERENCES
de-serialized and instantiated inside a rehabilitation 1. C. Abt. 1970. Serious Games. The Viking Press, New
exergame. York.
CONCLUSION AND FUTURE WORK 2. H. Afsarmanesh, S.S. Msanjila. 2010. ePAL Vision
In this paper we discussed about exergames to support 2020 for active ageing of senior professionals. In
rehabilitation of patient on LTC. While it is already very Proceedings of IFIP Advances in Information and
clear the importance of (video) games for rehabilitation, we Communication Technology (PRO-VE 2010), Vol.
are still missing a complete and streamlined solution to 336, pp. 60–72. DOI:10.1007/978-3-642-15961-9_6
dispense gaming activity as part of a therapy. In the context 3. J.W. Burke, M.D.J. McNeill, D.K. Charles, P.J.
of Care@Home, an infrastructure for remote administration Morrow, J.H. Crosbie, S.M. McDonough. 2010.
of LTC at home, we addressed here the problem of defining Augmented reality games for upper-limb stroke
exercises with a functional and easy-to-use application rehabilitation. In proceedings of 2nd International
Conference on Games and Virtual Worlds for Serious
Applications (VS-GAMES 2010). pp.75–78.
DOI:10.1109/VS-GAMES.2010.21.
4. J.W. Burke, M.D.J. McNeill, D.K. Charles, P.J.
Morrow, J.H. Crosbie, S.M. McDonough. 2009.
Serious games for upper limb rehabilitation following
stroke. In proceedings of the 2009 Conference in
Games and Virtual Worlds for Serious Applications
(VS-GAMES 2009). pp.103–110. DOI: 10.1109/VS-
GAMES.2009.17
5. N. D’Aquaro, D. Maggiorini, G. Mancuso, L. A.
Ripamonti, 2011. Videogames and Elders: a New Path
in LTC?. In proceedings International Conference on
Human-Computer Interaction (HCII 2011).
DOI:10.1007/978-3-642-21799-9_28
6. D. Deponti, D. Maggiorini, C.E. Palazzi. 2009.
DroidGlove: an android-based application for wrist
rehabilitation. In proceedings of International
Figure 9: Early prototype game for legs exercise (this
Conference on Ultra Modern Telecommunications
application is using standard assets from Unity3D).
(UMSA 2009). DOI:10.1109/ICUMT.2009.5345442.
7. D. Deponti, D. Maggiorini, C.E. Palazzi. 2011. 12. D. Maggiorini, L.A. Ripamonti, E. Zanon. 2012.
Smartphone’s physiatrist serious game. In proceedings Supporting Seniors Rehabilitation Through Videogame
of 1st International Conference on Serious Games and Technology: a Distributed Approach. In proceedings
Applications for Health (SeGAH 2011). DOI: International Workshop on Games and Software
10.1109/SeGAH.2011.6165434. Engineering (GAS 2012).
8. I. Di Loreto, G. Abdelkader, N. Hocine. 2011. Mixed DOI:10.1109/GAS.2012.6225920.
reality serious games for post-stroke rehabilitation. in 13. B. Maurer, F. Bergner, P. Kober, R. Baumgartner.
proceedings of 5th International Conference on 2012. Improving rehabilitation process after total knee
Pervasive Computing Technologies for Healthcare, replacement surgery through visual feedback and
pp.530–537. enhanced communication in a serious game. In
DOI:10.4108/icst.pervasivehealth.2011.246102. proceedings of the 30th ACM International Conference
9. C. Jeavans. 2004. Will we still be working at 70?. on Design of Communication. pp.355–356.
http://news.bbc.co.uk/2/hi/uk_news/4016969.stm DOI:10.1145/2379057.2379124.
(accessed April 3, 2018). 14. S. Saini, D.R.A. Rambli, S. Sulaiman, M.N. Zakaria,
10. E. lores, G. Tobon, E. Cavallaro, F.I. Cavallaro, J.C. S.R. Mohd Shukri. 2012. A low-cost game framework
Perry, T. Keller. 2008. Improving patient motivation in for a home-based stroke rehabilitation system. In
game development for motor deficit rehabilitation. In proceedings of 2012 International Conference on
proceedings of the 2008 International Conference on Computer and Information Science (ICCIS 2012).
Advances in Computer Entertainment Technology pp.55–60. DOI:10.1109/ICCISci.2012.6297212.
(ACE 2008). pp.381–384. DOI: 15. Serious Games Initiative.
10.1145/1501750.1501839 http://www.seriousgames.org/ (accessed April 3, 2018)
11. D. Maggiorini, L.A. Ripamonti, A. Scambia. 2012. 16. M. Stranges. 2008. Immigration as a remedy for
Videogame Technology to Support Seniors. In population decline? An overview of the European
proceedings International ICST Conference on countries. In European Papers on the New Welfare,
Simulation Tools and Techniques (SIMUTOOLS Special Issue on the Counter Ageing Society, Paper
2012). DOI:10.4108/icst.simutools.2012.247782 no. 8.