=Paper=
{{Paper
|id=Vol-3623/AIxAS_2023_paper_6
|storemode=property
|title=Enhancing upper limb mobility through gamified tasks and azure kinect: a preliminary study in post-stroke subjects
|pdfUrl=https://ceur-ws.org/Vol-3623/AIxAS_2023_paper_6.pdf
|volume=Vol-3623
|authors=Claudia Ferraris,Gianluca Amprimo,Luca Vismara,Alessandro Mauro,Giuseppe Pettiti
|dblpUrl=https://dblp.org/rec/conf/aixas/FerrarisAVMP23
}}
==Enhancing upper limb mobility through gamified tasks and azure kinect: a preliminary study in post-stroke subjects==
https://ceur-ws.org/Vol-3623/AIxAS_2023_paper_6.pdf
Enhancing upper limb mobility through gamified tasks
and Azure Kinect: a preliminary study in post-stroke
subjects
Claudia Ferraris1,*, Gianluca Amprimo1,2, Luca Vismara3, Alessandro Mauro3,4 and
Giuseppe Pettiti1
1 Consiglio Nazionale delle Ricerche (Institute of Electronics, Information Engineering and Telecommunication), Corso
Duca degli Abruzzi 24, Torino, 10129, Italy
2 Politecnico di Torino (Department of Control and Computer Engineering), Corso Duca degli Abruzzi 24, Torino, 10129,
Italy
3 Istituto Auxologico Italiano (Department of Neurology and Neurorehabilitation, S. Giuseppe Hospital), Strada Cadorna
90, Piancavallo, 28824, Italy
4 Università degli Studi di Torino (Department of Neurosciences), Via Cherasco 15, Torino, 10126, Italy
Abstract
Stroke is a leading cause of long-term disability as well as death worldwide, where aging is among the
most significant nonmodifiable factors. Motor impairments related to post-stroke hemiplegia, resulting
from the loss of specific brain functions following the acute event, commonly affect both upper and
lower limbs, leading to a deterioration in perceived quality of life as daily activities become unsafe and
difficult to perform. Various rehabilitation strategies and therapies are commonly adopted in the
hospital setting during the post-acute phase to recover, at least partially, major motor functions and
improve physical mobility to ensure the patients’ safety in daily life. However, these functions should be
stimulated continuously and frequently through maintenance activities in order not to lose the level of
functional recovery achieved and avoid subsequent hospitalizations for new rehabilitation treatments.
This paper proposes the use of gamified tasks in a virtual environment to enhance upper limb mobility.
Gamified tasks are performed using a single RGB-D camera-based vision system (specifically, Microsoft
Azure Kinect DK) suitable for easy deployment in home environments. Non-invasive body tracking
models are employed to capture 3D upper limb trajectories in real time and measure, through objective
parameters, the unilateral and bilateral movements required by each task. Preliminary results on a small
cohort of post-stroke subjects show a general progress in upper limb mobility and coordination, in
agreement with an improvement in some clinical severity scores and tests. This suggests that the
proposed solution is suitable for continuous stimulation of upper limb function and performance
monitoring over time in the home environment, contributing to the improvement of the patient's
general motor condition and increased physical well-being in daily life.
Keywords
Upper limb rehabilitation, Azure Kinect, home monitoring system, Artificial Intelligence 1
1. Introduction
Annual reports on stroke show an increasing incidence in the global population despite advances
in prevention, treatment, and wellness [1], [2]. Among the known risk conditions, age represents
one of the more critical non-modifiable factors causing incidence to double with age [3]. The
physical and neurological consequences of the acute event cause long-term functional deficits,
leading to a significant burden on the healthcare systems [4] and reduced quality of life for stroke
Proceedings Acronym: AIxAS 2023: Fourth Italian Workshop on Artificial Intelligence for an Ageing Society, 6-9
November 2023, Rome, Italy
claudia.ferraris@cnr.it (C. Ferraris); gianlucaamprimo@cnr.it (G. Amprimo); lucavisma@hotmail.com (L.
Vismara); alessandro.mauro@unito.it (A. Mauro); giuseppe.pettiti@cnr.it (G. Pettiti)
0000-0001-5381-4794 (C. Ferraris); 0000-0003-4061-8211 (G. Amprimo); 0000-0001-9034-7101 (L. Vismara);
0000-0001-9072-7454 (A. Mauro); 0000-0003-0547-0143 (G. Pettiti)
© 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
survivors [5]. Motor disabilities are common after stroke, limiting overall mobility with a direct
impact on activities in daily life and active social involvement [6]. One of the more disabling
conditions is the hemiparesis of the contralateral upper limb, which affects more than 80% of
stroke survivors, as acute or chronic limitation of mobility, control, and coordination and impairs
everyday daily actions (e.g., reaching and picking) [7].
Various complex rehabilitation treatments are promptly activated after the acute phase to
recover lost functions, activate compensatory strategies, and increase patients’ autonomy in daily
life. For example, rehabilitation protocols focus on gait, posture, and balance to avoid fall risks
and ensure patient safety [8]-[10]. Regarding upper limbs, several studies have shown that
exercise therapies play a crucial role in stroke rehabilitation, and several ad-hoc strategies are
planned (goal-oriented, task-oriented, repetitive task training) with different duration, workload,
and feedback to suit the patient’s condition [6]. For example, bilateral training is a recent
rehabilitation strategy that relies on the knowledge that the non-paretic upper limb can stimulate
the movement of the paretic upper limb during simultaneous movements, with significant
benefits on motor coordination [11].
In recent years, various technological solutions have been proposed for upper limb
rehabilitation of post-stroke addressing various degrees of motor impairment severity: these
include assistive devices [12] and robots [13][14], and more innovative methodologies such as
virtual reality [15][16], serious/exergames/gamification [13][17], and camera-based solutions
[18]-[22].
Along this line of research, we present an activity monitoring system, based on gamified tasks
and suitable for home environments, to stimulate upper limb mobility and promote
improvement/maintenance of motor functions with a focus on range of motion, motor control,
and coordination. To this end, gamified tasks propose unilateral and bilateral movements, with
configuration settings that take into account the subjects’ motor conditions. This work used one
of the platforms developed in the REHOME project [23], the Motor Rehabilitation and Exergames
platform (MREP) [24], which implements a suite of assessment tasks, gamified tasks, and
rehabilitative exergames for monitoring upper and lower limb performance in subjects with
motor and cognitive impairments originating from neurological disorders. Specifically, this
preliminary study considered two of the available gamified tasks and one of the assessment tasks
(i.e., walking) to focus only on upper limb mobility in post-stroke subjects and evaluate the
potential benefits for arm swing during walking [25]. Preliminary results confirm an
improvement in overall upper limb mobility for both the paretic and non-paretic sides in terms
of speed and number of movements per minute, increased movement coordination, and an
implicit adaptation of the non-paretic side to the performance of the paretic side, as stated in [6].
Moreover, these results agree with the overall improvement shown by the clinical scales at the
end of the experimental protocol and an objective reduction of arm swing asymmetry during
walking. This confirms that the proposed solution can stimulate movements and detect (monitor)
changes in upper limb performance over time. Considering the increased incidence of stroke
events associated with aging and that residual deficits require continual and prolonged
rehabilitation treatments, the activity monitoring system could be a helpful support tool in an
aging society, particularly in decentralizing specific healthcare services from hospitals to home
settings. In addition, it is important to note that, in this paper, we reported a specific application
case study of a system particularly suitable for the elderly population. Its main features and
versatility also make it relevant for healthy aging applications as a tool to stimulate motor
function in contrast to the functional physiological decline related to aging, thus promoting
physical exercises and early detection of functional alterations.
2. Materials and Methods
2.1. Tasks and vision system characteristics
In this study, only a specific subset of the tasks and exergames offered by MREP was analyzed,
choosing from the motor tasks most suitable for the recruited post-stroke volunteers. The
experimental protocol involved the following tasks:
• Gait (G): This task allows assessment of alterations in gait patterns, including
spatiotemporal features, dynamic stability, and rhythmic arm swings. Subjects were asked to
walk on a 6-m long path, facing the RGB-D camera, to the best of their ability. This setting allows
the estimation of relevant gait parameters over a relatively short area to be suitable for home
settings, as in [26]. The 6 m path devoted to the G task is undoubtedly more suitable for hospital
or outpatient settings with greater available spaces. However, the walking ability is analyzed over
a shorter distance (4.0-4.5 m), which is more suitable for home scenarios. In addition, the G task
was included in the experimental protocol only to demonstrate the effects of exergames on arm
swing during walking: only the gamified tasks could be proposed for home protocols, maintaining
the G task only for pre- and post-treatment comparison in clinical settings.
• Lateral Weightlifting (LWL): This task, commonly performed in physiotherapeutic
sessions, is offered in a gamified version (i.e., in a virtual gym environment) to assess the mobility
of the left and right upper limbs, as in [27]. Subjects were asked to perform, facing the RGB-D
camera, a predetermined number of lateral adduction/abduction movements with the arms to
the best of their ability in terms of range of motion and speed. This task allows estimating some
relevant mobility parameters of lateral movements related to each arm separately or both arms
simultaneously.
• Frontal Weightlifting (FWL): This task is designed like LWL [27]. The difference lies in the
type of movement required. In fact, subjects were asked to perform, facing the RGB-D camera, a
predetermined number of frontal up/down movements with their arms to the best of their ability
in terms of range of motion and speed. This task allows the estimation of relevant mobility
parameters of frontal movements related to each arm separately or both arms simultaneously.
Including both frontal and lateral arm movements allows to emphasize the differences in
execution and control movements in the two directions, since post-stroke survivors exhibit more
difficulty in the lateral direction, as often evidenced [28][29]. In addition, simultaneous bilateral
arm movements have shown the potential to reactivate the damaged hemisphere, contributing to
increased strength and motor function of the paretic limb [30]. FWL and LWL can be performed
in standing or sitting position, 2.5-3 meters away from the RGB-D camera, to address the patient’s
dynamic instability and ensure safety while performing the task. In addition, both gamified tasks
are customizable through a configuration file (i.e., number of movements or minimum arm angle)
to cope with the patient’s condition. For now, the configuration file is set by therapists; however,
in the next future, it can be set by automatic artificial intelligence algorithms based on the
assessed performance of patients and progress over time. In addition, several gamification
elements were considered during the design phase of the gamified tasks to enhance patient
engagement and experience. For example, the game scenario (virtual gym environment) allows
participants to be immediately involved in the exercise to be performed (weightlifting) and to
identify with the virtual character (avatar). In addition, using the body to control the avatar
without external aids or devices encourages simple, active, and autonomous participation by
providing immediate visual feedback of the interaction with the game. Sounds enrich the game
by highlighting the movements performed, thus increasing emotional engagement. At the same
time, text and voice messages (via text-to-speech functions) complement the user interface and
guide participants in completing tasks and levels. In the future, gamified tasks will be enriched
with other gamification elements (point rewards, timed challenges, new levels) to reward
performance and encourage treatment continuity in the medium and long term.
As mentioned earlier, the selected tasks were proposed through a vision system based on the
Azure Kinect [31] and the non-invasive 3D body tracking library that leverages Deep Learning
methodologies [32]. SDKs for using Azure Kinect facilities, available in C++, were first ported to
the Unity (C#-based) environment, the game engine used to design and develop the entire MREP
exercises suite. The body tracking library was used both to capture body movements and interact
with the game scenario in real time and non-invasive manner by analyzing the trajectories of
specific joints among the 32 available that make up the 3D skeletal model. The user interface,
consisting of text messages and audio support, guides the user in the execution of all tasks but
also allows a supervisor to intervene, if necessary, by starting and stopping the proposed
exercises [27].
For this experimental study, a ZOTAC© ZBOX EN52060-V (16 GB RAM, NVIDIA GeForce RTX
2060 6GB, 9th generation 2.4GHz quad-core processor) was used to run the MREP software, while
Azure Kinect was configured as follows: 30 fps for both color and depth streams, 1080p
resolution for the depth stream, and Narrow Field of View (NFV) to capture body movements
farther from the camera and with a wider frontal viewing angle to ensure optimal body tracking
[33].
2.2. Participants and experimental protocol
For this preliminary study, a small cohort of 11 volunteer post-stroke participants was recruited
from the Division of Neurology and Neurorehabilitation at San Giuseppe Hospital (Istituto
Auxologico Italiano, Piancavallo, Verbania, Italy). Participants were post-acute or subacute
stroke, with hemiparesis on one side of the body (six on the right and five on the left), with minor
disability of the upper and lower limbs (ability to walk). The only exclusion criterion was
cognitive impairment with Mini-Mental State Examination (MMSE)<26. No exclusion criteria
related to age, sex, side, dominance, or therapy were adopted. The local ethics committee
approved the study as part of the REHOME project. All participants were instructed on the
experimental protocol and instrumentation. Then, they signed an informed consent before being
admitted to the study.
The experimental protocol included an initial clinical assessment session (T0) in which clinical
staff assessed general motor status using traditional scales and functional tests commonly used
in post-stroke. These included the Berg Balance Scale [34], Trunk Impairment Test scale (TIS)
[35], Time Up-and-Go test (TUG) [36], and shoulder joint mobility assessment [37]. In the same
session, the instrumental gait motor task (G) was proposed to participants to assess gait
information before starting the subsequent sessions based on gamified tasks. The same
assessment was repeated at the end of the gamified sessions to compare the motor condition
before and after the overall protocol (TF). The gamified sessions were organized over two weeks,
three sessions per week, for a total of six sessions (R1-R6).
The experimental protocol was administered to all participants under the same environmental
conditions and the supervision of the clinical staff. All participants were able to complete the
experimental protocol correctly and as planned, except for one subject who withdrew after the
second gamified session and was therefore excluded from the subsequent analysis.
2.3. Functional parameters and data analysis
Data analysis was performed with MATLAB® from the 3D trajectories of specific joints of the
skeletal model collected during the three selected motor tasks (i.e., G, LWL, and FWL). Initial pre-
processing was applied to all skeletal model joints, which included a resampling procedure
(50Hz) to remove frame rate jittering in the camera acquisition phase and low-pass filtering
(5Hz) to focus on the voluntary motion frequency band and remove high-frequency noise
interference. The resampled and filtered trajectories were then used to estimate ad-hoc
functional measures.
Concerning G, several traditional spatiotemporal parameters were estimated, as well as
parameters related to dynamic stability and arm swing during walking. The methodological
approach to gait analysis was the same as in [25][26], where forward and backward arm swing
trajectories were estimated with respect to the trunk segment in the walking direction. Mean gait
parameters were estimated at T0 and TF to detect the improvement in performance at the end of
the experimental protocol in agreement with the clinical tests.
For LWL and FWL, parameters were estimated from specific body segments determined using
some skeletal model joints. The following body segments were considered: upper limb segment
between the wrist and clavicle joints (UPPL); trunk segment between the neck and pelvis joints
(TRUNK); arm segment between the clavicle and elbow joints (ARM); and forearm segment
between the elbow and wrist joints (FORE). Angle measurements were determined between the
UPPL and TRUNK segments (upper limb angle) and between the ARM and FORE segments (elbow
angle). Based on the movements required by the gamified tasks, the upper limb angle was
estimated in the corresponding movement axes: sagittal axis for LWL (adduction-abduction
movements) and transversal axis for FWL (up-down movements). Figure 1 shows the location of
the joints and body segments involved in the data analysis for LWL and FWL.
Figure 1: Position of joints (and relative body segments) for the gamified tasks: pelvis
(magenta), neck (cyan), clavicles (orange), elbows (blue), and wrists (green).
Other secondary parameters, in particular speed and rate, were estimated from the primary
angular measures. Angular measures were estimated for paretic and non-paretic limbs for both
unilateral and bilateral execution. The complete list of functional parameters considered for this
study is given in Table 1.
Table 1
List of parameters and metrics considered for the study.
Task Parameter Meaning and unit
1
G SPEEDG Gait speed on the walking path (m/s)
STEPLG Step length (m)
STANCEG Stance phase (% of gait cycle)
TSWAYG1 Trunk medio-lateral sway (mm)
ARMSWG Maximum arm swing angle (deg)
ARMSYMG1 Symmetry of arm swing angle (-)
FWL UPANGFWL Max upper limb angle of flexion-extension movements (deg)
ELANGFWL Mean elbow angle (deg)
SPEEDFWL Mean speed (deg/s)
RATEFWL Number of movements per minute (mov/min)
1,2
SYNCFWL Synchronicity index (-)
SIMILFWL1,2 Similarity index (-)
LWL UPANGLWL Max upper limb angle of abduction-adduction movements (deg)
ELANGLWL Mean elbow angle (deg)
SPEEDLWL Mean speed (deg/s)
RATELWL Number of movements per minute (mov/min)
1,2
SYNCLWL Synchronicity index (-)
SIMILLWL1,2 Similarity index (-)
1 Parameters that refer to the overall task. All the other parameters are computed separately for the affected and non-affected side.
2 Parameters estimated only for the bilateral execution of FWL and LWL.
The ARMSYMG is an index assessed to highlight arm swing asymmetry during walking. It was
calculated as in [25]: more negative values indicate more pronounced asymmetry between the
maximum swing angles of the upper limbs.
The SYNC and SIMIL metrics (for both LWL and FWL tasks) aim to highlight the differences, in
terms of temporal and spatial execution, between the 3D trajectories of the upper limbs during
simultaneous bilateral movements. These summary indices provide an immediate indication to
monitor the improvement of motor control, symmetry, and coordination in post-stroke subjects
along with the other single-arm parameters. The SYNC metric is defined as in [27] and considers
the time lag between upper limb trajectories above and below the minimum angular threshold
configured for the exercise, finding correspondence in bilateral movement cycles. According to
its definition, values close to 0 indicate bilateral movements with good time synchronization;
increasing values indicate unsynchronized bilateral movements. The SIMIL metric refers to the
similarity of the 2D closed shapes that enclose the trajectories drawn by the upper limbs (in
particular, WRIST joint), according to the two main directions of motion, with respect to a
reference point (in this case, NECK joint). It is estimated using Procrustes analysis [38]
implemented in MATLAB (procrustes function) that returns an index of dissimilarity between the
shapes that enclose left and right trajectories. The scaling parameter of the procrustes function
was disabled to maintain information on any different excursion between the paretic and non-
paretic sides. According to its definition, values close to 0 indicate bilateral movements with good
shape similarity; increasing values indicate dissimilar shapes during bilateral movements. Figure
2 shows an example of the shapes drawn during the LWL simultaneous execution.
Mean parameters and metrics were estimated for the first and second weeks to detect
performance improvement and trends in the gamified tasks.
Figure 2: Example of enclosing shapes (green for left and cyan for right upper limbs) drawn
during simultaneous bilateral movements for LWL referred to left-right and up-down directions
(X and Y axes of Azure Kinect). A good bilateral execution (left) with similar shapes; an impaired
execution (right) with dissimilar shapes.
3. Results
3.1. Participants clinical and demographic data
The clinical and demographic characteristics of the participants who correctly completed the
experimental protocol (10 participants) are shown in Table 2.
Table 2
Clinical (T0) and demographic information about the participants
Information Value
Age (years) 72.0 ± 10.5
Gender (male / female) 8/2
Time from acute event (years) 6.3± 5.1
Affected side (left / right) 3/7
Weight (kg) 77.40 ± 14.30
Berg score (pts) 36.60 ± 15.74
TIS score (pts) 11.00 ± 3.83
TUG (s) 26.76 ± 15.82
Paretic Shoulder Mobility (deg) 132.22 ± 33.46
Five subjects habitually use assistive devices during walking (tripod, walking stick). However,
they were all able to amble without them during task G. The same participants preferred to
perform gamified sessions in a sitting position. All subjects performed the proposed instrumental
and gamified tasks correctly, as scheduled by the experimental protocol. At the end of the study,
20 G sessions, 60 LWL sessions, and 60 FWL sessions were available for data analysis. Regarding
LWL and FWL, it is important to remember that, for each gamified task, 60 trials were collected
for the non-paretic arm, 60 trials for the paretic arm, and 60 trials for both arms simultaneously.
One subject was not able to complete the bilateral tasks in most of the planned sessions: the data
were discarded, so only 54 trials were considered for the analysis of LWL and FWL bilateral tasks.
Data analysis revealed that all the clinical metrics (TIS, TUG, BERG, and paretic shoulder
mobility) indicate an overall improvement in motor performance at the end of the experimental
protocol (TF). Specifically, TIS score increased (i.e., improved) by 20.9% (TF=13.30±4.30 pts);
TUG time decreased (i.e., improved) by 13.4% (TF=23.17±15.28 s); BERG score increased (i.e.,
improved) by 9.8% (TF=23.17±15.92 pts); and paretic shoulder mobility increased (i.e.,
improved) by 11.76% (TF=147.78±29.49 deg).
3.2. Results on gait analysis
This analysis aims to show the differences in mean G parameters between T0 and TF at the end
of the experimental protocol. Table 3 shows the percentage changes over the cohort of
participants.
Table 3
Percentage change in mean gait (G) parameters over all participants: T0 vs. TF
Parameter T0 TF Var (%)
SPEEDG (m/sec) 0.50±0.25 0.45±0.23 -8.5%
STEPLG (m) 0.36±0.15 0.35±0.13 -2.9%1
STANCEG (%) 76.97±12.09 76.62±8.36 -0.4%1
TSWAYG (mm) 107.91±20.00 111.07±36.54 2.9%
ARMSWG (deg) 40.63±19.35 33.73±20.93 -16.5%1
ARMSYMG (-) -16.31±13.70 -12.54±9.76 -22.6%
1
Mean of paretic and non-paretic side.
The results reveal no substantial differences in gait patterns, whose parameters are relatively
stable. There is a slight reduction in walking speed (-8.5%) and step length (-2.9%), while the
stance phase shows minimal improvement (-0.4% in stance phase duration). The dynamic
stability (TSWAYG) shows a minimal inter-group worsening (+2.93% of instability). Regarding
the arm swing, the maximum swing angle decreased for both sides (-16.5% on average).
However, interestingly, this result is associated with a significant overall reduction in arm swing
asymmetry (T0=-16.21±13.70, TF=-12.54±9.76), suggesting lower amplitude but greater
coordination in arm swing movements. Considering that the gamified tasks stimulate upper limb
movements, the result on arm swing asymmetry is consistent with the proposed exercises and
seems to confirm an overall practical benefit for the upper limb. The result on asymmetry agrees
with the mean improvement in shoulder mobility assessed by clinicians at TF (T0=132.22 ± 33.46
deg vs. TF=147.78±29.49 deg). However, it should be considered that the participants had
different clinical pictures, and each of them responded differently to the experimental protocol.
This justifies the stable gait parameters, suggesting the need for ad-hoc gamified tasks for the
lower limbs and balance to appreciate the same improvement obtained on arm swing asymmetry.
3.3. Results on LWL and FWL: unilateral execution
This analysis aims to detect trends in FWL and LWL parameters between the first and second
weeks of the experimental protocol. Table 4 shows the percentage changes over the cohort of
participants.
Table 4
Trends of FWL and LWL parameters in all participants: unilateral movements
Paretic Arm Non-paretic Arm
Parameter Week 1 Week 2 Var (%) Week 1 Week 2 Var (%)
UPANGFWL (deg) 103.03 103.00 -0.1% 124.60 125.84 +1.0%
ELANGFWL (deg) 123.18 122.57 -0.5% 138.51 138.14 -0.3%
SPEEDFWL (deg/s) 62.04 74.16 +19.5% 78.88 83.41 +5.7%
RATEFWL (mov/min) 16.14 22.35 +38.5% 15.97 21.54 +34.9%
UPANGLWL (deg) 91.45 93.04 +1.7% 117.41 119.60 +1.9%
ELANGLWL (deg) 125.87 128.61 +2.2% 145.81 142.39 -2.3%
SPEEDLWL (deg/s) 61.71 68.88 +11.6% 79.82 89.68 +12.3%
RATELWL (mov/min) 18.88 22.64 +19.9% 19.38 25.18 +29.9%
As expected, the mean parameters estimated from upper limb movements show a significant
difference between the paretic and non-paretic sides for FWL and LWL. The comparison shows a
significant improvement in the velocity parameters, both in terms of movement speed and
number of movements per minute. The improvement is substantial for both gamified tasks,
especially for RATEFWL and RATELWL. In contrast, the other parameters show negligible variation
between the two weeks. The results also suggest that the frontal movements (in FWL) allowed
higher upper limb angles (UPANGFWL > UPANGLWL) than the LWL. On the contrary, the lateral
movements (in LWL) facilitated the maintenance of adequate upper limb extension, as suggested
by the higher elbow angles (ELANGLWL > ELANGFWL). The results confirm a positive trend for all
participants in upper limb motor performance for both the paretic and non-paretic sides,
suggesting that prolonged treatment could produce many benefits to upper limb mobility with
positive effects on overall motor condition. Again, the results, particularly on the paretic arm,
agree with the average improvement in shoulder mobility assessed by clinicians at the end of the
experimental protocol (T0=132.22 ± 33.46 deg vs. TF=147.78±29.49 deg).
3.4. Results on LWL and FWL: bilateral execution
This analysis aims to detect trends in FWL and LWL parameters between the first and second
weeks of the experimental protocol. Table 5 shows the percentage changes over the cohort of
participants.
Table 5
Trends of FWL and LWL parameters in all participants: bilateral movements
Paretic Arm Non-paretic Arm
Parameter Week 1 Week 2 Var (%) Week 1 Week 2 Var (%)
UPANGFWL (deg) 105.08 110.22 +4.9% 118.16 119.36 +1.0%
ELANGFWL (deg) 128.57 128.31 -0.3% 137.95 135.23 -2.0%
SPEEDFWL (deg/s) 62.64 82.19 +31.2% 61.46 87.07 +41.7%
RATEFWL (mov/min) 15.50 21.34 +37.7% 15.67 21.53 +37.4%
UPANGLWL (deg) 85.45 82.06 -4.0% 109.14 106.56 -2.4%
ELANGLWL (deg) 125.04 129.13 +3.3% 139.79 140.92 +0.8%
SPEEDLWL (deg/s) 50.83 62.83 +23.6% 68.86 85.02 +23.5%
RATELWL (mov/min) 17.53 22.62 +29.0% 17.90 22.87 +27.8%
The results in Table 5 confirm the same outcome observed for unilateral execution, with
relevant improvement in movement speed and number of movements in a more complex
execution that demands motor control and coordination. This suggests that also bilateral tasks
confirm previous indications of increased shoulder joint mobility evidenced by clinical evaluation
in TF. Another significant outcome derives from the analysis of SYNC and SIMIL metrics (Table
6).
Table 6
Bilateral execution: metrics
Metric Week 1 Week 2 Var (%)
SYNCFWL 0.27 0.14 -49.2%
SIMILFWL 0.21 0.21 -0.2%
SYNCLWL 0.28 0.31 +7.5%
SIMILLWL 0.98 1.03 +4.6%
Table 6 highlights a significant improvement (SYNC index reduced by 49.2%) in temporal
synchronization of bilateral movements for FWL and a slight deterioration for LWL (+7.5%).
However, in both gamified tasks, the SYNC index is relatively low (<0.4). In contrast, only a slight
improvement (-0.2%) in the SIMIL index has been observed for FWL and a slight deterioration
(+4.6%) for LWL. Conversely from FWL, in LWL, the SIMIL index suggests more dissimilarities
between the shapes drawn during the movements, denoting a greater general difficulty in
coordinating movements during simultaneous lateral execution.
Another interesting outcome can be observed by comparing the upper limb performance
during unilateral and bilateral execution (Table 7).
Table 7
Bilateral execution: comparison of UPANG for unilateral and bilateral execution
UPANGFWL UPANGLWL
Side Unilateral Bilateral Var (%) Unilateral Bilateral Var (%)
Paretic (Week 1) 103.03 105.08 +2.0% 91.45 85.45 -6.6%
Paretic (Week 2) 103.00 110.22 +7.0% 93.04 82.06 -11.8%
Non-paretic (Week 1) 124.60 118.16 -5.2% 117.41 109.14 -7.0%
Non-paretic (Week 2) 125.84 119.36 -5.2% 119.60 106.56 -10.9%
As Table 7 shows, during the bilateral execution, the maximum angle of the upper limb is less
than the angle of unilateral execution for all conditions examined except for the paretic arm in
FWL. This emphasizes the greater complexity of simultaneous movement execution and control.
In addition, a form of implicit adaptation emerges from the analysis, in which the non-paretic arm
appears to adapt to the performance of the paretic. This behavior could be reversed by extending
gamified sessions for a longer period. However, the results seem to confirm a positive trend for
all participants in upper limb motor performance, even in bilateral execution, suggesting that
prolonged treatment could produce many benefits for upper limb control and coordination, with
consequent positive effects on overall motor condition.
4. Conclusions
This study investigated the potential of gamified tasks (i.e., exergames) as an easy-to-use and
engaging tool for improving upper limb mobility. As a case study, a small cohort of post-stroke
subjects was involved in a two-week experimental protocol that included six training sessions
with gamified tasks in a virtual environment to enhance user experience and engage participants
in a fun and playful real-world scenario. The gamified tasks were proposed through a vision
system based on a single RGB-D camera (specifically, Microsoft Azure Kinect DK) and its
innovative body tracking algorithm that relies on deep learning approaches. This solution was
developed as part of the REHOME project, with the primary objective of designing a
telerehabilitation and telemonitoring platform, thus suitable for the home environment and for
people with motor and cognitive deficits related to neurological disorders.
Post-stroke is one of the physical conditions that could benefit from this type of solution.
Stroke survivors promptly undergo in-hospital rehabilitation after the acute phase to begin
recovery of motor functions impaired by the event as soon as possible. Despite this, most patients
would need continuous and frequent maintenance activities to avoid losing the functional
recovery achieved, but this is not feasible in a hospital setting. Telemonitoring and
telerehabilitation solutions could fill this gap, and exergames could prove to be important in
ensuring continuity of treatment, facilitating the execution of specific physical exercises,
stimulating the achievement of new rehabilitation goals, and ensuring greater adherence to
treatment through a fun and engaging approach.
This study focused on upper limb mobility, stimulated through two gamified tasks requiring
the execution of arm-lifting movements. To exert joint mobility, frontal and lateral lifting
movements were included to stress the upper limb motor function differentially. In addition, the
gamified tasks proposed unilateral (i.e., with only one arm at a time) and bilateral (i.e., with
simultaneous and synchronized movements of both arms) execution modes to solicit not only
range of motion but also motor control and coordination. Another relevant feature of the gamified
tasks is their reconfigurability according to the subject's motor condition: when motor function
improves, a higher level can be set (e.g., the number of movements required or the minimum
amplitude of movements) or, conversely, the level can be reduced if the patient shows difficulty.
During the experimental study, the clinical supervisor increased the game level for some
participants by augmenting the number of arm-lifting movements for paretic and non-paretic
arms. In contrast, the game level remained unchanged for other more impaired participants
throughout the treatment.
Regarding motor condition, the clinical evaluation at the end of the experimental protocol
indicates an overall improvement in the participants, resulting from the clinical scales and tests.
The improvement relates to several motor functions, including shoulder joint mobility, posture
(TIS scale), balance (BERG scale), and walking (TUG test), as discussed in Section 3.1.
The results for gamified tasks follow this trend as expected, especially for upper limb mobility,
since gamified tasks exclusively solicit the upper limbs. Regarding unilateral execution, the most
significant improvement is related to velocity parameters in terms of speed and rate (i.e., number
of movements per minute) for both frontal and lateral execution. The parameters related to upper
limb and elbow angles are relatively stable (Table 4). However, this result is also clinically
relevant, as it was obtained with a significant increase in execution speed. Regarding bilateral
execution, the same trend was observed, with significant improvement in velocity parameters
and stability in angle parameters (Table 5). In addition, the synchronization metric for FWL
shows a relevant improvement in the second week: the same is not true for LWL (Table 6),
probably due to the greater difficulty in motor coordination during lateral execution. Finally, the
comparison of unilateral and bilateral executions highlights an implicit adaptation of the non-
paretic arm to the performance of the paretic arm, as suggested by a lower upper limb angle
(Table 7).
The improvements in upper limb performance reflect the significant reduction in arm swing
asymmetry during walking. In contrast, the other traditional gait analysis parameters appear
stable at the end of the two weeks: this was expected, however, since only the upper limbs were
directly stressed by this experimental protocol. Moreover, the results seem to indicate only
partial improvement of motor condition and in specific domains, in contrast to clinical
assessments that show overall improvement. However, two aspects must be kept in mind: 1)
instrumental assessment measures and quantifies specific parameters and does not provide a
qualitative assessment of performance as with clinical scales; 2) a more extended protocol would
probably be needed to appreciate the same improvements in terms of measurement of individual
parameters.
Nevertheless, the results obtained are positive and encouraging, especially from the
perspective of using the proposed solution as a tool for monitoring and training/maintenance of
motor function in the home environment. However, it will be necessary to extend the analysis to
a larger group of subjects, not necessarily post-stroke, and over a more extended period to
confirm the effectiveness of the proposed solution. In addition, in future studies, we will evaluate
the possibility of automatically configuring gamified tasks through artificial intelligence
algorithms that consider the subject's condition and motor performance to adjust game levels
appropriately, avoiding emotional stress (anxiety, distrust, demoralization) but stimulating the
subject to improve constantly. Future developments will also include the integration of new
gamified tasks for hand dexterity to enhance the full motor function of the upper limb. As
mentioned earlier, the purpose of this study was to evaluate a trend toward improvement in
upper limb motor function using ad-hoc gamified tasks, continuing the exploration of the
potential of such innovative approaches to support traditional physiotherapy treatments, as
evidenced by several studies and reviews in the literature [39]-[42]. However, a point-by-point
comparison with other studies is not possible, mainly because of the different protocols,
participants, games, and motor functions elicited. In addition, many of these studies are clinical
trials that measure the effectiveness of exergames in improving motor performance only through
pre- and post-treatment clinical scales and not through the comparison of functional parameters
estimated directly from the exergames, as in our case. In conclusion, specific and quantifiable
potential benefits emerge from the presented study, especially for remote follow-up, in line with
the state of the art, current trends, and future perspectives highlighted by several recent studies
in the literature.
Acknowledgements
This work was supported by the project “ReHOME – ICT solutions for tele-rehabilitation of
cognitive and motor disabilities in neurological disorders”, grant from ROP Piemonte (Italy), POR
FESR 2014-2020.
References
[1] J. Kim, T. Thayabaranathan, G.A. Donnan, et al. ”Global Stroke Statistics 2019”. International
Journal of Stroke (2020), 15(8):819-838.
[2] A. Bustamante, T. García-Berrocoso, N. Rodriguez, V. Llombart, M. Ribó, C. Molina, and J.
Montaner. ”Ischemic stroke outcome: A review of the influence of post-stroke complications
within the different scenarios of stroke care”. Eur J Intern Med. (2016) 29:9-21.
[3] A.K. Boehme, C. Esenwa, and M.S. Elkind. ”Stroke Risk Factors, Genetics, and Prevention”. Circ
Res. (2017) 120(3):472-495.
[4] M. Katan and A. Luft. ”Global Burden of Stroke”. Semin Neurol. (2018) 38(2):208-211.
[5] M.J.M. Ramos-Lima, I.C. Brasileiro, T.L. Lima, and P. Braga-Neto. ”Quality of life after stroke:
impact of clinical and sociodemographic factors”. Clinics (2018) 73: e418.
[6] S.M. Hatem, G. Saussez, M. Della Faille, V. Prist, X. Zhang, D. Dispa, and Y. Bleyenheuft.
”Rehabilitation of Motor Function after Stroke: A Multiple Systematic Review Focused on
Techniques to Stimulate Upper Extremity Recovery”. Front Hum Neurosci. (2016)
13:10:442.
[7] Y. Bleyenheuft and A.M. Gordon. ”Precision grip in congenital and acquired hemiparesis:
similarities in impairments and implications for neurorehabilitation”. Front. Hum.
Neurosci. (2014) 8:459.
[8] C. Beyaert, R. Vasa, and G.E. Frykberg. ”Gait post-stroke: Pathophysiology and rehabilitation
strategies”. Neurophysiol Clin. (2015) 45(4-5): 335-55.
[9] A. Darekar, B.J. McFadyen, A. Lamontagne, and J. Fung. ”Efficacy of virtual reality-based
intervention on balance and mobility disorders post-stroke: a scoping review”. J Neuroeng
Rehabil. (2015) 10: 12:46.
[10] C.L. Yang, R.A. Creath, L. Magder, M.W. Rogers, and S. McCombe Waller. ”Impaired posture,
movement preparation, and execution during both paretic and nonparetic reaching
following stroke.” J Neurophysiol. (2019) 121(4): 1465-1477
[11] H. Akremi, J. Higgins, R. Aissaoui, and S. Nadeau. ”Bilateral motor coordination during upper
limb symmetric pushing movements at two levels of force resistance in healthy and post-
stroke individuals.” Hum Mov Sci. (2022) 81: 102913
[12] I.M. Barbosa, P.R. Alves, and Z.C. Silveira. ”Upper limbs’ assistive devices for stroke
rehabilitation: a systematic review on design engineering solutions.” J Braz. Soc. Mech. Sci.
Eng. (2021) 43: 236
[13] A. Adomavičienė, K. Daunoravičienė, R. Kubilius, L. Varžaitytė, and J. Raistenskis. ”Influence
of New Technologies on Post-Stroke Rehabilitation: A Comparison of Armeo Spring to the
Kinect System”. Medicina (2019), 55: 98.
[14] H.M. Qassim, and W.Z. Wan Hasan. ”A Review on Upper Limb Rehabilitation Robots”. Appl.
Sci. (2020), 10: 6976.
[15] R. G. Lupu, N. Botezatu, F. Ungureanu, D. Ignat, and A. Moldoveanu, "Virtual reality based
stroke recovery for upper limbs using leap motion," 2016 20th International Conference on
System Theory, Control and Computing (ICSTCC), Sinaia, Romania, 2016. 295-299.
[16] W.-S. Kim, S. Cho, J. Ku, Y. Kim, K. Lee, et al. ”Clinical Application of Virtual Reality for Upper
Limb Motor Rehabilitation in Stroke: Review of Technologies and Clinical Evidence”. J. Clin.
Med. (2020), 9: 3369.
[17] Reis, A. et al. ”Developing a System for Post-Stroke Rehabilitation: An Exergames Approach”.
In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. Users
and Context Diversity. UAHCI 2016. Lecture Notes in Computer Science (2016), 9739.
Springer, Cham.
[18] A. Dash, A. Yadav, A. Chauhan, and U. Lahiri. ”Kinect-Assisted Performance-Sensitive Upper
Limb Exercise Platform for Post-stroke Survivors”. Front Neurosci. (2019) 13: 228.
[19] A. Scano, F. Molteni, and L. Molinari Tosatti. ”Low-Cost Tracking Systems Allow Fine
Biomechanical Evaluation of Upper-Limb Daily-Life Gestures in Healthy People and Post-
Stroke Patients”. Sensors (2019). 19(5): 1224.
[20] M. Weiss Cohen, and D. Regazzoni. ”Hand rehabilitation assessment system using leap
motion controller.” AI & Soc (2020) 35: 581–594.
[21] L. Vismara, C. Ferraris, V. Votta, et al. ”Tele-Monitoring and Tele-Rehabilitation of the Hand
in Hemiplegic Patients: A Preliminary Study”. In: Bettelli, A., Monteriù, A., Gamberini, L. (eds)
Ambient Assisted Living. ForItAAL 2020. Lecture Notes in Electrical Engineering (2022) vol
884. Springer, Cham.
[22] G. Fuertes Muñoz, R.A. Mollineda, J. Gallardo Casero, and F. Pla. ”A RGBD-Based Interactive
System for Gaming-Driven Rehabilitation of Upper Limbs”. Sensors (2019) 19(16): 3478.
[23] C. Ferraris, et al. "Telerehabilitation of cognitive, motor and sleep disorders in neurological
pathologies: the REHOME project" 2022 IEEE Symposium on Computers and
Communications (ISCC), Rhodes, Greece (2022):1-6.
[24] C. Ferraris, I. Ronga, R. Pratola, G. Coppo, et al. ”Usability of the REHOME Solution for the
Telerehabilitation in Neurological Diseases: Preliminary Results on Motor and Cognitive
Platforms.” Sensors (2022) 22: 9467.
[25] C. Ferraris, G. Amprimo, G. Masi, L. Vismara, et al. ”Evaluation of Arm Swing Features and
Asymmetry during Gait in Parkinson's Disease Using the Azure Kinect Sensor”. Sensors
(2022) 22(16):6282.
[26] V. Cimolin, L. Vismara, C. Ferraris, G. Amprimo, G. Pettiti, et al. ”Computation of Gait
Parameters in Post Stroke and Parkinson's Disease: A Comparative Study Using RGB-D
Sensors and Optoelectronic Systems.” Sensors (2022) 22(3): 824.
[27] G. Amprimo, G. Masi, L. Priano, et al. ”Assessment Tasks and Virtual Exergames for Remote
Monitoring of Parkinson’s Disease: An Integrated Approach Based on Azure
Kinect.” Sensors (2022) 22: 8173.
[28] G. Marchesi, G. Ballardini, L. Barone, P. Giannoni, et al. ”Modified Functional Reach Test:
Upper-Body Kinematics and Muscular Activity in Chronic Stroke Survivors.” Sensors (2021)
22(1): 230.
[29] A. Prat-Luri, P. Moreno-Navarro, J.A. García, D. Barbado, F.J. Vera-Garcia, J.L.L. Elvira. ”Do
Initial Trunk Impairment, Age, Intervention Onset, and Training Volume Modulate the
Effectiveness of Additional Trunk Exercise Programs after Stroke? A Systematic Review with
Meta-Analyses.” Int J Environ Res Public Health. (2020) 17(23): 8714.
[30] R. Bruchez, M. Jequier Gygax, S. Roches, J. Fluss, D. Jacquier, et al. ”Mirror therapy in children
with hemiparesis: a randomized observer-blinded trial”. Dev Med Child Neurol. (2016)
58(9): 970-8.
[31] Azure Kinect DK. URL: https://azure.microsoft.com/it-it/products/kinect-dk/ (accessed on
September 2023)
[32] Liu, Z. 3D Skeletal Tracking on Azure Kinect. 2019. URL: https://www.microsoft.com/en-
us/research/uploads/prod/2020/01/AKBTSDK.pdf (accessed on September 2023).
[33] L.F. Yeung, Z. Yang, K.C. Cheng, D. Du, and R.K. Tong. ”Effects of camera viewing angles on
tracking kinematic gait patterns using Azure Kinect, Kinect v2 and Orbbec Astra Pro v2.” Gait
Posture (2021) 87: 19-26.
[34] M. Kudlac, J. Sabol, K. Kaiser, C. Kane, and R.S. Phillips. ”Reliability and Validity of the Berg
Balance Scale in the Stroke Population: A Systematic Review.” Phys. Occup. Ther.
Geriatr. (2019) 37: 196–221
[35] G. Verheyden, A. Nieuwboer, J. Mertin, R. Preger, C. Kiekens and W. De Weerdt. ”The Trunk
Impairment Scale: a new tool to measure motor impairment of the trunk after stroke.” Clin
Rehabil. (2004) 18(3): 326-34.
[36] Z. Sawacha, E. Carraro, P. Contessa, A. Guiotto, S. Masiero and C. Cobelli. ”Relationship
between clinical and instrumental balance assessments in chronic post-stroke hemiparesis
subjects.” J Neuroeng Rehabil. (2013) 10: 95.
[37] J.M. Soucie, C. Wang, A. Forsyth, S. Funk, M. Denny, K.E. Roach and D. Boone. ”Range of motion
measurements: reference values and a database for comparison studies.” Haemophilia.
(2011) 17(3): 500-7.
[38] D.G. Kendall. “A survey of the statistical theory of shape” Statist. Sci. (1989) 4: 87–120.
[39] M. Rodríguez-Hernández, B. Polonio-López, A.I. Corregidor-Sánchez, J.L. Martín-Conty, A.
Mohedano-Moriano, J.J. Criado-Álvarez. ”Effects of Specific Virtual Reality-Based Therapy for
the Rehabilitation of the Upper Limb Motor Function Post-Ictus: Randomized Controlled
Trial.” Brain Sci. (2021) 11(5): 555.
[40] N. Norouzi-Gheidari, A. Hernandez, P.S. Archambault, J. Higgins, L. Poissant, D. Kairy.
”Feasibility, Safety and Efficacy of a Virtual Reality Exergame System to Supplement Upper
Extremity Rehabilitation Post-Stroke: A Pilot Randomized Clinical Trial and Proof of
Principle.” Int J Environ Res Public Health. (2019) 17(1): 113.
[41] P.P.B. Henrique, E.L. Colussi, A.C.B. De Marchi. ”Effects of Exergame on Patients' Balance and
Upper Limb Motor Function after Stroke: A Randomized Controlled Trial.” J Stroke
Cerebrovasc Dis. (2019) 28(8): 2351-2357.
[42] A. Gelineau, A. Perrochon, L. Robin, J.C. Daviet, S. Mandigout. ”Measured and Perceived Effects
of Upper Limb Home-Based Exergaming Interventions on Activity after Stroke: A Systematic
Review and Meta-Analysis.” Int J Environ Res Public Health. (2022) 19(15): 9112.