Alterations of gait and balance are a significant cause of falls, injuries, and consequent hospitalizations in the elderly. In addition to age-associated motor decline, other factors can impact gait and stability, including the motor dysfunctions caused by neurological diseases such as Parkinson's disease or hemiplegia after stroke. Monitoring changes and deterioration in gait patterns and balance is crucial for activating rehabilitation treatments and preventing serious consequences. This work presents a Kinect-based solution, suitable for domestic contexts, for assessing gait and balance in individuals at risk of falling. The system captures body movements during home acquisition sessions scheduled by clinicians at definite times of the day and automatically estimates specific functional parameters to objectively characterize the subjects' performance. The system includes a graphical user interface designed to ensure usability in unsupervised contexts: the human-computer interaction mainly relies on natural body movements to support the self-management of the system, if the motor conditions allow it. This work presents the system's features and facilities, and the preliminary results on healthy volunteers' trials.
Changes in gait patterns, balance disorders, and alteration of postural stability are among the most
evident effects of the functional decline linked to aging [
In addition to age-related factors, other comorbidities can adversely affect gait and stability,
including neurological pathologies characterized by movement disorders. In particular, the
dysfunctions in motor control and coordination cause changes in gait and postural stability linked to
the progression of the disease (for example, in Parkinson's disease [
This study presents an optical solution based on the new Microsoft Kinect Azure DK to evaluate gait and postural stability. With respect to the aforementioned approaches, ours considers an integrated analysis based on static (i.e., during the standing task) and dynamic (i.e., during the walking task) conditions, which allows for a more comprehensive evaluation of postural alterations, balance dysfunctions, and changes in the habitual motor behaviors. Moreover, recent validation studies have demonstrated the high potential of the new Kinect sensor versus previous device models and other commercial optical sensors, both in terms of performance (i.e., accuracy and reliability) as in [27] and of the real-time body tracking algorithm (i.e., accuracy and robustness) as in [28]. Last but not least, our approach is designed to be suitable for domestic contexts or, at least, semi-supervised scenarios (e.g., private patients’ associations). This aspect could allow to fill the gap between seldom “in hospital” testing and continuous “at home” monitoring, which, at the state of the art, can be easily achieved only through wearable devices [29]. The system is able to capture body movements during gait and postural stability tasks executed by the subjects in front of the Kinect Azure camera at definite times of the day, according to medical prescription, and then estimate sets of functional parameters to characterize performance in at-risk subjects automatically. Dedicated human-computer interaction and user interfaces, based on natural body movements and the same tracking algorithm, are part of the system to allow for high usability and self-management at home and without specific technological skills. Preliminary results on a small group of healthy volunteers indicate the solution’s ability to intercept changes in gait patterns and postural stability, suggesting the possibility of using it as an intelligent and automatic instrument to remotely monitor alterations of these functionalities in at-risk pathological and non-pathological populations.
The Kinect-based system has been designed to consider some specific needs of unsupervised environments, as the domestic contexts, including non-invasiveness, portability, and limited available spaces. In addition, to ensure high usability, particular attention has been paid to the development of the human-computer interaction and the graphical user interface to support subjects and caregivers in the acquisition of specific gait and balance tasks.
The system hardware is composed of a mini-pc, an RGB-Depth camera (i.e., Microsoft Kinect Azure), and a monitor. Microsoft Kinect Azure records synchronized colour and depth streams at up to 30 fps, with different camera modes available [27]. A default configuration consisting of 1080p resolution for the colour stream, Narrow Field of View (NFW) for the depth stream, and 30 fps framerate for both was employed for this study.
The RGB-Depth sensor is connected to the mini-pc: for this study, a ZOTAC ZBOX EN52060-V was used. The mini-pc model was chosen to fulfill the real-time processing requirements of the body tracking algorithm provided by the Azure Kinect Body Tracking Software Development Kit (SDK). An off-the-shelf monitor was employed to visualize the system Graphical User Interface (GUI) and provide visual feedback during tasks execution.
The system software is based on the Azure Kinect Sensor SDK and the Azure Kinect Body Tracking SDK. The former includes all the Application Programming Interfaces (APIs) for managing the device and recording, processing, and storing the video streams; the latter includes the tracking algorithm used for the 3D reconstruction of body movements. Unlike the previous Kinect model, the new Azure Kinect tracking algorithm implements a Deep Learning (DL) architecture that exploits the Part Affinity Field (PAF) [30] to compute 32 body joints that connect in a 2D skeleton. The 2D skeleton is then uplifted to a 3D model, exploiting the information from the depth camera. The developers trained the learning model on tons of real and synthetic images [31], so the tracking algorithm can be directly used on the RGB-D stream through the routines available in the SDK. Fig. 1 shows the physical setup of the system.
The acquisition software was developed in Unity, as it contains all the tools for an easy and effective design of the GUI and the Augmented Reality (AR) setting in which the user performs the required task. Unity, however, enables interaction and behaviour control of its objects only through C# scripting. Hence, a porting of the two SDK mentioned above (written in C++) was achieved by implementing a C# middleware.
The GUI was designed for two possible scenarios: user-alone or user-supported by a clinician/caregiver. As an example, Fig. 2 shows the main GUI implemented for the user-supported scenario. In the user-supported scenario, the caregiver is in charge of the Human-Computer Interaction, selecting with a mouse which task to execute (stability evaluation or gait) and when to start and to stop the recordings. This use case was designed to support subjects with severe motor disabilities who need help in using and interacting with the system, limiting their involvement in executing motor tasks. On the contrary, in the user-alone scenario, the subject is able to self-manage the system and is guided in the use and execution of motor tasks by textual and vocal messages provided by the user interface. The vocal messages are delivered through a Text-To-Speech (TTS) solution for Unity based on Microsoft Speech API 5.3. Interaction with GUI objects (e.g., buttons) is realized by tracking the dominant (or the healthiest) hand, which is configurable before software execution through a JSON configuration file. All the interactive objects are adequately scaled to be visible for patients with reduced sight (as typical in elderly) and are organised on the screen to be easily reachable, thus limiting the range of motion required in the perspective of system usage by subjects with reduced mobility.
In addition, the system saves the skeletal model data at the end of each task execution in a JSON file: this file contains the position and rotation of the body joints according to the 3D reference system and a confidence estimation of the tracking accuracy. These data are then processed offline using custom-written MATLAB scripts to extract relevant gait and postural parameters. 2.2.
For this preliminary study, the aim was to recruit a small number of healthy subjects to verify the capability of the system to quantify relevant postural and gait features. Moreover, the subjects were asked to simulate some motor anomalies (e.g. short steps, slow pace, dangling walking), to investigate if the system was able to detect them without direct involvement, at this stage, of neurological patients. Eventually, eight healthy volunteers (4 male and 4 female) were recruited. The average age is 49.5±16.4 years (range: 25-65 years). Subjects performed the tasks under the supervision of technical personnel and after being properly instructed on the system and experimental procedure. Stability tasks were executed in the user-alone scenario, whereas the user-supported scenario was used for gait tasks in which the caregiver controls the beginning and end of the execution.
The acquisitions were performed in a home setting to simulate the expected working conditions of the system. Each session included: postural stability task (initial); walking towards the 3D camera along a 5m-long straight path (repeated three times); postural stability task (final). Each subject performed three sessions with a different pace and gait pattern: the first, with a natural pace at normal speed (NW); the second, with slower speed and shorter steps (SW); the third, with a dandling and low-speed walking (DW). The last two sessions were used to simulate some typical behaviour of elderly and subjects with altered gait due to neurological diseases: the altered gait patterns can be faithfully replicated also by healthy subjects of a different age group than the older target. It is important to remember that our goal, at this stage, is to verify the potential of the proposed system in detecting alterations of the motor pattern that are typical in elderly subjects with and without neurological pathologies. Instead, the postural stability tasks were included to assess changes in balance conditions before and after walking: postural stability task consists in standing still with eyes open for 30 seconds, in front of the camera at about 2 m away.
All participants were able to perform all planned walking and postural stability trials. In total, 69 walking and 48 postural stability trials were collected: for three subjects, only two walking trials (instead of the three expected) in the SW condition were considered valid due to external disturbance. 2.3.
The pre-processing phase involves resampling and filtering techniques of the recorded data. First, the 3D trajectories of the joints are resampled at 30 Hz and then filtered using a Butterworth low pass filter (third order, 10 Hz cutoff) to remove jitter and high-frequency noise. Some joints of the skeletal model are then used to characterize gait and postural stability. The 3D body center of mass (COMBODY) is estimated for both analyses, as in [32][33]: the COMBODY is the 3D weighted average of six body segments relating to head, trunk, arms, and legs.
As in [33], a custom-written MATLAB script calculates the 3D COMBODY distance from the 3D camera to determine the gait analysis time window (GATW) and estimate spatiotemporal gait parameters inside the virtual gait analysis path (VGAP). The GATW starts when the subject enters the VGAP (about 5m from the camera) and ends when the subject leaves the VGAP established to capture the total body (about 2m from the camera). Additionally, the 3D torso center of mass (COMTORSO) is estimated to evaluate mediolateral and vertical sways during walking as potential indices of lateral and forward instability under dynamic conditions, thus avoiding the effects of head and arm movements. Instead, the ankle joints of the skeletal model are used by the step segmentation algorithm to identify each step and the related gait pattern features. Table 1 shows the list of gait parameters automatically estimated by the analysis scripts.
For the analysis of postural stability, as in [32], the movements of the COMBODY are used to estimate the maximum sways along the anteroposterior (i.e., forward and backward) and mediolateral (i.e., right and left) directions and other related functional parameters compared to the initial position. Table 2 shows the list of functional parameters related to postural stability and COMBODY sways automatically estimated by the analysis scripts.
Meaning Maximum AP sway range
Total AP sway range Maximum AP sway speed Maximum ML sway range
Total ML sway range Maximum ML sway speed
Maximum sway area
This section presents and discusses some preliminary results. The analysis of walking trials on normal and simulated abnormal conditions indicates that the system is able to quantify differences in walking parameters and altered patterns, a highly probable condition in elderly subjects with and without pathologies related to motor dysfunctions. The postural stability analysis, before and after walking trials, suggests that the system can detect differences in balance parameters, for example after ordinary quotidian activities such as walking, that, however, represent challenging motor actions for frail subjects and could aggravate the risk of falls. 3.1.
The dataset for gait analysis contains 69 trials, divided into 24 NW, 21 SW, and 24 DW, respectively. Each trial was automatically analyzed to estimate spatiotemporal and COMTORSO parameters. In order to verify the ability of the system to capture the main features of gait patterns, the following figures show an example of NW (Fig. 3), SW (Fig. 4), and DW (Fig. 5) gait trials of the same subject, and the outcome of the step segmentation algorithm. The graphical representation of walking trajectories and estimated parameters provides qualitative information on gait patterns, including the quantity, length, and duration of steps (for the right and left leg), the trajectory and mediolateral sways of the center of mass during walking. The visual comparison highlights significant differences between the three trials. A greater number of shorter steps characterize the SW trial, whereas the DW trial is characterized by more significant lateral sway than normal walking (NW). The qualitative analysis suggests that the system is capable of intercepting differences in gait patterns.
The visual analysis suggested we carry out a quantitative evaluation of the three types of walking.
To this end, we grouped the gait trials according to the session type and computed the average value
of each parameter to support the qualitative analysis with objective measures. Table 3 shows the mean
and standard deviation values concerning the three types of sessions.
Fig. 6 shows a graphical representation of the average parameters for the three types of sessions
according to Table 3. A normalization procedure has been applied to avoid different scaling issues by
considering the minimum and maximum per-parameter values on all the trials collected. Each
parameter was scaled in the [
The bar chart highlights significant quantitative differences between the three sessions. On average, the SW and DW trials show greater values for SN and GCN, denoting a higher number of steps and gait cycles than NW due to shorter steps both in length (SL) and time (ST) parameters, as expected. The same occurs for speed (SP) and cadence (CD) parameters that show slower values than NW due to the simulated abnormal condition during walking. The stance and swing phases of the gait cycle are relevant both as time and percentage parameters: the stance phase is slower when no abnormal condition is present (NW); on the contrary, the swing phase increases in normal (NW) and decreases in abnormal condition (SW and DW) as expected. The SW and DW sessions are similar for spatiotemporal parameters, confirming compliance with the dandling and slower walking requirements established by the acquisition protocol. Nevertheless, SW and DW sessions differ in mediolateral sways that are more significant in DW sessions than in SW sessions. These outcomes confirm the ability of the system to quantitatively characterize different gait patterns, intercepting features that could be associated with gait disorders linked to pathological conditions. 3.2.
The dataset for postural stability analysis contains 48 trials, divided into 24 before walking and 24 after walking. Each trial was automatically analyzed to estimate COMBODY parameters. The postural stability analysis before and after three gait trials aims to highlight alteration due to fatigue, a widespread condition in neurological diseases that enhance balance disorders.
The postural stability trials were grouped in “before” and “after” walking trials: the average value
of each parameter was computed to support the analysis with objective measures. A normalization
procedure has been applied to avoid different scaling issues by considering the minimum and
maximum per-parameter values on all the trials collected. Each parameter was scaled in the [
The bar chart highlights an increase of all parameters related to COMBODY after walking, in the anteroposterior and mediolateral directions, compared to before walking trials. The percentage increase, estimated on normalized parameters, ranges from 51% for APR to over 100% for MLR and SA. The other parameters show a percentage increase of about 60%. These outcomes confirm the system’s ability to quantitatively measure different stability conditions in healthy subjects, intercepting features that could be associated with fatigue and alteration of balance related to pathological conditions.
This work proposes a solution for the automatic and remote evaluation of gait patterns and balance dysfunctions due to age-related motor deterioration or the presence of neurological comorbidities characterized by movement disorders (e.g., Parkinson’s disease or post-stroke). Since gait and balance disorders are closely associated with a high risk of falling, especially in the elderly, early detection of any changes is essential to activate training and rehabilitation protocols to prevent serious consequences.
The proposed solution consists of a vision-based system built around the new Microsoft Kinect Azure DK, the last model of Kinect technologies, featuring improved performance and body tracking accuracy compared to its predecessors and other commercial devices, as demonstrated by recent studies. The solution captures the 3D trajectories of body movements during walking and postural stability tasks using a deep learning approach in real-time, allowing the estimation of spatiotemporal, postural and center of mass parameters. The system is also equipped with dedicated interactive GUIs, implemented in Unity, to support user-alone and user-supported scenarios. Thanks to the limited hardware equipment and the ease of use, the system is thought as an evaluation tool that could be employed in unsupervised or semi-supervised environments, like home settings and patients’ associations. Clinicians could disseminate the system to at-risk patients and prescribe them to perform the tasks in front of the 3D camera in different sessions throughout the day, to identify abnormalities or to evaluate, for example, variations due to pharmacological therapy. We recruited a group of healthy volunteers to test the implemented solution and the ability to intercept changes in both gait patterns and postural stability after walking. To this end, the acquisition protocol involved three types of walking to be analyzed. Preliminary results on gait analysis indicate the system’s ability to detect alterations in gait patterns and mediolateral sways, highlighting average differences between sessions at normal pace, slower pace, and dangling walk. This outcome suggests the system could intercept gait alterations typical of the elderly and subjects with neurological diseases. Even though this was achieved from simulated alterations, we expect that the system will prove effective also on pathological subjects, in which this kind of altered conditions are accentuated. To prove this aspect, we are currently planning to carry out a more comprehensive study including also the target group of subjects, to correlate the results that will be obtained to standardize clinical evaluation.
Furthermore, postural stability analysis denotes an average worsening of the parameters relating to mediolateral and anteroposterior sways of the body center of mass after the walking tests. This behavior suggests the system’s ability to intercept changes in balance in healthy subjects due to fatigue, which is also common in the elderly and subjects with pathological conditions. Based on this result and thanks to the system scalability, we plan to improve the analysis of stability dysfunctions by introducing additional exercises for automatic balance evaluation according to dedicated clinical reference scales, for example, the Berg Balance Scale (BBS) as in [20]. However, as the proposed system mainly addresses unsupervised or semi-supervised scenarios, only a subset of the BBS tasks will be considered and implemented to avoid risks for the patients’ safety in unsupervised scenarios.
In conclusion, the non-invasiveness, portability, and easy-to-use facilities make the solution a feasible tool for remote monitoring of gait and balance dysfunctions in the home environment on subjects at risk of falling.
This work was partially supported by ReHOME Project – ICT solutions for tele-rehabilitation of cognitive and motor disabilities in neurological diseases. Grants from Regione Piemonte (Italy), F.E.S.R. 2014/2020, F.S.E. 2014/2020
[19] M.A. Eltoukhy, C. Kuenze, J. Oh, J.F. Signorile. “Validation of Statis and Dynamic Balance Assessment Using Microsoft Kinect for Young and Elderly Populations.” IEEE J Biomed Health Inform. 22.1 (2018):147-153 [20] D. Bacciu, S. Chessa, C. Gallicchio, A. Micheli, L. Pedrelli, E. Ferro, L. Fortunati, D. La Rosa, F. Palumbo, F. Vozzi, O. Parodi. “A learning system for automatic Berg Balance Scale score estimation.” Engineering Applications of Artificial Intelligence 66 (2017): 60-74. doi:10.1016/j.engappai.2017.08.018. [21] W.J.A. Grooten, L. Sandberg, J. Ressman, N. Diamantoglou, E. Johansson, E. Rasmussen-Barr. “Reliability and validity of a novel Kinect-based software program for measuring posture, balance and side-bending.” BMC Musculoskelet Disord 19 (2018). doi:10.1186/s12891-0171927-0 [22] B. Dehbandi, A. Barachant, A.H. Smeragliuolo, J.D. Long, S.J. Bumanlag, V. He, A. Lampe, D.
Putrino. “Using data from the Microsoft Kinect 2 to determine postural stability in healthy subjects: A feasibility trial.” PloSOne 12.2 (2017). doi:10.1371/journal.pone.0170890 [23] A.K. Mishra, M. Skubic, B.W. Willis, T.M. Guess, S.S. Razu, C. Abbott, A.D. Gray. “Examining methods to estimate static body sway from the Kinect V2. 0 skeletal data: implications for clinical rehabilitation.” in: Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth ’17, Barcelona, Spain, 2017 [24] S. Springer, G. Yogev Seligmann. “Validity of the Kinect for Gait Assessment: A Focused
Review.” Sensors 16.2 (2016). doi:10.3390/s16020194 [25] Álvarez, Ignacio et al. “Validity and sensitivity of instrumented postural and gait assessment using low-cost devices in Parkinson's disease.” Journal of neuroengineering and rehabilitation vol. 17,1 149. 11 Nov. 2020, doi:10.1186/s12984-020-00770-7 [26] Q. Li, Y. Wang, A. Sharf, Y. Cao, C. Tu, B. Chen, S. Yu. “Classification of gait anomalies from
Kinect.” The Visual Computer 34.2 (2018): 229-241 [27] M. Tölgyessy, M. Dekan, L. Chovanec, P. Hubinský. “Evaluation of the Azure Kinect and Its
Comparison to Kinect V1 and Kinect V2.” Sensors 21.2 (2021). doi:10.3390/s21020413 [28] M. Antico, N. Balletti, G. Laudato, A. Lazich, M. Notarantonio, R. Oliveto, S. Ricciardi, S.
Scalabrino, J. Simeone. “Postural control assessment via Microsoft Azure Kinect DK: An evaluation study.” Comput Methods Programs Biomed. 209 (2021). doi:10.1016/j.cmpb.2021.106324 [29] P. Barsocchi et al., "Detecting User's Behavior Shift with Sensorized Shoes and Stigmergic Perceptrons," (2019) IEEE 23rd International Symposium on Consumer Technologies (ISCT), pp. 265-268, doi: 10.1109/ISCE.2019.8901007. [30] Z. Cao, G. Hidalgo, T. Simon, S. -E. Wei, Y. Sheikh. "OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields." IEEE Transactions on Pattern Analysis and Machine Intelligence 43.1 (2021): 172-186 [31] Z. Liu, 3D Skeletal Tracking on Azure Kinect, 2019. URL: https://www.microsoft.com/enus/research/uploads/prod/2020/01/AKBTSDK.pdf (Last Access: September 28, 2021) [32] C. Ferraris, R. Nerino, A. Chimienti, et al. “Feasibility of Home-Based Automated Assessment of Postural Instability and Lower Limb Impairments in Parkinson's Disease.” Sensors 19.5 (2019). doi:10.3390/s19051129 [33] C. Ferraris, V. Cimolin, L. Vismara, V. Votta, G. Amprimo, R. Cremascoli, M. Galli, R. Nerino, A. Mauro, L. Priano. “Monitoring of Gait Parameters in Post-Stroke Individuals: A Feasibility Study Using RGB-D Sensors.” Sensors 21.17 (2021). doi:10.3390/s21175945