For chronic patients, rehabilitation can reduce disability-related pain and improve functional capacity and quality of life. Every day, chronic patients should perform rehabilitation exercises, at home under the therapist's guidance. There are many problems related to the home rehabilitation scenario; most are connected to the patient experience. Sometimes, patients need to plug in components or have to face tricky procedures before the system run. The idea behind the work is to build a home rehabilitation system using a smartphone or tablet PCs ready to be used. Smartphone-based computer vision tools have shown potential for practical application in the field of telerehabilitation. To provide greater accessibility, there is a need to reduce the use of sensors and ensure the accuracy of monitoring without compromising the user experience. In this paper, we propose the validation of a novel smartphone-based pose detection tool during the performance of a rehabilitation exercise in which the elbow extension angle needs to be calculated as a metric. The tool allows real-time analysis of the exercise, although further eforts are needed to improve its accuracy.
Life expectancy has increased significantly in European countries in recent decades [
In this scenario, alternative rehabilitation models, such as telerehabilitation, have been created that use digital solutions to improve adherence and patient engagement.
Telerehabilitation is the provision of remote rehabilitation services using ICT technologies [
With recent advances in Artificial Intelligence, computer vision systems can track body movements in three-dimensional space using a single RGB camera, normally built into cell phones. The most commonly used computer vision tools for pose detection and body tracking are OpenPose [14], Mask R-CNN [15], Google’s MediaPipe or BlazePose [16, 17], Alpha-Pose [18], all available as open source. The previous tools have been used to develop smarthpone or web applications that can guide patients in performing exercises autonomously [19, 20]. These applications demonstrate that it is possible to verify the correct execution of a movement once completed and that, under stable environmental conditions, they can go so far as to provide guidance on specific joint angles. However, there is a lack of evaluation of the validity of the data provided for many of these systems [21, 22]. In fact, OpenPose is the only tool for which the ability to detect key points of the human skeleton during the performance of rehabilitation exercises has been validated, although the accuracy of detection is highly dependent on ambient lighting, the relative visibility of body joints, and the relative motion of the patient relative to the camera [22]. Unfortunately, OpenPose is not available for cell phones and thus poses a technological barrier and risk to adherence, especially in the case of elderly people. In contrast, Mediapipe’s Blazepose system can be used on Android and iOS and thus has the potential for the development of telerehabilitation solutions that can be used on mobile and can be easily integrated with gamification techniques to increase user motivation due to the availability of a dedicated plugin for Unity applications.
The purpose of this paper is to explore the accuracy of Blazepose in tracking and measuring body movements during the performance of telerehabilitation exercises performed in a home environment via a Unity app installed on a personal cell phone.
In this paper we describe the app and present the results obtained when measuring elbow angles during the arm extension in a classic post-stroke exercise. It is the subject of forthcoming papers to both evaluate the user experience and measure the accuracy of Blazepose during the execution of more complex exercises.
We have developed a telerehabilitation system that the patient can manage from a Unity application installed on the personal device. At the same time, a web application is available for the physician to function as a control room. Finally, patient and treatment session data are stored in the cloud.
The control room will allow the physician to schedule patient treatment sessions and monitor user performance and activity. The physician can modify treatment sessions at any time by adding or removing exercises or changing the frequency of training. In this way, the physician can tailor the session to the user’s specific abilities and needs.
The user application is structured with a login page and a menu. The menu allows the user to choose from the training sessions proposed by the physician. Each session consists of a calibration phase, one or more exercises to be performed, and a visual analog scale to measure pain and fatigue at the end of the session.
Figure 1 summarizes the combination of the previous steps. Basically, the treatment session can be considered as consisting of two cycles. The macrocycle handles the calibration phase, the final fatigue assessment, and the exercise sequence. Some exercises may be suggested by the physician as a warm-up or cool-down phase. The microcycle manages specific exercises by counting the number of times the exercise must be performed (series) or the number of times each movement must be repeated (repetitions). In addition, during the microcycle, an algorithm compares the user’s movements and poses with the movements and poses expected during the exercise. At the same time, it measures the distances between body joints and joint angles according to the exercise requirements. All this data is used to feed a virtual coach that translates the algorithm’s evaluations into audio feedback, motivating the user, encouraging him or her to improve previous performance, and suggesting corrections when unexpected movements are encountered.
As already anticipated, the proposed telerehabilitation system is based on the computer vision tool BlazePose. Blazepose is a pose detection model created by Google that, given an image or video frame, finds and returns the x, y, and z coordinates of 33 key points of the skeleton. BlazePose consists of two diferent machine-learning models: a detector and an estimator. The detector removes the human region from the input image or frame, while the estimator inserts a 256x256 resolution image of the recognized person and returns the key points [17]. This architecture enables real-time inference and, together with its lightweight nature, makes BlazePose favorable for smartphone applications.
Importantly, OpenPose is usually better than BlazePose at providing appropriate inference of key points from motion videos [23]. However, OpenPose is slower than Blazepose and cannot analyze real-time video. Therefore, BlazePose remains, according to us, the best choice for smartphone-based telerehabilitation applications where real-time feedback is needed to promote correct exercise execution.
As anticipated in the previous section, each exercise, as well as the calibration step, is analyzed by an algorithm. This algorithm is a computation of body joints extrapolated by BlazePose. In fact, an exercise can be thought of as a sequence of steps or poses. Each pose is characterized by a specific set of useful information for the clinician, such as joint angles, relative positions of arms and hands, distances, etc. While performing the exercise, the patient’s movement is captured by the smartphone camera and transmitted and processed by BlazePose at a rate of 30 frames per second. For each video frame, BlazePose returns body joints as 3D arrays and that can be organized to estimate the user’s pose, calculate joint angles or measure relative distances. The algorithm compares the temporal evolution of this information with exercise-specific and previously defined indications with an expert. In this way, the algorithm is able not only to assess whether the exercise is being performed correctly but also to identify which specific body part needs to be corrected. As already mentioned, this algorithm’s evaluations return to the user as an audio indication from a virtual coach.
The app can be autonomously used by the user and the execution of the treatment session does not require the presence of the physician, even remotely. For this reason, a calibration step is performed once the treatment session starts to ensure the app’s best functionality. The calibration step first asks the user to leave the cell phone in a stable position and to stand in front of the camera. Then the system compares the visible body joints with the body joints that the first exercise of the session needs to track. If all the body joints are visible the system asks the user to put himself in the initial position provided by the exercise. Some exercises could need to measure some specific distances or angles to be used as a benchmark during the execution (as an example the distance between hands when the arms are stretched out sideways).
The not visible body joints or the errors in the user pose are returned by the system as an audio message from a virtual coach that suggests moving away or approaching the camera to improve the visibility of body joints or guides the user to assume the correct position. The calibration is automatically required when one of the following conditions occurs: • the visibility of some body joints to track is lower than a threshold; • the next exercise in the treatment session begins and a new starting position must be checked; • the system recognizes a substantial change in the user’s position that could impair motion tracking.
It is known that patients who do not adhere to the prescribed exercise program can prolong
the duration of treatment, negatively impact the therapeutic relationship and make treatment
less efective. It can also impact health care providers with increased waiting time and poor
eficiency [ 24, 25]. Factors that may influence adherence to home exercise rehabilitation have
been discussed in numerous articles [
Some of these are related to the patient’s motivation, such as perceived barriers (e.g., forgetting to exercise, not having time, not getting back into the daily routine, work schedules), the individual’s belief in his or her ability to perform a task (self-eficacy), levels of pain during physical exercises, and psychological well-being (depression as a barrier has strong supporting evidence). Other characteristics are related to communication and education, such as information received, support from friends and family, therapist feedback and supervision during the session, and monitoring of progress information. Finally, some characteristics are specific to home rehabilitation treatment, such as goal setting, enjoyment during treatment, and avoidance of dificulties in using technological aids or fitness equipment.
Given the many factors that could influence patient adherence to treatment, the proposed telerehabilitation system aims to improve the entire patient experience thanks to 3 main features: • the system can be run on smartphones without any additional equipment (although the possibility of connecting some wearable sensors in the future to improve monitoring is not ruled out), and the treatment session can be performed without any intervention from the physician or other figures. • The Unity plugin of BlazePose is light and allows the app to run ofline. • The system also features a virtual coach that provides real-time audio feedback based on the algorithm’s ratings. The role of the virtual coach is to improve user engagement and ensure the correct execution of the exercise without annoying the user with wrong or redundant feedback.
Given the importance of the virtual coach in both improving and afecting the user experience, it is important to better explain how it works. Feedback from the virtual coach can remind you of the next movement, increase or decrease the speed of execution, allow time in stationary positions, correct patient position, encourage more or less arm extension, and suggest modifications to improve visibility. The feedback of the virtual coach is mainly based on the algorithm’s rating and can be one of these types: • encouraging feedback, for example when a repetition is completed or a series is concluded; • fixing feedback, when the algorithm encounters a sequence of movements not expected (for example when the computed angle is reducing while it is expected an extension or the arm raised is the right instead of the left); • motivating feedback, when, according to exercise-specific instructions, the algorithm computes some metrics (joint angles, leg gaps, ...) that could be improved (for example "try to increase the extension of your angle", ...); • guiding feedback, with instructions on the next movement when the algorithm encounters an unexpected stable position for more than 1 second (assuming the user does not know how to move); • historical feedback, returned when the algorithm measures an improvement or worsening in the performance of an exercise with respect to the performance of previous treatment sessions.
In this paper, we do not validate the user experience of the proposed telerehabilitation system, which will be the subject of a forthcoming paper. Indeed, since the system is based on the use of BlazePose, whose accuracy in a home environment while performing telerehabilitation exercises is not yet validated, our aim is to preliminary measure the accuracy of the computer vision tool and, subsequently, of the user experience.
To validate the use of a smartphone-based pose detection tool such as BlazePose for rehabilitation treatment, we focused on a post-stroke exercise: elbow joint extension. The exercise starts with the wrist near the shoulder. The patient then performs elbow extension by moving the wrist away from the shoulder in the same plane as the trunk of the body. Once maximum extension is reached, he or she holds the position for 2 seconds, then returns to the starting position. The exercise is repeated 3 times before ending a set.
The validation process involves 2 phases. The first phase wants to compare the elbow angle under two diferent conditions: when the patient is frontal and when he or she is rotated 45° relative to the camera. The second phase wants to verify, in what was found to be the best measurement condition among the two previous ones, the ability of the system to measure the angle correctly.
Participating in the study were 10 healthy volunteers who were asked, for the first phase, to perform the exercise independently and, with each repetition, to increase the elbow angle until they reached 180° in the last repetition. For the second phase, volunteers were asked to stand facing the camera (as we will see the best condition for measurement) and to extend the arm up to 180° in each repetition (data not shown). The average value of the maximum angle measured by the algorithm in each repetition and for each user was then calculated.
It is important to note that although the benchmark exercise is simple, it is suficient for proper validation of the algorithm because the assessment of correctness of movements is based on geometric analysis of body joints that can be easily extended to more complex exercises
The validation of BlazePose is composed of 2 phases. For the first phase, figure 2 shows the average, second by second, of elbow angles computed in each frame both when the volunteers were frontal and when they were rotated 45° relative to the camera. As shown in the figure, at a twist of 45° the calculation is incorrect, indicating a maximum angle of 147° when the volunteers reached about 180°.
It is mainly due to the partial overlapping of body joints when the user is rotated 45° relative to the camera. For each exercise is then necessary to identify the best condition to measure metrics and to ensure, during the calibration step, that this condition is satisfied. Then, the algorithm, by elaborating deep estimations of body joints, will be able to recognize rotations and changes in the user’s conditions, providing a new calibration. On the other hand, there exists a threshold between the two extremal conditions here presented (frontal view vs 45° view) beyond which the loss of accuracy is significant. In the future, it will be critical that the algorithm can recognize this threshold and maybe elaborate data to reduce the efects of rotation, in this way reducing the number of re-calibration steps and thus improving the user experience.
For the second validating phase, volunteers repeated the exercise reaching 180° in each repetition. Nevertheless, the algorithm computed an average maximum elbow angle of 172.1° (standard deviation 1.8°). It is important to say that a diference of about 8°− 10° in the correct computation could be unacceptable in some specific rehabilitation treatments. On the other hand, according to [27], evaluation therapists tend to underestimate the range of motion by 9.41° on average for any joint movement of the upper limb. Therefore, with the results obtained in this approach, it can be concluded that the proposed telerehabilitation system is an adequate tool for evaluating patient performance in rehabilitation programs, at least for those exercises that involve upper limb movements. Further, as mentioned in the introduction, OpenPose is, at the moment, the only tool whose accuracy has been validated for rehabilitation exercises. In particular, in [28] it has been shown that, when OpenPose is used to estimate a real elbow angle of 180°, the computed angle is less than 175°, with a performance very close to our results (compare figure 7 in [28]).
Recent developments in computer vision and machine learning techniques have improved the accuracy of human posture estimation, showing the potential practical application in the field of telerehabilitation.
In order to ensure greater accessibility, there is a need to reduce the use of sensors and equipment when performing home rehabilitation, and smartphone-based posture detection tools are a promising solution. However, it is crucial to pay attention to the accuracy of patient monitoring without compromising the user experience.
In this paper, we propose a novel validation of a smartphone-based pose detection tool (BlazePose) during the performance of a rehabilitation exercise in which some metrics (elbow angle) need to be calculated.
In general, the system is very fast in processing calculations and is able to guide the user in real time. Further, the accuracy in computing the elbow angle is very close to both other tools like OpenPose and general therapist performance. To summarise, even if it is important to further validate the system both on more complex exercises and about the user experience, this preliminary study shows that the proposed system is a promising and acceptable smartphonebased tool for upper limb rehabilitation with the potential to improve user experience without afecting the accuracy of measurements. [7] A. Vakanski, J. Ferguson, S. Lee, Mathematical modeling and evaluation of human motions in physical therapy using mixture density neural networks, Journal of physiotherapy & physical rehabilitation 1 (2016). [8] R. Sodhi, H. Benko, A. Wilson, Lightguide: projected visualizations for hand movement guidance, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2012, pp. 179–188. [9] M. Sousa, J. Vieira, D. Medeiros, A. Arsenio, J. Jorge, Sleevear: Augmented reality for rehabilitation using realtime feedback, in: Proceedings of the 21st international conference on intelligent user interfaces, 2016, pp. 175–185. [10] N. M. da Silva Ribeiro, D. D. Ferraz, É. Pedreira, Í. Pinheiro, A. C. da Silva Pinto, M. G. Neto, L. R. A. Dos Santos, M. G. G. Pozzato, R. S. Pinho, M. R. Masruha, Virtual rehabilitation via nintendo wii® and conventional physical therapy efectively treat post-stroke hemiparetic patients, Topics in stroke rehabilitation 22 (2015) 299–305. [11] A. Ficklscherer, J. Stapf, K. M. Meissner, T. Niethammer, M. Lahner, M. Wagenhäuser, P. E.
Müller, M. F. Pietschmann, Testing the feasibility and safety of the nintendo wii gaming console in orthopedic rehabilitation: a pilot randomized controlled study, Archives of Medical Science 12 (2016) 1273–1278. [12] D. Anton, I. Berges, J. Bermúdez, A. Goñi, A. Illarramendi, A telerehabilitation system for the selection, evaluation and remote management of therapies, Sensors 18 (2018) 1459. [13] D. Koulouris, A. Menychtas, I. Maglogiannis, An iot-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming, Sensors 22 (2022) 3181. [14] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, Y. A. Sheikh, Openpose: Realtime multiperson 2d pose estimation using part afinity fields, IEEE Transactions on Pattern Analysis and Machine Intelligence (2019). [15] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969. [16] A. Ahmadyan, T. Hou, J. Wei, L. Zhang, A. Ablavatski, M. Grundmann, Instant 3d object tracking with applications in augmented reality, arXiv preprint arXiv:2006.13194 (2020). [17] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang, M. Grundmann, Blazepose:
On-device real-time body pose tracking, arXiv preprint arXiv:2006.10204 (2020). [18] H.-S. Fang, S. Xie, Y.-W. Tai, C. Lu, Rmpe: Regional multi-person pose estimation, in:
Proceedings of the IEEE international conference on computer vision, 2017, pp. 2334–2343. [19] F. Rosique, F. Losilla, P. J. Navarro, Applying vision-based pose estimation in a telerehabilitation application, Applied Sciences 11 (2021) 9132. [20] T. R. Toelle, D. A. Utpadel-Fischler, K.-K. Haas, J. A. Priebe, App-based multidisciplinary back pain treatment versus combined physiotherapy plus online education: a randomized controlled trial, NPJ digital medicine 2 (2019) 1–9. [21] M. Ota, H. Tateuchi, T. Hashiguchi, T. Kato, Y. Ogino, M. Yamagata, N. Ichihashi, Verification of reliability and validity of motion analysis systems during bilateral squat using human pose tracking algorithm, Gait & posture 80 (2020) 62–67. [22] J. Adolf, J. Dolezal, P. Kutilek, J. Hejda, L. Lhotska, Single camera-based remote physical therapy: Verification on a large video dataset, Applied Sciences 12 (2022) 799. [23] S. Mroz, N. Baddour, C. McGuirk, P. Juneau, A. Tu, K. Cheung, E. Lemaire, Comparing the quality of human pose estimation with blazepose or openpose, in: 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART), IEEE, 2021, pp. 1–4. [24] M. A. Holden, K. L. Haywood, T. A. Potia, M. Gee, S. McLean, Recommendations for exercise adherence measures in musculoskeletal settings: a systematic review and consensus meeting (protocol), Systematic reviews 3 (2014) 1–6. [25] M. F. Pisters, C. Veenhof, F. G. Schellevis, J. W. Twisk, J. Dekker, D. H. De Bakker, Exercise adherence improving long-term patient outcome in patients with osteoarthritis of the hip and/or knee, Arthritis care & research 62 (2010) 1087–1094. [26] D. Collado-Mateo, A. M. Lavín-Pérez, C. Peñacoba, J. Del Coso, M. Leyton-Román, A. LuqueCasado, P. Gasque, M. Á. Fernández-del Olmo, D. Amado-Alonso, Key factors associated with adherence to physical exercise in patients with chronic diseases and older adults: an umbrella review, International journal of environmental research and public health 18 (2021) 2023. [27] C. Lavernia, M. D’Apuzzo, M. D. Rossi, D. Lee, Accuracy of knee range of motion assessment after total knee arthroplasty, The Journal of arthroplasty 23 (2008) 85–91. [28] Ó. G. Hernández, V. Morell, J. L. Ramon, C. A. Jara, Human pose detection for roboticassisted and rehabilitation environments, Applied Sciences 11 (2021) 4183.