=Paper=
{{Paper
|id=Vol-3932/paper3
|storemode=property
|title=Leveraging Multimodal Monitoring in Plan-Based Robot-Aided Rehabilitation (short paper)
|pdfUrl=https://ceur-ws.org/Vol-3932/paper3.pdf
|volume=Vol-3932
|authors=Christian Tamantini,Alessandro Umbrico,Andrea Orlandini
|dblpUrl=https://dblp.org/rec/conf/f4mr/TamantiniUO24
}}
==Leveraging Multimodal Monitoring in Plan-Based Robot-Aided Rehabilitation (short paper)==
https://ceur-ws.org/Vol-3932/paper3.pdf
Leveraging Multimodal Monitoring in Plan-Based
Robot-Aided Rehabilitation
Christian Tamantini1 , Alessandro Umbrico1,∗ and Andrea Orlandini1
1
Institute of Cognitive Sciences and Technologies, National Research Council of Italy, 00196 Rome, Italy
Abstract
Robotic rehabilitation systems typically rely on kinematic data to assess exercise execution, focusing on pose
accuracy without evaluating the physical impact on the patient. This paper explores the integration of physiolog-
ical monitoring into an automated planning system for robot-aided rehabilitation. Specifically, we assess whether
Heart Rate (HR) can provide meaningful feedback on the experimented physical workload while executing
physical tasks. Ten healthy participants completed a session of 10 exercises, with cumulative exercise intensity
(Σ𝐼) and HR recorded. A Pearson correlation analysis revealed an average strong correlation of 0.63 across
all participants, indicating that HR reliably reflects physical effort. The results suggest that incorporating HR
feedback allows the system to dynamically adjust exercise intensity in real-time, ensuring that patients are neither
over- nor under-challenged.
Keywords
Robot-aided rehabilitation, Physical rehabilitation, Automated Planning, Task Planning
1. Introduction
Robotic systems have become essential tools in rehabilitation, assisting with physical recovery by
delivering personalized, repeatable, and adaptive interventions. These systems include exoskeletons [1]
and end-effector robots providing continuous physical assistance [2], as well as socially assistive robots
that offer motivational support and corrective feedback [3]. Their integration into healthcare has been
especially beneficial for conditions such as stroke and musculoskeletal disorders [4].
In current robotic rehabilitation platforms, automated planning systems can select and sequence
exercises based on clinical goals [5]. A notable example is the NAOTherapist platform, widely applied in
pediatric robot-aided rehabilitation [6]. These systems predominantly rely on kinematic data to assess
how closely a patient’s upper limb configuration matches the assigned pose, typically demonstrated by
the robot. This assessment often uses error metrics such as normalized Euclidean distance [7]. Based
on the patient’s performance, the system adjusts the difficulty of the exercises, determining the level of
strictness in evaluating execution accuracy [8]. However, this focus on static upper limb configurations
is limiting, as most rehabilitation exercises involve dynamic, full-body movements. It is important to
acknowledge that kinematic monitoring is widely used in the literature because it offers immediate
feedback on task execution, making it effective for managing the flow of rehabilitation sessions.
Nevertheless, focusing solely on kinematic data has significant limitations, as it fails to capture the
overall physical impact of the rehabilitation session. While kinematic monitoring effectively ensures
that individual movements are executed correctly, it does not account for the cumulative physical
effort required throughout the session. This information gap can lead to exercises that either under- or
over-challenge the patient, potentially slowing their rehabilitation progress. In contrast, incorporating
physiological monitoring allows for tracking the patient’s physical workload during the execution of
the exercises [9] and psychophysiological state in longitudinal studies [10].
Workshop on Advanced AI Methods and Interfaces for Human-Centered Assistive and Rehabilitation Robotics (a Fit4MedRob
event) - AIxIA 2024
∗
Corresponding author.
Envelope-Open christian.tamantini@cnr.it (C. Tamantini); alessandro.umbrico@cnr.it (A. Umbrico); andrea.orlandini@cnr.it
(A. Orlandini)
Orcid 0000-0001-6238-2241 (C. Tamantini); 0000-0002-1184-5944 (A. Umbrico); 0000-0001-6458-5202 (A. Orlandini)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Therefore, this paper aims to evaluate whether the integration of physiological monitoring into a
multimodal patient monitoring system can provide valuable insights into the patient’s physical workload.
These insights can be leveraged by an automated planning-based robot-aided rehabilitation system to
more effectively tailor therapy sessions, ensuring that the prescribed exercises align with the patient’s
actual physical effort and overall exertion. By combining both kinematic and physiological data, the
system has the potential to deliver a more adaptive and personalized rehabilitation experience.
2. Materials and Methods
The block diagram reported in Fig. 1 illustrates the structure of a multimodal rehabilitation system
that integrates both kinematic and physiological monitoring to optimize exercise sessions tailored to
individual patients.
Figure 1: Block scheme of the proposed approach to exploit multimodal monitoring in automated planning-
based robot-aided rehabilitation.
Session inputs provide the initial parameters for the rehabilitation session, including clinical goals and
desired intensity. These inputs form the basis for personalizing the session to meet specific therapeutic
objectives. Based on these inputs, the Session Planner generates an optimized sequence of exercises
using automatic planning techniques. Its role is to ensure that the therapeutic session is in line with the
predefined clinical objectives by dynamically adjusting the exercise sequence if the monitored physical
exertion of the patient deviates from the planned intensity. This module stores a list of exercises labeled
with the intensity level ranging in [1, 5] that can be used for the generation of the current session [11].
The Session Manager monitors the execution of the planned exercises, interacting with both the
patient and the monitoring systems to ensure smooth progression throughout the session. It manages
the session in real-time by controlling the Feedback system according to the data collected from the
monitoring systems. Specifically, it compares the prescribed task with the patient’s actual performance
and provides the task completion percentage as feedback. Once the task is performed for the required
duration, the system advances to the next exercise, maintaining a structured flow throughout the
rehabilitation session.
At the heart of the system is Multimodal Monitoring, which consists of two key components: Physi-
ological Monitoring and Kinematic Monitoring. The former tracks the physiological responses of the
patient which provides essential, slow-varying feedback on the physical effort required by the patient
during the execution of the exercise. This information allows the system to adjust the sequence and
intensity of the exercises to ensure that the prescribed levels of physical exertion match the clinical
goals. Among the measurable physiological metrics, HR is a reliable indicator of exertion, offering
insights into the patient’s response to the exercises. Wrist-worn devices, such as smartwatches, are
ideal for continuous HR tracking due to their reliability and non-intrusive design.
The latter captures the movements of the patient in real-time, ensuring that the exercises are correctly
performed, for the correct amount of time. This data helps to manage the therapy session in real-time
providing immediate feedback on task performance and facilitating the correct session flow. Whenever
the patient executes the assigned task for a determined amount of time correctly, the Session Manager
administers the subsequent exercise.
2.1. Experimental Evaluation
In this experiment, 10 healthy participants (31.4 ± 5.1 years old, 9 males and 1 female) were recruited.
The experimental protocol was designed to assess the effect of various rehabilitation exercise plans
on the participants’ physiological responses, with a particular focus on HR as an indicator of physical
effort. Each participant completed a rehabilitation session consisting of 10 exercises, selected from the
database managed by the Session Planner. Each exercise was performed for 30 seconds. Figure 2 shows a
representative participant performing the exercises while being monitored by a multimodal monitoring
system. More in detail, the Garmin Vivosmart 4 wristband was used to monitor HR without disrupting
the patient’s comfort or movement during rehabilitation sessions, collecting data at 1 Hz. The TIAGo
robot’s head-mounted camera (ASUS Xtion) and the Mediapipe pose algorithm allow for kinematics
monitoring. They were used to capture and track the coordinates of the total body anatomical landmarks
at 30 Hz. A classification model, trained on data from four participants across 23 exercises, extracted
from the ”PhysioTherapy eXercises” database [12] due to their capacity to elicit a range of intensities
and body districts, utilizes a Support Vector Machine with a radial basis function kernel to identify the
task currently performed by the participant running at 15 Hz.
Figure 2: Experimental setup used in the current study.
To quantify the relationship between the prescribed exercise intensity and the participants’ physiolog-
ical responses, we computed the cumulative intensity for each session (Σ𝐼). Specifically, the cumulative
intensity at the 𝑖-th exercise, Σ𝐼 (𝑖), was calculated as the sum of the intensities of all exercises up to that
point:
𝑖
Σ𝐼 = ∑ [𝐼 (𝑎𝑗 )] (1)
𝑗=1
where 𝐼 (𝑎𝑗 ) represents the intensity of the 𝑗-th exercise. This cumulative measure reflects the progressive
workload imposed by the exercises throughout the session.
We then computed the Pearson correlation coefficient between Σ𝐼 and the HR data collected during
the session to assess how closely the actual physical workload experienced by the participants is aligned
with the planned intensity of the exercises. By examining this relationship, we aimed to determine
whether physiological monitoring can provide reliable feedback on the intensity of the administered
exercises, offering valuable insights for adjusting and optimizing future rehabilitation sessions. The
statistical significance level was set at 𝑝-value= 0.05.
3. Results and Discussion
Figure 3A illustrates the mean Σ𝐼 and HR along with their standard deviations across the 10 participants
during the execution of the 10 administered physical exercises. It is worth observing that both metrics
increase steadily as the session progresses. Σ𝐼 rises consistently with each exercise, reflecting the
progressively demanding nature of the session. Similarly, HR shows a positive trend, indicating an
increase in physical exertion as participants advance through the proposed exercises. The shaded
areas around the lines represent the standard deviation, showing a moderate level of variability in both
intensity and HR responses across participants.
A) B)
Figure 3: A) Cumulative intensity (Σ𝐼) of the administered exercises and Heart Rate (HR) expressed in [bpm]
averaged on the enrolled participants. B) Linear correlation coefficients for each subject between Σ𝐼 and HR.
The 𝑝-values are written in white inside each corresponding bar. The dotted line highlights the mean correlation
(𝜌 ̄ = 0.63).
Figure 3B shows the linear correlation coefficients (𝜌) and p-values between Σ𝐼 and HR computed for
each subject. Most of the enrolled subjects (S1, S2, S3, S5, S8, S10) show significant positive correlations,
with values ranging from 0.63 to 0.88, indicating that heart rate tends to increase with higher exercise
intensity. Subjects S4 and S6 exhibit moderate correlations close to the significance threshold, while S7
and S9 display weaker, non-significant correlations, with p-values above 0.05.
Overall, these results suggest that for the majority of participants, there is a strong and significant
relationship between Σ𝐼 and HR, reinforcing the utility of physiological monitoring to track the physical
workload during rehabilitation sessions. The weaker, non-significant correlations for some subjects
may indicate individual variability in response to exercise intensity.
Given the established link between cumulative exercise intensity and physical workload, the in-
tegration of HR-based physiological feedback into an automated planning system for robot-aided
rehabilitation could significantly enhance the personalization of therapy sessions. In particular, real-
time HR monitoring allows the system to dynamically assess whether a patient is being over- or
under-challenged with respect to their clinical goals. When a patient’s HR indicates excessive exertion,
the system could leverage this feedback to re-plan or adjust the session in real-time, ensuring that the
therapeutic exercises remain within the proper intensity range.
4. Conclusions
This study demonstrated how integrating physiological monitoring can provide crucial insights into
assessing the actual physical workload during robot-aided rehabilitation sessions. The results showed a
significant correlation between the cumulative intensity of the prescribed exercises and the HR for most
participants. On average, a correlation of 0.63 was found across all participants, indicating a strong
relationship between exercise intensity and physiological response. This reinforces the importance of
incorporating physiological measurements into automated planning systems, moving beyond the sole
reliance on kinematic data.
The ability to use real-time HR feedback allows the system to dynamically re-plan rehabilitation
sessions, adjusting exercise intensity to avoid overloading or under-challenging the patient with respect
to the assigned clinical goals. This flexibility introduces a new level of personalization in treatment,
improving the effectiveness of therapy and ensuring better alignment with the patient’s needs.
Future efforts will focus on exploring combinations of physiological metrics to assess both physical
and cognitive workload, providing a more comprehensive understanding of patient exertion during
rehabilitation. Additionally, this higher-level feedback will be integrated into the session planner to
tailor rehabilitation sessions according to specific therapist requirements. Validation of this approach
will be necessary to ensure its capability to deliver personalized therapeutic strategy.
Acknowledgments
This work was supported by the Italian Ministry of Research, under the complementary actions to the
NRRP “Fit4MedRob - Fit for Medical Robotics” Grant PNC0000007, (CUP: B53C22006990001).
References
[1] O. Coser, C. Tamantini, P. Soda, L. Zollo, Ai-based methodologies for exoskeleton-assisted
rehabilitation of the lower limb: a review, Frontiers in Robotics and AI 11 (2024) 1341580.
[2] H. I. Krebs, D. J. Edwards, B. T. Volpe, Forging mens et manus: The mit experience in upper
extremity robotic therapy, in: Neurorehabilitation Technology, Springer, 2022, pp. 597–621.
[3] M. MacNeil, E. Hirslund, L. Baiocco-Romano, A. Kuspinar, P. Stolee, A scoping review of the use
of intelligent assistive technologies in rehabilitation practice with older adults, Disability and
Rehabilitation: Assistive Technology 19 (2024) 1817–1848.
[4] C. Tamantini, F. Cordella, C. Lauretti, F. S. di Luzio, M. Bravi, F. Bressi, F. Draicchio, S. Sterzi,
L. Zollo, Patient-tailored adaptive control for robot-aided orthopaedic rehabilitation, in: 2022
international conference on robotics and automation (ICRA), IEEE, 2022, pp. 5434–5440.
[5] J. C. González, J. C. Pulido, F. Fernández, A three-layer planning architecture for the autonomous
control of rehabilitation therapies based on social robots, Cognitive Systems Research 43 (2017)
232–249.
[6] J. C. Pulido, J. C. González, C. Suárez-Mejías, A. Bandera, P. Bustos, F. Fernández, Evaluating
the child–robot interaction of the naotherapist platform in pediatric rehabilitation, International
Journal of Social Robotics 9 (2017) 343–358.
[7] J. C. Pulido, C. Suarez-Mejias, J. C. Gonzalez, A. D. Ruiz, P. F. Ferri, M. E. M. Sahuquillo, C. E. R.
De Vargas, P. Infante-Cossio, C. L. P. Calderon, F. Fernandez, A socially assistive robotic platform
for upper-limb rehabilitation: a longitudinal study with pediatric patients, IEEE Robotics &
Automation Magazine 26 (2019) 24–39.
[8] A. Martín, J. C. Pulido, J. C. González, Á. García-Olaya, C. Suárez, A framework for user adaptation
and profiling for social robotics in rehabilitation, Sensors 20 (2020) 4792.
[9] C. Tamantini, F. Cordella, N. L. Tagliamonte, I. Pecoraro, I. Pisotta, A. Bigioni, F. Tamburella,
M. Lorusso, M. Molinari, L. Zollo, A data-driven fuzzy logic method for psychophysiological
assessment: An application to exoskeleton-assisted walking, IEEE Transactions on Medical
Robotics and Bionics (2024).
[10] C. Tamantini, F. Cordella, F. Scotto di Luzio, C. Lauretti, B. Campagnola, F. Santacaterina, M. Bravi,
F. Bressi, F. Draicchio, S. Miccinilli, et al., A fuzzy-logic approach for longitudinal assessment of
patients’ psychophysiological state: an application to upper-limb orthopedic robot-aided rehabili-
tation, Journal of NeuroEngineering and Rehabilitation 21 (2024) 202.
[11] A. Umbrico, M. Benadduci, R. Bevilacqua, A. Cesta, F. Fracasso, E. Maranesi, A. Orlandini, G. Cortel-
lessa, Combining clinical and spatial constraints into temporal planning to personalize physical
rehabilitation, in: Proceedings of the International Conference on Automated Planning and
Scheduling, volume 33, 2023, pp. 532–540.
[12] J. Glinsky, L. Harvey, C. Sherrington, O. Katalinic, www. physiotherapyexercises. com–new
exercises and features to help physiotherapists prescribe home exercise programs, Physiotherapy
101 (2015) e1381.