In the digital era, Recommender Systems are a crucial component, commonly used in services such as music and movie streaming. Despite their widespread adoption, surprisingly little attention has been devoted to developing systems that can positively impact users' well-being and health. In an efort to combat the negative efects of a sedentary way of life on people's health and the subsequent rise in healthcare expenses, we introduce an encouraging approach, i.e. a recommender system that, through webcam-based monitoring of subject postures, suggests personalized exercise breaks to do directly near users' desks. Our system captures users' postures during work hours and employs 3D pose estimation to calculate key angles between shoulders, hips, and head. By identifying postural imbalances, we generate exercise recommendations using a Large Language Model (LLM). The system flags potential postural issues when angle thresholds are exceeded and prompts the LLM to provide tailored exercise suggestions. Our method's efectiveness is assessed by experts in the field. While the results are still preliminary, our approach deserves further investigation, with future developments likely to focus on enriching the data and refining the detection methods. The full-reproducible code is available at the following link: https://github.com/GaetanoDibenedetto/healthrecsys24
Extended periods of sitting have become unremarkable in modern lifestyles, but recent research
highlights the detrimental efects it can have on our health. Studies have shown that prolonged
sedentary behavior is linked to a higher risk of obesity and metabolic syndrome, characterized by
elevated blood pressure, high blood sugar, excessive abdominal fat, and unhealthy cholesterol levels
[
Building on our previous work focused on posture correction for ofice workers [
The main contributions of this work are summarized below: • 3D Keypoint Extraction Model: we propose a 3D Human Pose Estimation (HPE) keypoint extraction module able to detect the human pose from a single laptop camera; • Pose Classification Model : we develop a system able to detect wrong poses while sitting at a desk eficiently; • Angles Computation: we develop a vector-based angles computation module that calculates the angles between key body parts (shoulders, hips, and head) using 3D pose data. This helps detect postural imbalances by identifying deviations from normal alignment; • Recommendation Generation Module: a module that utilizes a LLM to generate personalized exercise recommendations based on detected posture issues. When key angle thresholds are exceeded, the system prompts the LLM to suggest corrective exercises tailored to the user’s specific posture; • Dataset publicly released: the dataset with the 3D keypoints extracted from frames, used for training and evaluating our model will be publicly released for replicability purposes on Zenodo1.
As highlighted in our previous work [
These limitations make it dificult to apply such systems in a typical daily routine. In our previous work, we introduced a posture classification approach tailored for ofice workers, which avoids the constraints imposed by related works. Specifically, we allowed users to record themselves while working, without imposing restrictions. In that study, we collected our own dataset and developed a system capable of posture classification using a multi-layer perceptron. The system ofered explanations based on data statistics and featured a personalized feedback module designed to correct user posture. Building on this, our current research aims to enhance the system by leveraging 3D HPE while maintaining the same dataset and classification technique. This approach shifts the focus towards creating a recommendation system to provide personalized exercise suggestions.
In today’s information-overloaded world, recommender systems have emerged as essential tools for navigating countless options. Whether we’re shopping, streaming, or socializing, these intelligent systems are constantly working behind the scenes to personalize our experiences. Generally, these systems work on a daily basis based on a user’s previous behavior in a system.
As discussed in [
A Physical Activity Recommender aims to recommend a daily routine of physical activities and workouts to the user, this could be based on data as user’s characteristics, his health status or other demographic information such as age and gender. One of the first work analyzed by us is RecFit [ 15], a context-aware recommender system, which systematically suggests physical activities based on the user’s context, e.g. risk tolerance, budget, location, weather, but not on user’s physical characteristics. However, it does not consider the user’s physical characteristics and is not specifically designed to address health concerns. Advancements in the field have incorporated more physical details into the recommendation process. For instance, in 2020, Ferretto et al. [16] developed a system for patients with arterial hypertension, generating personalized recommendations based on the patient’s age, gender, and physical condition. In 2021, Sengan et al. [17] proposed a system to prevent respiratory diseases, utilizing data on physical activity levels, heart rate, and respiratory rate. The most recent work we reviewed, DEEP-CARDIO [18], proposed in 2024, is a content-based system ofering physical and dietary recommendations for cardiac patients. Using Density-Based Clustering (DBSCAN), it categorizes patient data, such as class predictions for cardiovascular diseases, alongside age and gender. However, the recommendations are presented in a categorical format, which may be challenging for non-expert users to interpret, e.g., instructions like "Follow Type 1 diet plan. Exercise Tip: Walking stationary, cycling, rowing, or water aerobics".
We observed that while these systems produce positive outcomes for patients, they often present categorical outputs, which can be dificult for non-expert users to interpret . In contrast, our goal is to develop a system that not only delivers specific recommendations but also includes brief descriptions. This approach will help non-expert users easily understand the recommendations and the rationale behind them.
In recent years, following the introduction of Transformers [19], Large Language Models (LLMs) based on this architecture have been widely adopted across various machine-learning applications. Transformers have gained popularity primarily due to their attention mechanism, which improves the model’s ability to represent text by focusing on the semantics of surrounding words.
Similar works have been found with our aim in the area of recommendations generated by LLMs, but only based on movies or books, which are the most common applications in the recommendation area, thanks to the large availability of datasets. W.-C. Kang et al. [20] propose one of the studies that we explored, which analyzed multiple LLMs across diferent settings, i.e., zero-shot, few-shot, and fine-tuning, by feeding the LLM with a prompt representing the user profile based on their past item ratings. Another work proposed by Sanner et al. [21], focuses on sequential recommendations by incorporating item descriptions and user preferences. While these approaches, like ours, leverage LLMs for generating recommendations through prompting, these difers from our approach, which does not rely on an item-based dataset, due to the lack of data in our scenario. Instead, our system’s core lies in the Angles Computation module (Sec. 3.4), which detects anomalies in the user’s posture. These anomalies are prompt to the system to generate personalized physical exercise recommendations aimed at improving the user’s well-being.
Up to our knowledge, there is a noticeable gap in the literature related to the availability of a specific
public dataset within the domain of sitting poses. In order to bridge this gap, in our previous research,
we created our own dataset, where workers in ofice spaces have been recorded while sitting. No
constraints have been fixed in terms of camera angles or perspectives, minimum or maximum video
lengths, or number of videos. We gathered videos from 10 subjects, processed them to extract frames
and finally, they were annotated by humans. Some statistics of the dataset are reported in Tab. 1, all the
details about, collection, filtering and processing of data are already provided in [
Building on our previous 2D-based approach [
To guarantee consistency, we used the same architecture with same hyperparameters proposed with our 2D-based approach We used a Multilayer Perceptron architecture (Fig. 1), Adam optimiser and the BCE (Binary Cross Entropy) loss function, weighted with the class distribution in the training fraction. Training was conducted with a batch size of 20 on 1000 epochs with a learning rate of 0.001. The high number of training epochs is due to a slow learning process due to the high variability of the keypoints position. The best performing model based on the training loss was selected and saved to be used in the recommendation step.
The training phase of this architecture uses the keypoints extracted with MotionBERT previously described (Sec. 3.2). A data augmentation step is performed, considering the imbalance of the dataset, inserting the vertical flip of the correct postures. It is worth to note that only two subjects exhibited a quite balanced amount of correct posture data, leading to the consideration of a binary split: training and testing. Among these two subjects, the one with a more balance volume of data was designated as the sole subject for the test split, i.e. gd, while the remaining subjects were allocated to the training split. The results of the classification are shown in the Tab. 2.
We focus on computing specific angles related to the user’s posture, which are crucial for detecting imbalances and misalignments. The primary angles of interest are the inclination of the shoulders and hips, as well as the tilt of the head in both sideways and forward directions.
To compute these angles, we employ vector-based calculations. Using the 3D skeleton representation provided by MotionBERT, the position of each body part is defined as a point in 3D coordinate space. The vectors formed between these points allow us to determine their inclination relative to the coordinate axes. The angles are then derived using the dot product between these vectors and a reference axis, as detailed below:
Given two points representing diferent body parts, 1(1, 1, 1) and 2(2, 2, 2), we compute the vector between these points as v = 2 − 1. The general formula for computing the angle between two vectors A and B is: = cos− 1 ︂( A · B )︂
|A||B| where A · B is the dot product, and |A| and |B| represent the magnitudes of the respective vectors. The direction of rotation can be inferred using the cross product of the two vectors.
In our case, vector A corresponds to v, the vector connecting two body parts, and vector B is either
the -axis [
By analyzing these angles using vector-based methods and dot products, we aim to detect deviations from normal alignment that could indicate issues with posture. Unfortunately, we were unable to establish specific angle thresholds to definitively define "anomaly" for various body parts, also with the support of experts in the field. To address this, we propose a data-driven approach. After calculating all angles in our dataset, we will determine the average angle for each specific body part. This average will serve as a threshold to flag any angles that deviate significantly from the norm. This threshold-based approach will serve as the foundation for generating personalized recommendations in the subsequent stage.
(a) Human3.6M keypoints representation
(b) MMPose Extraction Module with the 3D HPE MotionBERT (c) Identify the 3D points of the shoulders (d) Example of the angle for the shoulder angle Figure 2: The figure illustrates an example of keypoints extracted from a frame, showcasing the computation of the shoulder inclination angle relative to the -axis. Subfigure (a) shows the representation of the keypoints in the HPE model used; Subfigure (b) presents the output of the Keypoint Extraction Module (Sec. 3.2); Subfigure (c) highlights the 3D pose with a focus on the shoulder keypoints; Subfigure (d) details the computation of the shoulder inclination angle (Sec. 3.4)
The angles, obtained as described previously, are used to detect anomalies in the user’s posture. These detected anomalies are inserted in a textual prompt to be used as input of an LLM. The prompt starts with the sentence "I’ve an incorrect posture caused by", than each anomaly is joint to the prompt with an "and" (except the first), and it ends with the sentence ", what exercise or stretching should I do?". If no anomalies are detected the prompt will be "If I had an incorrect posture what exercise or stretching should I do?". The system prompt provided to the model is "You are a medical assistant able to suggest a specific physical exercise for users with uneven body parts to prevent an increase in their unease". An example of the complete prompt provided to the LLM is shown in Listing 1.
< | b e g i n _ o f _ t e x t | > < | s t a r t _ h e a d e r _ i d | > system < | e n d _ h e a d e r _ i d | > You a r e a m e d i c a l a s s i s t a n t a b l e t o s u g g e s t a s p e c i f i c p h i s i c a l e x e r c i s e f o r u s e r s with uneven body p a r t s t o p r e v e n t an i n c r e a s e i n t h e i r u n e a s e . < | e o t _ i d | > < | s t a r t _ h e a d e r _ i d | > u s e r < | e n d _ h e a d e r _ i d | > I ’ ve an i n c o r r e c t p o s t u r e c a u s e d by S h o u l d e r I n c l i n a t i o n , what e x e r c i s e o r s t r e t c h i n g s h o u l d I do ? < | e o t _ i d | > < | s t a r t _ h e a d e r _ i d | > a s s i s t a n t < | e n d _ h e a d e r _ i d | > Our recommendation generation process is powered by Llama-3.1-8B-Instruct [25]. We deploy the model on a machine using 2 NVIDIA A16 16GB VRAM GPUs, which required from 50 to 70 seconds for the text generation. Using the transformers library2, we utilize the "text-generation" pipeline3 to generate the recommendations. The default hyperparameters from the pipeline are used, i.e. temperature=0.6, top-p=0.9, instead we set for the text generation length the "max_new_tokens=1024". An output example of the recommendation is shown in the Fig. 3.
Due to the limited availability of data, we were unable to conduct a quantitative evaluation. Instead, we conducted a preliminary evaluation by administering questionnaires to domain experts, with at least 5 years of background experience in the field, specifically two physiotherapists. While the number of enrolled experts was small, their insights were crucial for understanding the pipeline’s potential. They were asked to assess 10 diferent scenarios in which the user is seated in an incorrect posture. These scenarios were either manually selected by the evaluators from provided example images or uploaded by them using specific frames they found relevant. To facilitate this evaluation, we developed a website using a Gradio4 interface, allowing users to interact directly with our system. The interface presents a chat-based interaction where users can either upload their own image or select one from a preloaded list of examples. Upon submission of an image, the system processes it through the keypoint extraction module and displays the results in the chat. The system also identifies any posture anomalies, providing the respective inclination angles for each anomaly detected. Following this, the system generates personalized exercise recommendations based on the detected anomalies. After reviewing the recommendations, users are prompted to answer a set of five questions on a 1 to 5 scale, followed by an open-ended feedback question. An example of interaction with the interface is shown in Fig. 3. The expert feedback collected through this process forms the basis of our evaluation. The questionnaire provided to the experts consists of the following questions: • Q1 - Do you agree with the recommended exercises based on the user’s posture analysis? Rating: 1 (Totally Disagree) to 5 (Totally Agree)
• Q2 - Do the recommended exercises adequately address the user’s posture issue and imbalances?
Rating: 1 (Totally Inadequate) to 5 (Totally Adequate) • Q3 - Are the exercises clearly explained and easy to understand? Rating: 1 (Very Unclear) to 5 (Very
Clear) • Q4 - In your opinion, how efective would these exercises be in improving the user’s posture? Rating: 1 (Not Efective at All) to 5 (Highly Efective) • Q5 - Are the recommended exercises simple enough for the user to perform without expert supervision?
Rating: 1 (Not Simple at All) to 5 (Very Simple) • Q6 - What improvements would you suggest for this recommendation? With these questions, we evaluate the correctness (Q1, Q2, Q4), accuracy(Q1), adequacy(Q2), clarity(Q3), efectiveness(Q4) and the simplicity of the execution (Q5) of the recommendation. The average rating for each question, on a scale from 1 to 5, is presented in Tab. 3. Overall, while none of the average results were negative (i.e., below 3), they were also not exceptionally high, except for Q3, which exceeded a rating of 4. This indicates that the clarity of the exercise recommendations was appreciated, but there is significant room for improvement in other areas. In particular, the average score for the "correctness" criterion was 3.4. After discussions with the evaluators, it became clear that, also if they appreciated the concept behind the system, they noted that it was dificult to determine whether a recommended exercise is entirely appropriate or not, starting with limited user characteristics, such as a single image. They emphasized that many more variables need to be considered for an accurate recommendation.
From the expert open-feedback, particularly the insights provided by physiotherapists, we identified several areas for improvement; these are discussed below to better illustrate limitations and future work.
This study has introduced a system for physical activity recommendation based on assessment of sitting postures utilizing accessible cameras, vector-based angles computation, and a LLM. Despite the promising results, several limitations were identified through open feedback evaluations: • When the system detects uneven shoulders, it lacks to recommend exercises targeting the cervical region, which plays a significant role in connecting shoulder alignment and head posture; • When the subject is sliding down in their chair (example in Fig. 3), the system may only detect misalignment in the head’s inclination relative to the body; • The angles computation module does not recognize shoulder lifting and excessive shoulder closure; • The system should suggest to perform exercises in front of a mirror, which could help users maintain correct form.
As future work, based on these limitations, several ways for improvement have been identified. First of all, we need to improve the angle calculation module to detect a wider range of postural anomalies, enabling more accurate and varied exercise recommendations. Additionally, there is a need to explore the use of diferent LLMs to evaluate their performance in this recommendation task, and last but not least, a change of module that suggests the exercises, which could be based on a collected historical user data. These data are taken from the interactions of the current system, specifically combining the recommendations and the expert feedback: by storing the data and ratings provided by experts, the system could ofer better recommendations to users with similar postural issues, leveraging the best-rated advice from prior interactions. Although the experimental results presented in this paper are preliminary, we can see a great potential to improve them. The exercise descriptions generated by the LLMs were particularly appreciated for their clarity, highlighting the possibility of using LLMs to improve users’ understanding. In the future, by treating past recommendations and detected physical anomalies as items of a recommender system, we can refine the system in terms of correctness, adequacy, accuracy, and efectiveness.
The research is partially funded by PNRR - Mission 4 ("Education and research") – Component 2 ("From research to business"), Investment 3.3 ("Introduction of innovative doctorates that respond to the innovation needs of companies and promote the hiring of researchers by companies") D.M.n. 117/2023 CUP: H91I23000170007 and is supported by the co-funding of the European Union - Next Generation EU: NRRP Initiative, Mission 4, Component 2, Investment 1.3 – Partnerships extended to universities, research centers, companies, and research D.D. MUR n. 341 del 15.03.2022 – Next Generation EU (PE0000013 – “Future Artificial Intelligence Research – FAIR” - CUP: H97G22000210007). We extend my sincere gratitude to Naps Lab S.r.l.s.5 for their support and collaboration in the realisation of this research.
We would also like to thank Ilaria Vianello6 and Michele Dilillo7, who provided invaluable assistance in the evaluation phase of this research. Figure 3: Our system with Gradio Interface Sixth International Conference on eDemocracy & eGovernment (ICEDEG) (2019) 47–55. URL: https://api.semanticscholar.org/CorpusID:189824028. doi:10.1109/ICEDEG.2019.8734362. [15] Q. He, E. Agu, D. M. Strong, B. Tulu, Recfit: a context-aware system for recommending physical activities, in: S. K. S. Gupta, A. Banerjee (Eds.), Proceedings of the 1st Workshop on Mobile Medical Applications, MMA ’14, Memphis, Tennessee, USA, November 3-6, 2014, ACM, 2014, pp. 34–39.
URL: https://doi.org/10.1145/2676431.2676439. doi:10.1145/2676431.2676439. [16] L. R. Ferretto, E. A. Bellei, D. Biduski, L. C. P. Bin, M. M. Moro, C. R. Cervi, A. C. B. D. Marchi, A physical activity recommender system for patients with arterial hypertension, IEEE Access 8 (2020) 61656–61664. URL: https://doi.org/10.1109/ACCESS.2020.2983564. doi:10.1109/ACCESS. 2020.2983564. [17] S. Sengan, S. V, R. H. Jhaveri, V. Varadarajan, R. Setiawan, L. Ravi, A secure recommendation system for providing context-aware physical activity classification for users, Security and Communication Networks 2021 (2021) 1–15. doi:https://doi.org/10.1155/2021/4136909. [18] A. Yashudas, D. Gupta, G. C. Prashant, A. Dua, D. AlQahtani, A. S. K. Reddy, Deep-cardio: Recommendation system for cardiovascular disease prediction using iot network, IEEE Sensors Journal 24 (2024) 14539–14547. doi:10.1109/JSEN.2024.3373429. [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017, pp. 5998–6008. URL: https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. [20] W. Kang, J. Ni, N. Mehta, M. Sathiamoorthy, L. Hong, E. H. Chi, D. Z. Cheng, Do llms understand user preferences? evaluating llms on user rating prediction, CoRR abs/2305.06474 (2023). URL: https: //doi.org/10.48550/arXiv.2305.06474. doi:10.48550/ARXIV.2305.06474. arXiv:2305.06474. [21] S. Sanner, K. Balog, F. Radlinski, B. Wedin, L. Dixon, Large language models are competitive near cold-start recommenders for language- and item-based preferences, in: J. Zhang, L. Chen, S. Berkovsky, M. Zhang, T. D. Noia, J. Basilico, L. Pizzato, Y. Song (Eds.), Proceedings of the 17th ACM Conference on Recommender Systems, RecSys 2023, Singapore, Singapore, September 18-22, 2023, ACM, 2023, pp. 890–896. URL: https://doi.org/10.1145/3604915.3608845. doi:10.1145/ 3604915.3608845. [22] W. Zhu, X. Ma, Z. Liu, L. Liu, W. Wu, Y. Wang, Motionbert: A unified perspective on learning human motion representations, in: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, IEEE, 2023, pp. 15039–15053. URL: https://doi.org/10.1109/ ICCV51070.2023.01385. doi:10.1109/ICCV51070.2023.01385. [23] M. Contributors, Openmmlab pose estimation toolbox and benchmark, https://github.com/ open-mmlab/mmpose, 2020. [24] C. Ionescu, D. Papava, V. Olaru, C. Sminchisescu, Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments, IEEE Trans. Pattern Anal. Mach. Intell. 36 (2014) 1325–1339. URL: https://doi.org/10.1109/TPAMI.2013.248. doi:10.1109/TPAMI.2013.248. [25] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., The llama 3 herd of models, CoRR abs/2407.21783 (2024). URL: https: //doi.org/10.48550/arXiv.2407.21783. doi:10.48550/ARXIV.2407.21783. arXiv:2407.21783.