Health services personalization and early risk prediction represent the main research challenges in m-health systems, which can be achieved through the use of AI algorithms and tools applied to physiological and behavioural data collected by wearables and IoT devices in real-world settings. In this paper we present a summary of the results we obtained in our research activities in this area and future works, with particular attention to AI-empowered m-health systems as support for personalised rehabilitation services and malnutrition risk assessment, mobile sensing data analysis for disease detection and the identification of new health and behavioural markers that can support remote patient monitoring and the clinical practice.
bilitation therapies based on physiological data analysis, to the collection and analysis of mobile sensing data for The collection and analysis of mobile sensing data de- the fast screening of some diseases, and the integration rived from personal and wearable devices open the way of heterogeneous sensing data (both physiological and to the identification of predictive relationships between behavioural) aimed at identifying new digital markers for human habits, behaviour and health [1]. In addition, the individual health and well-being conditions. These syssame data, collected in-the-wild and for a medium or tems can be customised both for healthy ageing people, long period in a semi-continuous way, contribute to the aimed at maintaining a good autonomy and quality of generation of a Digital Phenotype [2] of the individual life, and for patients, in order to provide patient-centred that can be used to predict risks and early identify digital and integrated care pathways. markers of important diseases. This type of data includes The multidisciplinary characterisation of this research daily routines, sleep patterns, physical mobility, nutri- provides multiple impacts: (i) a technological impact with tion, cognitive functioning, speech production, social the definition of new AI-empowered decision support interactions, and many others, which can be selectively systems aimed to provide personalised feedback and thercollected based on the individual health profile. m-health apies; (ii) a medical impact, through the identification of applications, integrated with IoT and AAL systems, rep- new digital physiological and behavioural markers that resent the main infrastructure of the monitoring systems, can support the clinical diagnosis and the remote patient but today they must be enriched with AI algorithms to monitoring; (iii) a social impact by supporting people automatically detect specific health conditions, risky and with daily and unobtrusive monitoring instruments and adverse situations, and to implement personalized inter- personalised feedback, reducing the impact on the naventions. In addition, if the personal mobile device of the tional health systems and allowing caregivers to predict user is able to preliminary process this data directly on- health trajectories over time. board, it can avoid the transmission of a huge amount of In addition to the predictive performance analysis of sensitive data to the cloud, preserving the user’s privacy these systems, it is also essential to investigate how their and improving the system’s trustworthiness. results can be interpretable and explainable in order to
In our research activity we focus on diferent aspects in improve their acceptance and validation in the clinical this field: from the definition of customised clinical stud- practice and the users’ trust. To this aim we started also ies aimed at automatically defining personalised reha- investigating the latest Explainable AI (XAI) methodologies with particular attention to those used for time series and tabular data [3].
In this works we present the main results of our research activity divided in 3 main areas: • AI-empowered m-health systems for personalised rehabilitation and nutrition support in older adults • Smartphone-embedded sensing for the early de- ing several machine learning (ML) techniques exploiting tection of specific diseases the study protocol as implicit ground truth as stressful • Definition of behavioural and physiological mark- and non-stressful events’ labels. No self-report questioners from wearables and environmental sensors naires or clinical evaluations have been used to track the aimed at identifying the health status and early user perceived stress during the study. Then, in order to predicting risk conditions for active and healthy improve the stress detection resolution in terms of numageing. ber of detected stress levels, we evaluated few prominent ML and deep learning models for solving time series regression tasks [7]. We performed this analysis also on 2. Stress Detection and two available multimodal physiological dataset for stress Malnutrition Risk Assessment and afect detection: WESAD [ 8] and SWELL [9]. The considered models have been trained using stress scores Chronic and acute stress conditions and risk of malnutri- derived from diferent clinical questionnaires aimed at tion represent two important factors in frail older adults. detecting the perceived stress level severity over the time We decided to focus part of the research activity on these during multiple conditions. Models have been evaluated two specific health domains since they represent daily on each subject separately by using Leave-One-Subjectconditions that can characterise both healthy and frail Out (LOSO) cross validation scheme to test their generalolder adults and they can be also integrated with other isation capabilities. The obtained results demonstrated behavioural parameters for the definition of a general that the selected predictive models as well as the used health and well-being index. stress ground truth may provide an accurate and detailed
As far as the automatic stress detection is concerned, individual stress tracking in most cases. In addition, those we focused on the analysis of Heart Rate (HR), Heart Rate models could be integrated into a DSS module of the Variability (HRV) and Electrodermal Activity (EDA) as m-health solution for online stress monitoring and they reference physiological stress sensing data that can be could be further evaluated on new datasets collected from easily collected from wearable sensors, according to [4]. rehabilitation sessions of diferent target users, including Recently, other studies have introduced also EEG, Blink neuro-degenerative patients (e.g., Parkinson). Reflex through surface EMG, and eye tracking data anal- Malnutrition is a serious and prevalent health problem ysis [5], but the related instruments are invasive and they in the older population, and especially in hospitalised or are not easily accepted, even in clinical environments. institutionalised subjects and an accurate and early risk
The identification of acute stress events are important detection is essential for prevention. Also in this case AIin older adults, considering that they are commonly af- empowered m-health systems may lead to important imfected by chronic stress conditions in the management provements in terms of a more automatic, objective, and of their health status. Cognitive and motor rehabilitation continuous monitoring and assessment. We addressed activities, proposed to prevent physical and cognitive de- the challenges in this health domain by exploiting a simcline, can thus represent an additional stressful condition ple yet eficient m-health application, called DoEatWell that could mitigate the positive efects of the therapy. (DEW) [10], designed to collect information about nutriTherefore, the analysis of physiological signals during tional preferences and intake integrated with body comthe rehabilitation therapy execution can allow the defini- position data collected by a smart bioimpedance scale (i.e., tion of personalised protocols defined according to the body weight, body mass index, basal metabolic rate, bone automatic identification of the induced stress level. mass, body fat, water, and muscle percentage). DEW has
We developed a m-health system aimed at collecting been originally designed to be used by LTC care givers, and analysing those data during specific rehabilitation but it can be customised also for independent living sceprotocols that has been validated and evaluated through narios. It has been deployed in a LTC facility in Italy a pilot study we conducted with a group of frail MCI from March 2018, and we collected and analysed data older adults living in a long-term care (LTC) facility [6]. in multiple trial periods (approximately 2 years) from Each participant has been monitored through two wear- a total sample of 42 subjects. Feature engineering and able sensors (i.e., Zephyr BioHarness chest strap for HR extraction has been performed in collaboration with a and HRV and SHIMMER EDA sensor) during specific medical specialist in order to define suitable, multimodal training sessions based on standard clinical cognitive input predictors. We first focused on estimating the daily training alternated by a light physical exercise by using intake of the major macro-nutrients, namely cereals, ania cycle-ergometer. The proposed m-health system has mal proteins, vegetables, and fruit, which also represent been designed to highlight the short-time improvements the main components of the “Healthy Eating Plate” [11] on the cognitive performances generated by the proposed for the Mediterranean diet. For what concerns body comphysical exercise, and the related stress response. The position assessment, we initially computed only the fat ifrst step has been to perform binary stress detection us- mass index, which is highly correlated with the malnutrition risk. Finally, we also included some behavioural parameters characterizing the completeness and variability of the main daily meals. This data analysis is in line with the information generally requested by the reference clinical screening tools, yet providing a more objective and quantitative assessment. In order to validate our approach as a supervised learning task, we relied on the availability of a periodical clinical screening made by a healthcare professional on a monthly basis through the Mini Nutritional Assessment Short-Form (MNA-SF) tool Figure 1: The experimented COVID-19 detection approaches: [12]. (1) extraction of handcrafted acoustic features from the audio We investigated the performances of 6 benchmark ML al- waveform, which are then classified by a shallow ML model; gorithms, namely Logistic Regression (LR) with Least Ab- (2) usage of a pre-trained deep learning model as features solute Shrinkage and Selection Operator (LASSO) regular- extractor, in series with a shallow model to classify the deep isation, Support Vector Machine (SVM), k-Nearest Neigh- audio embeddings; and (3) fine-tuning of the pre-trained deep bors (k-NN), Classification And Regression Tree (CART), model for both features extraction and classification. Random Forest (RF), and AdaBoost (AB), as well as diferent data imbalance management techniques ranging from imbalanced training, dataset oversampling (SMOTE), to Finally, we also investigated the utility of XAI techcost-sensitive learning. We also included ad-hoc algo- niques to validate the proposed solution, both in terms rithms to directly classify from imbalanced data, such as of objective assessment of the agreement between explaRandom Undersampling Boosting (RUSBoost) and Bal- nations generated by diferent methods for each model anced Random Forest (BRF). We implemented a repeated, separately, and a preliminary clinical validation to verify stratified random hold-out partition ( 70% training, 30% that the input-target relationships learned for the most test), then we performed hyperparameter tuning through relevant predictors are in line with the current evidence10-fold CV along with Bayesian optimization algorithm based assessment. at each iteration. The resulting best model configuration Specifically we analysed the following XAI techniques: has been evaluated on the held back test set, considering SHAP, LIME, Anchors, and feature permutations. The a comprehensive set of evaluation metrics to correctly model-specific explanation consistency assessment highaccount for data imbalance. lighted that each model privileges very similar input Obtained results show that tree ensemble models (i.e., subsets to drive predictions, with 90% of pairwise comRF, AB, and RUSBoost) provide high accuracy and recall parisons among rankings showing a degree of overlap in detecting individual nutritional status by combining ≥ 3 for the top-5 features. In terms of clinical validation, nutritional intake, dietary habits, and body composition we demonstrated that the global reasoning of the best perdata, with median values of 94% and 92%, respectively. forming models adheres to the well-established domain Results also indicate that cost-sensitive learning is the knowledge and guidelines, without showing any severe most efective method to deal with data imbalance in AI bias. As a result, it can be considered “human-like” to our case study, as it also pushes the other models close a large extent, thus enhancing model clinical credibility. to the best performers. Instead, not considering body composition data, which is available only in a population subgroup, the classification performances worsen, even 3. Smartphone-embedded sensing if the sample increases. for preliminary disease
We also extended the previous analysis by integrating additional heterogeneous information including demo- detection graphic, anamnestic, and clinical data (e.g., age, chronic Personal mobile devices, like smartphones and smartdisease, therapies) stored in the DEW user health profile. watches, represent pervasive instruments for the colThe obtained classification results slightly improved in lection of user-generated data and signals (e.g., sound, case body composition is considered, maintaining RF and voices, images) that can be analysed to early detect spegradient boosting models as the best ones, with accuracy cific diseases. In the last few years, during COVIDfrom 95% to 96% and 1 from 93% to 94%. In addition, 19 pandemic, new m-health solutions have been prothe new models combining only clinical and nutritional posed to collect and analyse audio signals generated by data highlighted significant performance gains, with acsmartphone microphones, focusing mainly on human curacy gain ranging from +8.4% to +17.3%, whereas respiratory functions like breathing, speech, and coughF1 increases from +16.7% to +22.5%, thus reaching ing [13]. Schuller et al. [14] have been the pioneers competitive results also on a larger population. in the investigation of how the automatic analysis of also investigated the benefits and drawbacks of feature speech and audio data can contribute to fight the pan- extraction with respect to fine-tuning applied only to demic crisis, presenting the potential of Computer Au- the final fully-connected layers of the neural networks dition techniques (CA, i.e., computer-based speech and (see Figure 1). Experimental results showed that the finesound analysis) [15]. Subsequently, researchers investi- tuned models perform considerably worse than their use gated the efective applicability of those techniques in as feature extractors to input a shallow classifier since real scenarios even though the collection of objective their performances mainly depend on the similarity of the data from large populations represented a big challenge. pretraining and target tasks [28]. Since the considered In fact, initial studies focused on small patients’ cohorts respiratory sounds considerably difer from the original trying to automatically distinguish between COVID-19 model’s training data, which actually include heterogecough and cough sounds related to other pathologies [16], neous sounds extracted from YouTube videos, they do while others developed mobile and web apps to directly not allow an efective fine-tuning of the analyzed models. collect crowdsourced datasets from the population [17]. A possible solution for this issue could be the fine-tuning These projects allowed the sharing of important COVID- not only of the final classification layers but also part of 19 datasets, which opened the way to other researchers the convolutional components dedicated to the feature to validate new AI tools to improve the accuracy of the extraction. However, this requires a considerable amount proposed systems. Specifically, we identified 3 datasets: of data that is not currently available in public COVIDCOSWARA [18] and Virufy [19] that are publicly avail- 19 audio datasets [29]. Moreover, since we would like able, and the Cambridge dataset [20] which we can ac- to investigate the performances of the proposed modcess through a data transfer agreement between CNR els as components of a m-health system, we provided and Cambridge University for research purposes. also a preliminary evaluation of the models’ complexity
Classification methods proposed in the literature can considering the best trade-of between the classification be distinguished in three categories. Those relying performances and the model’s size. on hand-crafted acoustic features, like basic frequency- Inspired by recent applications of Deep Learning in based and temporal features [21], but also sets of fea- the dermatology field [ 30], we investigated also the use tures especially designed for voice and paralinguistic of CNNs to propose a novel m-health system aimed at applications (e.g., COMPARE [22])This approach is gen- detecting mpox (formerly known as Monkeypox) from erally outperformed in classification by deep learning skin lesion pictures captured by smartphone cameras. models [23]. Therefore, researchers proposed a second Adopting Transfer Learning to fine-tune diferent preapproach converting the audio files into a visual rep- trained CNNs, we identified MobileNetV3 [ 31] as the resentation (e.g., time-frequency spectrogram or Mel- best performing model for our use-case scenario. We spectrogram) that can be used as input to a Convolu- evaluated it on a combination of available skin lesions’ tional Neural Network (CNN) model for both features datasets preprocessed to obtain homogeneous picture as extraction and classification, thus relying directly on DL those provided by a smartphone camera. Experiments remodels [16]. Due to the scarcity of public COVID-19 sults present an average accuracy of 93%, 87% sensitivity, respiratory sound data, the DL models’ training has been and 78% specificity in the binary classification setting, performed on small-size datasets, typically composed and 88% accuracy, 88% sensitivity, and 96% specificity in of a few hundred samples, thus risking overfitting and distinguishing mpox from other diseases that produce providing unreliable results. Therefore, we decided to similar skin lesions, including, for example, acne and investigate a hybrid approach, based on the integration chickenpox. Finally, we applied Grad-CAM as an XAI of hand-crafted features and DL models, focusing on the algorithm to validate the model’s predictions and the performance of the recently proposed Look, Listen and quantization technique to reduce its memory footprint Learn (L3-Net) [24] embedding model and YAMNET [25], by 4x with negligible degradation in terms of accuracy. comparing the obtained results with those obtained by In this way the proposed solution results to be easily previous works focused on VGGish [26] as deep features deployable on commercial mobile devices, performing extraction model [27] We compared the performances the whole data processing on the user’s device and thus of the aforementioned deep audio embedding models supporting the fast detection of mpox. on the same datasets and with the same tasks, performing a series of subject-independent experiments, and we demonstrated that L3-Net overcomes the other models 4. Digital Phenotyping for Active in terms of standard metrics with diferent parameters’ and Healthy Ageing configurations.
All those solutions take advantage of the Transfer Previous activities mainly focus on single health domains, Learning concept to deal with the shortage of COVID- but the next step towards a comprehensive analysis and 19 audio data. Therefore, to complete our analysis, we classification of individual health and well-being status
This research activity has been partially funded by the European Commission under H2020-INFRAIA-20191SoBigData-PlusPlus project, by the European Union Next Generation EU, in the context of The National Recovery and Resilience Plan, Investment 1.5 Ecosystems of Innovation, Project Tuscany Health Ecosystem (THE), CUP: B83C22003920001. passes from the integration of heterogeneous data related to multiple health domains. This allows the identification of digital biomarkers that can support clinical evaluations and increase the potentialities of remote and m-health systems. In this area we focus on healthy older adults population, which can exploit the proposed solution as an early predictor of risky conditions and prevention of ageing diseases. To this aim, we proposed m-health systems aimed at collecting and analysing daily monitoring data related to physical activity, sleep patterns, nutrition, and social interactions. Most of this data can be collected by the use of commercial wearables, like smartwatches, and dedicated smart devices installed in the home environment, with a limited request of user’s interaction.
However, several research challenges are still open in this area, starting from the scarcity of data labeling inthe-wild up to the definition of significant features from each domain and their appropriate fusion for eficient and afordable risks prediction. In addition, personalised interventions (mainly based on behavioural suggestions) can be implemented after a risk prediction in order to observe potential behavioural changes. This activity needs a huge quantity of heterogeneous data from a consistent group of people to provide significant accuracy results.
To this aim, in the framework of a recently funded PNRR project called Tuscany Health Ecosystem, we are planning to set up a large scale pilot to validate and evaluate the proposed solutions. sound analysis could contribute in the sars-cov- D. Roelen, S. Schnieder, E. Bergelson, A. Cristia, 2 corona crisis, Frontiers in Digital Health 3 A. Seidl, A. S. Warlaumont, L. Yankowitz, E. Nöth, (2021). URL: https://www.frontiersin.org/article/ S. Amiriparian, S. Hantke, M. Schmitt, The IN10.3389/fdgth.2021.564906. doi:10.3389/fdgth. TERSPEECH 2019 Computational Paralinguistics 2021.564906. Challenge: Styrian Dialects, Continuous Sleepi[15] K. Qian, X. Li, H. Li, S. Li, W. Li, Z. Ning, S. Yu, ness, Baby Sounds & Orca Activity, in: Proc. InterL. Hou, G. Tang, J. Lu, F. Li, S. Duan, C. Du, speech 2019, 2019, pp. 2378–2382. doi:10.21437/ Y. Cheng, Y. Wang, L. Gan, Y. Yamamoto, B. W. Interspeech.2019-1122.
Schuller, Computer audition for healthcare: Op- [23] H. Purwins, B. Li, T. Virtanen, J. Schlüter, S.-Y. portunities and challenges, Frontiers in Digital Chang, T. Sainath, Deep learning for audio sigHealth 2 (2020). URL: https://www.frontiersin.org/ nal processing, IEEE Journal of Selected Topics in articles/10.3389/fdgth.2020.00005. doi:10.3389/ Signal Processing 13 (2019) 206–219. doi:10.1109/ fdgth.2020.00005. JSTSP.2019.2908700. [16] A. Imran, I. Posokhova, H. N. Qureshi, U. Ma- [24] R. Arandjelovic, A. Zisserman, Look, listen and sood, M. S. Riaz, K. Ali, C. N. John, M. I. Hus- learn, in: Proceedings of the IEEE International sain, M. Nabeel, Ai4covid-19: Ai enabled prelim- Conference on Computer Vision (ICCV), 2017. inary diagnosis for covid-19 from cough samples [25] D. Ellis, Yamnet: A pretrained audio event clasvia an app, Informatics in Medicine Unlocked 20 sifier, https://github.com/tensorflow/models/tree/ (2020) 100378. URL: https://www.sciencedirect.com/ master/research/audioset/yamnet, 2019. science/article/pii/S2352914820303026. doi:https: [26] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gem//doi.org/10.1016/j.imu.2020.100378. meke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, [17] J. Han, C. Brown, J. Chauhan, A. Grammenos, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, A. Hasthanasombat, D. Spathis, T. Xia, P. Cicuta, K. Wilson, Cnn architectures for large-scale auC. Mascolo, Exploring automatic covid-19 diagnosis dio classification, in: 2017 IEEE International via voice and symptoms from crowdsourced data, Conference on Acoustics, Speech and Signal Proin: ICASSP 2021 - 2021 IEEE International Con- cessing (ICASSP), 2017, pp. 131–135. doi:10.1109/ ference on Acoustics, Speech and Signal Process- ICASSP.2017.7952132. ing (ICASSP), 2021, pp. 8328–8332. doi:10.1109/ [27] M. G. Campana, A. Rovati, F. Delmastro, E. PaICASSP39728.2021.9414576. gani, L3-net deep audio embeddings to improve [18] N. Sharma, P. Krishnan, R. Kumar, S. Ramoji, covid-19 detection from smartphone data, in: 2022 S. R. Chetupalli, N. R., P. K. Ghosh, S. Gana- IEEE International Conference on Smart Computpathy, Coswara — a database of breath- ing (SMARTCOMP), 2022, pp. 100–107. doi:10. ing, cough, and voice sounds for covid-19 di- 1109/SMARTCOMP55677.2022.00029. agnosis, Interspeech 2020 (2020). URL: http:// [28] M. G. Campana, F. Delmastro, E. Pagani, dx.doi.org/10.21437/Interspeech.2020-2768. doi:10. Transfer learning for the eficient detection of 21437/interspeech.2020-2768. covid-19 from smartphone audio data, Perva[19] G. Chaudhari, X. Jiang, A. Fakhry, A. Han, J. Xiao, sive and Mobile Computing 89 (2023) 101754.
S. Shen, A. Khanzada, Virufy: Global applicabil- URL: https://www.sciencedirect.com/science/ ity of crowdsourced and clinical datasets for ai de- article/pii/S1574119223000123. doi:https: tection of covid-19 from cough, arXiv preprint //doi.org/10.1016/j.pmcj.2023.101754. arXiv:2011.13320 (2020). [29] G. Vrbančič, V. Podgorelec, Transfer learning with [20] C. Brown, J. Chauhan, A. Grammenos, J. Han, adaptive fine-tuning, IEEE Access 8 (2020) 196197– A. Hasthanasombat, D. Spathis, T. Xia, P. Cicuta, 196211. doi:10.1109/ACCESS.2020.3034343. C. Mascolo, Exploring automatic diagnosis of covid- [30] B. Shetty, R. Fernandes, A. P. Rodrigues, R. Chen19 from crowdsourced respiratory sound data, Pro- goden, S. Bhattacharya, K. Lakshmanna, Skin leceedings of the 26th ACM SIGKDD International sion classification of dermoscopic images using Conference on Knowledge Discovery & Data Min- machine learning and convolutional neural neting (2020). URL: http://dx.doi.org/10.1145/3394486. work, Scientific Reports 12 (2022) 18134. URL: 3412865. doi:10.1145/3394486.3412865. https://doi.org/10.1038/s41598-022-22644-9. doi:10. [21] T. Drugman, J. Urbain, T. Dutoit, Assessment of 1038/s41598-022-22644-9. audio features for automatic cough detection, in: [31] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, 2011 19th European Signal Processing Conference, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, 2011, pp. 1289–1293. Q. V. Le, H. Adam, Searching for mobilenetv3, in: [22] B. W. Schuller, A. Batliner, C. Bergler, F. B. Poko- Proceedings of the IEEE/CVF International Conferrny, J. Krajewski, M. Cychosz, R. Vollmann, S.- ence on Computer Vision (ICCV), 2019.