Introduction

Artificial Intelligence (AI) has proven itself as enabling factor for triggering great transformations of society [1,2,3,4]. However on the verge of the fifth industrial revolution, there are several challenges that involve the consolidation of AI arrival in sectors as medicine and people well-being. Indeed this paradigm shift towards AIdriven healthcare is not just a technological revolution; it represents a comprehensive reimagining of medical practices, enhancing the quality, efficiency, and accessibility of healthcare services. In this scenario our efforts are directed towards four research paths: (i) multimodal AI for precision medicine (section 2); (ii) multimodal AI to foster wellbeing (section 3); (iii) resilient AI (section 4); (iv) AI in robotics for healthcare (section 5). For each of these routes we provide a brief description of the developed solutions, highlighting solved problems and open challenges.

Multimodal AI enables precision medicine

The evolution of precision medicine marks a paradigm shift from the traditional "one-size-fits-all" approach in healthcare towards tailored therapeutic strategies that account for individual variability in genes, environment, and lifestyle. In this context, leveraging the variety of patient generated data (as images, clinical data, electronic health records etc.) can provide a significant boost to unlocking a holistic view of the patient. Towards this end multimodal AI provides the ultimate tool [5,6]: the integration is not merely additive, it's transformative, enabling the extraction of insights that would remain obscured under traditional, unimodal analysis. We are currently studying the potential of multimodal AI for precision medicine facing different challenges in different application domains: in the oncological domain we face challenges regarding data fusion and representation, with two projects on Non-Small Cell Lung Cancer (NSCLC) (sections 2. As a prior contribution, EHRs are vital resources for documenting patient clinical history and procedures, but are often challenging to process due to their unstructured nature. Natural Language Processing (NLP) tools, particularly Named Entity Recognition (NER) with the use of Transformer-based models, have proven effective in extracting meaningful information from EHRs [7]. Transformers excel at capturing contextual relationships between words and the still not thoroughly explored contextual embedding they create can enhance the understanding of the content itself. We propose the Hieararchical Embedding Attention for overall survivaL (HEAL), a methodology that leverages multi-class NER-driven representations from EHRs by weighting them with attentional mechanisms. The ability of emphasizing clinically relevant information within unstructured data, operating both at word and sentence levels, makes HEAL more interpretable for medical applications. In a NSCLC Overall Survival (OS) prediction case study, HEAL achieved an average 𝐶 𝑡𝑑 -index of 0.639 and a low standard deviation of 0.014 over 5 runs, showing a statistically significant superiority with respect to manually extracted clinical features.

Our second contribution, even if still at its preliminary steps, grounds on the fact that deep learning (DL) approaches have demonstrated significant value in automatically learning potentially relevant patterns from medical images, such as computed tomography (CT) [8]. Hence, in this study we explore a novel methodology for predicting OS in NSCLC patients using only CT images, aiming at a multitask architecture that encompasses prognostic factors like Progression-Free Survival (PFS) beyond predicting OS alone. The first steps in this direction include producing a soft attention weighted feature map for each input slice and highlighting the relevant slices crucial for predicting OS outcome.

PICTURE

PICTURE stands for "Pathological response AI-driven prediCTion after neoadjUvant theRapiEs in NSCLC". This project is based on the central hypothesis that heterogeneous medical data (i.e. radiological images, histology images, cytology and molecular data and EHRs) are consistent with the pathological complete response (pCR), so their combination using artificial intelligence (AI) can provide accurate pCR prediction in NSCLC patients. Indeed, albeit treating locally advanced NSCLC surgically is the mainstay, it is important to prevent post-surgery recurrence, and neoadjuvant therapy (NAT) has shown potential in enhancing overall survival rates and achieving a complete pathological response, that, if correctly evaluated before the treatment, can even avoid non necessary surgical resections.

PICTURE pursues three objectives: (i) pCr prediction through radiology imaging, histology, citology, molecular data, EHRs, and their combination; (ii) leveraging multimodal deep learning to make the performance of AI resilient and robust for pCR prediction signature; (iii) improving trust and transparency using explainable AI models. PICTURE also has the exploratory aim of transferring trained models to predict pCR for patients undergoing chemoimmunotherapy, tailoring treatments to the individual needs of patients.

Facing imbalance in Alzheimer's Disease diagnosis and prognosis

Alzheimer's disease (AD) is a progressive neurodegenerative condition with decline in cognitive function, and because of the lack of a cure, its early detection is paramount. Despite the recent progress in AI, challenges such as class imbalance, integration of multimodal data, and robust generalization remain pervasive. In response to this we introduce a novel methodology that leverages the strengths of ensemble learning while incorporating advanced fusion techniques. For each of the 4 modalities of the tabular ADNI database, we train a series of classifiers on varied class distributions followed by a late fusion strategy that integrates the different modalities to improve the results. Our framework is evaluated on two diagnostic tasks (binary and ternary) and four binary prognostic tasks (at 12, 24, 36, and 48 months) and compared with 12 stateof-the-art imbalanced data algorithms, achieving 97.04% g-mean on the binary diagnostic task and 90.81% g-mean on the 48-month prognostic task.

Multi-Dataset Multi-Task Learning for COVID-19 Prognosis

In COVID-19 context [9] in order to fight the scarcity of large, labelled chest radiographic images (CXR) datasets, we introduce a novel multi-dataset multi-task (MDMT) training framework, by integrating correlated datasets from disparate sources and assessing severity score to classify prognostic severity groups [10,11,12], instead of relying on datasets with multiple and correlated labelling schemes. As illustrated in figure 1 a deep CNN takes the images as input and branches into task-specific fully connected output networks, to end with a multi-task loss function incorporating an indicator function to exploit multi-dataset integration. Proceeding with a 5 cross-validation and leave one center out training, we evaluated the method across 18 different CNN backbone on prognosis classification task and fine-tuning from BRIXIA dataset to AIforCOVID dataset task. Best average performance with statistical robustness achieved: 68.6% accuracy, 66.6% F1-score and 68.5% g-mean for the 5 cross validation, and 65.7% accuracy, 64.3% F1-score and 66.0% g-mean for the leave-onecenter-out validation strategy. Future directions include new domains and the integration of XAI [6].

Multimodal AI to foster well-being

Stress, a response to physical and emotional demands, is crucial in determining individuals' well-being and if unmanaged can lead to conditions such as anxiety, depression and cardiovascular diseases. Also in this scenario multimodal AI offers a tool for proactive approach to health management, in order to provide real-time monitoring and interventions, thereby mitigating long-term health risks associated with chronic stress. We are targeting stress detection from two perspectives: first, we are focusing on maximising the stress level prediction accuracy within the shortest possible time, exploiting multimodal physiological time series data and Deep Reinforcement Learning (DRL); second, we are further expanding the multimodal view integrating information from video, audio and text with the physiological data. Robust and fast stress detection approaches can bring benefit in several contexts: from providing a targeted and more personal assistance to patients, to ensuring safety for workers, for instance, Air Traffic Controllers (ATC) that endure high levels of psychological pressure during their job impacting operational safety.

Our first approach [13] employs a new DRL model to identify stress indicators. We obtained this by leveraging a dynamic time observation window that expands each step of the learning process, asking the agent to choose either to continue observing or to classify based on the information gathered until that point, trying to minimize the amount of data required for decision-making. As depicted in the figure Figure 2, we adopted the Soft Actor-Critic algorithm for its effectiveness in handling continuous action spaces. In a Leave-One-Subject-Out approach with data augmentation on the Non-EEG public dataset we outperformed existing solutions, showing the power of DRL for early stress detection.

On top of this approach we are exploring the larger multimodal asset of the Ulm-Trier Social Stress Test dataset (ULM-TSST, MuSe 2022 challenge), containing 41 training, 14 validation and 14 test subjects, simulating a job interview scenario, with audio, video, text, and physiological data modalities, rated on arousal and valence stress parameters. The aim is to build a high performance architecture that leverages non invasive modalities for stress detection that can be employed in work environments. In both cases the scarcity of large datasets that provide a quantitative measurement of stress is still the main challenge: we will try to face it considering the construction of robust and specific acquisition protocols to test the effectiveness of the developed approaches in real-world scenarios.

Resilient AI

Due to the high stakes involved in healthcare decisions, the sensitivity of medical data, and the complexity of medical environments, AI systems should be designed to maintain their intended performance and integrity in the face of adversities, as data corruption, missing data, privacy leakage, or unexpected changes in their operating environment. This is the goal of Resilient AI, which is an aspect that cannot be left out when the aim is to integrate AI for augmenting the medical practice.

To meat this goal we are currently investigating three main aspects that fall under the Resilient AI umbrella: developing systems robust to missing data, the challenge of limited extension datasets and how to protect sensitive patients data.

With respect to the missing data challenge [14], although a variety of strategies exist for addressing this problem in health datasets, to overcome the obstacle to select the most suitable one and their dependency on the dataset's specifics, we developed a Transformer-based model [15] that applies masking to ignore the missing data, thus eliminating the need of imputation and deletion techniques and focusing directly only on the available features through self-attention. Moreover, we introduced a novel feature-identifying form of positional encoding to facilitate the integration of tabular data into a Transformer framework. This method was validated through an overall survival classification task, employing clinical data from the CLARO [16] project and improving the prediction accuracy.

In order to address the problem of working with datasets limited in the extension, particularly frequent in healthcare domain, Triplet networks, a subtype of the Siamese networks, emerge as a promising solution, comprising three identical networks operating concurrently. Throughout training of these three networks two inputs belong to the same class, whereas the third belongs to a distinct class, with the final objective to develop a feature space with two distinct clusters one per class by incorporating inter-class diversities alongside intra-class similarities and providing scenarios with limited data with more triplets compared to instances (Figure 3). In our study [17], using a private dataset of 86 CT scans, triplet networks surpass the plain deep networks in accurately predicting the histological subtypes of NSCLC patients. Currently, we are broadening the scope of our research including PET images alongside CT scans and adopting a multimodal strategy for the same classification. By integrating these complementary data we anticipate achieving a significant improvement and overcoming the challenges posed by limited data scenarios. Last but not least, the challenge related to patient privacy led us to explore Federated learning (FL). FL presents an innovative solution to the challenge of protecting sensitive patient data in artificial intelligence applications in healthcare, in fact enabling the training of a shared global model with a central server while ensuring data privacy within local institutions. On this basis we introduce a new token-based FL paradigm, revolutionizing the traditional approach with sequential or random passing of a token between clients during each epoch. This innovative method allows only the token owner to send the weights to the server, which redistributes them directly to all models. By eliminating local training epochs and allowing immediate transmission, this paradigm shift streamlines the process by circulating a single model among clients and also mitigating the need for an initial warm-up period, potentially paving the way for a decentralized system that reduces dependence on a central server and minimizes the number of parameters transmitted in each iteration. Results on the tabular part of the AIforCOVID dataset [18] composed of 6 hospitals show that the performance of the FL model does not deviate from that of its equivalent trained on all datasets aggregated into a single pool. The next steps will focus on integrating other modalities into the FL pipeline, such as CXR scans of the AIforCOVID dataset itself.

AI for healthcare robotics

The integration of robotics in healthcare settings exemplifies another dimension of AI's impact, automating routine tasks, assisting in surgeries with precision beyond human capability, and providing rehabilitation support to patients. This not only enhances service delivery but also alleviates the workload on healthcare professionals, allowing them to focus more on patient-centered care. In this scenario we pursue two aims: first, enhancing robotic surgery with real-time high precision localiza- For the first objective we focus on the laparoscopy use case, as one of the preferred surgical methods. Despite recent advancements in image acquisition it is still limited to rely on 2D images view: misinterpreting anatomical structures due to this limit is a common source of errors. In contrast, 3D imaging increases the accuracy of instrument manipulation, leads to better outcomes in surgery, and shortens the learning for trainees. Even several research in surgical 3D imaging has been explored, like camera-based tracking and mapping, Mosaicking, Structure from Motion, and Shape from Template, they often rely on simplifications that can limit their effectiveness. On this ground Simultaneous Localization and Mapping (SLAM) has shown promising results, as it aims to create a map of the environment while localizing the sensor position within it. Therefore we developed a robust deep learning SLAM pipeline to operate in real-time across diverse surgical settings by providing an immersive, interactive 3D environment (Figure 4), allowing for more precise and personalized interventions with the future possibility to be integrated with augmented reality displays.

For the second objective, we focus on the challenges in the field of lower limb robotics. It aims at supporting people with lower limb disabilities by enhancing movement, mobility, and providing targeted exercise. Technologies as exoskeletons, prosthetics, and rehabilitation robots, are particularly helpful for those with neurological issues, offering improved rehabilitation, independence, and tailored care. Effective use requires precise control settings to adapt walking patterns to different terrains. Challenges in this field involve the extensive need for sensors for terrain detection and the complexity in processing sensor data. Simplifying sensor requirements to accurately determine terrain and slope is critical for user-friendly, efficient operation, and safety. The aim of our work is to recognize the terrain on which an exoskeleton is walking and its inclination. Among several state-ofthe-art driven approaches, we achieved promising results using LSTM architectures with IMU data (0.94 of accuracy in Leave-one-out cross-validation), and CNN-LSTM architectures with EMG data (0.75 of accuracy). The fusion of IMU and EMG data didn't bring any significant improvement, as explanatory tests indicated that the best 20 contributing features belong to IMU. Next, by varying the number of sensors, and therefore features, we noticed that the best results are achieved by selecting the most relevant features, from one to three, according to SHAP (on a 3 subjects validation set), leading to 0.85, 0.89 and 0.93 of accuracy respectively. Lastly, we found that LSTM and CNN-LSTM are valid architectures for slope inclination prediction (MAE of 1.95°) and stair height (MAE of 15.65 mm), without significant differences in employing 3 or 4 sensors.