Advancements in generative artificial intelligence (AI) are setting the stage for transformative changes in medical imaging, particularly through the development of the Virtual Scanner. This innovative approach leverages resilient generative AI to synthesize radiological images, addressing critical challenges in the field such as data scarcity, patient exposure to radiation, and the limitations of current imaging technologies. By harnessing the power of Generative Adversarial Networks (GANs) and focusing on the resilience of these algorithms, the Virtual Scanner aims to enhance diagnostic accuracy, improve patient care, and fill gaps in multimodal datasets. Our research explores both unimodal and multimodal techniques, including GAN ensembles, latent augmentation, and advanced texture synthesis, to create robust and adaptable generative models. Through extensive experimentation and analysis, we demonstrate the potential of the Virtual Scanner to revolutionize medical diagnostics by providing a safer, more eficient, and comprehensive imaging solution. The implications of this work extend beyond immediate medical applications, ofering insights into the development of AI technologies capable of navigating the complexities of real-world data.
ated with radiation and contrast agents. Furthermore, the reliance on comprehensive multimodal imaging data In recent years, the intersection of artificial intelligence presents challenges in scenarios where certain modalities (AI) and healthcare has opened up novel possibilities for are unavailable or unsuitable for some patients, leading enhancing diagnostic accuracy, optimizing patient care, to gaps in the data that can hinder diagnostic processes and tailoring treatment plans towards precision medicine. and the development of AI models in healthcare [1, 2, 3]. One of the most promising developments in this domain The advancement of generative AI, particularly is the concept of the Medical Digital Twin, a virtual repre- through the deployment of Generative Adversarial Netsentation of a patient’s health status, enabling personal- works (GANs), ofers a novel solution to these challenges. ized medical interventions and predictive healthcare ana- By enabling the virtual generation of radiological images lytics. Central to the utility and efectiveness of Medical where real ones are unavailable or undesirable, AI not Digital Twins is the capability for detailed and accurate only mitigates the risks to patients but also bridges the radiological imaging, which provides a window into the data gaps in multimodal learning applications [4, 5, 6]. internal workings of the human body without invasive We introduce the concept of the Virtual Scanner as a corprocedures. nerstone of the Medical Digital Twin paradigm, aiming to
Radiological imaging, encompassing a range of modal- revolutionize the field of radiology by synthesizing highities such as X-rays, MRI, and CT scans, plays a pivotal ifdelity, modality-specific images through the power of role in the diagnosis, monitoring, and treatment planning AI, thus enhancing patient care and supporting radiolofor a myriad of health conditions. However, the acquisi- gists in delivering more accurate diagnoses. tion of these images often requires patients to undergo The scarcity of comprehensive radiological images multiple scans, exposing them to potential risks associ- presents significant challenges in medical diagnostics, afecting the eficacy of diagnostic processes and the deItal-IA 2024: 4th National Conference on Artificial Intelligence, orga- velopment of AI tools. This scarcity arises from limited nized by CINI, May 29-30, 2024, Naples, Italy access to advanced imaging technologies, concerns over * Corresponding author. radiation exposure, and the dificulty of compiling di$ 0v0a0le0r-i0o0.g0u2a-1r8ra6s0i-@74u4n7ic(Vam.Gpuusa.rirta(sVi.) Guarrasi) verse, multimodal datasets. Such challenges hinder the © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License creation of efective AI models for diagnostics, impacting Attribution 4.0 International (CC BY 4.0). their accuracy and real-world applicability. The Virtual Pareto multi-objective optimization problem that simulScanner, leveraging resilient generative AI algorithms, taneously covers the real training set, aiming to generate addresses these issues by synthesizing radiological im- high-quality GANs and using as few GANs as possible. ages to fill dataset gaps and reduce the need for repeated We tested out methodology across three distinct medscans. Resilient generative AI refers to the development ical datasets, employing 22 GANs with difering archiof models that not only excel in their designated tasks un- tectures, loss functions, and regularization techniques. der ideal conditions but also maintain their performance Moreover, we uniformly sampled each model every 20000 when confronted with data that deviate from the norm, training iterations, i.e., resulting in a total search space known as “data in the wild". Such resilience is crucial in of 110 models. The experiments showcase that using ensuring that the AI tools developed for medical imag- synthetic datasets generated from such an ensemble iming are robust against the variations inherent in patient proves the performances in classification downstream data across diferent demographics, equipment used, and tasks compared to single GANs and Naive selection appathological conditions. proaches, i.e., using all available 110 GANs or randomly
By embedding resilience at the core of our generative selecting a subset.
AI algorithms, we aim to create a foundation for the Virtual Scanner that is not only technologically advanced 2.1.2. LatentAugment but also reliable and efective across the spectrum of medical imaging needs. This approach positions our work not just as a technical achievement but as a meaningful contribution to the field of radiology, where the capacity to handle data in the wild can significantly enhance diagnostic processes and patient care.
Data Augmentation (DA) is a crucial strategy in AI to enhance the volume and diversity of training datasets, thereby mitigating the risk of overfitting and bolstering model generalization to unseen data. Standard DA methods in image recognition tasks transform the images via geometric rigid and non-rigid transformations using image processing primitives, such as translation, rotation, 2. Research Activities cropping, etc. However, such transformations rely on human experts with prior knowledge of the dataset and Embarking on the journey to realize the Virtual Scanner, fail to generate suficiently diverse synthetic data. GANs our investigation delves into a series of research activ- ofer a valuable addition to the available augmentation ities, as shown in Figure 1, each designed to push the techniques. However, GANs generate high-quality samboundaries of what’s possible with generative AI in the ples rapidly, but they sufer from poor mode coverage, ifeld of radiology [ 7, 8]. These activities are categorized i.e., the variation and variety of the samples that can be into two main areas: “Resilient Generative AI" and “Vir- generated, limiting their utility for DA purposes in the tual Scanner Applications", enveloping a diverse array of medical field. methodologies and applications aimed at enhancing the We propose LatentAugment [9], a DA strategy that generation and translation of medical imaging data. By overcomes the low diversity of GANs, opening up for use addressing many aspects of generative AI, from improv- in DA applications. LatentAugment addresses the threeing algorithm resilience to creating virtual modalities, fold challenge of producing synthetic samples that are these activities underscore our commitment to advancing not only of high fidelity and quality but also diverse and diagnostic capabilities and patient care through techno- rapidly generated. LatentAugment modifies the latent logical innovation. vectors of the real training set, moving them towards regions that maximize their diversity and fidelity. We 2.1. Resilient Generative AI applied LatentAugment to improve the performances of a downstream model performing of MRI-to-CT im2.1.1. GAN Ensemble age translation. The results showed LatentAugment’s In tackling the complexities of synthetic data generation superiority over common DA methods and naive GANwithin medical imaging, our research delves into opti- sampling, i.e., creating data sampling from the GAN’s mizing generative AI through the use of GAN ensembles. latent space without any control.
This strategy is born from the necessity to overcome inherent limitations in single-model GAN applications, 2.1.3. Paired vs. Unpaired Image Translation such as mode collapse and the inadequate representation Image-to-Image translation in medical imaging presents of real data distributions, a common obstacle in generat- a critical challenge due to the predominance of uning high-quality and diverse medical images. The core paired datasets, where the direct correspondence beof our approach lies in creating an ensemble of GANs tween source and target images is not established [10]. that jointly optimizes the visual quality and diversity of While paired methods, e.g., Pix2Pix, rely on direct mapsynthetic images from a set of GANs. We aim to solve a (FDCT) images to validate our model, demonstrating its An extensive quantitative and qualitative analysis unefectiveness in producing high-quality, denoised images derpins our research, including evaluations by profesthat closely approximate FDCT standards, thereby mit- sional radiologists on a novel CESM dataset comprising igating health risks associated with radiation without 1,138 images. This dataset has been made publicly availcompromising diagnostic integrity. able to foster ongoing research and development in the ifeld. Among the models tested, CycleGAN emerged 2.1.4. Homogenization as the most efective, showcasing its ability to produce high-quality synthetic recombined images that closely In the domain of lung CT imaging, the heterogeneity of mimic those obtained with traditional contrast-enhanced images stemming from varied scanners and reconstruc- techniques. tion kernels poses a significant challenge. This variability can severely impact the performance of automated anal- 2.2.2. Virtual Treatment Planning in Lung Cancer ysis tools, notably in tasks relying on deep learning models such as 3D Convolutional Neural Networks (CNNs), Monitoring the progression and response to therapy is which are crucial for predicting patient outcomes like fundamental in lung cancer treatment. Traditional apoverall survival rates. To address this challenge, our work proaches rely on a series of CT scans taken before and introduces an innovative approach based on StarGAN, during treatment to evaluate the eficacy of the intera state-of-the-art image-to-image translation generative ventions. In our previous work [12], we developed an model, for the homogenization of lung CT images. ODE-based Digital Twin by using patient-specific CT
Our objective is to transform disparate lung CT images, scans to train a deep reinforcement learning controller, regardless of their originating scanner types or recon- which can adapt to diferent tissue aggressiveness and struction kernels, into a standardized format that retains outperform the current radiotherapy clinical practice of critical diagnostic features while presenting a uniform uniform dose delivery. However, this methodology often appearance. By employing StarGAN, we leverage its ca- exposes patients to additional radiation and can be logispacity for multi-domain image translation to achieve the tically challenging. Our innovative research introduces goal of not only enhancing the quality of the dataset but a novel application of AI in virtual treatment planning, also to significantly improve the performance of down- leveraging conditioned CycleGANs to simulate the postream tasks. This approach paves the way for more tential progression of lung cancer treatment based on generalized and robust AI tools in medical diagnostics, varying doses. By conditioning the CycleGAN on speultimately contributing to better patient care and out- cific treatment doses, our model can generate virtual CT comes. scans that predict how the patient’s anatomy and the tumor itself might respond to diferent levels of treat2.2. Virtual Scanner Applications ment. This approach allows for the creation of a virtual time series of CT scans without the need for repeated 2.2.1. Virtual Contrast Enhancement (VCE) radiation exposure. The ability to accurately forecast the In the evolving landscape of medical imaging, Contrast treatment’s progression through these synthetic scans Enhanced Spectral Mammography (CESM) represents a ofers a significant advantage in personalizing treatment significant advancement, ofering detailed insights for plans, enabling more precise adjustments to therapy regbreast cancer diagnosis by utilizing a dual-energy tech- imens based on predicted outcomes. By reducing the nique that integrates both low and high-energy images. reliance on multiple physical CT scans and minimizing This method, however, necessitates the administration patient exposure to radiation, we pave the way for a more of an iodinated contrast medium and subjects patients to patient-centric approach to cancer treatment monitorhigher radiation doses than standard mammography, rais- ing. Additionally, the predictive insights gained from this ing concerns about potential side efects and increased technology could significantly enhance decision-making radiation exposure. processes in treatment planning, potentially improving
Addressing these critical limitations, our work intro- patient outcomes in lung cancer care. duces a novel approach to VCE in CESM using deep generative models [11]. By eliminating the need for contrast 2.2.3. Whole-Body Translation from CT to PET mediums and aiming to reduce radiation doses, this research not only mitigates the associated risks but also preserves the diagnostic benefits of CESM. Our methodology employs GAN, e.g., Pix2Pix or CycleGAN, to generate synthetic recombined images from solely low-energy images.
The integration of CT and PET scans is essential in oncological diagnostics, combining the structural clarity of CT with the metabolic insights of PET imaging. While CT scans provide detailed anatomical structure, PET scans offer a window into the metabolic activity within the body, making the combined PET/CT an invaluable tool in the diagnosis, staging, and management of cancer patients. ing processes. By integrating a multi-scale, diferentiable Despite their clinical significance, the dual-modality ap- GLCM into the loss function, we facilitate a deeper underproach of PET/CT scanning is not without drawbacks, standing and recognition of complex textural information e.g., additional radiation exposure and higher costs com- during the image generation phase. pared to CT-only scans. These limitations restrict the Furthermore, the incorporation of a self-attention layer widespread availability of PET/CT imaging in numerous represents a pivotal innovation in our methodology, enmedical centers globally, underscoring the need for alter- abling the dynamic synthesis of texture information native methods that can replicate the integrative insights across various scales. This approach not only enhances of PET/CT imaging while mitigating its drawbacks. the denoising capabilities of GANs but also ensures the
Recognizing the challenges inherent in translating CT preservation of essential textural details, thereby improvimages to PET, especially given the variability in trans- ing the diagnostic utility of the generated images. lation efectiveness across diferent anatomical regions, Extensive experimental validation of our approach our methodology introduces a district-specific approach. within the field of low-dose CT denoising, aimed at imDrawing from the current literature, which suggests the proving noisy CT scans while minimizing radiation expopotential for improved accuracy through organ-specific sure, underscores the eficacy of our proposed solution. networks, we propose a novel strategy that segments Utilizing three publicly available datasets, including both whole-body images into four major anatomical districts. simulated and real-world scenarios, our methodology Each district is then processed through independently demonstrates a notable improvement over traditional trained GANs to generate district-specific PET images. loss functions across a variety of GAN architectures. The final step involves stitching these district-specific PET images together to reconstruct a comprehensive 2.2.5. Report Generation whole-body PET scan.
Employing two GAN architectures, Pix2Pix and CycleGAN, our approach facilitates a comparative analysis to evaluate the efectiveness and precision of the image translation process. Through standard evaluation metrics, we quantify the quality of the generated images, highlighting the advantages of our district-specific translation methodology over traditional approaches that rely on a single GAN trained on entire whole-body images. This innovative strategy not only promises to reduce the time, cost, and radiation exposure associated with PET/CT imaging but also ofers a tailored approach that accounts for the unique characteristics of diferent anatomical regions.
Our work focuses on Automatic Medical Reporting (AMR) that, as fostered by escalating digitization of healthcare data and the mounting stress national healthcare systems, aims to produce diagnostic reports from biomedical data. The eforts are currently directed towards chest radiographs assessing solutions based on encoder-decoder and transformer-based models. Alongside all this, Quantum Artificial Intelligence represents a novel field whose theoretical superiority in data representation capabilities and processing speeds makes it the main technology we are forwarding our eforts to, with its numerous methodologies for healthcare, even if it still presents hardware immaturity, scalability issues, and substantial financial costs. Because its application in AMR is still unnavigated, we aim to develop an architecture that merges traditional encoder-decoder concepts with quantum computing, to transcribe features obtained using classical binary computation into quantum states, which are then entangled with quantum representations of the shifted predictions for computational and accuracy benefits. 2.2.4. Texture Loss
through denoising, the application of GANs emerges as a promising task. Yet, a critical challenge lies in the GAN-based algorithms’ capacity to accurately capture and replicate the intricate textural details inherent in medical images. This task’s complexity is significantly amplified by the diverse and complex relationships that define image textures, making conventional denoising 3. Future Directions and approaches inadequate for preserving or restoring fine- Conclusion grained textural fidelity.
Our research introduces a novel loss function tai- As we stand on the verge of a new era in medical imaging, lored to address these limitations by exploiting the multi- propelled by the advancements in generative AI, the jourscale textural properties captured by the Gray-Level Co- ney of our exploration is ongoing. The groundwork laid occurrence Matrix (GLCM) [13]. The GLCM, traditionally by the Virtual Scanner and the development of resilient utilized in image processing to quantify texture, is rede- generative AI algorithms opens a myriad of pathways for ifned in our work as a diferentiable module compatible future research and application. In the quest for further with the gradient-based optimization of the GAN train- innovation, it is essential to delve deeper into the integra
dents enrolled in the National Ph.D. in Artificial Intelligence, course on Health and life sciences, organized by Università Campus Bio-Medico di Roma. We acknowledge financial support from: i) PNRR MUR project PE0000013-FAIR; ii) PRIN 2022 MUR 20228MZFAA-AIDA (CUP C53D23003620008); iii) PRIN PNRR 2022 MUR P2022P3CXJ-PICTURE (CUP C53D23009280001); iv) FCS MISE (CUP B89J23000580005). This work was also partially supported by the following companies: Teleconsys S.p.A.. tion of AI with emerging imaging technologies, aiming to enhance the precision, eficiency, and accessibility of diagnostic tools. Future research will focus on refining the algorithms for even greater resilience [14], enabling them to adapt more seamlessly to the vast diversity of medical imaging data. Additionally, exploring the potential for AI-driven predictive analytics in patient treatment plans presents a promising frontier, where the insights garnered from virtual scans could inform more personalized and efective treatment strategies. Moreover, the ethical considerations and data privacy concerns associated with deploying AI in healthcare require ongoing attention. Ensuring the security of patient data and the unbiased application of AI tools remains paramount as we advance.
In conclusion, the exploration into generative AI and the Virtual Scanner represents a significant leap toward revolutionizing medical imaging. As we move forward, the presented research activities and the technologies developed will undoubtedly pave the way for a future where diagnostics are more accurate, treatments are more personalized, and patient care is enhanced at every level. [4] V. Guarrasi, N. C. D’Amico, R. Sicilia, E. Cordelli,
P. Soda, A multi-expert system to detect covid-19 cases in x-ray images, in: 2021 IEEE 34th International Symposium on Computer-Based Medical
Systems (CBMS), IEEE, 2021, pp. 395–400. [5] V. Guarrasi, N. C. D’Amico, R. Sicilia, E. Cordelli,
P. Soda, Pareto optimization of deep networks for covid-19 diagnosis from chest x-rays, Pattern
Recognition 121 (2022) 108242. [6] V. Guarrasi, P. Soda, Optimized fusion of cnns to diagnose pulmonary diseases on chest x-rays, in: International Conference on Image Analysis and
Processing, Springer, 2022, pp. 197–209. [7] V. Guarrasi, L. Tronchin, C. M. Caruso, A. Rofena,
G. Manni, F. Aksu, D. Paolo, G. Iannello, R. Sicilia, E. Cordelli, et al., Building an ai-enabled metaverse for intelligent healthcare: opportunities and challenges, in: Ital-IA 2023, Italia Intelligenza Artificiale Thematic Workshops, co-located with the 3rd CINI National Lab AIIS Conference on Artificial Intelligence (Ital IA 2023), Pisa, Italy, May 29-30, 2023,
CEUR-WS, 2023, pp. 134–139. [8] E. Cordelli, V. Guarrasi, G. Iannello, F. Rufini, R. Sicilia, P. Soda, L. Tronchin, Making ai trustworthy in multimodal and healthcare scenarios, Proceedings of the Ital-IA (2023). [9] L. Tronchin, M. H. Vu, P. Soda, T. Löfstedt, Latentaugment: Data augmentation via guided manipulation of gan’s latent space, arXiv preprint arXiv:2307.11375 (2023). [10] F. Di Feola, L. Tronchin, P. Soda, A comparative study between paired and unpaired Image Quality Assessment in Low-Dose CT Denoising, in: 2023 IEEE 36th International Symposium on ComputerBased Medical Systems (CBMS), IEEE, 2023, pp. 471– 476. [11] A. Rofena, V. Guarrasi, M. Sarli, C. L. Piccolo,
M. Sammarra, B. B. Zobel, P. Soda, A deep learning approach for virtual contrast enhancement in contrast enhanced spectral mammography, arXiv [1] G. Fiscon, F. Salvadore, V. Guarrasi, A. R. Garbuglia, preprint arXiv:2308.00471 (2023).
P. Paci, Assessing the impact of data-driven lim- [12] M. Tortora, E. Cordelli, R. Sicilia, M. Miele, P. Matitations on tracing and forecasting the outbreak teucci, G. Iannello, S. Ramella, P. Soda, Deep reindynamics of covid-19, Computers in biology and forcement learning for fractionated radiotherapy medicine 135 (2021) 104657. in non-small cell lung carcinoma, Artificial Intelli[2] V. Guarrasi, P. Soda, Multi-objective optimization gence in Medicine 119 (2021) 102137. determines when, which and how to fuse deep [13] F. D. Feola, L. Tronchin, V. Guarrasi, P. Soda, Multinetworks: An application to predict covid-19 out- Scale Texture Loss for CT denoising with GANs, comes, Computers in Biology and Medicine 154 2024. arXiv:2403.16640.
(2023) 106625. [14] C. M. Caruso, V. Guarrasi, S. Ramella, P. Soda, [3] V. Guarrasi, L. Tronchin, D. Albano, E. Faiella, A deep learning approach for overall survival D. Fazzini, D. Santucci, P. Soda, Multimodal ex- analysis with missing values, arXiv preprint plainability via latent shift applied to covid-19 strat- arXiv:2307.11465 (2023). ification, arXiv preprint arXiv:2212.14084 (2022).