Virtual Scanner: Leveraging Resilient Generative AI for Radiological Imaging in the Era of Medical Digital Twins Carolina Adornato1 , Cecilia Assolito1 , Ermanno Cordelli1 , Francesco Di Feola1,2 , Valerio Guarrasi1,* , Giulio Iannello1 , Lorenzo Marcoccia1 , Elena Mulero Ayllon1 , Rebecca Restivo1 , Aurora Rofena1 , Rosa Sicilia1 , Paolo Soda1,2 , Matteo Tortora1 and Lorenzo Tronchin1,2 1 Research Unit of Computer Systems and Bioinformatics, Department of Engineering, Università Campus Bio-Medico di Roma, Via Àlvaro del Portillo 21, Rome, 00128, Italy 2 Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, 90187, Sweden Abstract Advancements in generative artificial intelligence (AI) are setting the stage for transformative changes in medical imaging, particularly through the development of the Virtual Scanner. This innovative approach leverages resilient generative AI to synthesize radiological images, addressing critical challenges in the field such as data scarcity, patient exposure to radiation, and the limitations of current imaging technologies. By harnessing the power of Generative Adversarial Networks (GANs) and focusing on the resilience of these algorithms, the Virtual Scanner aims to enhance diagnostic accuracy, improve patient care, and fill gaps in multimodal datasets. Our research explores both unimodal and multimodal techniques, including GAN ensembles, latent augmentation, and advanced texture synthesis, to create robust and adaptable generative models. Through extensive experimentation and analysis, we demonstrate the potential of the Virtual Scanner to revolutionize medical diagnostics by providing a safer, more efficient, and comprehensive imaging solution. The implications of this work extend beyond immediate medical applications, offering insights into the development of AI technologies capable of navigating the complexities of real-world data. Keywords Medical Imaging, Generative Artificial Intelligence, Virtual Scanner, Resilient AI, Multimodal Learning, Radiology 1. Introduction ated with radiation and contrast agents. Furthermore, the reliance on comprehensive multimodal imaging data In recent years, the intersection of artificial intelligence presents challenges in scenarios where certain modalities (AI) and healthcare has opened up novel possibilities for are unavailable or unsuitable for some patients, leading enhancing diagnostic accuracy, optimizing patient care, to gaps in the data that can hinder diagnostic processes and tailoring treatment plans towards precision medicine. and the development of AI models in healthcare [1, 2, 3]. One of the most promising developments in this domain The advancement of generative AI, particularly is the concept of the Medical Digital Twin, a virtual repre- through the deployment of Generative Adversarial Net- sentation of a patient’s health status, enabling personal- works (GANs), offers a novel solution to these challenges. ized medical interventions and predictive healthcare ana- By enabling the virtual generation of radiological images lytics. Central to the utility and effectiveness of Medical where real ones are unavailable or undesirable, AI not Digital Twins is the capability for detailed and accurate only mitigates the risks to patients but also bridges the radiological imaging, which provides a window into the data gaps in multimodal learning applications [4, 5, 6]. internal workings of the human body without invasive We introduce the concept of the Virtual Scanner as a cor- procedures. nerstone of the Medical Digital Twin paradigm, aiming to Radiological imaging, encompassing a range of modal- revolutionize the field of radiology by synthesizing high- ities such as X-rays, MRI, and CT scans, plays a pivotal fidelity, modality-specific images through the power of role in the diagnosis, monitoring, and treatment planning AI, thus enhancing patient care and supporting radiolo- for a myriad of health conditions. However, the acquisi- gists in delivering more accurate diagnoses. tion of these images often requires patients to undergo The scarcity of comprehensive radiological images multiple scans, exposing them to potential risks associ- presents significant challenges in medical diagnostics, affecting the efficacy of diagnostic processes and the de- Ital-IA 2024: 4th National Conference on Artificial Intelligence, orga- velopment of AI tools. This scarcity arises from limited nized by CINI, May 29-30, 2024, Naples, Italy access to advanced imaging technologies, concerns over * Corresponding author. radiation exposure, and the difficulty of compiling di- $ valerio.guarrasi@unicampus.it (V. Guarrasi)  0000-0002-1860-7447 (V. Guarrasi) verse, multimodal datasets. Such challenges hinder the © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License creation of effective AI models for diagnostics, impacting Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings their accuracy and real-world applicability. The Virtual Pareto multi-objective optimization problem that simul- Scanner, leveraging resilient generative AI algorithms, taneously covers the real training set, aiming to generate addresses these issues by synthesizing radiological im- high-quality GANs and using as few GANs as possible. ages to fill dataset gaps and reduce the need for repeated We tested out methodology across three distinct med- scans. Resilient generative AI refers to the development ical datasets, employing 22 GANs with differing archi- of models that not only excel in their designated tasks un- tectures, loss functions, and regularization techniques. der ideal conditions but also maintain their performance Moreover, we uniformly sampled each model every 20000 when confronted with data that deviate from the norm, training iterations, i.e., resulting in a total search space known as “data in the wild". Such resilience is crucial in of 110 models. The experiments showcase that using ensuring that the AI tools developed for medical imag- synthetic datasets generated from such an ensemble im- ing are robust against the variations inherent in patient proves the performances in classification downstream data across different demographics, equipment used, and tasks compared to single GANs and Naive selection ap- pathological conditions. proaches, i.e., using all available 110 GANs or randomly By embedding resilience at the core of our generative selecting a subset. AI algorithms, we aim to create a foundation for the Vir- tual Scanner that is not only technologically advanced 2.1.2. LatentAugment but also reliable and effective across the spectrum of med- ical imaging needs. This approach positions our work Data Augmentation (DA) is a crucial strategy in AI to not just as a technical achievement but as a meaningful enhance the volume and diversity of training datasets, contribution to the field of radiology, where the capac- thereby mitigating the risk of overfitting and bolstering ity to handle data in the wild can significantly enhance model generalization to unseen data. Standard DA meth- diagnostic processes and patient care. ods in image recognition tasks transform the images via geometric rigid and non-rigid transformations using im- age processing primitives, such as translation, rotation, 2. Research Activities cropping, etc. However, such transformations rely on human experts with prior knowledge of the dataset and Embarking on the journey to realize the Virtual Scanner, fail to generate sufficiently diverse synthetic data. GANs our investigation delves into a series of research activ- offer a valuable addition to the available augmentation ities, as shown in Figure 1, each designed to push the techniques. However, GANs generate high-quality sam- boundaries of what’s possible with generative AI in the ples rapidly, but they suffer from poor mode coverage, field of radiology [7, 8]. These activities are categorized i.e., the variation and variety of the samples that can be into two main areas: “Resilient Generative AI" and “Vir- generated, limiting their utility for DA purposes in the tual Scanner Applications", enveloping a diverse array of medical field. methodologies and applications aimed at enhancing the We propose LatentAugment [9], a DA strategy that generation and translation of medical imaging data. By overcomes the low diversity of GANs, opening up for use addressing many aspects of generative AI, from improv- in DA applications. LatentAugment addresses the three- ing algorithm resilience to creating virtual modalities, fold challenge of producing synthetic samples that are these activities underscore our commitment to advancing not only of high fidelity and quality but also diverse and diagnostic capabilities and patient care through techno- rapidly generated. LatentAugment modifies the latent logical innovation. vectors of the real training set, moving them towards regions that maximize their diversity and fidelity. We 2.1. Resilient Generative AI applied LatentAugment to improve the performances of a downstream model performing of MRI-to-CT im- 2.1.1. GAN Ensemble age translation. The results showed LatentAugment’s In tackling the complexities of synthetic data generation superiority over common DA methods and naive GAN- within medical imaging, our research delves into opti- sampling, i.e., creating data sampling from the GAN’s mizing generative AI through the use of GAN ensembles. latent space without any control. This strategy is born from the necessity to overcome inherent limitations in single-model GAN applications, 2.1.3. Paired vs. Unpaired Image Translation such as mode collapse and the inadequate representation Image-to-Image translation in medical imaging presents of real data distributions, a common obstacle in generat- a critical challenge due to the predominance of un- ing high-quality and diverse medical images. The core paired datasets, where the direct correspondence be- of our approach lies in creating an ensemble of GANs tween source and target images is not established [10]. that jointly optimizes the visual quality and diversity of While paired methods, e.g., Pix2Pix, rely on direct map- synthetic images from a set of GANs. We aim to solve a Resilient AI GAN Ensemble Latent Augment GAN Training ! Optimisation W⇤ ↵lat N N multiplication Ensembles Multi-objective Inversion wk k·k22 Llat (wk ) "! ! !! AAACAnicbVDLSsNAFJ3UV62vqCtxM9iKdVOSgo9lUUGXFewDmlAmk0k7dJIJMxOhhOLGX3HjQhG3foU7/8ZJm4VWDwxzOOde7r3HixmVyrK+jMLC4tLySnG1tLa+sbllbu+0JU8EJi3MGRddD0nCaERaiipGurEgKPQY6Xijy8zv3BMhKY/u1DgmbogGEQ0oRkpLfXOv4nic+XIc6i91rghTaFK9Pjqu9M2yVbOmgH+JnZMyyNHsm5+Oz3ESkkhhhqTs2Vas3BQJRTEjk5KTSBIjPEID0tM0QiGRbjo9YQIPteLDgAv9IgWn6s+OFIUyW1JXhkgN5byXif95vUQF525KozhRJMKzQUHCoOIwywP6VBCs2FgThAXVu0I8RAJhpVMr6RDs+ZP/kna9Zp/WTm7r5cZFHkcR7IMDUAU2OAMNcAOaoAUweABP4AW8Go/Gs/FmvM9KC0beswt+wfj4Bv4ploU= (G ) 0 L ! !! ↵f 0 ⇤ addition w =w N 1 G x̃k D Lf (wk ) " x w k·k l2 -distance !! !! !|"| W w2 X ↵pix N L ··· ⋮ "! ! k·k22 Lpix (w ) k D trained discriminator ⋯ ! !|"| !! wK = w̃ (X ) ↵perc (G0 ) trained feature extractor AAACAnicbVDLSsNAFJ3UV62vqCtxM9iKdVOSgo9l0YUuK9gHNKFMJpN26CQTZiZCCcWNv+LGhSJu/Qp3/o2TNgttPTDM4Zx7ufceL2ZUKsv6NgpLyyura8X10sbm1vaOubvXljwRmLQwZ1x0PSQJoxFpKaoY6caCoNBjpOONrjO/80CEpDy6V+OYuCEaRDSgGCkt9c2DiuNx5stxqL/U8QlTaFK9OTmt9M2yVbOmgIvEzkkZ5Gj2zS/H5zgJSaQwQ1L2bCtWboqEopiRSclJJIkRHqEB6WkaoZBIN52eMIHHWvFhwIV+kYJT9XdHikKZLakrQ6SGct7LxP+8XqKCSzelUZwoEuHZoCBhUHGY5QF9KghWbKwJwoLqXSEeIoGw0qmVdAj2/MmLpF2v2ee1s7t6uXGVx1EEh+AIVIENLkAD3IImaAEMHsEzeAVvxpPxYrwbH7PSgpH37IM/MD5/AC/4lqU= N k·k22 Lperc (wk ) Reconstruction G trained generator update w Generation "! ! !|"| " ∗ !! !! !|"| L(wk ) ! !! x̃ !! !! !|"| ⋯ Homogenization ⋯ "! ! !! !|"| !! ! !∗ !! !! !|"| ⋯ ⋮ ⋯ Center A Paired vs Unpaired METODI – Image Translation Approccio proposto Dataset A NIQE Metric Trained Extraction I Regressor BRISQUE PaQ2PiQ !"" H MSTE Center B Dataset* pxv Dataset B GAN Real Generator Fake Generator Reconstructed hxw Low-Dose A→B Full-Dose B→A Low-Dose 𝛥H || ⋅ ||1 Real/Fake I pxv I Discriminator Center B H MSTE Dataset C Real Full-Dose H MSTE pxv Virtual Scanner Applications hxw pxv hxw 𝛥H Virtual Contrast Enhancement Texture Loss || ⋅ ||1 CC AAAB8HicbVDLTgJBEOzFF+IL9ehlIzHxRHaNQY9ELh4xkYeBDZkdBpgwM7uZ6TWSDV/hxYPGePVzvPk3DrAHBSvppFLVne6uMBbcoOd9O7m19Y3Nrfx2YWd3b/+geHjUNFGiKWvQSES6HRLDBFesgRwFa8eaERkK1grHtZnfemTa8Ejd4yRmgSRDxQecErTSQxfZE6a12rRXLHllbw53lfgZKUGGeq/41e1HNJFMIRXEmI7vxRikRCOngk0L3cSwmNAxGbKOpYpIZoJ0fvDUPbNK3x1E2pZCd67+nkiJNGYiQ9spCY7MsjcT//M6CQ6ug5SrOEGm6GLRIBEuRu7se7fPNaMoJpYQqrm91aUjoglFm1HBhuAvv7xKmhdlv1Ku3F2WqjdZHHk4gVM4Bx+uoAq3UIcGUJDwDK/w5mjnxXl3PhatOSebOYY/cD5/AOdlkH4= I pxv (a) (b) Gd1 ,𝜃1 h I H MSTE FFDM AAAB8nicbVBNSwMxEM3Wr1q/qh69BIvgqeyKVI9FpXgRKtgP2C4lm6ZtaDZZklmxLP0ZXjwo4tVf481/Y9ruQVsfDDzem2FmXhgLbsB1v53cyura+kZ+s7C1vbO7V9w/aBqVaMoaVAml2yExTHDJGsBBsHasGYlCwVrh6Hrqtx6ZNlzJBxjHLIjIQPI+pwSs5HeAPUFaq93cTbrFklt2Z8DLxMtICWWod4tfnZ6iScQkUEGM8T03hiAlGjgVbFLoJIbFhI7IgPmWShIxE6Szkyf4xCo93FfalgQ8U39PpCQyZhyFtjMiMDSL3lT8z/MT6F8GKZdxAkzS+aJ+IjAoPP0f97hmFMTYEkI1t7diOiSaULApFWwI3uLLy6R5VvYq5cr9eal6lcWRR0foGJ0iD12gKrpFddRAFCn0jF7RmwPOi/PufMxbc042c4j+wPn8AR13kSk= nxn H pxv 𝛥H MSTE MLO AAAB8XicbVBNS8NAEN3Ur1q/qh69BIvgqSQi1WPRiwfFCvYD21A220m7dLMJuxOxhP4LLx4U8eq/8ea/cdvmoK0PBh7vzTAzz48F1+g431ZuaXlldS2/XtjY3NreKe7uNXSUKAZ1FolItXyqQXAJdeQooBUroKEvoOkPLyd+8xGU5pG8x1EMXkj7kgecUTTSQwfhCdOb69txt1hyys4U9iJxM1IiGWrd4lenF7EkBIlMUK3brhOjl1KFnAkYFzqJhpiyIe1D21BJQ9BeOr14bB8ZpWcHkTIl0Z6qvydSGmo9Cn3TGVIc6HlvIv7ntRMMzr2UyzhBkGy2KEiEjZE9ed/ucQUMxcgQyhQ3t9psQBVlaEIqmBDc+ZcXSeOk7FbKlbvTUvUiiyNPDsghOSYuOSNVckVqpE4YkeSZvJI3S1sv1rv1MWvNWdnMPvkD6/MHpJGQ6g== hxw 𝛥H max( · ) I or I Gd2 ,𝜃2 h H or H || ⋅ ||1 demux mean( · ) c pxv mux 𝜉 . I pxv || · ||FF (a) hxw . nxn SA sum I H MSTE . H h AM MSTE Gdp ,𝜃v pxv (b) Gd1 ,𝜃1 h MSTE hxw pp x vv nxn gCC gMLO MSTLF max max(( ·· )) AAAB8nicbVBNS8NAEN3Ur1q/qh69LBbBU0lEqsdiLx4r2FpoQ9lsJ+3STTbsTsQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNirVHFpcSaU7ATMgRQwtFCihk2hgUSDhIRg3Zv7DI2gjVHyPkwT8iA1jEQrO0ErdYb+H8IRZozHtlytu1Z2DrhIvJxWSo9kvf/UGiqcRxMglM6bruQn6GdMouIRpqZcaSBgfsyF0LY1ZBMbP5idP6ZlVBjRU2laMdK7+nshYZMwkCmxnxHBklr2Z+J/XTTG89jMRJylCzBeLwlRSVHT2Px0IDRzlxBLGtbC3Uj5imnG0KZVsCN7yy6ukfVH1atXa3WWlfpPHUSQn5JScE49ckTq5JU3SIpwo8kxeyZuDzovz7nwsWgtOPnNM/sD5/AFk4pFY AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi1WPRiwfFCvYDmlA22027dLMJuxOxhP4NLx4U8eqf8ea/cdvmoNUHA4/3ZpiZFySCa3ScL6uwtLyyulZcL21sbm3vlHf3WjpOFWVNGotYdQKimeCSNZGjYJ1EMRIFgrWD0eXUbz8wpXks73GcMD8iA8lDTgkayRv0PGSPmN1c30565YpTdWaw/xI3JxXI0eiVP71+TNOISaSCaN11nQT9jCjkVLBJyUs1SwgdkQHrGipJxLSfzW6e2EdG6dthrExJtGfqz4mMRFqPo8B0RgSHetGbiv953RTDcz/jMkmRSTpfFKbCxtieBmD3uWIUxdgQQhU3t9p0SBShaGIqmRDcxZf/ktZJ1a1Va3enlfpFHkcRDuAQjsGFM6jDFTSgCRQSeIIXeLVS69l6s97nrQUrn9mHX7A+vgEi2JHE y 𝛥H hh x w w AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KolI9Vj04rEF+wFtKJvtpF272YTdjRBKf4EXD4p49Sd589+4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8N/PbT6g0j+WDyRL0IzqUPOSMGis1sn6p7FbcOcgq8XJShhz1fumrN4hZGqE0TFCtu56bGH9CleFM4LTYSzUmlI3pELuWShqh9ifzQ6fk3CoDEsbKljRkrv6emNBI6ywKbGdEzUgvezPxP6+bmvDGn3CZpAYlWywKU0FMTGZfkwFXyIzILKFMcXsrYSOqKDM2m6INwVt+eZW0LitetVJtXJVrt3kcBTiFM7gAD66hBvdQhyYwQHiGV3hzHp0X5935WLSuOfnMCfyB8/kD60+NCA== || || ⋅⋅ || ||11 demux demux mean mean(( ·· )) mux mux nxn 𝛥H 𝜉 𝜉 I or I I ||FF || || ·· || (c) pp xx vv (c) (a) (b) (b) hh SA sum h H or H Gdd11 ,𝜃 SA sum H I G ,𝜃11 Gd2 ,𝜃2 MSTE H Q Q c n n xx n n 𝛥H A AM AM . MSTE pxv A nxn hxw or I I or max( · ) K . CESM 𝛥H h h H or or H K AAAB8nicbVBNSwMxEM3Wr1q/qh69BIvgqeyKVI/FIngRKtoP2C4lm6ZtaDZZklmxLP0ZXjwo4tVf481/Y9ruQVsfDDzem2FmXhgLbsB1v53cyura+kZ+s7C1vbO7V9w/aBqVaMoaVAml2yExTHDJGsBBsHasGYlCwVrhqDb1W49MG67kA4xjFkRkIHmfUwJW8jvAniCtXd/fTrrFklt2Z8DLxMtICWWod4tfnZ6iScQkUEGM8T03hiAlGjgVbFLoJIbFhI7IgPmWShIxE6Szkyf4xCo93FfalgQ8U39PpCQyZhyFtjMiMDSL3lT8z/MT6F8GKZdxAkzS+aJ+IjAoPP0f97hmFMTYEkI1t7diOiSaULApFWwI3uLLy6R5VvYq5crdeal6lcWRR0foGJ0iD12gKrpBddRAFCn0jF7RmwPOi/PufMxbc042c4j+wPn8AS4ykTQ= Gdd22 ,𝜃 cc G pxv ,𝜃22 mean( · ) pxv 𝛥H' 𝛥H' . MSTLF || ⋅ ||1 𝛾 demux mux 𝜉 .. 𝛾 hh xx w w I n n xx n h || · ||F (a) pxv SA sum .. n V V Gdp ,𝜃v I H ∗∗ .. MSTE CC AAAB8HicbVDLTgJBEOzFF+IL9ehlIzHxRHaNQY9ELh4xkYeBDZkdBpgwM7uZ6TWSDV/hxYPGePVzvPk3DrAHBSvppFLVne6uMBbcoOd9O7m19Y3Nrfx2YWd3b/+geHjUNFGiKWvQSES6HRLDBFesgRwFa8eaERkK1grHtZnfemTa8Ejd4yRmgSRDxQecErTSQxfZE6a12rRXLHllbw53lfgZKUGGeq/41e1HNJFMIRXEmI7vxRikRCOngk0L3cSwmNAxGbKOpYpIZoJ0fvDUPbNK3x1E2pZCd67+nkiJNGYiQ9spCY7MsjcT//M6CQ6ug5SrOEGm6GLRIBEuRu7se7fPNaMoJpYQqrm91aUjoglFm1HBhuAvv7xKmhdlv1Ku3F2WqjdZHHk4gVM4Bx+uoAq3UIcGUJDwDK/w5mjnxXl3PhatOSebOYY/cD5/AOdlkH4= hh H MSTE AM Gddp ,𝜃 MSTE G ,𝜃v nxn p v (b) (c) (c) pxv (b) Gdd1 ,𝜃 hh MSTE SA hxw MSTE SA G ,𝜃1 pxv n n xx n n MSTLF max( · ) 1 1 𝛥H hxw Q Q || ⋅ ||1 mean( · ) n n xx n n 𝛥H A demux mux 𝜉 𝛥H A or II II or || · ||F (c) pxv K MLO (b) AAAB8XicbVBNS8NAEN3Ur1q/qh69BIvgqSQi1WPRiwfFCvYD21A220m7dLMJuxOxhP4LLx4U8eq/8ea/cdvmoK0PBh7vzTAzz48F1+g431ZuaXlldS2/XtjY3NreKe7uNXSUKAZ1FolItXyqQXAJdeQooBUroKEvoOkPLyd+8xGU5pG8x1EMXkj7kgecUTTSQwfhCdOb69txt1hyys4U9iJxM1IiGWrd4lenF7EkBIlMUK3brhOjl1KFnAkYFzqJhpiyIe1D21BJQ9BeOr14bB8ZpWcHkTIl0Z6qvydSGmo9Cn3TGVIc6HlvIv7ntRMMzr2UyzhBkGy2KEiEjZE9ed/ucQUMxcgQyhQ3t9psQBVlaEIqmBDc+ZcXSeOk7FbKlbvTUvUiiyNPDsghOSYuOSNVckVqpE4YkeSZvJI3S1sv1rv1MWvNWdnMPvkD6/MHpJGQ6g== h SA sum hh H or K Gd1 ,𝜃1 I H or H Gd2 ,𝜃2 H Q G d2 ,𝜃2 cc H 𝛾 𝛥H' 𝛥H' nxn 𝛥H A AM .. 𝛾 I I MSTE nxn Planning in Lung CancerVV nxn K .. Whole-Body Translation from CT to PET or Gd2 ,𝜃2 h c H or H pxv Virtual 𝛥H' Treatment .. ∗∗ MSTLF . 𝛾 SEGMENTATION hxw . GENERATION nxn STITCHING V ∗ Gdp ,𝜃v G dp ,𝜃v hh . Dose condition MSTE h SA Gdp ,𝜃v MSTE SA n n xx n (c) (continuous value) n (b) PATCH Gd1 ,𝜃1 hPATCH MSTE PATCH SA BOUNDING BOX EXTRACTION EXTRACTION nxn GENERATION FUSION Q nxn 𝛥H A 32 x 32 x 32 I or I 32 x 32 x 32 K G0 → X GX → 0 Gd2 ,𝜃2 h H or H . c 𝛾 𝛥H' . !!"→ %&" nxn V ∗ . h WHOLE-BODY IMAGE WHOLE-BODY SEGMENTATION Gdp ,𝜃v CT IMAGE PET IMAGE nxn MSTE SA 32 x 32 x 32 32 x 32 x 32 !!"→ %&" Lung CT0 Lung CTX Lung CT0 !"#$"%&'&() 32 x 32 x 32 32 x 32 x 32 real fake !!"→ %&" Dose 32 x 32 x 32 32 x 32 x 32 prediction !!"→ %&" DX Figure 1: Overview of the presented research activities. pings between paired images, unpaired techniques, e.g., translation methodologies. By integrating a novel paired CycleGAN, offer flexibility by ignoring this requirement. virtual loss function into the unpaired CycleGAN frame- Given the logistical and ethical complexities in acquiring work, we enhance the stability and accuracy of unpaired paired medical images, unpaired models have gained sig- image translation without necessitating direct image nificant attention despite the potential compromises in pairs. This innovation finds practical application in Low- performance attributed to training instability and vari- Dose Computed Tomography (LDCT) denoising, a pro- able outcomes. cess aimed at reducing radiation exposure while main- Our work introduces a novel approach to narrow the taining image quality. Through this approach, we lever- performance drop between paired and unpaired image age the abundance of unpaired LDCT and Full-Dose CT (FDCT) images to validate our model, demonstrating its An extensive quantitative and qualitative analysis un- effectiveness in producing high-quality, denoised images derpins our research, including evaluations by profes- that closely approximate FDCT standards, thereby mit- sional radiologists on a novel CESM dataset comprising igating health risks associated with radiation without 1,138 images. This dataset has been made publicly avail- compromising diagnostic integrity. able to foster ongoing research and development in the field. Among the models tested, CycleGAN emerged 2.1.4. Homogenization as the most effective, showcasing its ability to produce high-quality synthetic recombined images that closely In the domain of lung CT imaging, the heterogeneity of mimic those obtained with traditional contrast-enhanced images stemming from varied scanners and reconstruc- techniques. tion kernels poses a significant challenge. This variability can severely impact the performance of automated anal- 2.2.2. Virtual Treatment Planning in Lung Cancer ysis tools, notably in tasks relying on deep learning mod- els such as 3D Convolutional Neural Networks (CNNs), Monitoring the progression and response to therapy is which are crucial for predicting patient outcomes like fundamental in lung cancer treatment. Traditional ap- overall survival rates. To address this challenge, our work proaches rely on a series of CT scans taken before and introduces an innovative approach based on StarGAN, during treatment to evaluate the efficacy of the inter- a state-of-the-art image-to-image translation generative ventions. In our previous work [12], we developed an model, for the homogenization of lung CT images. ODE-based Digital Twin by using patient-specific CT Our objective is to transform disparate lung CT images, scans to train a deep reinforcement learning controller, regardless of their originating scanner types or recon- which can adapt to different tissue aggressiveness and struction kernels, into a standardized format that retains outperform the current radiotherapy clinical practice of critical diagnostic features while presenting a uniform uniform dose delivery. However, this methodology often appearance. By employing StarGAN, we leverage its ca- exposes patients to additional radiation and can be logis- pacity for multi-domain image translation to achieve the tically challenging. Our innovative research introduces goal of not only enhancing the quality of the dataset but a novel application of AI in virtual treatment planning, also to significantly improve the performance of down- leveraging conditioned CycleGANs to simulate the po- stream tasks. This approach paves the way for more tential progression of lung cancer treatment based on generalized and robust AI tools in medical diagnostics, varying doses. By conditioning the CycleGAN on spe- ultimately contributing to better patient care and out- cific treatment doses, our model can generate virtual CT comes. scans that predict how the patient’s anatomy and the tumor itself might respond to different levels of treat- 2.2. Virtual Scanner Applications ment. This approach allows for the creation of a virtual time series of CT scans without the need for repeated 2.2.1. Virtual Contrast Enhancement (VCE) radiation exposure. The ability to accurately forecast the treatment’s progression through these synthetic scans In the evolving landscape of medical imaging, Contrast offers a significant advantage in personalizing treatment Enhanced Spectral Mammography (CESM) represents a plans, enabling more precise adjustments to therapy reg- significant advancement, offering detailed insights for imens based on predicted outcomes. By reducing the breast cancer diagnosis by utilizing a dual-energy tech- reliance on multiple physical CT scans and minimizing nique that integrates both low and high-energy images. patient exposure to radiation, we pave the way for a more This method, however, necessitates the administration patient-centric approach to cancer treatment monitor- of an iodinated contrast medium and subjects patients to ing. Additionally, the predictive insights gained from this higher radiation doses than standard mammography, rais- technology could significantly enhance decision-making ing concerns about potential side effects and increased processes in treatment planning, potentially improving radiation exposure. patient outcomes in lung cancer care. Addressing these critical limitations, our work intro- duces a novel approach to VCE in CESM using deep gen- erative models [11]. By eliminating the need for contrast 2.2.3. Whole-Body Translation from CT to PET mediums and aiming to reduce radiation doses, this re- The integration of CT and PET scans is essential in onco- search not only mitigates the associated risks but also logical diagnostics, combining the structural clarity of CT preserves the diagnostic benefits of CESM. Our method- with the metabolic insights of PET imaging. While CT ology employs GAN, e.g., Pix2Pix or CycleGAN, to gener- scans provide detailed anatomical structure, PET scans of- ate synthetic recombined images from solely low-energy fer a window into the metabolic activity within the body, images. making the combined PET/CT an invaluable tool in the diagnosis, staging, and management of cancer patients. ing processes. By integrating a multi-scale, differentiable Despite their clinical significance, the dual-modality ap- GLCM into the loss function, we facilitate a deeper under- proach of PET/CT scanning is not without drawbacks, standing and recognition of complex textural information e.g., additional radiation exposure and higher costs com- during the image generation phase. pared to CT-only scans. These limitations restrict the Furthermore, the incorporation of a self-attention layer widespread availability of PET/CT imaging in numerous represents a pivotal innovation in our methodology, en- medical centers globally, underscoring the need for alter- abling the dynamic synthesis of texture information native methods that can replicate the integrative insights across various scales. This approach not only enhances of PET/CT imaging while mitigating its drawbacks. the denoising capabilities of GANs but also ensures the Recognizing the challenges inherent in translating CT preservation of essential textural details, thereby improv- images to PET, especially given the variability in trans- ing the diagnostic utility of the generated images. lation effectiveness across different anatomical regions, Extensive experimental validation of our approach our methodology introduces a district-specific approach. within the field of low-dose CT denoising, aimed at im- Drawing from the current literature, which suggests the proving noisy CT scans while minimizing radiation expo- potential for improved accuracy through organ-specific sure, underscores the efficacy of our proposed solution. networks, we propose a novel strategy that segments Utilizing three publicly available datasets, including both whole-body images into four major anatomical districts. simulated and real-world scenarios, our methodology Each district is then processed through independently demonstrates a notable improvement over traditional trained GANs to generate district-specific PET images. loss functions across a variety of GAN architectures. The final step involves stitching these district-specific PET images together to reconstruct a comprehensive 2.2.5. Report Generation whole-body PET scan. Employing two GAN architectures, Pix2Pix and Cy- Our work focuses on Automatic Medical Reporting cleGAN, our approach facilitates a comparative analysis (AMR) that, as fostered by escalating digitization of to evaluate the effectiveness and precision of the im- healthcare data and the mounting stress national health- age translation process. Through standard evaluation care systems, aims to produce diagnostic reports from metrics, we quantify the quality of the generated im- biomedical data. The efforts are currently directed to- ages, highlighting the advantages of our district-specific wards chest radiographs assessing solutions based on translation methodology over traditional approaches that encoder-decoder and transformer-based models. Along- rely on a single GAN trained on entire whole-body im- side all this, Quantum Artificial Intelligence represents ages. This innovative strategy not only promises to re- a novel field whose theoretical superiority in data rep- duce the time, cost, and radiation exposure associated resentation capabilities and processing speeds makes it with PET/CT imaging but also offers a tailored approach the main technology we are forwarding our efforts to, that accounts for the unique characteristics of different with its numerous methodologies for healthcare, even if anatomical regions. it still presents hardware immaturity, scalability issues, and substantial financial costs. Because its application in AMR is still unnavigated, we aim to develop an architec- 2.2.4. Texture Loss ture that merges traditional encoder-decoder concepts In the quest to enhance the quality of medical images with quantum computing, to transcribe features obtained through denoising, the application of GANs emerges as using classical binary computation into quantum states, a promising task. Yet, a critical challenge lies in the which are then entangled with quantum representations GAN-based algorithms’ capacity to accurately capture of the shifted predictions for computational and accuracy and replicate the intricate textural details inherent in benefits. medical images. This task’s complexity is significantly amplified by the diverse and complex relationships that define image textures, making conventional denoising 3. Future Directions and approaches inadequate for preserving or restoring fine- Conclusion grained textural fidelity. Our research introduces a novel loss function tai- As we stand on the verge of a new era in medical imaging, lored to address these limitations by exploiting the multi- propelled by the advancements in generative AI, the jour- scale textural properties captured by the Gray-Level Co- ney of our exploration is ongoing. The groundwork laid occurrence Matrix (GLCM) [13]. The GLCM, traditionally by the Virtual Scanner and the development of resilient utilized in image processing to quantify texture, is rede- generative AI algorithms opens a myriad of pathways for fined in our work as a differentiable module compatible future research and application. In the quest for further with the gradient-based optimization of the GAN train- innovation, it is essential to delve deeper into the integra- tion of AI with emerging imaging technologies, aiming [4] V. Guarrasi, N. C. D’Amico, R. Sicilia, E. Cordelli, to enhance the precision, efficiency, and accessibility of P. Soda, A multi-expert system to detect covid-19 diagnostic tools. Future research will focus on refining cases in x-ray images, in: 2021 IEEE 34th Inter- the algorithms for even greater resilience [14], enabling national Symposium on Computer-Based Medical them to adapt more seamlessly to the vast diversity of Systems (CBMS), IEEE, 2021, pp. 395–400. medical imaging data. Additionally, exploring the poten- [5] V. Guarrasi, N. C. D’Amico, R. Sicilia, E. Cordelli, tial for AI-driven predictive analytics in patient treatment P. Soda, Pareto optimization of deep networks plans presents a promising frontier, where the insights for covid-19 diagnosis from chest x-rays, Pattern garnered from virtual scans could inform more person- Recognition 121 (2022) 108242. alized and effective treatment strategies. Moreover, the [6] V. Guarrasi, P. Soda, Optimized fusion of cnns to ethical considerations and data privacy concerns asso- diagnose pulmonary diseases on chest x-rays, in: ciated with deploying AI in healthcare require ongoing International Conference on Image Analysis and attention. Ensuring the security of patient data and the Processing, Springer, 2022, pp. 197–209. unbiased application of AI tools remains paramount as [7] V. Guarrasi, L. Tronchin, C. M. Caruso, A. Rofena, we advance. G. Manni, F. Aksu, D. Paolo, G. Iannello, R. Sicilia, In conclusion, the exploration into generative AI and E. Cordelli, et al., Building an ai-enabled metaverse the Virtual Scanner represents a significant leap toward for intelligent healthcare: opportunities and chal- revolutionizing medical imaging. As we move forward, lenges, in: Ital-IA 2023, Italia Intelligenza Artificiale the presented research activities and the technologies Thematic Workshops, co-located with the 3rd CINI developed will undoubtedly pave the way for a future National Lab AIIS Conference on Artificial Intel- where diagnostics are more accurate, treatments are more ligence (Ital IA 2023), Pisa, Italy, May 29-30, 2023, personalized, and patient care is enhanced at every level. CEUR-WS, 2023, pp. 134–139. [8] E. Cordelli, V. Guarrasi, G. Iannello, F. Ruffini, R. Si- cilia, P. Soda, L. Tronchin, Making ai trustworthy in Acknowledgments multimodal and healthcare scenarios, Proceedings of the Ital-IA (2023). Aurora Rofena and Lorenzo Marcoccia are Ph.D. stu- [9] L. Tronchin, M. H. Vu, P. Soda, T. Löfstedt, La- dents enrolled in the National Ph.D. in Artificial Intel- tentaugment: Data augmentation via guided ma- ligence, course on Health and life sciences, organized nipulation of gan’s latent space, arXiv preprint by Università Campus Bio-Medico di Roma. We ac- arXiv:2307.11375 (2023). knowledge financial support from: i) PNRR MUR project [10] F. Di Feola, L. Tronchin, P. Soda, A comparative PE0000013-FAIR; ii) PRIN 2022 MUR 20228MZFAA-AIDA study between paired and unpaired Image Quality (CUP C53D23003620008); iii) PRIN PNRR 2022 MUR Assessment in Low-Dose CT Denoising, in: 2023 P2022P3CXJ-PICTURE (CUP C53D23009280001); iv) FCS IEEE 36th International Symposium on Computer- MISE (CUP B89J23000580005). This work was also par- Based Medical Systems (CBMS), IEEE, 2023, pp. 471– tially supported by the following companies: Teleconsys 476. S.p.A.. [11] A. Rofena, V. Guarrasi, M. Sarli, C. L. Piccolo, M. Sammarra, B. B. Zobel, P. Soda, A deep learn- References ing approach for virtual contrast enhancement in contrast enhanced spectral mammography, arXiv [1] G. Fiscon, F. Salvadore, V. Guarrasi, A. R. Garbuglia, preprint arXiv:2308.00471 (2023). P. Paci, Assessing the impact of data-driven lim- [12] M. Tortora, E. Cordelli, R. Sicilia, M. Miele, P. Mat- itations on tracing and forecasting the outbreak teucci, G. Iannello, S. Ramella, P. Soda, Deep rein- dynamics of covid-19, Computers in biology and forcement learning for fractionated radiotherapy medicine 135 (2021) 104657. in non-small cell lung carcinoma, Artificial Intelli- [2] V. Guarrasi, P. Soda, Multi-objective optimization gence in Medicine 119 (2021) 102137. determines when, which and how to fuse deep [13] F. D. Feola, L. Tronchin, V. Guarrasi, P. Soda, Multi- networks: An application to predict covid-19 out- Scale Texture Loss for CT denoising with GANs, comes, Computers in Biology and Medicine 154 2024. arXiv:2403.16640. (2023) 106625. [14] C. M. Caruso, V. Guarrasi, S. Ramella, P. Soda, [3] V. Guarrasi, L. Tronchin, D. Albano, E. Faiella, A deep learning approach for overall survival D. Fazzini, D. Santucci, P. Soda, Multimodal ex- analysis with missing values, arXiv preprint plainability via latent shift applied to covid-19 strat- arXiv:2307.11465 (2023). ification, arXiv preprint arXiv:2212.14084 (2022).