Virtual Scanner: Leveraging Resilient Generative AI for
Radiological Imaging in the Era of Medical Digital Twins
Carolina Adornato1 , Cecilia Assolito1 , Ermanno Cordelli1 , Francesco Di Feola1,2 ,
Valerio Guarrasi1,* , Giulio Iannello1 , Lorenzo Marcoccia1 , Elena Mulero Ayllon1 ,
Rebecca Restivo1 , Aurora Rofena1 , Rosa Sicilia1 , Paolo Soda1,2 , Matteo Tortora1 and
Lorenzo Tronchin1,2
1
Research Unit of Computer Systems and Bioinformatics, Department of Engineering, Università Campus Bio-Medico di Roma, Via Àlvaro del
Portillo 21, Rome, 00128, Italy
2
Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, 90187, Sweden
Abstract
Advancements in generative artificial intelligence (AI) are setting the stage for transformative changes in medical imaging,
particularly through the development of the Virtual Scanner. This innovative approach leverages resilient generative AI to
synthesize radiological images, addressing critical challenges in the field such as data scarcity, patient exposure to radiation,
and the limitations of current imaging technologies. By harnessing the power of Generative Adversarial Networks (GANs)
and focusing on the resilience of these algorithms, the Virtual Scanner aims to enhance diagnostic accuracy, improve patient
care, and fill gaps in multimodal datasets. Our research explores both unimodal and multimodal techniques, including
GAN ensembles, latent augmentation, and advanced texture synthesis, to create robust and adaptable generative models.
Through extensive experimentation and analysis, we demonstrate the potential of the Virtual Scanner to revolutionize medical
diagnostics by providing a safer, more efficient, and comprehensive imaging solution. The implications of this work extend
beyond immediate medical applications, offering insights into the development of AI technologies capable of navigating the
complexities of real-world data.
Keywords
Medical Imaging, Generative Artificial Intelligence, Virtual Scanner, Resilient AI, Multimodal Learning, Radiology
1. Introduction ated with radiation and contrast agents. Furthermore,
the reliance on comprehensive multimodal imaging data
In recent years, the intersection of artificial intelligence presents challenges in scenarios where certain modalities
(AI) and healthcare has opened up novel possibilities for are unavailable or unsuitable for some patients, leading
enhancing diagnostic accuracy, optimizing patient care, to gaps in the data that can hinder diagnostic processes
and tailoring treatment plans towards precision medicine. and the development of AI models in healthcare [1, 2, 3].
One of the most promising developments in this domain The advancement of generative AI, particularly
is the concept of the Medical Digital Twin, a virtual repre- through the deployment of Generative Adversarial Net-
sentation of a patient’s health status, enabling personal- works (GANs), offers a novel solution to these challenges.
ized medical interventions and predictive healthcare ana- By enabling the virtual generation of radiological images
lytics. Central to the utility and effectiveness of Medical where real ones are unavailable or undesirable, AI not
Digital Twins is the capability for detailed and accurate only mitigates the risks to patients but also bridges the
radiological imaging, which provides a window into the data gaps in multimodal learning applications [4, 5, 6].
internal workings of the human body without invasive We introduce the concept of the Virtual Scanner as a cor-
procedures. nerstone of the Medical Digital Twin paradigm, aiming to
Radiological imaging, encompassing a range of modal- revolutionize the field of radiology by synthesizing high-
ities such as X-rays, MRI, and CT scans, plays a pivotal fidelity, modality-specific images through the power of
role in the diagnosis, monitoring, and treatment planning AI, thus enhancing patient care and supporting radiolo-
for a myriad of health conditions. However, the acquisi- gists in delivering more accurate diagnoses.
tion of these images often requires patients to undergo The scarcity of comprehensive radiological images
multiple scans, exposing them to potential risks associ- presents significant challenges in medical diagnostics,
affecting the efficacy of diagnostic processes and the de-
Ital-IA 2024: 4th National Conference on Artificial Intelligence, orga- velopment of AI tools. This scarcity arises from limited
nized by CINI, May 29-30, 2024, Naples, Italy access to advanced imaging technologies, concerns over
*
Corresponding author.
radiation exposure, and the difficulty of compiling di-
$ valerio.guarrasi@unicampus.it (V. Guarrasi)
0000-0002-1860-7447 (V. Guarrasi) verse, multimodal datasets. Such challenges hinder the
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License creation of effective AI models for diagnostics, impacting
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
their accuracy and real-world applicability. The Virtual Pareto multi-objective optimization problem that simul-
Scanner, leveraging resilient generative AI algorithms, taneously covers the real training set, aiming to generate
addresses these issues by synthesizing radiological im- high-quality GANs and using as few GANs as possible.
ages to fill dataset gaps and reduce the need for repeated We tested out methodology across three distinct med-
scans. Resilient generative AI refers to the development ical datasets, employing 22 GANs with differing archi-
of models that not only excel in their designated tasks un- tectures, loss functions, and regularization techniques.
der ideal conditions but also maintain their performance Moreover, we uniformly sampled each model every 20000
when confronted with data that deviate from the norm, training iterations, i.e., resulting in a total search space
known as “data in the wild". Such resilience is crucial in of 110 models. The experiments showcase that using
ensuring that the AI tools developed for medical imag- synthetic datasets generated from such an ensemble im-
ing are robust against the variations inherent in patient proves the performances in classification downstream
data across different demographics, equipment used, and tasks compared to single GANs and Naive selection ap-
pathological conditions. proaches, i.e., using all available 110 GANs or randomly
By embedding resilience at the core of our generative selecting a subset.
AI algorithms, we aim to create a foundation for the Vir-
tual Scanner that is not only technologically advanced 2.1.2. LatentAugment
but also reliable and effective across the spectrum of med-
ical imaging needs. This approach positions our work Data Augmentation (DA) is a crucial strategy in AI to
not just as a technical achievement but as a meaningful enhance the volume and diversity of training datasets,
contribution to the field of radiology, where the capac- thereby mitigating the risk of overfitting and bolstering
ity to handle data in the wild can significantly enhance model generalization to unseen data. Standard DA meth-
diagnostic processes and patient care. ods in image recognition tasks transform the images via
geometric rigid and non-rigid transformations using im-
age processing primitives, such as translation, rotation,
2. Research Activities cropping, etc. However, such transformations rely on
human experts with prior knowledge of the dataset and
Embarking on the journey to realize the Virtual Scanner, fail to generate sufficiently diverse synthetic data. GANs
our investigation delves into a series of research activ- offer a valuable addition to the available augmentation
ities, as shown in Figure 1, each designed to push the techniques. However, GANs generate high-quality sam-
boundaries of what’s possible with generative AI in the ples rapidly, but they suffer from poor mode coverage,
field of radiology [7, 8]. These activities are categorized i.e., the variation and variety of the samples that can be
into two main areas: “Resilient Generative AI" and “Vir- generated, limiting their utility for DA purposes in the
tual Scanner Applications", enveloping a diverse array of medical field.
methodologies and applications aimed at enhancing the We propose LatentAugment [9], a DA strategy that
generation and translation of medical imaging data. By overcomes the low diversity of GANs, opening up for use
addressing many aspects of generative AI, from improv- in DA applications. LatentAugment addresses the three-
ing algorithm resilience to creating virtual modalities, fold challenge of producing synthetic samples that are
these activities underscore our commitment to advancing not only of high fidelity and quality but also diverse and
diagnostic capabilities and patient care through techno- rapidly generated. LatentAugment modifies the latent
logical innovation. vectors of the real training set, moving them towards
regions that maximize their diversity and fidelity. We
2.1. Resilient Generative AI applied LatentAugment to improve the performances
of a downstream model performing of MRI-to-CT im-
2.1.1. GAN Ensemble age translation. The results showed LatentAugment’s
In tackling the complexities of synthetic data generation superiority over common DA methods and naive GAN-
within medical imaging, our research delves into opti- sampling, i.e., creating data sampling from the GAN’s
mizing generative AI through the use of GAN ensembles. latent space without any control.
This strategy is born from the necessity to overcome
inherent limitations in single-model GAN applications, 2.1.3. Paired vs. Unpaired Image Translation
such as mode collapse and the inadequate representation
Image-to-Image translation in medical imaging presents
of real data distributions, a common obstacle in generat-
a critical challenge due to the predominance of un-
ing high-quality and diverse medical images. The core
paired datasets, where the direct correspondence be-
of our approach lies in creating an ensemble of GANs
tween source and target images is not established [10].
that jointly optimizes the visual quality and diversity of
While paired methods, e.g., Pix2Pix, rely on direct map-
synthetic images from a set of GANs. We aim to solve a
Resilient AI
GAN Ensemble Latent Augment
GAN Training ! Optimisation W⇤ ↵lat N
N multiplication
Ensembles Multi-objective Inversion wk k·k22 Llat (wk )
"! ! !!
AAACAnicbVDLSsNAFJ3UV62vqCtxM9iKdVOSgo9lUUGXFewDmlAmk0k7dJIJMxOhhOLGX3HjQhG3foU7/8ZJm4VWDwxzOOde7r3HixmVyrK+jMLC4tLySnG1tLa+sbllbu+0JU8EJi3MGRddD0nCaERaiipGurEgKPQY6Xijy8zv3BMhKY/u1DgmbogGEQ0oRkpLfXOv4nic+XIc6i91rghTaFK9Pjqu9M2yVbOmgH+JnZMyyNHsm5+Oz3ESkkhhhqTs2Vas3BQJRTEjk5KTSBIjPEID0tM0QiGRbjo9YQIPteLDgAv9IgWn6s+OFIUyW1JXhkgN5byXif95vUQF525KozhRJMKzQUHCoOIwywP6VBCs2FgThAXVu0I8RAJhpVMr6RDs+ZP/kna9Zp/WTm7r5cZFHkcR7IMDUAU2OAMNcAOaoAUweABP4AW8Go/Gs/FmvM9KC0beswt+wfj4Bv4ploU=
(G ) 0 L
! !! ↵f
0 ⇤
addition
w =w N
1
G x̃k D Lf (wk )
" x w k·k l2 -distance
!! !! !|"| W w2 X ↵pix
N L
···
⋮
"! ! k·k22 Lpix (w ) k
D trained discriminator
⋯ ! !|"|
!!
wK = w̃ (X ) ↵perc
(G0 ) trained feature extractor
AAACAnicbVDLSsNAFJ3UV62vqCtxM9iKdVOSgo9l0YUuK9gHNKFMJpN26CQTZiZCCcWNv+LGhSJu/Qp3/o2TNgttPTDM4Zx7ufceL2ZUKsv6NgpLyyura8X10sbm1vaOubvXljwRmLQwZ1x0PSQJoxFpKaoY6caCoNBjpOONrjO/80CEpDy6V+OYuCEaRDSgGCkt9c2DiuNx5stxqL/U8QlTaFK9OTmt9M2yVbOmgIvEzkkZ5Gj2zS/H5zgJSaQwQ1L2bCtWboqEopiRSclJJIkRHqEB6WkaoZBIN52eMIHHWvFhwIV+kYJT9XdHikKZLakrQ6SGct7LxP+8XqKCSzelUZwoEuHZoCBhUHGY5QF9KghWbKwJwoLqXSEeIoGw0qmVdAj2/MmLpF2v2ee1s7t6uXGVx1EEh+AIVIENLkAD3IImaAEMHsEzeAVvxpPxYrwbH7PSgpH37IM/MD5/AC/4lqU=
N
k·k22 Lperc (wk )
Reconstruction
G trained generator
update w
Generation "! ! !|"| " ∗ !! !! !|"| L(wk )
! !! x̃
!! !! !|"|
⋯
Homogenization
⋯
"! ! !! !|"| !!
!
!∗
!! !! !|"| ⋯
⋮
⋯
Center A
Paired vs Unpaired
METODI – Image Translation
Approccio proposto Dataset A
NIQE Metric
Trained Extraction
I
Regressor BRISQUE
PaQ2PiQ
!"" H
MSTE
Center B Dataset* pxv
Dataset B GAN
Real
Generator Fake
Generator
Reconstructed hxw
Low-Dose A→B Full-Dose B→A Low-Dose 𝛥H
|| ⋅ ||1
Real/Fake
I pxv
I
Discriminator
Center B H
MSTE
Dataset C
Real Full-Dose
H
MSTE pxv
Virtual Scanner Applications hxw pxv
hxw 𝛥H
Virtual Contrast Enhancement Texture Loss || ⋅ ||1
CC
AAAB8HicbVDLTgJBEOzFF+IL9ehlIzHxRHaNQY9ELh4xkYeBDZkdBpgwM7uZ6TWSDV/hxYPGePVzvPk3DrAHBSvppFLVne6uMBbcoOd9O7m19Y3Nrfx2YWd3b/+geHjUNFGiKWvQSES6HRLDBFesgRwFa8eaERkK1grHtZnfemTa8Ejd4yRmgSRDxQecErTSQxfZE6a12rRXLHllbw53lfgZKUGGeq/41e1HNJFMIRXEmI7vxRikRCOngk0L3cSwmNAxGbKOpYpIZoJ0fvDUPbNK3x1E2pZCd67+nkiJNGYiQ9spCY7MsjcT//M6CQ6ug5SrOEGm6GLRIBEuRu7se7fPNaMoJpYQqrm91aUjoglFm1HBhuAvv7xKmhdlv1Ku3F2WqjdZHHk4gVM4Bx+uoAq3UIcGUJDwDK/w5mjnxXl3PhatOSebOYY/cD5/AOdlkH4=
I pxv
(a) (b)
Gd1 ,𝜃1 h
I
H
MSTE
FFDM
AAAB8nicbVBNSwMxEM3Wr1q/qh69BIvgqeyKVI9FpXgRKtgP2C4lm6ZtaDZZklmxLP0ZXjwo4tVf481/Y9ruQVsfDDzem2FmXhgLbsB1v53cyura+kZ+s7C1vbO7V9w/aBqVaMoaVAml2yExTHDJGsBBsHasGYlCwVrh6Hrqtx6ZNlzJBxjHLIjIQPI+pwSs5HeAPUFaq93cTbrFklt2Z8DLxMtICWWod4tfnZ6iScQkUEGM8T03hiAlGjgVbFLoJIbFhI7IgPmWShIxE6Szkyf4xCo93FfalgQ8U39PpCQyZhyFtjMiMDSL3lT8z/MT6F8GKZdxAkzS+aJ+IjAoPP0f97hmFMTYEkI1t7diOiSaULApFWwI3uLLy6R5VvYq5cr9eal6lcWRR0foGJ0iD12gKrpFddRAFCn0jF7RmwPOi/PufMxbc042c4j+wPn8AR13kSk=
nxn
H
pxv 𝛥H
MSTE
MLO
AAAB8XicbVBNS8NAEN3Ur1q/qh69BIvgqSQi1WPRiwfFCvYD21A220m7dLMJuxOxhP4LLx4U8eq/8ea/cdvmoK0PBh7vzTAzz48F1+g431ZuaXlldS2/XtjY3NreKe7uNXSUKAZ1FolItXyqQXAJdeQooBUroKEvoOkPLyd+8xGU5pG8x1EMXkj7kgecUTTSQwfhCdOb69txt1hyys4U9iJxM1IiGWrd4lenF7EkBIlMUK3brhOjl1KFnAkYFzqJhpiyIe1D21BJQ9BeOr14bB8ZpWcHkTIl0Z6qvydSGmo9Cn3TGVIc6HlvIv7ntRMMzr2UyzhBkGy2KEiEjZE9ed/ucQUMxcgQyhQ3t9psQBVlaEIqmBDc+ZcXSeOk7FbKlbvTUvUiiyNPDsghOSYuOSNVckVqpE4YkeSZvJI3S1sv1rv1MWvNWdnMPvkD6/MHpJGQ6g==
hxw
𝛥H max( · )
I or I Gd2 ,𝜃2 h H or H
|| ⋅ ||1
demux
mean( · ) c pxv
mux 𝜉 .
I pxv
|| · ||FF (a)
hxw .
nxn
SA sum
I
H
MSTE
.
H
h
AM
MSTE
Gdp ,𝜃v
pxv (b)
Gd1 ,𝜃1 h
MSTE
hxw pp x vv
nxn
gCC gMLO
MSTLF
max
max(( ·· ))
AAAB8nicbVBNS8NAEN3Ur1q/qh69LBbBU0lEqsdiLx4r2FpoQ9lsJ+3STTbsTsQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNirVHFpcSaU7ATMgRQwtFCihk2hgUSDhIRg3Zv7DI2gjVHyPkwT8iA1jEQrO0ErdYb+H8IRZozHtlytu1Z2DrhIvJxWSo9kvf/UGiqcRxMglM6bruQn6GdMouIRpqZcaSBgfsyF0LY1ZBMbP5idP6ZlVBjRU2laMdK7+nshYZMwkCmxnxHBklr2Z+J/XTTG89jMRJylCzBeLwlRSVHT2Px0IDRzlxBLGtbC3Uj5imnG0KZVsCN7yy6ukfVH1atXa3WWlfpPHUSQn5JScE49ckTq5JU3SIpwo8kxeyZuDzovz7nwsWgtOPnNM/sD5/AFk4pFY AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQi1WPRiwfFCvYDmlA22027dLMJuxOxhP4NLx4U8eqf8ea/cdvmoNUHA4/3ZpiZFySCa3ScL6uwtLyyulZcL21sbm3vlHf3WjpOFWVNGotYdQKimeCSNZGjYJ1EMRIFgrWD0eXUbz8wpXks73GcMD8iA8lDTgkayRv0PGSPmN1c30565YpTdWaw/xI3JxXI0eiVP71+TNOISaSCaN11nQT9jCjkVLBJyUs1SwgdkQHrGipJxLSfzW6e2EdG6dthrExJtGfqz4mMRFqPo8B0RgSHetGbiv953RTDcz/jMkmRSTpfFKbCxtieBmD3uWIUxdgQQhU3t9p0SBShaGIqmRDcxZf/ktZJ1a1Va3enlfpFHkcRDuAQjsGFM6jDFTSgCRQSeIIXeLVS69l6s97nrQUrn9mHX7A+vgEi2JHE
y 𝛥H
hh x w
w
AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KolI9Vj04rEF+wFtKJvtpF272YTdjRBKf4EXD4p49Sd589+4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8N/PbT6g0j+WDyRL0IzqUPOSMGis1sn6p7FbcOcgq8XJShhz1fumrN4hZGqE0TFCtu56bGH9CleFM4LTYSzUmlI3pELuWShqh9ifzQ6fk3CoDEsbKljRkrv6emNBI6ywKbGdEzUgvezPxP6+bmvDGn3CZpAYlWywKU0FMTGZfkwFXyIzILKFMcXsrYSOqKDM2m6INwVt+eZW0LitetVJtXJVrt3kcBTiFM7gAD66hBvdQhyYwQHiGV3hzHp0X5935WLSuOfnMCfyB8/kD60+NCA==
||
|| ⋅⋅ ||
||11
demux
demux
mean
mean(( ·· ))
mux
mux
nxn 𝛥H
𝜉
𝜉
I or I
I ||FF
||
|| ·· || (c)
pp xx vv (c)
(a) (b)
(b)
hh SA sum
h H or H
Gdd11 ,𝜃
SA sum
H
I
G ,𝜃11
Gd2 ,𝜃2
MSTE H Q
Q c
n
n xx n
n 𝛥H A AM
AM .
MSTE
pxv A
nxn
hxw
or I
I or
max( · ) K
.
CESM 𝛥H
h h H or
or H K
AAAB8nicbVBNSwMxEM3Wr1q/qh69BIvgqeyKVI/FIngRKtoP2C4lm6ZtaDZZklmxLP0ZXjwo4tVf481/Y9ruQVsfDDzem2FmXhgLbsB1v53cyura+kZ+s7C1vbO7V9w/aBqVaMoaVAml2yExTHDJGsBBsHasGYlCwVrhqDb1W49MG67kA4xjFkRkIHmfUwJW8jvAniCtXd/fTrrFklt2Z8DLxMtICWWod4tfnZ6iScQkUEGM8T03hiAlGjgVbFLoJIbFhI7IgPmWShIxE6Szkyf4xCo93FfalgQ8U39PpCQyZhyFtjMiMDSL3lT8z/MT6F8GKZdxAkzS+aJ+IjAoPP0f97hmFMTYEkI1t7diOiSaULApFWwI3uLLy6R5VvYq5crdeal6lcWRR0foGJ0iD12gKrpBddRAFCn0jF7RmwPOi/PufMxbc042c4j+wPn8AS4ykTQ=
Gdd22 ,𝜃
cc
G
pxv
,𝜃22
mean( · )
pxv 𝛥H'
𝛥H' .
MSTLF
|| ⋅ ||1 𝛾
demux mux 𝜉 .. 𝛾
hh xx w
w
I n
n xx n h
|| · ||F (a)
pxv
SA sum
..
n V
V Gdp ,𝜃v
I
H ∗∗
..
MSTE
CC
AAAB8HicbVDLTgJBEOzFF+IL9ehlIzHxRHaNQY9ELh4xkYeBDZkdBpgwM7uZ6TWSDV/hxYPGePVzvPk3DrAHBSvppFLVne6uMBbcoOd9O7m19Y3Nrfx2YWd3b/+geHjUNFGiKWvQSES6HRLDBFesgRwFa8eaERkK1grHtZnfemTa8Ejd4yRmgSRDxQecErTSQxfZE6a12rRXLHllbw53lfgZKUGGeq/41e1HNJFMIRXEmI7vxRikRCOngk0L3cSwmNAxGbKOpYpIZoJ0fvDUPbNK3x1E2pZCd67+nkiJNGYiQ9spCY7MsjcT//M6CQ6ug5SrOEGm6GLRIBEuRu7se7fPNaMoJpYQqrm91aUjoglFm1HBhuAvv7xKmhdlv1Ku3F2WqjdZHHk4gVM4Bx+uoAq3UIcGUJDwDK/w5mjnxXl3PhatOSebOYY/cD5/AOdlkH4=
hh
H
MSTE
AM
Gddp ,𝜃
MSTE
G ,𝜃v
nxn
p v (b) (c)
(c)
pxv (b)
Gdd1 ,𝜃 hh
MSTE SA
hxw
MSTE SA
G ,𝜃1
pxv
n
n xx n
n
MSTLF
max( · )
1 1
𝛥H
hxw
Q
Q
|| ⋅ ||1 mean( · ) n
n xx n
n 𝛥H A
demux mux 𝜉 𝛥H A
or II
II or
|| · ||F (c)
pxv
K
MLO (b)
AAAB8XicbVBNS8NAEN3Ur1q/qh69BIvgqSQi1WPRiwfFCvYD21A220m7dLMJuxOxhP4LLx4U8eq/8ea/cdvmoK0PBh7vzTAzz48F1+g431ZuaXlldS2/XtjY3NreKe7uNXSUKAZ1FolItXyqQXAJdeQooBUroKEvoOkPLyd+8xGU5pG8x1EMXkj7kgecUTTSQwfhCdOb69txt1hyys4U9iJxM1IiGWrd4lenF7EkBIlMUK3brhOjl1KFnAkYFzqJhpiyIe1D21BJQ9BeOr14bB8ZpWcHkTIl0Z6qvydSGmo9Cn3TGVIc6HlvIv7ntRMMzr2UyzhBkGy2KEiEjZE9ed/ucQUMxcgQyhQ3t9psQBVlaEIqmBDc+ZcXSeOk7FbKlbvTUvUiiyNPDsghOSYuOSNVckVqpE4YkeSZvJI3S1sv1rv1MWvNWdnMPvkD6/MHpJGQ6g==
h SA sum hh H or K
Gd1 ,𝜃1
I H or H
Gd2 ,𝜃2
H Q
G d2 ,𝜃2 cc H 𝛾 𝛥H'
𝛥H'
nxn 𝛥H A AM .. 𝛾
I I
MSTE nxn
Planning in Lung CancerVV
nxn
K ..
Whole-Body Translation from CT to PET or Gd2 ,𝜃2 h
c
H or H
pxv Virtual
𝛥H' Treatment ..
∗∗
MSTLF
. 𝛾
SEGMENTATION
hxw .
GENERATION
nxn
STITCHING V ∗
Gdp ,𝜃v
G dp ,𝜃v
hh
.
Dose condition
MSTE
h
SA
Gdp ,𝜃v
MSTE SA
n
n xx n
(c) (continuous value)
n
(b)
PATCH Gd1 ,𝜃1 hPATCH
MSTE
PATCH
SA
BOUNDING BOX
EXTRACTION EXTRACTION
nxn GENERATION FUSION
Q
nxn 𝛥H A
32 x 32 x 32
I or I
32 x 32 x 32
K
G0 → X GX → 0
Gd2 ,𝜃2 h H or H
. c 𝛾 𝛥H'
.
!!"→ %&"
nxn V ∗
.
h
WHOLE-BODY IMAGE WHOLE-BODY
SEGMENTATION
Gdp ,𝜃v
CT IMAGE PET IMAGE
nxn
MSTE SA
32 x 32 x 32 32 x 32 x 32
!!"→ %&" Lung CT0 Lung CTX Lung CT0
!"#$"%&'&()
32 x 32 x 32 32 x 32 x 32
real
fake
!!"→ %&"
Dose
32 x 32 x 32 32 x 32 x 32 prediction
!!"→ %&" DX
Figure 1: Overview of the presented research activities.
pings between paired images, unpaired techniques, e.g., translation methodologies. By integrating a novel paired
CycleGAN, offer flexibility by ignoring this requirement. virtual loss function into the unpaired CycleGAN frame-
Given the logistical and ethical complexities in acquiring work, we enhance the stability and accuracy of unpaired
paired medical images, unpaired models have gained sig- image translation without necessitating direct image
nificant attention despite the potential compromises in pairs. This innovation finds practical application in Low-
performance attributed to training instability and vari- Dose Computed Tomography (LDCT) denoising, a pro-
able outcomes. cess aimed at reducing radiation exposure while main-
Our work introduces a novel approach to narrow the taining image quality. Through this approach, we lever-
performance drop between paired and unpaired image age the abundance of unpaired LDCT and Full-Dose CT
(FDCT) images to validate our model, demonstrating its An extensive quantitative and qualitative analysis un-
effectiveness in producing high-quality, denoised images derpins our research, including evaluations by profes-
that closely approximate FDCT standards, thereby mit- sional radiologists on a novel CESM dataset comprising
igating health risks associated with radiation without 1,138 images. This dataset has been made publicly avail-
compromising diagnostic integrity. able to foster ongoing research and development in the
field. Among the models tested, CycleGAN emerged
2.1.4. Homogenization as the most effective, showcasing its ability to produce
high-quality synthetic recombined images that closely
In the domain of lung CT imaging, the heterogeneity of mimic those obtained with traditional contrast-enhanced
images stemming from varied scanners and reconstruc- techniques.
tion kernels poses a significant challenge. This variability
can severely impact the performance of automated anal-
2.2.2. Virtual Treatment Planning in Lung Cancer
ysis tools, notably in tasks relying on deep learning mod-
els such as 3D Convolutional Neural Networks (CNNs), Monitoring the progression and response to therapy is
which are crucial for predicting patient outcomes like fundamental in lung cancer treatment. Traditional ap-
overall survival rates. To address this challenge, our work proaches rely on a series of CT scans taken before and
introduces an innovative approach based on StarGAN, during treatment to evaluate the efficacy of the inter-
a state-of-the-art image-to-image translation generative ventions. In our previous work [12], we developed an
model, for the homogenization of lung CT images. ODE-based Digital Twin by using patient-specific CT
Our objective is to transform disparate lung CT images, scans to train a deep reinforcement learning controller,
regardless of their originating scanner types or recon- which can adapt to different tissue aggressiveness and
struction kernels, into a standardized format that retains outperform the current radiotherapy clinical practice of
critical diagnostic features while presenting a uniform uniform dose delivery. However, this methodology often
appearance. By employing StarGAN, we leverage its ca- exposes patients to additional radiation and can be logis-
pacity for multi-domain image translation to achieve the tically challenging. Our innovative research introduces
goal of not only enhancing the quality of the dataset but a novel application of AI in virtual treatment planning,
also to significantly improve the performance of down- leveraging conditioned CycleGANs to simulate the po-
stream tasks. This approach paves the way for more tential progression of lung cancer treatment based on
generalized and robust AI tools in medical diagnostics, varying doses. By conditioning the CycleGAN on spe-
ultimately contributing to better patient care and out- cific treatment doses, our model can generate virtual CT
comes. scans that predict how the patient’s anatomy and the
tumor itself might respond to different levels of treat-
2.2. Virtual Scanner Applications ment. This approach allows for the creation of a virtual
time series of CT scans without the need for repeated
2.2.1. Virtual Contrast Enhancement (VCE) radiation exposure. The ability to accurately forecast the
treatment’s progression through these synthetic scans
In the evolving landscape of medical imaging, Contrast
offers a significant advantage in personalizing treatment
Enhanced Spectral Mammography (CESM) represents a
plans, enabling more precise adjustments to therapy reg-
significant advancement, offering detailed insights for
imens based on predicted outcomes. By reducing the
breast cancer diagnosis by utilizing a dual-energy tech-
reliance on multiple physical CT scans and minimizing
nique that integrates both low and high-energy images.
patient exposure to radiation, we pave the way for a more
This method, however, necessitates the administration
patient-centric approach to cancer treatment monitor-
of an iodinated contrast medium and subjects patients to
ing. Additionally, the predictive insights gained from this
higher radiation doses than standard mammography, rais-
technology could significantly enhance decision-making
ing concerns about potential side effects and increased
processes in treatment planning, potentially improving
radiation exposure.
patient outcomes in lung cancer care.
Addressing these critical limitations, our work intro-
duces a novel approach to VCE in CESM using deep gen-
erative models [11]. By eliminating the need for contrast 2.2.3. Whole-Body Translation from CT to PET
mediums and aiming to reduce radiation doses, this re- The integration of CT and PET scans is essential in onco-
search not only mitigates the associated risks but also logical diagnostics, combining the structural clarity of CT
preserves the diagnostic benefits of CESM. Our method- with the metabolic insights of PET imaging. While CT
ology employs GAN, e.g., Pix2Pix or CycleGAN, to gener- scans provide detailed anatomical structure, PET scans of-
ate synthetic recombined images from solely low-energy fer a window into the metabolic activity within the body,
images. making the combined PET/CT an invaluable tool in the
diagnosis, staging, and management of cancer patients. ing processes. By integrating a multi-scale, differentiable
Despite their clinical significance, the dual-modality ap- GLCM into the loss function, we facilitate a deeper under-
proach of PET/CT scanning is not without drawbacks, standing and recognition of complex textural information
e.g., additional radiation exposure and higher costs com- during the image generation phase.
pared to CT-only scans. These limitations restrict the Furthermore, the incorporation of a self-attention layer
widespread availability of PET/CT imaging in numerous represents a pivotal innovation in our methodology, en-
medical centers globally, underscoring the need for alter- abling the dynamic synthesis of texture information
native methods that can replicate the integrative insights across various scales. This approach not only enhances
of PET/CT imaging while mitigating its drawbacks. the denoising capabilities of GANs but also ensures the
Recognizing the challenges inherent in translating CT preservation of essential textural details, thereby improv-
images to PET, especially given the variability in trans- ing the diagnostic utility of the generated images.
lation effectiveness across different anatomical regions, Extensive experimental validation of our approach
our methodology introduces a district-specific approach. within the field of low-dose CT denoising, aimed at im-
Drawing from the current literature, which suggests the proving noisy CT scans while minimizing radiation expo-
potential for improved accuracy through organ-specific sure, underscores the efficacy of our proposed solution.
networks, we propose a novel strategy that segments Utilizing three publicly available datasets, including both
whole-body images into four major anatomical districts. simulated and real-world scenarios, our methodology
Each district is then processed through independently demonstrates a notable improvement over traditional
trained GANs to generate district-specific PET images. loss functions across a variety of GAN architectures.
The final step involves stitching these district-specific
PET images together to reconstruct a comprehensive 2.2.5. Report Generation
whole-body PET scan.
Employing two GAN architectures, Pix2Pix and Cy- Our work focuses on Automatic Medical Reporting
cleGAN, our approach facilitates a comparative analysis (AMR) that, as fostered by escalating digitization of
to evaluate the effectiveness and precision of the im- healthcare data and the mounting stress national health-
age translation process. Through standard evaluation care systems, aims to produce diagnostic reports from
metrics, we quantify the quality of the generated im- biomedical data. The efforts are currently directed to-
ages, highlighting the advantages of our district-specific wards chest radiographs assessing solutions based on
translation methodology over traditional approaches that encoder-decoder and transformer-based models. Along-
rely on a single GAN trained on entire whole-body im- side all this, Quantum Artificial Intelligence represents
ages. This innovative strategy not only promises to re- a novel field whose theoretical superiority in data rep-
duce the time, cost, and radiation exposure associated resentation capabilities and processing speeds makes it
with PET/CT imaging but also offers a tailored approach the main technology we are forwarding our efforts to,
that accounts for the unique characteristics of different with its numerous methodologies for healthcare, even if
anatomical regions. it still presents hardware immaturity, scalability issues,
and substantial financial costs. Because its application in
AMR is still unnavigated, we aim to develop an architec-
2.2.4. Texture Loss
ture that merges traditional encoder-decoder concepts
In the quest to enhance the quality of medical images with quantum computing, to transcribe features obtained
through denoising, the application of GANs emerges as using classical binary computation into quantum states,
a promising task. Yet, a critical challenge lies in the which are then entangled with quantum representations
GAN-based algorithms’ capacity to accurately capture of the shifted predictions for computational and accuracy
and replicate the intricate textural details inherent in benefits.
medical images. This task’s complexity is significantly
amplified by the diverse and complex relationships that
define image textures, making conventional denoising 3. Future Directions and
approaches inadequate for preserving or restoring fine- Conclusion
grained textural fidelity.
Our research introduces a novel loss function tai- As we stand on the verge of a new era in medical imaging,
lored to address these limitations by exploiting the multi- propelled by the advancements in generative AI, the jour-
scale textural properties captured by the Gray-Level Co- ney of our exploration is ongoing. The groundwork laid
occurrence Matrix (GLCM) [13]. The GLCM, traditionally by the Virtual Scanner and the development of resilient
utilized in image processing to quantify texture, is rede- generative AI algorithms opens a myriad of pathways for
fined in our work as a differentiable module compatible future research and application. In the quest for further
with the gradient-based optimization of the GAN train- innovation, it is essential to delve deeper into the integra-
tion of AI with emerging imaging technologies, aiming [4] V. Guarrasi, N. C. D’Amico, R. Sicilia, E. Cordelli,
to enhance the precision, efficiency, and accessibility of P. Soda, A multi-expert system to detect covid-19
diagnostic tools. Future research will focus on refining cases in x-ray images, in: 2021 IEEE 34th Inter-
the algorithms for even greater resilience [14], enabling national Symposium on Computer-Based Medical
them to adapt more seamlessly to the vast diversity of Systems (CBMS), IEEE, 2021, pp. 395–400.
medical imaging data. Additionally, exploring the poten- [5] V. Guarrasi, N. C. D’Amico, R. Sicilia, E. Cordelli,
tial for AI-driven predictive analytics in patient treatment P. Soda, Pareto optimization of deep networks
plans presents a promising frontier, where the insights for covid-19 diagnosis from chest x-rays, Pattern
garnered from virtual scans could inform more person- Recognition 121 (2022) 108242.
alized and effective treatment strategies. Moreover, the [6] V. Guarrasi, P. Soda, Optimized fusion of cnns to
ethical considerations and data privacy concerns asso- diagnose pulmonary diseases on chest x-rays, in:
ciated with deploying AI in healthcare require ongoing International Conference on Image Analysis and
attention. Ensuring the security of patient data and the Processing, Springer, 2022, pp. 197–209.
unbiased application of AI tools remains paramount as [7] V. Guarrasi, L. Tronchin, C. M. Caruso, A. Rofena,
we advance. G. Manni, F. Aksu, D. Paolo, G. Iannello, R. Sicilia,
In conclusion, the exploration into generative AI and E. Cordelli, et al., Building an ai-enabled metaverse
the Virtual Scanner represents a significant leap toward for intelligent healthcare: opportunities and chal-
revolutionizing medical imaging. As we move forward, lenges, in: Ital-IA 2023, Italia Intelligenza Artificiale
the presented research activities and the technologies Thematic Workshops, co-located with the 3rd CINI
developed will undoubtedly pave the way for a future National Lab AIIS Conference on Artificial Intel-
where diagnostics are more accurate, treatments are more ligence (Ital IA 2023), Pisa, Italy, May 29-30, 2023,
personalized, and patient care is enhanced at every level. CEUR-WS, 2023, pp. 134–139.
[8] E. Cordelli, V. Guarrasi, G. Iannello, F. Ruffini, R. Si-
cilia, P. Soda, L. Tronchin, Making ai trustworthy in
Acknowledgments multimodal and healthcare scenarios, Proceedings
of the Ital-IA (2023).
Aurora Rofena and Lorenzo Marcoccia are Ph.D. stu-
[9] L. Tronchin, M. H. Vu, P. Soda, T. Löfstedt, La-
dents enrolled in the National Ph.D. in Artificial Intel-
tentaugment: Data augmentation via guided ma-
ligence, course on Health and life sciences, organized
nipulation of gan’s latent space, arXiv preprint
by Università Campus Bio-Medico di Roma. We ac-
arXiv:2307.11375 (2023).
knowledge financial support from: i) PNRR MUR project
[10] F. Di Feola, L. Tronchin, P. Soda, A comparative
PE0000013-FAIR; ii) PRIN 2022 MUR 20228MZFAA-AIDA
study between paired and unpaired Image Quality
(CUP C53D23003620008); iii) PRIN PNRR 2022 MUR
Assessment in Low-Dose CT Denoising, in: 2023
P2022P3CXJ-PICTURE (CUP C53D23009280001); iv) FCS
IEEE 36th International Symposium on Computer-
MISE (CUP B89J23000580005). This work was also par-
Based Medical Systems (CBMS), IEEE, 2023, pp. 471–
tially supported by the following companies: Teleconsys
476.
S.p.A..
[11] A. Rofena, V. Guarrasi, M. Sarli, C. L. Piccolo,
M. Sammarra, B. B. Zobel, P. Soda, A deep learn-
References ing approach for virtual contrast enhancement in
contrast enhanced spectral mammography, arXiv
[1] G. Fiscon, F. Salvadore, V. Guarrasi, A. R. Garbuglia, preprint arXiv:2308.00471 (2023).
P. Paci, Assessing the impact of data-driven lim- [12] M. Tortora, E. Cordelli, R. Sicilia, M. Miele, P. Mat-
itations on tracing and forecasting the outbreak teucci, G. Iannello, S. Ramella, P. Soda, Deep rein-
dynamics of covid-19, Computers in biology and forcement learning for fractionated radiotherapy
medicine 135 (2021) 104657. in non-small cell lung carcinoma, Artificial Intelli-
[2] V. Guarrasi, P. Soda, Multi-objective optimization gence in Medicine 119 (2021) 102137.
determines when, which and how to fuse deep [13] F. D. Feola, L. Tronchin, V. Guarrasi, P. Soda, Multi-
networks: An application to predict covid-19 out- Scale Texture Loss for CT denoising with GANs,
comes, Computers in Biology and Medicine 154 2024. arXiv:2403.16640.
(2023) 106625. [14] C. M. Caruso, V. Guarrasi, S. Ramella, P. Soda,
[3] V. Guarrasi, L. Tronchin, D. Albano, E. Faiella, A deep learning approach for overall survival
D. Fazzini, D. Santucci, P. Soda, Multimodal ex- analysis with missing values, arXiv preprint
plainability via latent shift applied to covid-19 strat- arXiv:2307.11465 (2023).
ification, arXiv preprint arXiv:2212.14084 (2022).