Deep learning has revolutionized artificial intelligence by enabling the extraction of intricate representations from large datasets. Generative models have emerged as powerful tools for data synthesis by mimicking the distributions of training data. Despite their advancements, these models encounter critical concerns, including biases, privacy risks, and the authenticity of the generated data. These challenges underscore the necessity for incorporating fairness, expert insights, and comprehensive evaluations into their development. Applications like medical imaging pose challenges due to scarce high-quality data and demanding requirements for condition-specific synthesis. Furthermore, the management of high-dimensional low-sample size (HDLSS) data accentuates the demand for sophisticated representation learning techniques, enabling the generation of efective synthetic data from limited clinical datasets. The complexity of longitudinal medical data, characterized by intricate temporal correlations, further challenges existing methodologies, revealing their limitations. In the light of above, my doctorate research path intends to focus on two main objectives: (i) employ cutting-edge techniques to advance beyond current state-of-the-art in data synthesis, and (ii) bridge the gap between privacy, fairness and generating meaningful synthetic data leveraging on XAI and HCI for further robustness.
Deep learning has revolutionized artificial intelligence by enabling the extraction of intricate
patterns from vast datasets. However, in many fields, including healthcare, the scarcity of
labeled data remains a significant bottleneck despite the increasing availability of large datasets.
This challenge is particularly acute in healthcare due to the limited availability of patient
cohorts and the high dimensionality of data, such as neuroimaging with millions of voxels.
Traditional statistical analyses may be unreliable [
Balancing Realism and Variability While imposing constraints to enhance the realism and clinical validity of synthesized medical images is crucial, it is important to acknowledge the potential trade-of between realism and variability. Overly strict restrictions on the generative models may limit the diversity and representative capacity of the generated samples, introducing bias into the synthetic data. Ensuring a high degree of realism is essential for the efective utilization of synthetic data in medical applications. However, if the generative process is excessively constrained, the models may fail to capture the true underlying distribution of the data, leading to an incomplete or skewed representation of the target domain. This could result in synthetic samples that do not accurately reflect the inherent diversity and variability present in real clinical data, potentially hindering the generalization capabilities of models trained on such data.
To address this challenge, it is crucial to strike a delicate balance between imposing constraints for realism and allowing suficient flexibility for variability. One approach could involve incorporating domain knowledge and expert feedback through human-computer interaction (HCI) techniques. By leveraging the expertise of medical professionals, the generative process can be guided to prioritize clinically relevant features and constraints, while still allowing for a certain degree of variability within acceptable bounds.
Moreover, the integration of explainable AI (XAI) methodologies can provide insights into the generative models’ decision-making processes, enabling a more clear understanding of the factors influencing the generated samples. Through XAI-guided analysis, it may be possible to identify and adjust specific model components or parameters that contribute to unrealistic artifacts or limited variability, without compromising overall realism.
Additionally, iterative refinement cycles with domain experts could be implemented, wherein generated samples undergo evaluation for realism and diversity, leading to adjustments in the generative process. This iterative approach can help fine-tune the balance between realism and variability, ensuring that the synthetic data accurately reflects the complexities and nuances of real clinical data while maintaining representative diversity.
Image augmentation techniques constitute a fundamental subset of DA methods that manipulate existing image samples through various transformations. These can range from basic operations such as elastic transformations which deform image spatial dimensions through displacement ifelds, introducing localized shape variations without constraints on parallelism or aspect ratios. However, elastic warping may produce anatomically implausible samples. On the other hand, erasing transformations replace selected image regions with constant intensity values or random noise, while pixel-level techniques adjust attributes like brightness, contrast, saturation, and noise. These augmentations mainly modify original data, potentially limiting generalization. Augmented samples also tend to strongly correlate with originals.
Synthetic generation ofers a diferent approach to overcome the limitations of traditional
techniques. Unlike manipulating existing data, these methods create entirely new samples
from scratch, potentially introducing greater diversity and complexity. Specialized models,
tailored to specific modalities and tasks, further enhance those capabilities. While promising,
synthetic techniques require increased computational resources and more complex architectures
compared to basic transformations. A crucial challenge remains to ensure the visual fidelity and
realism of generated samples, as unrealistic artifacts can negatively impact model performance.
Generative Adversarial Networks (GANs). GANs [
Variational autoencoders (VAEs). VAEs [
Difusion Models (DMs). Recent academic literature has notably expanded the use of DMs for
image synthesis [
While DMs progress rapidly, their application in medical contexts lags, highlighting a gap
between state-of-the-art (SotA) generative models in general domains and those in medical fields.
There is a pursuit for computational eficiency in general-purpose generative models.
Wuerstchen [
State Space Models (SSMs). Parallel to DMs, research explores computationally eficient
architectures. Mamba [
Explainable Artificial Intelligence (XAI). XAI has emerged as a critical field in the era of deep learning, aiming to address the opaque and complex nature of modern machine learning models. As these models become increasingly sophisticated and are deployed in sensitive domains, the need for transparency, interpretability, and accountability has become fundamental. XAI techniques aim to clarify how these models make decisions, helping users and researchers understand why they make certain predictions, recognize any biases, and ensure compliance with ethical and regulatory guidelines. In the context of synthetic data generation, XAI plays a pivotal role in ensuring the reliability and trustworthiness of the generated data.
As previously mentioned, synthetic DA is fundamental in fields with scarce, sensitive, or
costly real-world data. Among GANs, VAEs, SSMs and DMs, the latter have gained significant
attention due to their ability to generate high-fidelity samples by approximating complex
realworld data distributions, yet their complex architectures and iterative generation processes pose
challenges in terms of interpretability and explainability. Integrating XAI techniques into DMs
for synthetic data generation can provide valuable insights into the model’s decision-making
process, as proposed by Park et al. [
Although initially designed for general-purpose images, its application within the medical
domain proves advantageous for ensuring accurate representation and generation of relevant
anatomical structures and pathological features. An initial strategy for developing forthcoming
frameworks may involve the adaptation of tools such as DF-RISE and DF-CAM [
Reinforcement Learning from Human Feedback (RLHF). The synergy between XAI and
DMs can be further amplified through RLHF [
In this task, RLHF can be used to fine-tune DMs for generating synthetic medical images
that align with medical standards and requirements. The process involves training a DM to
generate medical images based on specific classes. Human evaluators (e.g. domain experts)
then provide feedback on the generated images, indicating which ones better align with the
task requirements, such as improving class-image alignment, or refining aesthetic quality. In
this work [
This research work aims to investigate the capabilities and the limits of modern generative models in the context of HDLSS domains. Specifically, the primary objective of this study is to develop a comprehensive framework capable of efectively addressing the fundamental constraints associated with scenarios characterized by HDLSS data such as medical imaging. This framework will focus on systematically addressing challenges related to privacy, fairness, and consistency in discrete segments. To ensure the reliability and robustness of the methodology, the incorporation of Human-Computer Interaction (HCI) techniques can facilitate systematic validation of significant outcomes. By harnessing the expertise of domain specialists, these techniques can guide both the generation of synthetic data and the final output, thereby validating the integrity and credibility of the framework. Moreover, it is imperative to devise a solution that transcends limitations inherent to specific environments by leveraging reproducible datasets and extending its applicability to a diverse array of real-world datasets, as real-world data may exhibit unpredictable or anomalous behavior.
RQ1: What are the current limitations of using modern generative models for sensitive domains? Modern generative models have immense potential to benefit healthcare, but their adoption remains limited. Key challenges include: ensuring the clinical validity of synthesized images; handling multimodal data; scarce annotated datasets; protecting privacy; explaining outputs; computational eficiency; and robustness to data distribution shifts. A comprehensive examination of these limitations can help identify critical gaps and requirements, informing the development of tailored methodologies, validation procedures, guidelines, and best practices.
Specifically, how to ensure that data from the same patient or source remains entirely within either the training or validation set, preventing any overlap or leakage between the two? Additionally, how to establish appropriate similarity measures that go beyond human visual evaluation to objectively assess whether the synthetic data accurately captures the true underlying distribution of the target domain?
To address this, it is necessary to develop robust evaluation protocols that involve splitting the limited dataset into two distinct subsets (training and validation) in a principled manner, ensuring that data from the same source (e.g., patient) is consistently allocated to either subset. This strict separation is crucial to maintain the validity of the DA process and subsequent model evaluation, restricting the generation of synthetic data solely to the training subset.
However, rather than relying solely on human visual inspection, which can be subjective and limited, we must establish quantitative similarity measures that can objectively evaluate the extent to which the synthetic data accurately represents the characteristics and nuances present in the validation set. These measures should be designed to assess the ability of models trained on the augmented data to generalize and accurately label or process the validation data, thereby indirectly evaluating the quality and representativeness of the synthetic samples.
Finally, it is essential to acknowledge that in certain scenarios, the available data may be so scarce that confident label prediction becomes impossible without introducing additional real data into the dataset. In such extremely data-limited situations, relying solely on augmentation techniques is unlikely to be suficient and may, in fact, aggravate bias rather than address it. To make meaningful progress, it is crucial to recognize the inherent limitations of DA and its inability to circumvent the fundamental issue of inadequate real data in certain contexts. Identifying and quantifying the data scarcity thresholds beyond which augmentation alone becomes inefective would represent a significant step forward.
By addressing these challenges and adhering to strict train-test separation principles, we can ensure a rigorous and reliable evaluation of synthetic data generation methods in sensitive domains, ultimately enabling the development of more efective and trustworthy DA techniques for applications with limited and scarce data.
RQ2: Can the combination of XAI and HCI in generative models efectively enhance
the quality and robustness of new data?
The integration of XAI into DA is still limited. Apart from XAI-guided classification tasks, some
work [
Deep generative models for medical image augmentation address the fundamental challenge of scarce training data in healthcare applications. While traditional augmentation provides some benefits, its eficacy remains limited. This research focuses on progressing from foundational models like GANs and VAEs to more advanced techniques including difusion models and SSMs. Key advantages of generative augmentation include producing realistic synthetic data and capturing the true underlying distribution. However, limitations persist. The dual objectives of this work are: (i) employing cutting-edge advances to push beyond the current SotA in medical image synthesis, and (ii) addressing the interplay between privacy, fairness, and the generation of meaningful synthetic data by leveraging XAI and HCI for enhanced robustness.