Generative intelligence has now become the pinnacle of research in artificial intelligence. Generative intelligence systems allow you to generate texts, images, sounds, etc. Generative intelligence systems are based on deep neural network models that are trained on large samples of data.
Consequently, a variety of generative intelligence systems have emerged that transform text into image, image into image, image into text, sound into text, text into sound, sound into sound. Textto-image transformation takes place on a fixed set of data. For this purpose, a transformer was used, which autoregressively simulates text and graphic tokens [1]. The Codex GPT language model, which is trained on GitHub, makes it possible to write code in Python. The paper [3] analyzes the opportunities and risks of fundamental models, such as language, vision, reasoning. In addition, the analysis of technical principles -the architecture of models, learning algorithms, data, is carried out. The impact of generative intelligence on society has also been studied.
Other papers [4] investigated a family of neurospeech models for LaMDA dialogue applications. The model generates responses based on learning from known sources. The authors investigated the LaMDA system in education.
Generative intelligence has also found applications in medicine. The paper investigates the use of generative intelligence in oncology, in particular for generating cytological images of breast cancer.
Breast cancer is one of the most common cancers among women worldwide. Early diagnosis and accurate determination of the stage of disease development are key factors for successful treatment and reduction of mortality. Cytological, histological and immunohistochemical images are used to detect pathologies. These images are a class of biomedical images. Cytological analysis of images of cell preparations is one of the diagnostic methods, which allows the detection of pathological changes at the cellular level [5].
To train automatic systems for diagnosing breast cancer, large and high-quality datasets are needed that reflect the variety of possible pathological changes. Datasets of cytological images of breast cancer have the following features: Diversity of cell structures: normal cells, different types of atypical and malignant cells. Image variability: changes in color, lighting, focus, etc. Annotations & markups: availability of expert markup for supervised learning.
The available datasets of real images are limited and poorly annotated. Therefore, an actual problem is the generation of biomedical images in oncology. This provides the necessary accuracy in the classification of biomedical images. To solve this problem, the paper uses the means of generative intelligence: GAN and diffusion models.
Researchers in their works have developed a number of approaches to solving the problem of generating biomedical images. In particular, the article discusses the problems of creating medically significant fine-grained images of pulmonary adenocarcinomas using Stable Diffusion models [6]. The authors show how these models can be used to generate images with a limited number of samples, which is important for medical research where data can be scarce.
Other papers present the analysis of diffusion models in medical imaging [7]. The authors consider modern methods and approaches in the processing of medical images using deep learning, in particular diffusion models, which can significantly improve the quality of diagnostics.
The paper [8] presents a novel generative model that uses Langevin dynamics to generate samples by estimating gradients in data distribution with the addition of Gaussian noise. This avoids problems with low-dimensional manifolds and improves sample quality.
The paper explores how computer vision models trained on large sets of images from the Internet automatically learn human social biases, such as racism and sexism [9]. This question becomes important in the context of the ethical use of generative models.
The authors' article [10] describes the process of synthetic data generation in digital pathology using diffusion models. The authors present a comprehensive approach to assessing the quality of the generated images, which can be useful for educational purposes.
An article by A. Radford, J.W. Kim, C. Hallacy and other authors describes the CLIP model, which is trained on large datasets of images and texts to perform a variety of computer vision tasks without special training for each task [11]. The model demonstrates the ability to zero-learn on many datasets, which opens up new possibilities for application.
The authors R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer describe latent diffusion models for high-resolution image generation [12]. They use autoencoders to reduce the dimensionality of the data, which allows for a reduction in computational costs without losing image quality.
The article [13] presents the Imagen model, which is a text-to-image diffusion model with a high level of photorealism. The model uses large language models to encode the text, which greatly improves the quality of the samples.
A paper by other researchers describes the use of diffusion probabilistic models for the synthesis of histopathological images, which is important for pathology research [14].
The paper [15] presents diffusion probabilistic models used to generate high-quality images. These models demonstrate high-quality samples on various datasets, such as CIFAR10 and LSUN. Thus, the analysis of literature sources indicates significant progress in the development of image synthesis methods, in particular, through the use of diffusion models and GANs. This opens up new possibilities for improving the quality and diversity of synthesized images in medical imaging.
In the paper [16], researchers consider a deep learning approach using non-stationary thermodynamics. They represent diffusion probabilistic models that gradually break down the structure in the data through the diffusion process and then train the reverse process to reconstruct the structure, creating a flexible and computationally efficient generative model.
In the paper, the authors investigate diffusion models that are superior to generative adversarial networks (GANs) in image synthesis tasks [17]. They demonstrate that diffusion models can achieve high quality image samples, surpassing current generative models.
The paper [18] presents the use of cascading diffusion models to generate high-quality images. The cascade diffusion model consists of several stages, where each subsequent stage increases the resolution of the image.
The authors T. Karras, S. Laine, and T. Aila describe a new generator architecture for generative adversarial networks (GANs) that borrows ideas from stylistic transference [19]. This architecture allows for automatic and uncontrolled separation of high-level attributes from stochastic variations in generated images.
The paper describes a new approach to variational autoencoders (VAEs) for image generation [20]. The NVAE network uses deep-cut convolutions and batch normalization to improve the quality of generated images.
The paper [21] describes a novel approach to generative modeling that uses stochastic differential equations (SDE) to transform a data distribution to a simple noise distribution and vice versa. The model achieves high results in image generation and demonstrates the capabilities for solving inverse problems.
The authors of another paper [22] developed a method for filling images using diffusion probabilistic denoiseing models (DDPM). Based on this method, diverse and semantically meaningful images can be generated, surpassing current GAN-based methods
In [23], the authors describe improvements to diffusion probabilistic denoiseing models for image generation. They use accuracy and completeness metrics to compare images. Experiments have shown that diffusion models achieve higher completeness at similar values of the FID metric.
When creating datasets of cytological images, it is important to standardize the annotation, as it ensures high quality, reliability and compatibility of data for their further use in machine learning and diagnostic processes. In addition, proper annotation increases the efficiency of training AI models, as well-defined labels reduce error rates in the learning process and help algorithms better recognize cell features and pathological changes. When segmenting and annotating objects on cytological images, it is important to adhere to the image annotation formats used in the PASCAL VOC [32] and COCO [33] datasets.
The APCData dataset [34] consists of cytological images of the cervix, developed in collaboration with the laboratory of anatomical pathology and cytology, located in Rivera, Uruguay. The set includes 425 images divided into 6 classes. The cells are labeled using bounding boxes and centers of the nuclei.
The dataset consists of 425 images of 2048 x 1532 pixels, corresponding to 73 diagnosed with Papanicolaou test. A total of 3619 cells were annotated. The images were taken using the Olympus CX40RF100 microscope and the Olympus LC30 Optical Microscope camera. Images are processed using Olympus L.Cmicro software. Bounding boxes were created for cells in an appropriate format for use with the YOLO convolutional neural network architecture.
The UFSC OCPap dataset [35] contains 9797 annotated images of 1200x1600 pixels in size, obtained from 5 slides with diagnosed oral tissue cancer and 3 healthy samples. The slides are provided by the Hospital Dental Center of the University Hospital of the Federal University of Santa Catarina. The dataset contains binary kernel masks and cell annotations in Json format. The images are divided into subsets of training, validation, and testing. The images were taken using an Axio Scan.Z1 microscope and a Hitachi HV-F202SCL camera. Dataset images are derived from virtual slides measuring 214,000 x 161,000 pixels (0.111 μm x 0.111 μm per pixel). For annotation, medical specialists used LabelMe and LabelBox tools.
The authors have developed a database of cytological images of breast cancer [36]. The image was obtained using a laboratory setup that includes a Delta Optical microscope, a Tucsen digital CMOS camera with a resolution of 8 megapixels. The sources of microscope slides and diagnostic information are provided by the Department of Pathological Anatomy with the Sectional Course of Forensic Medicine of the Ternopil National Medical University. The database consists of 14 related tables. The table of studies includes basic information about each study, its title, the object of the study, as well as references to the patient and doctor associated with this study.
All images of cytological samples are divided into 4 classes. The database supports several user roles: physician, expert, administrator. The database contains information about the segmentation algorithm used. For each cell there are the following features: area, perimeter, contour height, contour width, contour circularity, center coordinates, main axis of inertia, minor axis of inertia, angle of inclination of the main axis, Feret diameter, coordinates of the bounding rectangle, roundness, compactness.
As you know, the architecture of modern GANs consists of a generator and a discriminator [37].
The generator and discriminator architectures are based on cells. A cell consists of nodes performing an append operation and operations between them.
The following operations are used in the generator cell: convolution by kernel 11, 33, 55; separable convolution by kernel 33; zero; skip connection. The cell architecture remains the same for the entire generator model. In contrast to the generator, the set of operations in the discriminator cell is extended by two operations: the maximum pooling by the kernel 33 and the average pooling by the 33 core. The architecture of the generator is shown in Figure 1 and described in Table 1 and Table 2 L18: Output 64643 The discriminator architecture is shown in Figure 2 and described in Table 3 and Table 4. The generator takes a noise vector from a Gaussian distribution of 1×128 as input, and outputs an image of 64×64×3.
The number of nodes in the generator and discriminator cells is 4 and 5 respectively. There are two skip connection operations in the generator cell, and 3 in the discriminator cell. There is also a zero operation in the discriminator cell, which is not present in the generator. The Self-Attention operation is applied 2 times in both the generator and the discriminator. However, in the generator, this operation is placed towards the end of the network, and in the discriminator, on the contrary, it is closer to the beginning.
Generator CELLG Cell and Upsample Block Structure 1. training based on its dataset of images in the Hypernetwork environment; 2. the process of making noise of the initial 𝐼 𝐶 dataset; 3. noise reduction process.
Let's detail the steps. The initial dataset is transformed to a latency space: 𝐼 𝐶 → 𝑍 0𝐶 . Based on 𝑍 0𝐶 , we calculate the noise value at each step t as follows:
where 𝑎 𝑡 is the coefficient that determines the noise rate at step t. The value of step t is selected from the range 𝑡 ∈ [0, 𝑇] where T is the number of steps; 𝜀 𝑡 -is the value of random Gaussian noise at step t. Value 𝜀 𝑡 calculated according to the expression:
where N is a normal distribution law with a expected value of E = 0 and a variance of D = 1. L19: L17 + L18 11 L20: Output 11 The noise reduction value is calculated according to the expression:
where 𝜀 𝑡 is the estimated noise value at step t; 𝛼 𝑡 -the coefficient that determines the noise level in the previous step t; 𝛽 𝑡 is a coefficient that controls the level of noise reduction.
After performing the noise reduction process (after traversing t=T steps), a 𝑍 1𝐶 vector is formed in the latency space. The encoder then transforms 𝑍 1𝐶 into a set of 𝐼 𝐶𝐷 images, with 𝐼 𝐶𝐷 ≫ 𝐼 𝐶 . The quality of the generated images is checked by IS and FID metrics.
Two main metrics are used to assess the quality of synthesized images: the IS metric and the FID metric.
The IS metric is based on the Google Inception V3 neural network model for color image classification. This metric was tested on the ImageNet dataset with a capacity of 1.2 million RGB images, which are divided into 1000 classes.
The analytic expression for the metric is as follows:
This configuration provides high performance for creating AI-generated images, allowing you to effectively use the capabilities of the Stable Diffusion model to generate high-quality results.
Experiment results. FID metric value -0.63 (class 1 -0.54, class 2 -0.6, class 3 -0.7, class 4 -0.68). The value of the IS metric is 3.99.
An example of real images is shown in Figure 4. An example of synthetic images is shown in Figure 5.
Let's analyze the conducted computer experiments using GAN and Stable Diffusion. The results of comparison of synthesized cytological images quality using the developed GAN architecture and other known architectures are given in Table 5. The advantages of GAN are as follows:
1. The ability to generate high-quality, realistic images, video, and audio.
2. The ability to control the synthesis process (from the smallest details to common features in the image). 3. Relatively high speed of image synthesis, which is synthesized in one pass (forward pass) of the neural network.
The disadvantages of GAN are as follows:
1. Significant computing resources and the need for expertise to learn effectively, making them less accessible. 2. Collapse mode, where the generator begins to produce a limited number of images, which reduces the variety of synthetic images. 3. The learning process is complex and long because GAN consists of two neural networks competing with each other.
The advantages of diffusion models are as follows:
1. The ability to produce high-quality images that often surpass GAN in terms of realism and variety. 2. The ability to work with complex data distributions, which makes diffusion models universal for different areas. 3. A simpler learning process compared to GAN, which avoids the problem of collapse.
The disadvantages of diffusion models are as follows:
1. Significant computing resources for training and generation, which may limit the availability of use. 2. Data generation using an iterative process is quite resource-intensive compared to the forward pass method used by GAN.
Diffusion models transform noise distribution into data distribution through a diffusion process, gradually improving the generated image. This process provides a high degree of control over the generation process, as the model can be stopped at any point to obtain different levels of detail.
However, GANs generate data in a single step, where the generator creates the image and the discriminator evaluates it. Although this process is faster, it can lead to collapse mode, where the generator produces a limited number of images.
Consequently, GAN is built using the concept of competition between a generator and a discriminator to create realistic images, while diffusion models transform noise into images through an iterative process of diffusion (noise reduce). Diffusion models involve careful tuning of hyperparameters and longer training times. In addition, both approaches require a large amount of training data to perform optimally.
As a result, the tools for synthesizing cytological images have been developed and their comparison has been conducted in the work.
At the same time, the following results were obtained:
1. A new GAN architecture has been developed, which, unlike existing architectures, uses the Self-Attention mechanism in the generator and discriminator, which made it possible to improve the quality of synthesized images. The developed architecture for image synthesis supports the mechanism of image synthesis by labels (conditional generation), which is not relevant for the above architectures and approaches. 2. A new algorithm for the synthesis of cytological images based on diffusion models has been developed. In the Stable Diffusion environment, an algorithm for synthesizing cytological images was implemented, which made it possible to synthesize a sufficient sample of images for CNN training. Consequently, generation based on the diffusion model in the Stable Diffusion environment showed better results compared to generation based on GAN.
Therefore, further research will be the development of new diffusion models for generating histological and immunohistochemical images.
The authors of another publication [24] developed an algorithm for stochastic variational Bayesian inference. This approach allows you to train model parameters without using iterative inference schemes.
The authors of this publication have been analyzing biomedical images for over twenty years under the guidance of Professor Oleh Berezsky. A number of publications reflect methods, algorithms, and software tools for analyzing cytological, histological and immunohistochemical images [25][26][27][28][29][30][31]. This is the result of a creative collaboration of researchers from West Ukrainian National University and Ivan Horbachevsky Ternopil National Medical University.
Given: the set of real cytological images of 𝐼 𝐶 . Image synthesis will be carried out on the basis of GAN and networks that are built on DMN diffusion models. After generating by means of GAN, we get a set of 𝐼 𝐶𝐺 images. Using DMN, we get a set of 𝐼 𝐶𝐷 images. In addition, we are given two metrics: IS and FID.
It is necessary to find the 𝑀 𝐼𝑆 and 𝑀 𝐹𝐼𝐷 distances between the set of real 𝐼 𝐶 cytological images and the sets of 𝐼 𝐶𝐺 and 𝐼 𝐶𝐷 synthetic images using the IS and FID metrics, i.e.:
where 𝐸 is the math expected value; 𝑥~𝑝 𝑔 shows what 𝑥 an image synthesized from a distribution -𝑝 𝑔 (𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟 𝑑𝑖𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛); 𝐷 𝐾𝐿 is the Kullback-Leibler distance between the conditional probability distribution and the marginal distribution 𝑝(𝑦) [38].
Discriminator CELLD Cell and Downsample block structure
The IS metric measures the average Kullback-Leibler distance between a conditional distribution 𝑝(𝑦|𝑥) and a marginal class distribution 𝑝(𝑦). The minimum value of the metric is 1, and the maximum value is the number of classes.
The FID metric compares the distributions of original and synthetic data. Based on this metric, the distance between images is calculated as follows:
), where (𝑚 𝑟 𝐶 𝑟 ) and (𝑚 𝑔 𝐶 𝑔 ) are the average and covariance of the real and synthesized data distributions respectively, 𝑇𝑟 − sum of the diagonal elements of the matrix. Therefore, the smaller the value of the metric, the smaller the distance between the distributions, that is, the images are more similar to each other [39]. The FID metric is sensitive to distortion in images (shift, noise, etc.).
Computer experiments on the synthesis of cytological images were carried out using GAN and Stable Diffusion.
To conduct computational experiments, a training set of cytological images was used, which was published on the Zenodo platform [40].
Images from the training dataset have been transformed to a resolution of 64×64 pixels (the original resolution is 3264×2448). The initial number of images is around 100, which is not enough. Therefore, the dataset is expanded to 800 images by applying affine transformations. By applying this technique, the dataset was balanced -it contains the same number (200 images) for each class. To extend the training dataset, Rudi own library with default parameters [41] was used. Images are randomly rotated, flipped, scaled. All operations were applied with a probability of 50%.
Hardware. The Python programming language and the Pytorch framework were used to write the code. A virtual machine with the following configuration was used for experiments: 16 GB RAM, 10 vCPU x 2.2 GHz, Nvidia Tesla V100 GPU 16 GB (13.2 TFLOPS).
Training Options. In experiments, Hinge Loss was used as a loss function and Adam optimizer (betas = 0.5, 0.999). A technique called the Two Time-scale Update Rule is also used, which involves the use of different learning norms for the generator and the discriminator. Accordingly, the learning rate of the generator is 0.0001, and the discriminator is 0.0004. For all convolutional, deconvolutional, and linear layers, the spectral normalization technique was applied in both models, which allows to stabilize the learning process. Batch size -128, number of iterations -100,000. Training time ~13.6 GPU hours.
Experiment results. The FID metric value is 3.39 (Class 1 -3.42, Class 2 -3.42, Class 3 -3.35, Class 4 -3.37), and the IS metric value is 3.95
Examples of synthesized images are shown in Figure 3. Training Options. To train the model, the Linear loss function and the Adam optimizer were used. 768, 1024, 320, 640, 1280 layers with linear activation and initialization of Normal weights were chosen as the hypermodel structure. Batch size was set to 1 and Gradient Accumulation Steps to 1. Gradient Clipping with a value of 0.1 was used to stabilize learning. The training took place with a learning norm for the hypermodel of 0.00001. The total number of iterations was 20,000 steps, and the size of the images was fixed at 512x512 pixels. The training was carried out using text prompts based on a style_filewords.txt template. The intermediate results of the images were saved in the log directory every 100 steps.
Hardware. For the experiments, the infrastructure from Jarvis Labs was used, which has the following computing resources:
The authors have not employed any Generative AI tools.