A wide variety of biomedical image data, as well as methods for generating training images using basic deep neural networks, were analyzed. Additionally, all platforms for creating images were analyzed, considering their characteristics. The article develops a method for generating artificial biomedical images based on GAN. GAN architecture has been developed for biomedical image synthesis. The data foundation and module for generating training images were designed and implemented in a software system. A comparison of the generated image database with known databases was made.
2. When creating synthetic images, the true state of the pathology is known, which allows you to accurately evaluate the effectiveness of the algorithms and analyze their accuracy.
3. Using synthetic data, large volumes of images can be generated for testing and validating algorithms at different scales.
4. Public access makes these and other datasets available to the public, facilitating data sharing and research collaboration.
Therefore, an actual problem is the generation of biomedical images in oncology. This provides the necessary accuracy in the classification of biomedical images. To solve this problem, GAN was used in this work.
In their fundamental work, Ian Goodfellow and others developed the concept of Generative
Adversarial Networks (GANs) [
In work [
Synthesizing medical images using GANs involves not only extension data sets, but also ensuring
the confidentiality of patient information. Work [
The authors of article [
Other research [
The article [
The authors of work [
To synthesize high-resolution images of skin lesions, the researchers compared several GAN
network architectures. A classifier was successfully trained based on the synthesized samples [
The method has been developed in [16] that allow synthesizing histopathological images. Based on specific tissue types, cancer subtypes, and known reference image data, the authors synthesize histopathological images.
The authors of the article [17] used an improved conditional GAN architecture, which they called HistoGAN. The researchers used the self-attention module and other methods to stabilize the learning process and improve the quality of the synthesized images.
The authors of the article [18] propose a generative competitive network with additional regularization based on loss of sharpness to generate realistic histopathological images. The authors additionally introduce a sharpening loss that enhances the contrast of pixels on the contours of the nuclei.
Researchers in [19] propose a vision transformer-based GAN model for synthetically augmenting a set of histopathological images.
The authors of this work have developed a many of CADs for automatic diagnosis in oncology [2024].
Therefore, GANs have significant potential in biomedical image synthesis. However, the application of GANs is influenced by training complexity, data quality, ethical issues, and clinical relevance.
The conducted analysis showed that there is a problem of increasing datasets for the classification of biomedical images. To solve of this problem, it is necessary to: - analyze the typical training sets of images; - develop the artificial image synthesis method; - develop the software synthesis and image storage; - conduct the computer experiments.
World practice has shown that the creation and storage of data sets is a pressing problem. Creating datasets is a separate task. A large number of specialists from different countries are involved in the creation of data sets. Data sets are stored on well-known platforms. This is also a separate task. Therefore, we will first analyze the training data sets.
The main characteristics TND [25] are as follows: 1. Image volume – the number of images of normal and pathological tissues of the human body. 2. Image format. The most efficient way to present images is through WSI (Whole Slide Imaging) multi-scale whole slide imaging. To train algorithms, TND must contain instructions or expert segmentation of micro-objects. This segmentation is maintained through contour coordinates and binary masks.
Automated microscopy systems are used to create TND. For example, the Automated Slide Analysis Platform (ASAP) is an open platform for visualization, annotation, and automated analysis of WSI images. ASAP is built on the basis of well-known open source software OpenSlide, Qt and OpenCV.
Camelyon16, Tupac16 are TDSs playing an important role in the development of modern digital histopathology and cancer diagnosis. Based on large volumes of tissue imaging, these kits provide researchers and clinicians with a unique opportunity to study and develop advanced machine learning algorithms for automated analysis of histological data.
The Camelyon16 dataset [26] was built to detect lymph node metastases for breast cancer. It includes about 400 gigabytes of lymphatic images, which are used to train and test the algorithms.
The TUPAC16 dataset [27, 28] was created to investigate the morphological characteristics and detect cancerous abnormalities in the mammary glands. The set includes 490 gigabytes of images. The Camelyon17 dataset is an extended version of Camelyon16, containing even more images and clinical data.
Platforms are formed based on data sets. Let's analyze the main functions of the platforms. The main functions of the platforms are: 1. Data storage and organization. Platforms allow you to load, store and organize data in hierarchical structures. Versioning, GitHub integration, visit and download statistics can also be supported. 2. Availability and exchange. Data can be shared with the global community. 3. Methods and description of data sets. The platforms allow you to provide detailed descriptions and metadata for each dataset. Metadata includes DOI, author information (ORCID), keywords, project and research grant information, source citations, organization type, publisher type. 4. Licenses and access control. Users can set licenses and rules for accessing datasets. The following are available for public sets: Creative Commons, GPL, Open Data Commons, Community Data License. Collaboration functionality is also available, allowing multiple users to jointly own and maintain a private or publicly accessible dataset. 5. Tools for analysis and rendering. Some platforms provide tools for analyzing, processing and visualizing data directly on the platform. 6. Support for data formats. Platforms can support different data formats, including text, images, video, audio and others. In particular, the available formats are XML, PDF, HTML, EXCEL, CSV, JSON, RDF, DOC, ZIP, BigQuery.
Comparative characteristics of known platforms are presented in Table 1.
A method for synthesizing artificial biomedical databases using GANs has been developed in this research.
GAN is a neural network consisting of two neural networks: a generator and a discriminator, trained simultaneously using a competitive process. The generator tries to create realistic data, the discriminator distinguishes genuine data from artificially generated ones. Over time, the generator improves its ability to create realistic data by learning from feedback from the discriminator.
Thus, our method combines data loading, data augmentation, GAN training, image generation, and image quality evaluation.
To generate the image, use the GAN. The architecture of the generator and discriminator is based on ResNet Block. Also in the generator and discriminator there is a self-attention mechanism. The architecture of GenBlock and the generator is shown in Figures 2 and 3.
The architecture of the DiscBlock and discriminator is shown in Figures 4 and 5.
The following training parameters were used during generation: 1. Optimizer – Adam. 2. Generator learning rate 1e-4. 3. Discriminator learning rate 4e-4. 4. Loss function – Hinge Loss. 5. Epochs – 100,000.
6. Batch size – 96.
The implementation of the software is based on the use of the Google Cloud Platform (GCP) cloud infrastructure. This approach allows efficient use of data storage resources. The program infrastructure is shown in Figure 6.
The software implementation has two main files: train.py and generate.py. The train.py file is for training a PyTorch GAN model. This file defines the architecture and parameters of the GAN model. To train the model, we use the Vertex AI service, which provides infrastructure and tools for efficient model training (with GPU support).
The FID and IS metrics are used to evaluate the performance of the trained model.
After graduation, the model is uploaded to Cloud Storage. Next, the trained GAN model is deployed on the Vertex AI platform by generating a final URL. This address can be accessed to create images. The deployment process involves specifying the necessary computing resources and loading our model code into a Docker container for efficient use.
Once the model is deployed, the master data is stored in a Cloud SQL instance. This database is a central repository for storing the following data: Model endpoint URLs, Image scoring metrics. This allows you to track model versions and associated performance characteristics.
The generate.py file is used to create new images. Users can specify the number of images to create. An optional parameter is the identifier of the model used for image synthesis. By default, the last model added to the database is used. For example, to generate 1000 images, we use model ID 5. To do this, just run the program by calling python generate.py 1000. 5. This script interacts with the URL endpoint of the deployed model, sending requests to generate images.
The generated images are obtained and stored in GCP cloud storage. At the same time, information about the created images is written to the cloud SQL instance of PostgreSQL. The database schema is shown in Figure 7.
Thus, our software infrastructure integrates Python, PyTorch, Google Cloud Platform services such as Vertex AI, Cloud Storage, and PostgreSQL Cloud to facilitate GAN model training, deployment, and management. The generate.py script provides a convenient interface for creating new images while maintaining a record of the model's performance and generated content in cloud storage and a database. This architecture makes it possible to efficiently generate images for various purposes.
A dataset was used to conduct computer experiments. The dataset is a sample of histological images measuring 64 by 64 pixels. The dataset is divided into three classes. The total number of images is 185. This sample is expanded to approximately 700 images by applying a set of affine transformations (random rotation, translation, scaling).
An example of the start date of the set is given in Table 2.
For the experiments we used the GCP n1-standard-4 virtual machine: 15 GB RAM, 4 vCPU, Nvidia Tesla V100 GPU 16 GB (13.2 TFLOPS). The network training process lasted about 11 hours. The Inception Score (IS) and FID metrics were used to evaluate the network. The metrics values are as follows: IS – 3.025, FID – 68. As a result of the experiments, 2000 artificial images were generated for each class. The resolution of the generated images is 64 by 64 pixels. An example of these images is shown in Figure 8.
The results of the experiments are stored in the Cloud SQL database. Examples of records from the database are shown in Figure 9.
9. References [15] K. Tero, T. Aila, S. Laine, J. Lehtinen, Progressive Growing of GANs for Improved Quality,
Stability, and Variation. ArXiv abs/1710.10196 (2017) [16] L. Hou, Ayush Agarwal, D. Samaras, T. Kurç, Rajarsi R. Gupta, J. Saltz, Unsupervised
Histopathology Image Synthesis. ArXiv. abs/1712.05021 (2017) [17] Y. Xue et al. Selective Synthetic Augmentation with HistoGAN for Improved Histopathology
Image Classification. Medical Image Anal. 67 (2021) 101816 doi:10.1016/j.media.2020.101816. [18] S. Butte, H. Wang, M. Xian, A. Vakanski Sharp-GAN: Sharpness Loss Regularized GAN for Histopathology Image Synthesis. in: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2021, pp. 1-5 [19] M. Li, C. Li, C. Peng, B.C. Lovell Conditioned Generative Transformers for Histopathology Image
Synthetic Augmentation. (2022) doi:10.48550/arXiv.2212.09977 [20] O. Berezsky, O. Pitsun, G. Melnyk, T. Datsko, I. Izonin, B. Derysh, An Approach toward Automatic Specifics Diagnosis of Breast Cancer Based on an Immunohistochemical Image.
Journal of Imaging 9(1) (2023) 12. doi: 10.3390/jimaging9010012 [21] O. Berezsky, O. Pitsun, P. Liashchynskyi, B. Derysh, N. Batryn, Computational Intelligence in Medicine, volume 149 of Lecture Notes on Data Engineering and Communications Technologiesthis, Springer, Cham, 2023, pp. 488–510. doi: 10.1007/978-3-031-16203-9_28 [22] O. Berezsky, P. Liashchynskyi, O. Pitsun, P. Liashchynskyi, M. Berezkyy Comparison of Deep Neural Network Learning Algorithms for Biomedical Image Processing. CEUR Workshop Proceeding, 2022, 3302, pp. 135–145. [23] O.M. Berezsky, P.B. Liashchynskyi, Comparison of generative adversarial networks architectures for biomedical images synthesis. Applied Aspects of Information Technology. 4(3) (2021) 250– 260. doi: 10.15276/aait.03.2021.4. [24] O. Berezsky, O.Pitsun, N. Batryn, T. Datsko, K. Berezska, L. Dubchak Modern automated microscopy systems in oncology, in: Proceedings of the 1st International Workshop on Informatics & Data-Driven Medicine, Lviv, Ukraine, 28-30 november 2018, pp. 311-325 [25] The CAMELYON16 challenge. URL: https://camelyon16.grand-challenge.org/Data/ [26] B. E. Bejnordi, M Veta, P Johannes van Diest, et al. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. Journal of the American Medical Assosiation. 318 (2017) 2199-2210. doi:10.1001/jama.2017.14585 [27] Veta, Mitko, et al. "Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge." Medical image analysis. 54 (2019): 111-121. doi: 10.1016/j.media.2019.02.012. [28] Tumor Proliferation Assessment Challenge. URL: https://tupac.grand-challenge.org [29] Liashchynskyi P. Rudi library. URL: https://github.com/liashchynskyi/rudi.