Automatic Segmentation of Immunohistochemical Images based on U-NET Architectures Oleh Berezskya, Oleh Pitsuna, Bohdan Derysha, Tamara Datskob, Kateryna Berezkaa, Nadiya Savkaa a West Ukrainian National University, 11 Lvivska st., Ternopil, 46001, Ukraine b I.Ya. Horbachevsky Temopil State Medical University, m.Voli, 1, Ternopil, 46001, Ukraine Abstract Biomedical (immunohistochemical) images of breast cancer are analyzed in this paper. Related work on automatic image segmentation is reviewed. The authors analyzed the architectures of convolutional neural networks of the U-net type for automatic segmentation of immunohistochemical images. Examples of neural network architectures that make it possible to receive more accurate and better image segmentation are given. A modified neural network architecture for segmentation of immunohistochemical images is developed. Computer experiments were carried out according to different numbers of epochs and iterations. ROC-curves are constructed to assess the quality of segmentation of known and modified network architectures of the U-net type. Keywords 1 Breast cancer, automated diagnosis, CNN, immunohistochemical analysis. 1. Introduction Cancer ranks as the second or third cause of human mortality. Early diagnosis is the only way to prevent and timely treat cancer. With the increasing growth of computer power, it has become possible to use modern information technologies such as artificial intelligence to diagnose diseases. Methods and tools of artificial intelligence are widely used in medicine. Traditional methods, such as knowledge engineering, are applied to present expertise in various fields of medicine. With the development of INTERNET technologies, a new field emerged, that is, telemedicine. Telemedicine makes it possible to attract experts from different countries remotely [1]. Breast cancer ranks as the first cause of women mortality in the world. Pathomorphological diagnosis is the main method of research and treatment. The following biomedical images are used in oncology for diagnosis: cytological, histological and immunohistochemical. Therefore, computer vision methods and algorithms are used to process these images [2-4]. Computer vision methods and algorithms are used at different stages of image processing: pre- processing, segmentation, classification, etc. Segmentation is one of the key stages in the processing of biomedical images. The main purpose of segmentation is to clearly identify the nuclei of tissue cells in the image. Neural networks are widely used in medicine [4]. Recently, U-net technology has been used increasingly, which makes it possible to train neural networks for a specific type of images. First, this technology was designed for use in medicine. Therefore, the development and training of a neural network for segmentation of immunohistochemical images is an urgent task. IDDM-2021: 4th International Conference on Informatics & Data-Driven Medicine, November 19–21, 2021 Valencia, Spain; EMAIL: ob@tneu.edu.ua (A. 1); o.pitsun@tneu.edu.ua (A. 2); dbb@tneu.edu.ua (A. 3); datsko_t@tdmu.edu.ua (B.1); k.berezka@wunu.edu.ua(A. 4); nadya_savka@ukr.net(A. 5); ORCID: : 0000-0001-9931-4154 (A. 1); 0000-0003-0280-8786 (A. 2); 0000-0002-7215-9032 (A. 3), 0000-0001-9283-2629 (B. 1), 0000- 0002-9632-4004(A. 4); 0000-0003-4182-7867(A. 5); ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) The aim of this work is to conduct a comparative analysis of the quality of immunohistochemical image segmentation using existing and modified architectures of U-net type networks. 2. Literature review Let us review some new articles on automatic image segmentation. Manual and automatic segmentation is investagated, its advantages and disadvantages are highlighted in paper [5]. The study was conducted with the involvement of experts in the field of histology and pathology. The method of automatic segmentation is based on the threshold color. The developed method makes it possible to segment positive and negative cells based on the Ki67 biomarker with an average accuracy of 90%. Article [6] presents a new segmentation and nuclei counting method that can automatically count nuclei using a modified single-pass super pixel segmentation method. This method was tested on a large sample of immunohistochemical images. Experiments showed sufficient accuracy and speed. The authors in paper [7] described the automatic analysis technique of images of oral cancer tissue areas stained with immunoglobulin P53. Tissue images are segmented by the entropy threshold, and cluster cells are found by the watershed method applied selectively. For this purpose, the color indices of each nucleus and the method of reference vectors are used. This is followed by a classification assessment. Article [8] presents a method of segmentation and classification of medical images based on a hierarchical transformation and a cascade model of nonlinear mappings. The authors offered to use the proposed method for classifying cell nuclei. Mostly, all cancer cells are studied by diagnosticians, who are undoubtedly highly qualified specialists, but such an assessment is a subjective one. An attempt was made in paper [9] to solve the problem of subjectivity of assessment on the example of immunohistochemical images based on the biomarker KI-67. The result is an integrated, universal method of automatic cancer diagnosis based on immunohistochemical images. The basis for assessing breast cancer is the tumor's response to a particular type of protein. An effective algorithm for automatic detection of carcinoma cells was presented in article [10]. This formed the basis of automatic segmentation and classification of cancer using biomarkers. The combination of histological and immunohistochemical images allows one to make an accurate diagnosis. Immunohistochemical images based on biomarkers were classified using neural networks DCNN and VGG. New machine learning strategies were applied to train these networks in [11]. The methods of automatic detection of cancer cells are analyzed in paper [12]. The developed algorithm selected sets of images that contain ganglion cells. These images were then examined and evaluated by five pathologists. The authors of paper [13] developed accurate diagnosis methods of lymphoma using machine learning methods. Semi-automatic histopathological analysis of adenocarcinoma tumor cells is presented in article [14]. Article [15] presented the developed U-net convolutional neural network for segmentation of nuclei on a public data set. A recurrent residual U-network (R2U-Net) was proposed, which demonstrated high performance on different sets of medical images (retinal blood vessels, skin cancer, and lung segmentation) in segmentation tasks. Scientists developed and trained U-net to segment immunohistochemical images previously coloured. This network was then applied to the raw slides to create templates [16]. The proposed approach is based on CNN and applies for pixel-by-pixel segmentation of information areas. This became possible with the parallel use of GPU processors [17]. The results presented in the article can be beneficial for clinical use. The authors tested the effectiveness of four methods based on deep learning. Testing was performed on different areas of the slide image: normal tissues, with clusters of immune cells and artifacts [18]. F. Mahmood, R. Chen, D. Borders and others proposed to supplement the existing data sets of medical images using CGAN. Competitive U-Net with spectral normalization was then used to increase learning stability. This paired network not only generates new images, but also looks for the optimal loss function. Paper [19] presents machine learning methods for effective segmentation, classification and quantification of breast cancer images. It is shown that the presented algorithm is better than the superpixel classification, which is based on CNN. Paper [20] compares the efficiency of segmentation of several architectures (U-Net, Mask R-CNN, Cellpose, U-Net ResNet) and traditional segmentation algorithms on the example of fluorescent images of cell nuclei. A new method of segmentation was developed in paper [21], based on a deep learning network. It is aimed at accurate selection of the stained area of the nuclei of breast tissue images. Morphological post-processing of segmented micro-objects splits overlapping nuclei. To improve the results of a separate classifier, an ensemble method is used, which integrates the solutions of three models of machine learning for the final assessment of cancer. The researchers developed a tandem of an artificial neural network and U-Net for recognising and generating interference. In article, the developed network achieves better results compared to the semantic segmentation of deep neural networks. The authors of this paper analyzed segmentation methods on different types of neural networks. For conducting research, deep convolutional networks - U-Net, Mask R-CNN and the developed network (GB U-Net) are compared [22]. Thus, based on the analysis of literature review, the following approaches to the segmentation of biomedical images are identified: 1. Traditional methods of segmentation. 2. Application of CNN and deep learning. 3. The use of U-Net type networks. 3. Problem statement U-net technology is considered to be a modern approach to image segmentation in medicine according to the analyzed literature sources. The segmentation phase is very important, as it makes it possible to prepare the image for processing at a high level of computer vision, in particular classification and diagnosis. The objectives of this work are as follows. 1. Analyze immunohistochemical images of breast cancer. 2. Develop a modification of the neural network architecture of U-net type. 3. Compare different architectures of U-net type networks for segmentation of immunohistochemical images. 4. Analyze computer experiments. 4. Analysis of immunohistochemical images of breast cancer. Immunohistochemical examenations clarify the diagnosis, which is made on the basis of histological studies. The specified diagnosis is made with use of biomarkers. There is a microscopic description of immunohistochemical images of breast cancer [23]. A tumor was found in the breast tissue, the morphological structure of which corresponds to moderately differentiated invasive ductal carcinoma (G2) (Figure 1). Figure 1: G2 invasive ductal carcinoma Immunohistochemical images were obtained on the basis of immunohistochemical studies. Estrogen receptor α (DAKO, clone EP1) - positive reaction in 59% of tumor cells (PS = 4), moderate intensity (IS = 2). TS = 4 + 2 = 6 - positive result (Figure 2). Figure 2: Reaction to the Estrogen receptors Progesterone receptor (DAKO, clone PgR 636) - specific color is not deteсted. TS = 0 - negative result (Figure 3). Figure 3: Reaction to the Progesterone receptors Oncoprotein c-erbB-2 / neu (HER-2 / neu) (DAKO, polyclonal) - specific color is not detected, 0 points - a negative result (Figure 4). Figure 4: Reaction to the biomarker HER-2 / neu Ki-67 antigen (DAKO, clone MIB-1) -positive reaction of significant intensity in 44% of tumor cells (Figure 5). Figure 5: Reaction to the biomarker Ki-67 Moderately differentiated (G2) invasive ductal carcinoma of the breast (code ICD-O code : 8500/3).According to the results of immunohistochemical staining of luminal type B tumor, HER-2 / neu negative: • estrogen (ER +) - sensitive (59% of cells ++); • progesterone (PR-) - negative; • HER-2 / neu - negative (0 points); • 44% of cells are positive (+++) for the marker of proliferative activity of Ki-67. 5. Generalized approach to automatic image segmentation. In [24] the authors present a generalized approach to diagnosis based on immunohistochemical images. One of the important stages of this approach is the segmentation stage for determining the relative area and average brightness of cell nuclei. As it is known, U-net is a convolutional neural network, which was designed mainly for the segmentation of biomedical images. This architecture is based on a sequence of convolutional layers. The architecture of the U-network is characterized by certain features, in particular in the transmission of information in the downlink and uplink. Unlike other topologies of convolutional networks, this topology does not make use of fully connected layers, but only convolutional ones. The basic architecture of the U-net network is shown in Figure 6. The structure of the U-net is described in more detail in [25]. Figure 6: Basic structure of U-net This architecture consists of two main parts: narrowing (left) and expansion (right). The narrowing path is a typical convolutional neural network architecture and consists of a convolution operation and a RELu function. This reduces the dimensionality of the image. Each step in the expansion phase consists of operations that increase the discretization of the property map. The neural network is trained by the method of stochastic gradient descent, based on input immunohistochemical images and segmentation maps. In the process of U-net learning it is necessary to determine the degree of similarity between the generated segmented image and the segmented person. To do this, the Dice coefficient is used. A generalized approach to segmentation of immunohistochemical images is shown in Figure 7. Saved Neural model of network neural training Grayscaling network Input images Segmentatio n using trained Test image networks Result Figure 7: Generalized approach to immunohistochemical segmentation Learning process includes the following stages: 1. Download images into memory for training sampling. 2. Convert an image to a grayscale image. 3. Adjust the structure and parameters of the neural network. 4. The learning process. 5. Save learning outcomes in a separate file. The testing phase is as follows: 1. Download the test sample. 2. Download a file with a trained network. 3. The segmentation process. 6. U-net architectures Examples of U-nets used for segmentation of immunohistochemical images are shown in Figure 8. In this case, examples of the encoder architecture of the basic U-net neural network (a) and the modified neural network architecture (b) are given. Conv 1024X1024 Conv 1024X1024 Conv 512X512 Conv 512X512 Conv 256X256 Conv 256X256 Conv 128X128 Conv 128X128 Conv 64X64 Conv 64X64 Conv 32X32 a) Type A b)Type B Figure 8: U-net encoder architectures Unlike convolutional neural network for classification of images, the U-net type network does not have enough flexibility to modify the architecture. The U-net type network consists of descending and ascending parts that are interconnected. The modified architecture, which is shown in Figure 3 (b), has an additional convolution layer of 32 x 32 pixels. This modification was made to increase the accuracy of segmentation of immunohistochemical images by complicating the neural network architecture. 7. Computer experiments The database of images was used for computer experiments [26]. ROC curves are constructed to assess the quality of segmentation. Quantitative interpretation of ROC is done due to AUC indicator — the area bounded by the ROC curve and the axis of the share of false positive classifications. To determine the accuracy of the classification, the pixel value (black / white) on the resulting image and the corresponding value on the image processed by the expert are used. Table 1 shows the results of the segmentation accuracy assessment for the U-net type network architecture (Type A). Table 1 Comparative analysis of segmentation accuracy assessment of convolutional neural network architecture (Type A) A number of A number ROC AUC,% iterations of epochs 200 1 68 300 1 65 100 2 61 300 2 49 Table 2 shows the results of the segmentation accuracy assessment for the U-net type network architecture (Type B). Table 2 Comparative analysis of segmentation accuracy assessment of convolutional neural network architecture (Type B) A number of A number ROC AUC,% iterations of epochs 300 1 79 200 2 85 These U-net network architectures are implemented on the basis of Python programming language, Tensorflow and Keras libraries using the Linux Mint operating system. The image dataset consists of 150 immunohistochemical images, divided in the proportion of 70% (training sample) / 30% (test sample). The comparison shows that the best result was obtained using the architecture of a neural network of type B with the following parameters: the number of iterations – 200, the number of epochs – 2. 8. Conclusions 1. The analysis of relevant work is carried out. It is shown that automatic segmentation is a relevant task. Automatic segmentation algorithms are analyzed; their advantages and disadvantages are highlighted. The necessity of using convolutional neural networks of U-net type is substantiated. 2. The histological image of breast cancer is analyzed and the diagnosis is specified on the basis of the analysis of immunohistochemical images. 3. The typical U-net architecture is analyzed and its modification is developed. U-net architectures are compared on the basis of immunohistochemical images. 4. Based on computer experiments, it was found that the best result was obtained using the architecture of the neural network of type B with the following parameters: the number of iterations – 200, the number of epochs – 2 and the accuracy of segmentation is 85%. The results of automatic segmentation are used for preliminary diagnosis of cancer. 9. References [1] R. Wootton, J. Craig, V. Patterson, Introduction to Telemedicine, 2nd ed., CRC Press, 2017. [2] O. Pitsun, O. Berezsky, L. Dubchak, K. Berezka, T. Dolynyuk, B. Derish, Cytological Images Clustering of Breast Pathologies, in: Proceedings of the IEEE International Conference «Computer Science and Information Technologies» CSIT’2020, Zbarazh-Lviv. Ukraine, 23-26 September 2020, Vol. 1, pp. 62-65. doi: 10.1109/CSIT49958.2020.9321867. [3] O. Berezsky, O. Pitsun, T. Dolynyuk, L. Dubchak, N. Savka, G. Melnyk, V. Teslyuk, Cytological Image Classification Using Data Reduction, in: Proceedings of the II International Workshop Informatics & Data-Driven Medicine (IDDM 2019), Lviv, Ukraine, 11-13 November 2019. ceur- ws.org/Vol-2488/paper2.pdf [4] Ivan Izonin, Roman Tkachenko, Ivanna Dronyuk, Pavlo Tkachenko, Michal Gregus, Mariia Rashkevych. Predictive modeling based on small data in clinical medicine: RBF-based additive input-doubling method[J]. Mathematical Biosciences and Engineering, 2021, 18(3): 2599-2613. doi: 10.3934/mbe.2021132 [5] F. A. Dzulkifli, M. Y. Mashor, H. Jaafar, Colour thresholding-based automatic Ki67 counting procedure for immunohistochemical staining in meningioma, International Journal of Computational Vision and Robotics. 11(3) (2021) 279–298. doi: 10.1504/IJCVR.2021.115160. [6] J. Shu, J. Liu, Y. Zhang, H. Fu, M. Ilyas, G. Faraci, ... & G. Qiu, Marker controlled superpixel nuclei segmentation and automatic counting on immunohistochemistry staining images, Bioinformatics. 36(10) (2020) 3225–3233. doi: 10.1093/bioinformatics/btaa107 [7] K. S. Hameed, K. S. Abubacker, A. Banumathi, & G. Ulaganathan, Immunohistochemical analysis of oral cancer tissue images using support vector machine, Measurement. 173 (2021) 108476. https://doi.org/10.1016/j.measurement.2020.108476 [8] Y. Feng, L. Zhang, Z. Yi, Breast cancer cell nuclei classification in histopathology images using deep neural networks, Int J Comput Assist Radiol Surg. 13(2) (2018 Feb) 179-191. doi: 10.1007/s11548-017-1663-9. Epub 2017 Aug 31. PMID: 28861708. [9] S. Razavi et al., An Automated and Accurate Methodology to Assess Ki-67 Labeling Index of Immunohistochemical Staining Images of Breast Cancer Tissues, in: Proceedings of the 25th International Conference on Systems, Signals and Image Processing (IWSSIP), 2018, pp. 1–5. doi: 10.1109/IWSSIP.2018.8439184. [10] J. Oscanoa, F. Doimi, R. Dyer, J. Araujo, J. Pinto and B. Castaneda, Automated segmentation and classification of cell nuclei in immunohistochemical breast cancer images with estrogen receptor marker, in: Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2016, pp. 2399–2402. doi: 10.1109/EMBC.2016.7591213. [11] P. Pradhan, K. Köhler, S. Guo, O. Rosin, J. Popp, A. Niendorf, & T. Bocklitz, Data Fusion of Histological and Immunohistochemical Image Data for Breast Cancer Diagnostics using Transfer Learning, in: Proceedings of the International Conference on Pattern Recognition Applications and Methods, V. 1: ICPRAM, ISBN 978-989-758-486-2, 2021, February, p. 495–506. doi: 10.5220/0010225504950506. [12] A. Greenberg, A. Aizic, A. Zubkov, S. Borsekofsky, R. R. Hagege, & D. Hershkovitz, Automatic ganglion cell detection for improving the efficiency and accuracy of hirschprung disease diagnosis, Scientific reports. 11(1) (2021) 1–9. doi: 10.1038/s41598-021-82869-y [13] C. Syrykh, A. Abreu, N. Amara, A. Siegfried, V. Maisongrosse, F. X. Frenois, ... & P. Brousset, Accurate diagnosis of lymphoma on whole-slide histopathology images using deep learning, NPJ digital medicine. 3(1) (2020) 1–8. doi: 10.1038/s41746-020-0272-0 [14] D. Kazdal, E. Rempel, C. Oliveira, M. Allgäuer, A. Harms, K. Singer, M. Kriegsmann, Conventional and semi-automatic histopathological analysis of tumor cell content for multigene sequencing of lung adenocarcinoma, Translational Lung Cancer Research. 10(4) (2021) 1666. doi: 10.21037/tlcr-20-1168 [15] M. Z. Alom, C. Yakopcic, T. M. Taha and V. K. Asari, Nuclei Segmentation with Recurrent Residual Convolutional Neural Networks based U-Net (R2U-Net), in: Proceedings of the IEEE National Aerospace and Electronics Conference, 2018, pp. 228–233. doi: 10.1109/NAECON.2018.8556686. [16] W. Bulten, P. Bándi, J. Hoven, R. van de Loo, J. Lotz, N. Weiss, ... & G. Litjens, Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard, Scientific report.s 9(1) (2019) 1–10. doi: 10.1038/s41598-018-37257-4 [17] Z. Swiderska-Chadaj, T. Markiewicz, J. Gallego, G. Bueno, B. Grala, & M. Lorent, Deep learning for damaged tissue detection and segmentation in Ki-67 brain tumor specimens based on the U- net model, Bulletin of the Polish Academy of Sciences: Technical Sciences. (2018) 849–856. doi: 10.24425/bpas.2018.125932. [18] Z. Swiderska-Chadaj, H. Pinckaers, M. van Rijthoven, M. Balkenhol, M. Melnikova, O. Geessink, ... & F. Ciompi, Learning to detect lymphocytes in immunohistochemistry with deep learning. Medical image analysis. 58, 101547 (2019). doi: 10.1016/j.media.2019.101547 [19] F. D. Khameneh, S. Razavi, & M. Kamasak, Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network, Computers in biology and medicine. 110 (2019) 164–174. doi: 10.1016/j.compbiomed.2019.05.020 [20] F. Kromp et al., Evaluation of Deep Learning Architectures for Complex Immunofluorescence Nuclear Image Segmentation, in: Proceedings of the IEEE Transactions on Medical Imaging. 40(7) (July 2021) 1934–1949. doi: 10.1109/TMI.2021.3069558. [21] L. B. Mahanta, E. Hussain, N. Das, L. Kakoti, & M. Chowdhury, IHC-Net: A fully convolutional neural network for automated nuclear segmentation and ensemble classification for Allred scoring in breast pathology, Applied Soft Computing. 103, 107136. (2021). doi:10.1016/j.asoc.2021.107136 [22] A. Lagree, M. Mohebpour, N. Meti, K. Saednia, F. I. Lu, E. Slodkowska, W. T. Tran, A review and comparison of breast tumor cell nuclei segmentation performances using deep convolutional neural networks, Scientific Reports. 11(1) (2021) 1–11. doi: 10.1038/s41598-021-87496-1. [23] C. R. Taylor, L. Rudbeck, Immunohistochemical Staining Methods, Dako, 2013, 208 p. [24] O. Berezsky, O. Pitsun, T. Datsko, B. Derysh, I. Tsmots, V. Tesluk, Specified diagnosis of breast cancer on the basis of immunogistochemical images analysis, in: Proceedings of the III International Conference on Informatics & Data-Driven Medicine (IDDM-2020), Lviv, Ukraine, 19-21 November 2020, pp. 129-135. ceur-ws.org/Vol-2753/short5.pdf [25] Yu. Weng, NAS-Unet: Neural Architecture Search for Medical Image Segmentation, Special section on advanced optical imaging for extreme environment, IEEE Access, 2019, Vol.7, pp. 44247-44257. doi: 10.1109/ACCESS.2019.2908991. [26] Certificate for copyright to a work №75359 “Database of digital histological and cytological images of precancerous and cancerous states of the breast «BPCI2100».” Berezsky, O., Melnyk, G., Verbovyi S., Pitsun, O., Nykolyuk, V., Datsko, T. Registration date 14.12.2017 State Enterprise "Ukrainian Intellectual Property Institute" (Ukrpatent) https://ukrpatent.org.