=Paper=
{{Paper
|id=Vol-3038/paper3
|storemode=property
|title=Automatic Segmentation of Immunohistochemical Images based on U-NET Architectures
|pdfUrl=https://ceur-ws.org/Vol-3038/paper3.pdf
|volume=Vol-3038
|authors=Oleh Berezsky,Oleh Pitsun,Bohdan Derysh,Tamara Datsko,Kateryna Berezka ,Nadiya Savka
|dblpUrl=https://dblp.org/rec/conf/iddm/BerezskyPDDBS21
}}
==Automatic Segmentation of Immunohistochemical Images based on U-NET Architectures ==
https://ceur-ws.org/Vol-3038/paper3.pdf
Automatic Segmentation of Immunohistochemical Images based
on U-NET Architectures
Oleh Berezskya, Oleh Pitsuna, Bohdan Derysha, Tamara Datskob, Kateryna Berezkaa, Nadiya
Savkaa
a
West Ukrainian National University, 11 Lvivska st., Ternopil, 46001, Ukraine
b
I.Ya. Horbachevsky Temopil State Medical University, m.Voli, 1, Ternopil, 46001, Ukraine
Abstract
Biomedical (immunohistochemical) images of breast cancer are analyzed in this paper.
Related work on automatic image segmentation is reviewed. The authors analyzed the
architectures of convolutional neural networks of the U-net type for automatic segmentation
of immunohistochemical images. Examples of neural network architectures that make it
possible to receive more accurate and better image segmentation are given. A modified
neural network architecture for segmentation of immunohistochemical images is
developed. Computer experiments were carried out according to different numbers of
epochs and iterations. ROC-curves are constructed to assess the quality of segmentation of
known and modified network architectures of the U-net type.
Keywords 1
Breast cancer, automated diagnosis, CNN, immunohistochemical analysis.
1. Introduction
Cancer ranks as the second or third cause of human mortality. Early diagnosis is the only way to prevent
and timely treat cancer.
With the increasing growth of computer power, it has become possible to use modern information
technologies such as artificial intelligence to diagnose diseases. Methods and tools of artificial
intelligence are widely used in medicine. Traditional methods, such as knowledge engineering, are
applied to present expertise in various fields of medicine. With the development of INTERNET
technologies, a new field emerged, that is, telemedicine. Telemedicine makes it possible to attract
experts from different countries remotely [1].
Breast cancer ranks as the first cause of women mortality in the world. Pathomorphological diagnosis
is the main method of research and treatment.
The following biomedical images are used in oncology for diagnosis: cytological, histological and
immunohistochemical. Therefore, computer vision methods and algorithms are used to process these
images [2-4].
Computer vision methods and algorithms are used at different stages of image processing: pre-
processing, segmentation, classification, etc. Segmentation is one of the key stages in the processing of
biomedical images. The main purpose of segmentation is to clearly identify the nuclei of tissue cells in
the image.
Neural networks are widely used in medicine [4]. Recently, U-net technology has been used
increasingly, which makes it possible to train neural networks for a specific type of images. First, this
technology was designed for use in medicine. Therefore, the development and training of a neural
network for segmentation of immunohistochemical images is an urgent task.
IDDM-2021: 4th International Conference on Informatics & Data-Driven Medicine, November 19–21, 2021 Valencia, Spain;
EMAIL: ob@tneu.edu.ua (A. 1); o.pitsun@tneu.edu.ua (A. 2); dbb@tneu.edu.ua (A. 3); datsko_t@tdmu.edu.ua (B.1);
k.berezka@wunu.edu.ua(A. 4); nadya_savka@ukr.net(A. 5);
ORCID: : 0000-0001-9931-4154 (A. 1); 0000-0003-0280-8786 (A. 2); 0000-0002-7215-9032 (A. 3), 0000-0001-9283-2629 (B. 1), 0000-
0002-9632-4004(A. 4); 0000-0003-4182-7867(A. 5);
©️ 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
The aim of this work is to conduct a comparative analysis of the quality of immunohistochemical
image segmentation using existing and modified architectures of U-net type networks.
2. Literature review
Let us review some new articles on automatic image segmentation.
Manual and automatic segmentation is investagated, its advantages and disadvantages are
highlighted in paper [5]. The study was conducted with the involvement of experts in the field of
histology and pathology. The method of automatic segmentation is based on the threshold color. The
developed method makes it possible to segment positive and negative cells based on the Ki67 biomarker
with an average accuracy of 90%.
Article [6] presents a new segmentation and nuclei counting method that can automatically count
nuclei using a modified single-pass super pixel segmentation method. This method was tested on a large
sample of immunohistochemical images. Experiments showed sufficient accuracy and speed.
The authors in paper [7] described the automatic analysis technique of images of oral cancer tissue
areas stained with immunoglobulin P53. Tissue images are segmented by the entropy threshold, and
cluster cells are found by the watershed method applied selectively. For this purpose, the color indices
of each nucleus and the method of reference vectors are used. This is followed by a classification
assessment.
Article [8] presents a method of segmentation and classification of medical images based on a
hierarchical transformation and a cascade model of nonlinear mappings. The authors offered to use the
proposed method for classifying cell nuclei.
Mostly, all cancer cells are studied by diagnosticians, who are undoubtedly highly qualified
specialists, but such an assessment is a subjective one. An attempt was made in paper [9] to solve the
problem of subjectivity of assessment on the example of immunohistochemical images based on the
biomarker KI-67. The result is an integrated, universal method of automatic cancer diagnosis based on
immunohistochemical images.
The basis for assessing breast cancer is the tumor's response to a particular type of protein. An
effective algorithm for automatic detection of carcinoma cells was presented in article [10]. This formed
the basis of automatic segmentation and classification of cancer using biomarkers.
The combination of histological and immunohistochemical images allows one to make an accurate
diagnosis. Immunohistochemical images based on biomarkers were classified using neural networks
DCNN and VGG. New machine learning strategies were applied to train these networks in [11].
The methods of automatic detection of cancer cells are analyzed in paper [12]. The developed
algorithm selected sets of images that contain ganglion cells. These images were then examined and
evaluated by five pathologists.
The authors of paper [13] developed accurate diagnosis methods of lymphoma using machine
learning methods.
Semi-automatic histopathological analysis of adenocarcinoma tumor cells is presented in article
[14].
Article [15] presented the developed U-net convolutional neural network for segmentation of nuclei
on a public data set. A recurrent residual U-network (R2U-Net) was proposed, which demonstrated
high performance on different sets of medical images (retinal blood vessels, skin cancer, and lung
segmentation) in segmentation tasks.
Scientists developed and trained U-net to segment immunohistochemical images previously
coloured. This network was then applied to the raw slides to create templates [16].
The proposed approach is based on CNN and applies for pixel-by-pixel segmentation of information
areas. This became possible with the parallel use of GPU processors [17].
The results presented in the article can be beneficial for clinical use. The authors tested the
effectiveness of four methods based on deep learning. Testing was performed on different areas of the
slide image: normal tissues, with clusters of immune cells and artifacts [18].
F. Mahmood, R. Chen, D. Borders and others proposed to supplement the existing data sets of
medical images using CGAN. Competitive U-Net with spectral normalization was then used to increase
learning stability. This paired network not only generates new images, but also looks for the optimal
loss function.
Paper [19] presents machine learning methods for effective segmentation, classification and
quantification of breast cancer images. It is shown that the presented algorithm is better than the
superpixel classification, which is based on CNN.
Paper [20] compares the efficiency of segmentation of several architectures (U-Net, Mask R-CNN,
Cellpose, U-Net ResNet) and traditional segmentation algorithms on the example of fluorescent images
of cell nuclei.
A new method of segmentation was developed in paper [21], based on a deep learning network. It
is aimed at accurate selection of the stained area of the nuclei of breast tissue images. Morphological
post-processing of segmented micro-objects splits overlapping nuclei. To improve the results of a
separate classifier, an ensemble method is used, which integrates the solutions of three models of
machine learning for the final assessment of cancer.
The researchers developed a tandem of an artificial neural network and U-Net for recognising and
generating interference. In article, the developed network achieves better results compared to the
semantic segmentation of deep neural networks.
The authors of this paper analyzed segmentation methods on different types of neural networks. For
conducting research, deep convolutional networks - U-Net, Mask R-CNN and the developed network
(GB U-Net) are compared [22].
Thus, based on the analysis of literature review, the following approaches to the segmentation of
biomedical images are identified:
1. Traditional methods of segmentation.
2. Application of CNN and deep learning.
3. The use of U-Net type networks.
3. Problem statement
U-net technology is considered to be a modern approach to image segmentation in medicine according
to the analyzed literature sources. The segmentation phase is very important, as it makes it possible to
prepare the image for processing at a high level of computer vision, in particular classification and
diagnosis. The objectives of this work are as follows.
1. Analyze immunohistochemical images of breast cancer.
2. Develop a modification of the neural network architecture of U-net type.
3. Compare different architectures of U-net type networks for segmentation of
immunohistochemical images.
4. Analyze computer experiments.
4. Analysis of immunohistochemical images of breast cancer.
Immunohistochemical examenations clarify the diagnosis, which is made on the basis of histological
studies. The specified diagnosis is made with use of biomarkers. There is a microscopic description of
immunohistochemical images of breast cancer [23].
A tumor was found in the breast tissue, the morphological structure of which corresponds to
moderately differentiated invasive ductal carcinoma (G2) (Figure 1).
Figure 1: G2 invasive ductal carcinoma
Immunohistochemical images were obtained on the basis of immunohistochemical studies.
Estrogen receptor α (DAKO, clone EP1) - positive reaction in 59% of tumor cells (PS = 4),
moderate intensity (IS = 2). TS = 4 + 2 = 6 - positive result (Figure 2).
Figure 2: Reaction to the Estrogen receptors
Progesterone receptor (DAKO, clone PgR 636) - specific color is not deteсted. TS = 0 - negative
result (Figure 3).
Figure 3: Reaction to the Progesterone receptors
Oncoprotein c-erbB-2 / neu (HER-2 / neu) (DAKO, polyclonal) - specific color is not detected, 0
points - a negative result (Figure 4).
Figure 4: Reaction to the biomarker HER-2 / neu
Ki-67 antigen (DAKO, clone MIB-1) -positive reaction of significant intensity in 44% of tumor cells
(Figure 5).
Figure 5: Reaction to the biomarker Ki-67
Moderately differentiated (G2) invasive ductal carcinoma of the breast (code ICD-O code :
8500/3).According to the results of immunohistochemical staining of luminal type B tumor, HER-2 /
neu negative:
• estrogen (ER +) - sensitive (59% of cells ++);
• progesterone (PR-) - negative;
• HER-2 / neu - negative (0 points);
• 44% of cells are positive (+++) for the marker of proliferative activity of Ki-67.
5. Generalized approach to automatic image segmentation.
In [24] the authors present a generalized approach to diagnosis based on immunohistochemical
images. One of the important stages of this approach is the segmentation stage for determining the
relative area and average brightness of cell nuclei.
As it is known, U-net is a convolutional neural network, which was designed mainly for the
segmentation of biomedical images. This architecture is based on a sequence of convolutional layers.
The architecture of the U-network is characterized by certain features, in particular in the transmission
of information in the downlink and uplink. Unlike other topologies of convolutional networks, this
topology does not make use of fully connected layers, but only convolutional ones.
The basic architecture of the U-net network is shown in Figure 6. The structure of the U-net is
described in more detail in [25].
Figure 6: Basic structure of U-net
This architecture consists of two main parts: narrowing (left) and expansion (right). The narrowing
path is a typical convolutional neural network architecture and consists of a convolution operation and
a RELu function. This reduces the dimensionality of the image. Each step in the expansion phase
consists of operations that increase the discretization of the property map. The neural network is trained
by the method of stochastic gradient descent, based on input immunohistochemical images and
segmentation maps.
In the process of U-net learning it is necessary to determine the degree of similarity between the
generated segmented image and the segmented person. To do this, the Dice coefficient is used.
A generalized approach to segmentation of immunohistochemical images is shown in Figure 7.
Saved
Neural
model of
network
neural
training
Grayscaling network
Input images
Segmentatio
n using
trained
Test image networks
Result
Figure 7: Generalized approach to immunohistochemical segmentation
Learning process includes the following stages:
1. Download images into memory for training sampling.
2. Convert an image to a grayscale image.
3. Adjust the structure and parameters of the neural network.
4. The learning process.
5. Save learning outcomes in a separate file.
The testing phase is as follows:
1. Download the test sample.
2. Download a file with a trained network.
3. The segmentation process.
6. U-net architectures
Examples of U-nets used for segmentation of immunohistochemical images are shown in Figure 8.
In this case, examples of the encoder architecture of the basic U-net neural network (a) and the modified
neural network architecture (b) are given.
Conv 1024X1024 Conv 1024X1024
Conv 512X512 Conv 512X512
Conv 256X256 Conv 256X256
Conv 128X128 Conv 128X128
Conv 64X64 Conv 64X64
Conv 32X32
a) Type A b)Type B
Figure 8: U-net encoder architectures
Unlike convolutional neural network for classification of images, the U-net type network does not
have enough flexibility to modify the architecture. The U-net type network consists of descending and
ascending parts that are interconnected. The modified architecture, which is shown in Figure 3 (b), has
an additional convolution layer of 32 x 32 pixels. This modification was made to increase the accuracy
of segmentation of immunohistochemical images by complicating the neural network architecture.
7. Computer experiments
The database of images was used for computer experiments [26].
ROC curves are constructed to assess the quality of segmentation. Quantitative interpretation of
ROC is done due to AUC indicator — the area bounded by the ROC curve and the axis of the share
of false positive classifications. To determine the accuracy of the classification, the pixel value (black /
white) on the resulting image and the corresponding value on the image processed by the expert are
used.
Table 1 shows the results of the segmentation accuracy assessment for the U-net type network
architecture (Type A).
Table 1
Comparative analysis of segmentation accuracy assessment of convolutional neural network
architecture (Type A)
A number of A number ROC AUC,%
iterations of epochs
200 1 68
300 1 65
100 2 61
300 2 49
Table 2 shows the results of the segmentation accuracy assessment for the U-net type network
architecture (Type B).
Table 2
Comparative analysis of segmentation accuracy assessment of convolutional neural network
architecture (Type B)
A number of A number ROC AUC,%
iterations of epochs
300 1 79
200 2 85
These U-net network architectures are implemented on the basis of Python programming language,
Tensorflow and Keras libraries using the Linux Mint operating system.
The image dataset consists of 150 immunohistochemical images, divided in the proportion of 70%
(training sample) / 30% (test sample).
The comparison shows that the best result was obtained using the architecture of a neural network
of type B with the following parameters: the number of iterations – 200, the number of epochs – 2.
8. Conclusions
1. The analysis of relevant work is carried out. It is shown that automatic segmentation is a relevant
task. Automatic segmentation algorithms are analyzed; their advantages and disadvantages are
highlighted. The necessity of using convolutional neural networks of U-net type is substantiated.
2. The histological image of breast cancer is analyzed and the diagnosis is specified on the basis of
the analysis of immunohistochemical images.
3. The typical U-net architecture is analyzed and its modification is developed. U-net architectures
are compared on the basis of immunohistochemical images.
4. Based on computer experiments, it was found that the best result was obtained using the
architecture of the neural network of type B with the following parameters: the number of iterations –
200, the number of epochs – 2 and the accuracy of segmentation is 85%.
The results of automatic segmentation are used for preliminary diagnosis of cancer.
9. References
[1] R. Wootton, J. Craig, V. Patterson, Introduction to Telemedicine, 2nd ed., CRC Press, 2017.
[2] O. Pitsun, O. Berezsky, L. Dubchak, K. Berezka, T. Dolynyuk, B. Derish, Cytological Images
Clustering of Breast Pathologies, in: Proceedings of the IEEE International Conference
«Computer Science and Information Technologies» CSIT’2020, Zbarazh-Lviv. Ukraine, 23-26
September 2020, Vol. 1, pp. 62-65. doi: 10.1109/CSIT49958.2020.9321867.
[3] O. Berezsky, O. Pitsun, T. Dolynyuk, L. Dubchak, N. Savka, G. Melnyk, V. Teslyuk, Cytological
Image Classification Using Data Reduction, in: Proceedings of the II International Workshop
Informatics & Data-Driven Medicine (IDDM 2019), Lviv, Ukraine, 11-13 November 2019. ceur-
ws.org/Vol-2488/paper2.pdf
[4] Ivan Izonin, Roman Tkachenko, Ivanna Dronyuk, Pavlo Tkachenko, Michal Gregus, Mariia
Rashkevych. Predictive modeling based on small data in clinical medicine: RBF-based additive
input-doubling method[J]. Mathematical Biosciences and Engineering, 2021, 18(3): 2599-2613.
doi: 10.3934/mbe.2021132
[5] F. A. Dzulkifli, M. Y. Mashor, H. Jaafar, Colour thresholding-based automatic Ki67 counting
procedure for immunohistochemical staining in meningioma, International Journal of
Computational Vision and Robotics. 11(3) (2021) 279–298. doi: 10.1504/IJCVR.2021.115160.
[6] J. Shu, J. Liu, Y. Zhang, H. Fu, M. Ilyas, G. Faraci, ... & G. Qiu, Marker controlled superpixel
nuclei segmentation and automatic counting on immunohistochemistry staining images,
Bioinformatics. 36(10) (2020) 3225–3233. doi: 10.1093/bioinformatics/btaa107
[7] K. S. Hameed, K. S. Abubacker, A. Banumathi, & G. Ulaganathan, Immunohistochemical analysis
of oral cancer tissue images using support vector machine, Measurement. 173 (2021) 108476.
https://doi.org/10.1016/j.measurement.2020.108476
[8] Y. Feng, L. Zhang, Z. Yi, Breast cancer cell nuclei classification in histopathology images using
deep neural networks, Int J Comput Assist Radiol Surg. 13(2) (2018 Feb) 179-191. doi:
10.1007/s11548-017-1663-9. Epub 2017 Aug 31. PMID: 28861708.
[9] S. Razavi et al., An Automated and Accurate Methodology to Assess Ki-67 Labeling Index of
Immunohistochemical Staining Images of Breast Cancer Tissues, in: Proceedings of the 25th
International Conference on Systems, Signals and Image Processing (IWSSIP), 2018, pp. 1–5. doi:
10.1109/IWSSIP.2018.8439184.
[10] J. Oscanoa, F. Doimi, R. Dyer, J. Araujo, J. Pinto and B. Castaneda, Automated segmentation and
classification of cell nuclei in immunohistochemical breast cancer images with estrogen receptor
marker, in: Proceedings of the 38th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society (EMBC), 2016, pp. 2399–2402. doi:
10.1109/EMBC.2016.7591213.
[11] P. Pradhan, K. Köhler, S. Guo, O. Rosin, J. Popp, A. Niendorf, & T. Bocklitz, Data Fusion of
Histological and Immunohistochemical Image Data for Breast Cancer Diagnostics using Transfer
Learning, in: Proceedings of the International Conference on Pattern Recognition Applications and
Methods, V. 1: ICPRAM, ISBN 978-989-758-486-2, 2021, February, p. 495–506. doi:
10.5220/0010225504950506.
[12] A. Greenberg, A. Aizic, A. Zubkov, S. Borsekofsky, R. R. Hagege, & D. Hershkovitz, Automatic
ganglion cell detection for improving the efficiency and accuracy of hirschprung disease diagnosis,
Scientific reports. 11(1) (2021) 1–9. doi: 10.1038/s41598-021-82869-y
[13] C. Syrykh, A. Abreu, N. Amara, A. Siegfried, V. Maisongrosse, F. X. Frenois, ... & P. Brousset,
Accurate diagnosis of lymphoma on whole-slide histopathology images using deep learning, NPJ
digital medicine. 3(1) (2020) 1–8. doi: 10.1038/s41746-020-0272-0
[14] D. Kazdal, E. Rempel, C. Oliveira, M. Allgäuer, A. Harms, K. Singer, M. Kriegsmann,
Conventional and semi-automatic histopathological analysis of tumor cell content for multigene
sequencing of lung adenocarcinoma, Translational Lung Cancer Research. 10(4) (2021) 1666. doi:
10.21037/tlcr-20-1168
[15] M. Z. Alom, C. Yakopcic, T. M. Taha and V. K. Asari, Nuclei Segmentation with Recurrent
Residual Convolutional Neural Networks based U-Net (R2U-Net), in: Proceedings of the IEEE
National Aerospace and Electronics Conference, 2018, pp. 228–233. doi:
10.1109/NAECON.2018.8556686.
[16] W. Bulten, P. Bándi, J. Hoven, R. van de Loo, J. Lotz, N. Weiss, ... & G. Litjens, Epithelium
segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry
as reference standard, Scientific report.s 9(1) (2019) 1–10. doi: 10.1038/s41598-018-37257-4
[17] Z. Swiderska-Chadaj, T. Markiewicz, J. Gallego, G. Bueno, B. Grala, & M. Lorent, Deep learning
for damaged tissue detection and segmentation in Ki-67 brain tumor specimens based on the U-
net model, Bulletin of the Polish Academy of Sciences: Technical Sciences. (2018) 849–856. doi:
10.24425/bpas.2018.125932.
[18] Z. Swiderska-Chadaj, H. Pinckaers, M. van Rijthoven, M. Balkenhol, M. Melnikova, O. Geessink,
... & F. Ciompi, Learning to detect lymphocytes in immunohistochemistry with deep learning.
Medical image analysis. 58, 101547 (2019). doi: 10.1016/j.media.2019.101547
[19] F. D. Khameneh, S. Razavi, & M. Kamasak, Automated segmentation of cell membranes to
evaluate HER2 status in whole slide images using a modified deep learning network, Computers
in biology and medicine. 110 (2019) 164–174. doi: 10.1016/j.compbiomed.2019.05.020
[20] F. Kromp et al., Evaluation of Deep Learning Architectures for Complex Immunofluorescence
Nuclear Image Segmentation, in: Proceedings of the IEEE Transactions on Medical Imaging. 40(7)
(July 2021) 1934–1949. doi: 10.1109/TMI.2021.3069558.
[21] L. B. Mahanta, E. Hussain, N. Das, L. Kakoti, & M. Chowdhury, IHC-Net: A fully convolutional
neural network for automated nuclear segmentation and ensemble classification for Allred scoring
in breast pathology, Applied Soft Computing. 103, 107136. (2021).
doi:10.1016/j.asoc.2021.107136
[22] A. Lagree, M. Mohebpour, N. Meti, K. Saednia, F. I. Lu, E. Slodkowska, W. T. Tran, A review
and comparison of breast tumor cell nuclei segmentation performances using deep convolutional
neural networks, Scientific Reports. 11(1) (2021) 1–11. doi: 10.1038/s41598-021-87496-1.
[23] C. R. Taylor, L. Rudbeck, Immunohistochemical Staining Methods, Dako, 2013, 208 p.
[24] O. Berezsky, O. Pitsun, T. Datsko, B. Derysh, I. Tsmots, V. Tesluk, Specified diagnosis of breast
cancer on the basis of immunogistochemical images analysis, in: Proceedings of the III
International Conference on Informatics & Data-Driven Medicine (IDDM-2020), Lviv, Ukraine,
19-21 November 2020, pp. 129-135. ceur-ws.org/Vol-2753/short5.pdf
[25] Yu. Weng, NAS-Unet: Neural Architecture Search for Medical Image Segmentation, Special
section on advanced optical imaging for extreme environment, IEEE Access, 2019, Vol.7, pp.
44247-44257. doi: 10.1109/ACCESS.2019.2908991.
[26] Certificate for copyright to a work №75359 “Database of digital histological and cytological
images of precancerous and cancerous states of the breast «BPCI2100».” Berezsky, O., Melnyk,
G., Verbovyi S., Pitsun, O., Nykolyuk, V., Datsko, T. Registration date 14.12.2017 State Enterprise
"Ukrainian Intellectual Property Institute" (Ukrpatent) https://ukrpatent.org.