=Paper=
{{Paper
|id=Vol-2740/20200063
|storemode=property
|title=Tumor Nuclei Detection in Histopathology Images Using R-CNN
|pdfUrl=https://ceur-ws.org/Vol-2740/20200063.pdf
|volume=Vol-2740
|authors=Daria Hlavcheva,Vladyslav Yaloveha,Andrii Podorozhniak,Heorhii Kuchuk
|dblpUrl=https://dblp.org/rec/conf/icteri/HlavchevaYPK20
}}
==Tumor Nuclei Detection in Histopathology Images Using R-CNN==
https://ceur-ws.org/Vol-2740/20200063.pdf
Tumor Nuclei Detection in Histopathology Images Using
R – CNN
Daria Hlavcheva1[0000-0001-6990-6845], Vladyslav Yaloveha1[0000-0001-7109-9405],
Andrii Podorozhniak1[0000-0002-6688-8407], Heorhii Kuchuk1[0000-0002-2862-438Х]
1 National Technical University “Kharkiv Polytechnic Institute”, Kharkiv, Ukraine
dariahlavcheva@gmail.com, vladyslavyaloveha@gmail.com,
andriipodorozhniak@gmail.com, kuchuk56@ukr.net
Abstract. Breast cancer is the most common in the world and its rates could be
increased from 2 million in 2018 to 3 million in 2040, and the death rate from
600 thousand to nearly 1 million per year. Histopathological analysis is used for
diagnosis of almost all cancer types. Nowadays histopathological tissue analysis
and evaluating the microscopic appearance of a biopsied tissue sample are pro-
vided by a pathologist. The paper is devoted to the problem of histopathological
analysis automatization using a region-based convolutional neural network (R-
CNN). The purpose of the research is to automatizate the tumor nuclei detection
in the histopathological images, because detection can be used as qualitative
and quantitative analysis. In the research breast cancer histopathological anno-
tation and diagnosis dataset is used (BreCaHAD). The classification accuracy
for SVM classifier, which uses features, extracted by CNN, is 0.96. The object
detection heatmap was built. It is obtained that the average precision for tumor
nuclei detection is 0.338. The theory of deep learning neural networks and
mathematical statistics methods are used in the research.
Keywords: object detection, histopathological analysis, region-based convolu-
tional neural network, deep learning.
1 Introducton
Cancer rates continue to increase globally, undergoing tremendous physical, emotion-
al and financial strains on individuals, families, communities and the health care sys-
tem.
Cancer is the second leading cause of death worldwide, accounting for 9.6 million
deaths in 2018, according to the World Health Organization report. In total, approxi-
mately 1 in 6 deaths occurred by cancer [1]. Breast cancer is the most common in the
world, accounting for 11.6% of cases and 6.6% of deaths [2].
In Ukraine cancer is the second cause of death in 2018 [3]. In 2018 there were 170
thousand cancer cases and almost 100 thousand deaths. As in the world, breast cancer
is the most common in Ukraine – 20 thousand cases in 2018 [4].
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The Cancer Tomorrow World Health Organization estimates breast cancer rates
will increase from 2 million in 2018 to 3 million in 2040, and the death rate from 600
thousand to nearly 1 million per year [5].
In particular, much in cancer diagnosis and treatment depends on proper analysis
of medical images.
Nowadays the deep learning approaches is widely used for image analyzing includ-
ing medical images for cancer diagnosis [6, 7]. Convolutional neural networks (CNN)
is a type of deep learning neural networks, which can be used for processing different
types of medical images:
─ detection of mitosis in histopathological images in the diagnosis of breast cancer
using CNN and Data Augmentation obtained the value of metric F1 = 0.480 on the
MITOSATYPIA-14 dataset test data [8];
─ determination of lung cancer incidence by computer tomography imaging from the
JSRT dataset with an AUC 0.722 accuracy [9];
─ determination of brain cancer incidence by magnetic resonance imaging with an
accuracy of classification of 0.986 on the validation data [10];
─ classification of medical images for the presence of skin lesions by the integrated
use of AlexNet, VGG-16, ResNet-18 CNNs on the ISIC dataset and the resulting
AUC accuracy of 0.907 [11];
─ diagnosis of breast cancer from thermal images from the DMR-IR dataset using
SeResNet18 with a classification accuracy of 0.90 [12];
─ classification of mammograms using CNN with the resulting classification accura-
cy for VGG16 - 0.84 (25 epochs) and Resnet50 - 0.97 (198 epochs) [13].
Human histology is a fundamental medical and biological science that studies the
microscopic structure and vital functions of the tissues that make up the body, ie the
tissue level of the organization of living things. Collection of material for histological
examination is performed by biopsy - the removal of a fragment of the studied organ
(biopsy) from a living organism for in vivo diagnosis [14].
To visualize the various components of the tissue under a microscope, the sections
are stained with one or more substances. The purpose of staining is to reveal the cellu-
lar components; dyes provide contrast. Hematoxylin-eosin (H&E) staining has been
used by pathologists for over a hundred years. Hematoxylin stains the cells of the
nucleus blue, while Eosin stains the cytoplasm and connective tissue pink [15].
Imaging analysis of histopathological specimens is used to obtain the definitive di-
agnosis of most cancers. The current practice of histopathology has several limita-
tions: it takes a long time, and may be different in the analysis by different
pathologists. This is why using computer-aided methods for histopathological analy-
sis can help with these issues.
The main tasks for automatization in histopathological analysis are [16]:
─ mitosis detection: the number of mitoses correlates strongly with the aggressive-
ness of the cancer, as they cause faster cell division;
─ image classification: defining a set of characteristics for a particular type of fabric
by analyzing basic tissue primitives;
─ localization or segmentation of a structure: localization and delineation around
specific tissue structures such as cell nuclei or glands.
In [17], the following datasets were used to train the neural network for detecting
mitosis: MITOS-ATYPIA-14, ICPR-2012, AMIDA-13. In addition to the features
that are directly allocated by the network, the researchers have designed their own
features that were given to the CNN fully-connected layers. The accuracy on the test
images was 0.92.
An example of histopathological images segmentation in the diagnosis of breast
cancer is given in [18]. The researchers used images from the Warwick-QU dataset,
which were preprocessed to highlight angles. The LinkNet neural network was used
for segmentation, with additional error metrics added to increase the resulting accura-
cy. The resulting metric F1 was 0.912.
In [19], classification, segmentation, and detection were performed on histopatho-
logical images. The classification dataset consisted of 20,000 images, and the classifi-
cation was performed in four classes, resulted AUC metric is 0.961. Segmentation
was conducted on the basis of the 2018 Data Science Bowl grand challenges dataset
and accuracy was obtained on a test data is 0.922. In the detecting of all types of his-
topathological objects using SC-CNN, the F1 metric was 0.83.
2 R-CNN
The detection task involves locating the objects and determining their belonging to a
particular class. So, the task of the system is to formulate the basic patterns that char-
acterize a particular class and localize their objects in the image.
In the research R-CNN has been used for detection of tumor nuclei in histopatho-
logical images. Because the cells in the histopathological images have fixed size, a
detection algorithm was to easily focus on the selection of small regions for detection.
System for object detection based on R-CNN consists of [20]:
1. Region proposal algorithm (can be implemented using various algorithms).
2. Deep learning neural network to extract features from the input region.
3. Classifier (Support Vector Machine – SVM), analyze features provided by the deep
learning neural network.
Convolutional Neural Networks. Nowadays CNN demonstrate good results at seg-
mentation, classification, detection and related tasks [21].
Typical CNN structure consists of: pairs of Convolutional and MaxPooling layers
and several fully-connected layers [22, 23].
Units in a convolutional layer are organized in feature maps, each unit is connected
to local patches in the previous layer feature maps through a set of weights The result
of this local weighted sum is then passed through a non-linearity such as a ReLU:
0, x 0;
f ( x) (1)
x, x 0.
Such an approach determine objects features in local areas because it difficult to de-
termine the relationship between objects at larger distances (relative to image sizes).
At longer distances, pixels have less correlation with each other than at short distanc-
es [24].
How a feature map performs a convolution operation depends on several parame-
ters [25]:
1. Kernel size. Typically, feature cards are square and their size matches the size of
the image and determines which features can be captured by the filters.
2. Padding. This is a field of zero values that are placed around the image so the out-
put of the current convolution layer does not become smaller after performing the
convolution operation
3. Stride. Indicates how many pixels the feature card will be displaced the next time it
is applied to the image. In some architectures (GoogLeNet), this approach is used
instead of pooling to reduce the original dimension of the layer.
According to these parameters output size of convolutional layer can be calculated by
equation:
n 2p f
1, (2)
s
where n – input size;
p – padding;
f – kernel size;
s – stride.
Pooling layers is used for decreasing the number of parameters before the next lay-
er [26]. In the research MaxPooling has been used:
a j max ainn u n, n ,
N N
(3)
where n n – size of input feature map;
u n, n – input feature map;
N N – pooling size.
The common problem of deep learning neural networks is overfitting. The most
common approaches to avoid overfitting is dropout [27] and data augmentation [28].
The key idea of dropout is to randomly drop units (along with their connections) from
the neural network during training. This prevents units from co-adapting too much.
Data augmentation is done by simple image transformations such as vertical and hori-
zontal flip, regions selection.
Support Vector Machine. The SVM was developed throughout the framework of the
statistical approach theory. This method was successfully used for recognizing ap-
pearance, processing of biological data, medical diagnostics [29].
Objective of the algorithm is to find optimal hyperplane two separate classes in
given dataset. Objects are classified by SVM according to formula:
a ( x ) sign i i yi xi , x w0 , (4)
where a( x) – linear classifier;
xi – support vector;
i – support vectors own number;
yi – training data value on support vector;
xi , x – vector’s scalar product;
w0 – normalizing parallel shift of a hyperplane.
Non-max suppression. NMS algorithm is an important part of the object detection. It
is used for decreasing the number of resulted bounding boxes if they overlap each
other.
In NMS all objects found are sorted according to their probability detection algo-
rithm. The found object M with maximum probability is selected, and all other objects
with a threshold (𝑁𝑡 – intersection over union) overlap with M are deleted. The algo-
rithm is recursively repeated for all detected objects. If the overlapping of objects is
less than the threshold, then both objects remain [30].
3 Data Preparation
In the research BreCaHAD (breast cancer histopathological annotation and diagnosis)
dataset has been used [31]. The images were obtained from archived surgical patholo-
gy example cases which have been archived for teaching purposes. Nottingham Grad-
ing System is an international grading system for breast cancer recommended by the
World Health Organization, where the assessment of three morphological features
(tubule formation, nuclear pleomorphism, and mitotic count) is used for scoring to
decide on the final grade of the cancer case.
The task associated with this dataset is to automatically classify histological struc-
tures in these hematoxylin and eosin (H&E) stained images into six classes, namely
mitosis, apoptosis, tumor nuclei, non-tumor nuclei, tubule, and non-tubule.
According to the assigned task in our research only tumor-nuclei annotations have
been used.
Data was divided into two parts: dataset for classification training, validation and
testing and dataset for detection testing.
For classification CNN and SVM have been used. According to tumor nuclei size
input image size for CNN is 48×48 pixels. Images from primary dataset were sliced
and divided into two classes: “tumor” and “healthy”. “Tumor” class images have
tumor nuclei allocated in the center of the images. “Healthy” images don’t contain
any part of tumor nuclei. Total images count is 40,000 (20,000 images – “tumor”,
20,000 images – “healthy”). Examples of the classification images are presented in
the Fig. 1.
Classification dataset was divided into 3 parts:
─ training data (70% – 28,000 images);
─ validation data (15% – 6,000 images);
─ testing data (15% – 6,000 images).
For detection testing quarters of 8 different images from primary dataset were used
(resulting size – 680×512) for better demonstration of results. On these images detec-
tion metrics were calculated.
a) b)
Fig. 1. Examples of the classification images (a – “tumor” images, b – “healthy” images)
4 Results and Discussions
4.1 Detection algorithm
The part of the R-CNN algorithm is region proposals. Every proposed regions needs
to be resized to the CNN input size. Because the task of the research is the detection
of cells, which, by their biological properties, cannot have a large size difference, a
fixed-size floating-window algorithm was choosed for region selection. The general
scheme of the R-CNN adapted to the task of tumor nuclei detection in histopathologi-
cal images algorithm is shown in the Fig. 2.
A 512×680 pixels image is processed by the detection algorithm using a floating-
window approach (48×48 window size according to the neural network input size)
with fixed step 16. The selected regions are fed to the neural network input, which
highlights 128 features in each region. The highlighted features are submitted to the
classifier in form of SVM, according to the classifier output region belongs to a par-
ticular class. When all regions are analyzed, those which belonged to the “tumor”
class are processed by the NMS algorithm with an overlap value of 𝑁𝑡 = 0.3. The
remaining regions are the result of the detection algorithm.
Fig. 2. R-CNN algorithm for nuclei detection in histopathological images
4.2 Classification training
R-CNN for classifying regions use CNN and SVM, and training is carried out in 2
stages:
1. CNN training using as input data image region;
2. SVM training using as input data features rom the last fully-connected CNN layer.
The input image size for CNN is 48×48 pixels. The convolutional part of the net-
work is implemented by 3 convolutional layers and 2 MaxPooling layers. Then the
information is processed by the Flatten layer to prepare data before fully-connected
part of the CNN, implemented in the form of perceptron. The Dropout layer has been
used for preventing of overfitting. The total amount of parameters is 149,729.
Resulted CNN architecture is presented in the Fig.3.
Fig. 3. CNN architecture in the research
For developing and implementing of the CNN Python 3.6 has been used with addi-
tional modules:
─ Numpy (v. 1.16.4) is a module for working with multidimensional arrays;
─ Tensorflow (v. 1.12.0) is a backend for neural network developing;
─ Keras (v.2.2.4) API is a set of functions for easier implementing of neural net-
works.
The CNN was trained using Coogle Colaboratory IDE [32]. This environment
gives an opportunity to use GPU Tesla K80 for calculations. The CNN was trained
during 50 epochs with batch size 16.
Accuracy and loss values for every epoch on training and validation data are pre-
sented in the Fig.4, 5.
The maximum accuracy and minimum loss values on training and validation data
are presented at the Table 1.
Table 1. The highest accuracy and lowest loss value for training and validation data
Maximum Accuracy / Epoch Minimal Loss / Epoch
Training data 0.969 / 50 0.091 / 50
Validation data 0.961 / 16 0.114 / 14
Fig. 4. Accuracy and loss values on training data
Fig. 5. Accuracy and loss values on validation data
After the CNN training, it is necessary to train the SVM. The inputs for the SVM
are the features highlighted on the last fully-connected CNN layer. In the architecture
shown in the Fig.3, this layer contains 128 parameters, these parameters are input data
for the SVM. The output of the SVM method gives 2 numbers: the probability of
belonging to each of the classes (“tumor”, “healthy”).
For developing and implementing of the CNN Python 3.6 and scikit-learn module
have been used.
Resulted accuracy on test data for 2 stages of R-CNN training after running of
SVM hyper parameters [33] is shown at the Table 2.
Table 2. Resulted accuracy on test data
Training stage Resulted accuracy
CNN 0.957
SVM 0.960
4.3 Detection results and discussions
Using R-CNN algorithm images from detection dataset were analyzed and exam-
ples are shown in the Fig.6. For every image heatmap was built.
Fig. 6. Examples of heatmaps and tumor nuclei detection (white squares – ground truth objects,
black squares – detected objects)
For detection accuracy metrics Precision×Recall (PR) curve has been built (Fig.7). To
find the area under PR curve, all points of the curve are interpolated according to the
formulas 5, 6. In the research area under PR (Average Precision) is 0.338.
AP r 0 rn 1 rn pinterp (rn 1 ),
1
(5)
pinterp (rn1 ) max p(r ) , (6)
r ;r rn 1
where rn – recall value in the current point;
rn1 – recall value in the next point;
p(r ) – precision value in the current point.
Fig. 7. PR curve for tumor object detection in histopathology images
The tasks of detecting nuclei of all cell types were performed in [34, 35] and obtained
greater accuracy than in the present study. Authors of [34] obtained the value of met-
ric F1 0.77-0.81 depending on the parameters of training and image processing. In
[35] the value of AP 0.82 was obtained for detecting all cell types. Such a difference
in the results can be explained by the fact that the study carried out the detection of
only tumor nuclei, which is more difficult because the system needs to determine the
characteristics of only a certain type of cells.
5 Conclusions
Cancer is one of the leading causes of death in the world. Deep learning methods are
widely used in cancer diagnosis and show good results at medical images analyzing.
Histopathology analysis is used in almost all cancer type diagnosis. Process of such
type of analysis takes a lot of time, so it’s automatization could be a useful tool for
pathologist.
In that paper R-CNN has been used for tumor nuclei detection in the histopatholog-
ical images for quantitative analysis.
BreCaHAD dataset can be effectively used for solving the task of tumor nuclei
detection. It is shown that using of SVM for classifying features, extracted by CNN
from histopathological images with size 48×48, obtained 0.96 accuracy. Detection
system, which consist of: floating window regions proposals, classifier and post-
processing algorithm NMS obtained AP 0.338. Results have been compared with re-
lated works.
In further research obtained results will be compared with other others detection
algorithms and optimize parameters of detection algorithm backend neural network.
It has been proven that developed detection system can be used as additional in-
strument for tumor nuclei detection in histopathology images.
References
1. WHO, Cancer, https://www.who.int/news-room/fact-sheets/detail/cancer, last accessed
2020/02/15.
2. WHO Globocan All Cancers, https://gco.iarc.fr/today/data/factsheets/cancers/39-All-
cancers-fact-sheet.pdf, last accessed 2010/01/15.
3. The State Statistics Service of Ukraine and the Ukrainian Center for Public Data, Causes
of Death in Ukraine. https://socialdata.org.ua/death/, last accessed 2020/02/15.
4. WHO, Global Cancer Observatory, Cancer Statistics: Ukraine,
https://gco.iarc.fr/today/data/factsheets/populations/804-ukraine-fact-sheets.pdf, last ac-
cessed 2020/02/15.
5. Cancer, WHO International Agency for research Cancer Tomorrow.
https://gco.iarc.fr/tomorrow/home, last accessed 2020/02/15.
6. Kourou, Konstantina, et al.: Machine Learning Applications in Cancer Prognosis and Pre-
diction. Computational and structural biotechnology journal 13, 8-17 (2015).
7. Hlavcheva, D., Yaloveha V., Podorozhniak., A.: Application of Convolutional Neural
Network for Histopathological Analysis. Advanced information Systems 4(3), 69-73
(2019).
8. Tellez, Balkenhol, et al.: Whole-Slide Mitosis Detection in H&E Breast Histology Using
Phh3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks. IEEE
transactions on medical imaging 37(9), 2126-36 (2018).
9. Wang, Xiaohua, et al.: Vasculature Surrounding a Nodule: A Novel Lung Cancer Bi-
omarker. Lung Cancer 114, 38-43 (2017).
10. Sawant, Bhandari, et al.: "Brain Cancer Detection from Mri: A Machine Learning Ap-
proach (Tensorflow)." Brain 5 (4), (2018).
11. Mahbod, Schaefer, et al.: Skin Lesion Classification Using Hybrid Deep Neural Networks.
ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Pro-
cessing (ICASSP), (2019).
12. Zuluaga-Gomez, Juan, et al.: A Cnn-Based Methodology for Breast Cancer Diagnosis Us-
ing Thermal Images. arXiv preprint arXiv:1910.13757, (2019).
13. Shen, Li, et al.: Deep Learning to Improve Breast Cancer Early Detection on Screening
Mammography. arXiv preprint arXiv:1708.09427, (2017).
14. Bykov V.: Cytology and General Histology. 254, (2002).
15. Fox, H.: Is H&E morphology coming to an end? Journal of clinical pathology 53(1), 38-
40, (2000).
16. Jimenez-del-Toro, Otálora, et al.: Analysis of Histopathology Images: From Traditional
Machine Learning to Deep Learning. Biomedical Texture Analysis. Elsevier, 281-314
(2017).
17. Monjoy, Chakrabortt, et al.: Efficient Deep Learning Model for Mitosis Detection Using
Breast Histopathology Images. Computerized Medical Imaging and Graphics 64, 29-40
(2018)
18. Rezaei, Safiye, et al. Gland Segmentation in Histopathological Images by Deep Neural
Network. arXiv preprint arXiv:1911.00909, (2019).
19. Alom, Md Zahangir, et al.: Microscopic Nuclei Classification, Segmentation and Detection
with Improved Deep Convolutional Neural Network (Dcnn) Approaches. arXiv preprint
arXiv:1811.03447 (2018).
20. Girshick, Donahue, et al..: Rich Feature Hierarchies for Accurate Object Detection and
Semantic Segmentation. Proceedings of the IEEE conference on computer vision and pat-
tern recognition. (2014).
21. Liu, X., Deng, Z., Yang, Y.: Recent Progress in Semantic Image Segmentation. Artificial
Intelligence Review 52.2, 1089-106, (2019).
22. Kuchuk, Podorozhniak, et al.: Application of Deep Learning in the Processing of the Aero-
space System's Multispectral Images. Handbook of Research on Artificial Intelligence Ap-
plications in the Aviation and Aerospace Industries. IGI Global, 134-147 (2020).
23. Hlavcheva, Yaloveha et al.: Fire Hazard Research of Forest Areas Based on the Use of
Convolutional and Capsule Neural Networks. IEEE 2nd Ukraine Conference on Electrical
and Computer Engineering (UKRCON), (2019).
24. LeCun, Y., Yoshua, B., Hinton, G.: Deep Learning. nature 521.7553, 436-44 (2015).
25. Murphy, J.: An Overview of Convolutional Neural Network Architectures for Deep Learn-
ing. Microway Inc, (2016).
26. Scherer, D., Muller, A., Behnke, S.: Evaluation of Pooling Operations in Convolutional
Architectures for Object Recognition. International conference on artificial neural net-
works, Springer, (2010).
27. Srivastava, Nitish, et al.: Dropout: A Simple Way to Prevent Neural Networks from Over-
fitting. The journal of machine learning research 15(1), 1929-58 (2014).
28. Frühwirth‐Schnatter, S.: Data Augmentation and Dynamic Linear Models. Journal of time
series analysis 15(2), 183-202 (1994).
29. Evgeniou, T., Massimiliano P.: Workshop on Support Vector Machines: Theory and Ap-
plications. Support Vector Machines: Theory and Applications, 1 (2001).
30. Rosenfeld, A., Thurston, M.: Edge and Curve Detection for Visual Scene Analysis. IEEE
Transactions on computers 100(5), 562-69 (1971).
31. Aksac, Alper, et al.: Brecahad: A Dataset for Breast Cancer Histopathological Annotation
and Diagnosis. BMC research notes 12(1), 1-3 (2019).
32. Google Colaboratory, https://colab.research.google.com/, last accessed 2020/02/15.
33. Hlavcheva, D., Yaloveha V., et al.: A Comparison of Classifiers Applied to the Problem of
Biopsy Images Analysis Advanced Information Systems 2(4), 12-16 (2020).
34. Höfener, Henning, et al.: Deep Learning Nuclei Detection: A Simple Approach Can Deliv-
er State-of-the-Art Results. Computerized Medical Imaging and Graphics 70, 43-52
(2018).
35. Yuan, Pengyu, et al.: Phasetime: Deep Learning Approach to Detect Nuclei in Time Lapse
Phase Images. Journal of clinical medicine 8(8), 1159 (2019).