Breast cancer is the most common in the world and its rates could be increased from 2 million in 2018 to 3 million in 2040, and the death rate from 600 thousand to nearly 1 million per year. Histopathological analysis is used for diagnosis of almost all cancer types. Nowadays histopathological tissue analysis and evaluating the microscopic appearance of a biopsied tissue sample are provided by a pathologist. The paper is devoted to the problem of histopathological analysis automatization using a region-based convolutional neural network (RCNN). The purpose of the research is to automatizate the tumor nuclei detection in the histopathological images, because detection can be used as qualitative and quantitative analysis. In the research breast cancer histopathological annotation and diagnosis dataset is used (BreCaHAD). The classification accuracy for SVM classifier, which uses features, extracted by CNN, is 0.96. The object detection heatmap was built. It is obtained that the average precision for tumor nuclei detection is 0.338. The theory of deep learning neural networks and mathematical statistics methods are used in the research.
Cancer rates continue to increase globally, undergoing tremendous physical, emotional and financial strains on individuals, families, communities and the health care system.
Cancer is the second leading cause of death worldwide, accounting for 9.6 million deaths in 2018, according to the World Health Organization report. In total, approximately 1 in 6 deaths occurred by cancer [1]. Breast cancer is the most common in the world, accounting for 11.6% of cases and 6.6% of deaths [2].
In Ukraine cancer is the second cause of death in 2018 [3]. In 2018 there were 170 thousand cancer cases and almost 100 thousand deaths. As in the world, breast cancer is the most common in Ukraine – 20 thousand cases in 2018 [4].
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The Cancer Tomorrow World Health Organization estimates breast cancer rates will increase from 2 million in 2018 to 3 million in 2040, and the death rate from 600 thousand to nearly 1 million per year [5].
In particular, much in cancer diagnosis and treatment depends on proper analysis of medical images.
Nowadays the deep learning approaches is widely used for image analyzing including medical images for cancer diagnosis [6, 7]. Convolutional neural networks (CNN) is a type of deep learning neural networks, which can be used for processing different types of medical images: ─ detection of mitosis in histopathological images in the diagnosis of breast cancer using CNN and Data Augmentation obtained the value of metric F1 = 0.480 on the
MITOSATYPIA-14 dataset test data [8]; ─ determination of lung cancer incidence by computer tomography imaging from the
JSRT dataset with an AUC 0.722 accuracy [9]; ─ determination of brain cancer incidence by magnetic resonance imaging with an
accuracy of classification of 0.986 on the validation data [10]; ─ classification of medical images for the presence of skin lesions by the integrated use of AlexNet, VGG-16, ResNet-18 CNNs on the ISIC dataset and the resulting
AUC accuracy of 0.907 [11]; ─ diagnosis of breast cancer from thermal images from the DMR-IR dataset using
SeResNet18 with a classification accuracy of 0.90 [12]; ─ classification of mammograms using CNN with the resulting classification accuracy for VGG16 - 0.84 (25 epochs) and Resnet50 - 0.97 (198 epochs) [13].
Human histology is a fundamental medical and biological science that studies the microscopic structure and vital functions of the tissues that make up the body, ie the tissue level of the organization of living things. Collection of material for histological examination is performed by biopsy - the removal of a fragment of the studied organ (biopsy) from a living organism for in vivo diagnosis [14].
To visualize the various components of the tissue under a microscope, the sections are stained with one or more substances. The purpose of staining is to reveal the cellular components; dyes provide contrast. Hematoxylin-eosin (H&E) staining has been used by pathologists for over a hundred years. Hematoxylin stains the cells of the nucleus blue, while Eosin stains the cytoplasm and connective tissue pink [15].
Imaging analysis of histopathological specimens is used to obtain the definitive diagnosis of most cancers. The current practice of histopathology has several limitations: it takes a long time, and may be different in the analysis by different pathologists. This is why using computer-aided methods for histopathological analysis can help with these issues.
The main tasks for automatization in histopathological analysis are [16]: ─ mitosis detection: the number of mitoses correlates strongly with the aggressive
ness of the cancer, as they cause faster cell division; ─ image classification: defining a set of characteristics for a particular type of fabric
by analyzing basic tissue primitives; ─ localization or segmentation of a structure: localization and delineation around
specific tissue structures such as cell nuclei or glands.
In [17], the following datasets were used to train the neural network for detecting mitosis: MITOS-ATYPIA-14, ICPR-2012, AMIDA-13. In addition to the features that are directly allocated by the network, the researchers have designed their own features that were given to the CNN fully-connected layers. The accuracy on the test images was 0.92.
An example of histopathological images segmentation in the diagnosis of breast cancer is given in [18]. The researchers used images from the Warwick-QU dataset, which were preprocessed to highlight angles. The LinkNet neural network was used for segmentation, with additional error metrics added to increase the resulting accuracy. The resulting metric F1 was 0.912.
In [19], classification, segmentation, and detection were performed on histopathological images. The classification dataset consisted of 20,000 images, and the classification was performed in four classes, resulted AUC metric is 0.961. Segmentation was conducted on the basis of the 2018 Data Science Bowl grand challenges dataset and accuracy was obtained on a test data is 0.922. In the detecting of all types of histopathological objects using SC-CNN, the F1 metric was 0.83. 2
R-CNN The detection task involves locating the objects and determining their belonging to a particular class. So, the task of the system is to formulate the basic patterns that characterize a particular class and localize their objects in the image.
In the research R-CNN has been used for detection of tumor nuclei in histopathological images. Because the cells in the histopathological images have fixed size, a detection algorithm was to easily focus on the selection of small regions for detection.
System for object detection based on R-CNN consists of [20]: 1. Region proposal algorithm (can be implemented using various algorithms). 2. Deep learning neural network to extract features from the input region. 3. Classifier (Support Vector Machine – SVM), analyze features provided by the deep
learning neural network.
Convolutional Neural Networks. Nowadays CNN demonstrate good results at segmentation, classification, detection and related tasks [21].
Typical CNN structure consists of: pairs of Convolutional and MaxPooling layers and several fully-connected layers [22, 23].
Units in a convolutional layer are organized in feature maps, each unit is connected to local patches in the previous layer feature maps through a set of weights The result of this local weighted sum is then passed through a non-linearity such as a ReLU:
0, x 0; f (x)
x, x 0.
Such an approach determine objects features in local areas because it difficult to determine the relationship between objects at larger distances (relative to image sizes).
At longer distances, pixels have less correlation with each other than at short distances [24].
How a feature map performs a convolution operation depends on several parameters [25]: 1. Kernel size. Typically, feature cards are square and their size matches the size of
the image and determines which features can be captured by the filters. 2. Padding. This is a field of zero values that are placed around the image so the output of the current convolution layer does not become smaller after performing the convolution operation 3. Stride. Indicates how many pixels the feature card will be displaced the next time it is applied to the image. In some architectures (GoogLeNet), this approach is used instead of pooling to reduce the original dimension of the layer.
According to these parameters output size of convolutional layer can be calculated by equation: (1) (2) (3) where n – input size; p – padding; f – kernel size; s – stride.
n 2 p f s
1 , a j max ainnu n, n ,
NN
Pooling layers is used for decreasing the number of parameters before the next layer [26]. In the research MaxPooling has been used: where n n – size of input feature map; u n, n – input feature map; N N – pooling size.
The common problem of deep learning neural networks is overfitting. The most common approaches to avoid overfitting is dropout [27] and data augmentation [28].
The key idea of dropout is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much.
Data augmentation is done by simple image transformations such as vertical and horizontal flip, regions selection.
Support Vector Machine. The SVM was developed throughout the framework of the statistical approach theory. This method was successfully used for recognizing appearance, processing of biological data, medical diagnostics [29].
Objective of the algorithm is to find optimal hyperplane two separate classes in given dataset. Objects are classified by SVM according to formula: a(x) sign i i yi xi , x w0 , (4) where a(x) – linear classifier; xi – support vector; i – support vectors own number; yi – training data value on support vector; xi , x – vector’s scalar product; w0 – normalizing parallel shift of a hyperplane.
Non-max suppression. NMS algorithm is an important part of the object detection. It is used for decreasing the number of resulted bounding boxes if they overlap each other.
In NMS all objects found are sorted according to their probability detection algorithm. The found object M with maximum probability is selected, and all other objects with a threshold ( – intersection over union) overlap with M are deleted. The algorithm is recursively repeated for all detected objects. If the overlapping of objects is less than the threshold, then both objects remain [30]. 3
In the research BreCaHAD (breast cancer histopathological annotation and diagnosis) dataset has been used [31]. The images were obtained from archived surgical pathology example cases which have been archived for teaching purposes. Nottingham Grading System is an international grading system for breast cancer recommended by the World Health Organization, where the assessment of three morphological features (tubule formation, nuclear pleomorphism, and mitotic count) is used for scoring to decide on the final grade of the cancer case.
The task associated with this dataset is to automatically classify histological structures in these hematoxylin and eosin (H&E) stained images into six classes, namely mitosis, apoptosis, tumor nuclei, non-tumor nuclei, tubule, and non-tubule.
According to the assigned task in our research only tumor-nuclei annotations have been used.
Data was divided into two parts: dataset for classification training, validation and testing and dataset for detection testing.
For classification CNN and SVM have been used. According to tumor nuclei size input image size for CNN is 48×48 pixels. Images from primary dataset were sliced and divided into two classes: “tumor” and “healthy”. “Tumor” class images have tumor nuclei allocated in the center of the images. “Healthy” images don’t contain any part of tumor nuclei. Total images count is 40,000 (20,000 images – “tumor”, 20,000 images – “healthy”). Examples of the classification images are presented in the Fig. 1.
Classification dataset was divided into 3 parts: ─ training data (70% – 28,000 images); ─ validation data (15% – 6,000 images); ─ testing data (15% – 6,000 images).
For detection testing quarters of 8 different images from primary dataset were used (resulting size – 680×512) for better demonstration of results. On these images detection metrics were calculated.
a) b)
Detection algorithm The part of the R-CNN algorithm is region proposals. Every proposed regions needs to be resized to the CNN input size. Because the task of the research is the detection of cells, which, by their biological properties, cannot have a large size difference, a fixed-size floating-window algorithm was choosed for region selection. The general scheme of the R-CNN adapted to the task of tumor nuclei detection in histopathological images algorithm is shown in the Fig. 2.
A 512×680 pixels image is processed by the detection algorithm using a floatingwindow approach (48×48 window size according to the neural network input size) with fixed step 16. The selected regions are fed to the neural network input, which highlights 128 features in each region. The highlighted features are submitted to the classifier in form of SVM, according to the classifier output region belongs to a particular class. When all regions are analyzed, those which belonged to the “tumor” class are processed by the NMS algorithm with an overlap value of = 0.3. The remaining regions are the result of the detection algorithm. R-CNN for classifying regions use CNN and SVM, and training is carried out in 2 stages: 1. CNN training using as input data image region; 2. SVM training using as input data features rom the last fully-connected CNN layer.
The input image size for CNN is 48×48 pixels. The convolutional part of the network is implemented by 3 convolutional layers and 2 MaxPooling layers. Then the information is processed by the Flatten layer to prepare data before fully-connected part of the CNN, implemented in the form of perceptron. The Dropout layer has been used for preventing of overfitting. The total amount of parameters is 149,729.
Resulted CNN architecture is presented in the Fig.3.
For developing and implementing of the CNN Python 3.6 has been used with additional modules: ─ Numpy (v. 1.16.4) is a module for working with multidimensional arrays; ─ Tensorflow (v. 1.12.0) is a backend for neural network developing; ─ Keras (v.2.2.4) API is a set of functions for easier implementing of neural networks.
The CNN was trained using Coogle Colaboratory IDE [32]. This environment gives an opportunity to use GPU Tesla K80 for calculations. The CNN was trained during 50 epochs with batch size 16.
Accuracy and loss values for every epoch on training and validation data are presented in the Fig.4, 5.
The maximum accuracy and minimum loss values on training and validation data are presented at the Table 1.
After the CNN training, it is necessary to train the SVM. The inputs for the SVM are the features highlighted on the last fully-connected CNN layer. In the architecture shown in the Fig.3, this layer contains 128 parameters, these parameters are input data for the SVM. The output of the SVM method gives 2 numbers: the probability of belonging to each of the classes (“tumor”, “healthy”).
For developing and implementing of the CNN Python 3.6 and scikit-learn module have been used.
Resulted accuracy on test data for 2 stages of R-CNN training after running of SVM hyper parameters [33] is shown at the Table 2.
Using R-CNN algorithm images from detection dataset were analyzed and examples are shown in the Fig.6. For every image heatmap was built. For detection accuracy metrics Precision×Recall (PR) curve has been built (Fig.7). To find the area under PR curve, all points of the curve are interpolated according to the formulas 5, 6. In the research area under PR (Average Precision) is 0.338.
AP 1r0 rn1 rn pinterp (rn1), pinterp (rn1)
max p(r ) , r ;rrn1 (5) (6) where rn – recall value in the current point; rn1 – recall value in the next point; p(r ) – precision value in the current point. The tasks of detecting nuclei of all cell types were performed in [34, 35] and obtained greater accuracy than in the present study. Authors of [34] obtained the value of metric F1 0.77-0.81 depending on the parameters of training and image processing. In [35] the value of AP 0.82 was obtained for detecting all cell types. Such a difference in the results can be explained by the fact that the study carried out the detection of only tumor nuclei, which is more difficult because the system needs to determine the characteristics of only a certain type of cells. 5
Cancer is one of the leading causes of death in the world. Deep learning methods are widely used in cancer diagnosis and show good results at medical images analyzing. Histopathology analysis is used in almost all cancer type diagnosis. Process of such type of analysis takes a lot of time, so it’s automatization could be a useful tool for pathologist.
In that paper R-CNN has been used for tumor nuclei detection in the histopathological images for quantitative analysis.
BreCaHAD dataset can be effectively used for solving the task of tumor nuclei detection. It is shown that using of SVM for classifying features, extracted by CNN from histopathological images with size 48×48, obtained 0.96 accuracy. Detection system, which consist of: floating window regions proposals, classifier and postprocessing algorithm NMS obtained AP 0.338. Results have been compared with related works.
In further research obtained results will be compared with other others detection algorithms and optimize parameters of detection algorithm backend neural network.
It has been proven that developed detection system can be used as additional instrument for tumor nuclei detection in histopathology images. 14. 15. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35.
Bykov V.: Cytology and General Histology. 254, (2002).
Fox, H.: Is H&E morphology coming to an end? Journal of clinical pathology 53(1), 3840, (2000).
Jimenez-del-Toro, Otálora, et al.: Analysis of Histopathology Images: From Traditional Machine Learning to Deep Learning. Biomedical Texture Analysis. Elsevier, 281-314 (2017).
Monjoy, Chakrabortt, et al.: Efficient Deep Learning Model for Mitosis Detection Using
Breast Histopathology Images. Computerized Medical Imaging and Graphics 64, 29-40
(2018)
Rezaei, Safiye, et al. Gland Segmentation in Histopathological Images by Deep Neural
Network. arXiv preprint arXiv:1911.00909, (
Alom, Md Zahangir, et al.: Microscopic Nuclei Classification, Segmentation and Detection with Improved Deep Convolutional Neural Network (Dcnn) Approaches. arXiv preprint arXiv:1811.03447 (2018).
Girshick, Donahue, et al..: Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition. (2014).
Liu, X., Deng, Z., Yang, Y.: Recent Progress in Semantic Image Segmentation. Artificial
Intelligence Review 52.2, 1089-106, (
Kuchuk, Podorozhniak, et al.: Application of Deep Learning in the Processing of the
Aerospace System's Multispectral Images. Handbook of Research on Artificial Intelligence
Applications in the Aviation and Aerospace Industries. IGI Global, 134-147 (2020).
Hlavcheva, Yaloveha et al.: Fire Hazard Research of Forest Areas Based on the Use of
Convolutional and Capsule Neural Networks. IEEE 2nd Ukraine Conference on Electrical
and Computer Engineering (UKRCON), (
LeCun, Y., Yoshua, B., Hinton, G.: Deep Learning. nature 521.7553, 436-44 (2015). Murphy, J.: An Overview of Convolutional Neural Network Architectures for Deep Learning. Microway Inc, (2016).
Scherer, D., Muller, A., Behnke, S.: Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition. International conference on artificial neural networks, Springer, (2010).
Srivastava, Nitish, et al.: Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The journal of machine learning research 15(1), 1929-58 (2014).
Frühwirth‐Schnatter, S.: Data Augmentation and Dynamic Linear Models. Journal of time series analysis 15(2), 183-202 (1994).
Evgeniou, T., Massimiliano P.: Workshop on Support Vector Machines: Theory and Applications. Support Vector Machines: Theory and Applications, 1 (2001).
Rosenfeld, A., Thurston, M.: Edge and Curve Detection for Visual Scene Analysis. IEEE Transactions on computers 100(5), 562-69 (1971).
Aksac, Alper, et al.: Brecahad: A Dataset for Breast Cancer Histopathological Annotation
and Diagnosis. BMC research notes 12(1), 1-3 (
Google Colaboratory, https://colab.research.google.com/, last accessed 2020/02/15. Hlavcheva, D., Yaloveha V., et al.: A Comparison of Classifiers Applied to the Problem of Biopsy Images Analysis Advanced Information Systems 2(4), 12-16 (2020).
Höfener, Henning, et al.: Deep Learning Nuclei Detection: A Simple Approach Can Deliver State-of-the-Art Results. Computerized Medical Imaging and Graphics 70, 43-52 (2018).
Yuan, Pengyu, et al.: Phasetime: Deep Learning Approach to Detect Nuclei in Time Lapse
Phase Images. Journal of clinical medicine 8(8), 1159 (