Automatic layers recognition of the wall of the stomach and colon on whole slide images is an extremely urgent task in digital pathology as it can be used for automatic determining the depth of invasion of the digestive tract tumors. In this paper we propose a new CNN-based method of automatic tissue type recognition on whole slide histological images. We also describe an efective pipeline of training that uses 2 diferent training datasets. The proposed method of automatic tissue type recognition achieved 0.929 accuracy and 0.903 balanced accuracy on CRC-VAL-HE-7K dataset for 9-types classification and 0.98 accuracy and 0.926 balanced accuracy on the test subset of whole slide images from PATH-DTMSU dataset for 5-types classification. The developed method makes it possible to classify the areas corresponding to the gastric own mucous glands in the lamina propria and also to distinguish the tubular structures of a highly diferentiated gastric adenocarcinoma with normal glands.
Determining the depth of invasion (T) of the digestive tract tumors (stomach and colon tumors) is one of the most important tasks of surgical pathology since the depth of invasion is a reliable highly significant negative prognostic factor. If determining the depth of invasion of the digestive tract tumors at advanced stages is a relatively simple task for a pathologist, then the determination of foci of microinvasion of adenocarcinoma in polyps with low and high grade dysplasia is a rather dificult task, which can be tried to be solved using of deep learning based methods.
The key task for developing these algorithms is learning to recognize the layers of the wall
of the stomach and colon on whole slide images (WSI), namely the lamina propria, muscularis
mucosa, submucosa, own muscle layer, subserous layer, serous membrane and adjacent areas
adipose tissue. Recognition of tumors and layers of the stomach wall in gastric cancer is an
example of one of the most urgent problems in the digital pathology of the digestive tract tumors.
The papers available in this area are mainly devoted to the recognition and classification tasks
for endoscopic images obtained during gastroscopy. For example, it has been shown that the
accuracy of deep learning-based algorithms for predicting the depth of invasion of early gastric
cancer is 71.43%, which is slightly higher compared to the opinion of endoscopists of 64.41% [
Whole slide histological images recognition in gastric cancer using deep learning algorithms
is used to estimate the density of infiltration by various types of immune cells [
Thus, the existing algorithms based on deep learning are so far aimed only to solve specific
narrow problems in relation to histological images of gastric cancer. While there are already
quite efective algorithms for recognizing the depth of invasion and the layers of the intestinal
wall in colorectal cancer [
In this work we use three diferent image datasets that are developed for the purpose of WSI segmentation and tissue type recognition.
The first dataset is NCT-CRC-HE-100K [
The second dataset is CRC-VAL-HE-7K [
The third dataset is the subset of PATH-DT-MSU dataset [
It is also worth noting that area of annotated regions of whole slide images in PATH-DT-MSU dataset is relatively small compared to the area of WSI images. The main reason of this is the necessity of choosing only the regions with "clear" texture that is most characteristic to the one of the corresponding 5 classes. Also an objective reason of this fact is the complexity and laboriousness of WSI annotation process.
At the moment, the direct implementation of segmentation models for whole slide image
processing using fully-supervised deep learning methods is almost impossible due to the necessity
of preparing accurate full pixel-wise annotations of whole slide images. The common way of
getting the approximate result of automatic tissue segmentation is splitting WSI image into
small patches and predicting the class label for each patch using the classification model. Since
the impressive resolution of WSI images this way provides a tissue segmentation map with the
resolution which is appropriate from the medical point of view [
In this paper we propose a CNN-based model for automatic tissue type recognition of whole slide histological images based on the patch classification approach. We also propose a special pipeline for training the model using several diferent datasets to maximize the eficiency of the model.
Image patches from NCT-CRC-HE-100K dataset are available in two versions: with and without
color-normalization using Macenko’s method [
NCT-CRC-HE-100K dataset. This allows us to use CRC-VAL-HE-7K dataset for validation of the model trained on NCT-CRC-HE-100K without any modifications.
The whole slide images from the PATH-DT-MSU dataset are also split into train and test subsets. The train subset contains 5 annotated images, the test subset contains 2 annotated images.
The annotations of WSI images from PATH-DT-MSU dataset made with Aperio ImageScope software can easily be exported in the XML markup format. In order to train the patch classification model and make our PATH-DT-MSU dataset compatible with NCT-CRC-HE-100K and CRC-VAL-HE-7K we extract patches from the WSI images in accordance with annotations.
For the train subset we use the window of 320 × 320 pixels, which is moved over each WSI image with stride of 160 pixels. At each position we calculate whether the window intersects any of the annotation polygons and if the intersection happens the area of intersection is calculated. If the area of intersection is larger than 0.75 of the window area, the patch corresponding to the current position of the window is extracted and saved.
For the test subset the process of patch extraction is the same except that the window size is 224 × 224 pixels and the stride is 112 pixels. The diferent size for the windows is chosen in order to augment the training set of extracted patches with rotation. The stride of the moving window is always half of the window size.
Moreover, due to the 2 times diference in the magnification level of images from NCT-CRCHE-100K and PATH-DT-MSU datasets, all the described above patch extraction procedure from PATH-DT-MSU WSI images is performed on the half of the largest scale (at 20x magnification level).
The described patch extraction procedure with the current annotations of WSI images from PATH-DT-MSU dataset allowed us to extract 70871 patch from training subset and 14462 patches from test subset. Some examples from the extracted set of patches are shown in Fig.2.
It is also necessary to mention that all three datasets used in this work are imbalanced (the number of patches per class for all used datasets is given below): • NCT-CRC-HE-100K. ADI - 10407, BACK - 10566, DEB - 11512, LYM - 11557, MUC - 8896,
MUS - 13536, N0RM - 8763, STR - 10446, TUM - 14317, • CRC-VAL-HE-7K. ADI - 1338, BACK - 847, DEB - 339, LYM - 634, MUC - 1035, MUS - 592,
NORM - 741, STR - 421, TUM - 1233, • Train subset of PATH-DT-MSU. AT - 15207, BG - 45977, LP - 3911, MM - 675, TUM - 5101, • Test subset of PATH-DT-MSU. AT - 1491, BG - 11252, LP - 590, MM - 211, TUM - 918.
To overcome the class imbalance we use the simple oversampling technique that guarantees that the number of patches of each class fed into the CNN model during training is equal.
In this work we use a CNN-based model for automatic tissue type recognition of whole slide
histological images based on the patch classification approach. Since the amount of data in the
obtained datasets is limited, we chose the DenseNet architecture [
DenseNet is the further development of the ResNet architecture [
To get the most eficient model of tissue type recognition in this work we use the transfer learning principle gradually turning the DenseNet model to be able to classify patches from the target PATH-DT-MSU whole slide images.
Thus, the training of the proposed model consists of two main steps: • take the DenseNet model pretrained on ImageNet dataset and fine tune it on the NCT
CRC-HE-100K dataset,
• fine tune the obtained DenseNet model on the PATH-DT-MSU dataset.
These steps will be split into three phases of training and will be described in detail in the next section of the paper.
DenseNet-121 version from [
Since all used in this work histological datasets are imbalanced concerning the number of patches per class (tissue type), we use the oversampling technique while training CNN-based model. Particularly the data batches that are fed into CNN are created by generator in the following way: • randomly select one of the classes, • randomly select a patch from the list existing patches corresponding to the selected class, • augment the patch (diferent augmentation methods are used at diferent phases of training), • repeat until the batch is collected.
This technique equalizes the amount of patches for each class passed through the CNN during training thus making the classification more efective and robust.
During the validation step all image patches from the validation set are passed through the CNN, accuracy, balanced accuracy and confusion matrix are calculated based on the retrieved prediction.
The first phase of training consists in fine tuning of the DenseNet-121 model that is pretrained on the ImageNet dataset. Due to the enormous amount of diferent types of images in ImageNet dataset the head of the pretrained DenseNet-121 can be used as a universal image feature extractor. We replace the last fully-connected layer with the new one with 9 outputs in correspondence with the number of classes in NCT-CRC-HE-100K dataset, freeze all head layers and train the obtained modified DenseNet for 20 epochs with the initial learning rate of 2 · 10− 5. At this step the augmentation with random flip and 90 degrees turn is applied. The validation is performed on CRC-VAL-HE-7K dataset. We reached the accuracy value of 0.919 and balanced accuracy of 0.8816 at this phase.
The DenseNet obtained at the previous phase now represent an universal feature extractor tuned for tissue type classification. In order to better adapt it for working with histological images at the second phase of training we unfreeze all layers in the model and further train it with the same NCT-CRC-HE-100K dataset for 30 epochs with the initial learning rate of 10− 4. The augmentation and validation are the same as at the previous phase. At this point we reached the accuracy value of 0.929 and balanced accuracy of 0.903.
After the previous phase of training the DenseNet model is much more adapted to work with histological images. Finally we fine tune it on patches from whole slide images of PATH-DTMSU dataset. We replace the last fully-connected layer with the new one with 5 outputs in correspondence with the number of classes in PATH-DT-MSU dataset, freeze all head layers and train the model for 8 epochs with the initial learning rate of 10− 5. Herewith the data augmentation besides the random flip includes the random rotation with subsequent crop to the 224 × 224 size. The validation is performed on the test set of patches extracted from PATH-DT-MSU whole slide images. At this phase we finally achieved the accuracy value of 0.98 and balanced accuracy of 0.926. The obtained confusion matrix is shown in Table.1.
In order to perform the visual inspection of the efectiveness of the trained CNN model in the task of automatic tissue type recognition we apply it to the whole slide images from the test subset of PATH-DT-MSU. Each WSI image is split into patches of size 224 × 224 pixels without overlap, for each patch the trained model predicts the class label, after that all the predictions are combined into a matrix, which is then visualized as a semi-transparent layer imposed on the source WSI image.
The visualization of the proposed method of tissue type recognition is shown in Fig.5. The developed algorithm makes it possible to classify with suficient accuracy the areas corresponding to the gastric own mucous glands in the lamina propria and also makes it possible to distinguish the tubular structures of a highly diferentiated gastric adenocarcinoma with normal glands. Less accurate results obtained with respect to the muscularis mucosa can be explained by a similar structure with longitudinal smooth muscle fibers in the own muscle layer of the stomach.
Several experiments with alternative models and training techniques were tested within this work.
Diferent shallow CNN models with reduced number of parameters (compared to DenseNet121) without pretraining were only able to provide the balanced accuracy value of 0.76 on PATH-DT-MSU dataset.
Direct fine tuning of the pretrained DenseNet-121 model on NCT-CRC-HE-100K dataset in one step instead of the proposed two phases of training and further training the model on PATH-DT-MSU dataset allowed us to get the balanced accuracy value of 0.9 on PATH-DT-MSU dataset.
Replacing the third phase of training with two phases similar with the first two phases (first training fully connected layer, then training whole model) did not improve the achieved level of accuracy. This can be explained by the small size of the training set extracted from whole slide images of PATH-DT-MSU dataset.
The proposed CNN-based method for automatic tissue type recognition in whole slide histological images was implemented using Python 3 programming language and open-source software library for machine learning Tensorflow 2 [ 12]. The training of CNN model was performed on a personal computer with AMD Ryzen 9 5900HS with 32 Gb RAM and Nvidia GeForce 3080 GPU.
Processing whole slide images was done using slideio library [13], shapely library [14] was used for geometric calculations during patch extraction.
After preliminary translation and saving the WSI images from PATH-DT-MSU dataset as numpy arrays, the inference time of applying the proposed model of automatic tissue type recognition with full visualization of the results takes about 1.5 minutes on the mentioned above PC configuration, which is good enough for practical use by pathologists.
In this paper we proposed a new CNN-based method of automatic tissue type recognition in whole slide histological images and a special pipeline of training CNN model within three phases using diferent histological datasets. The proposed model achieved accuracy value of 0.929 and balanced accuracy of 0.903 on CRC-VAL-HE-7K dataset and accuracy value of 0.98 and balanced accuracy of 0.926 on the test subset of patches from WSI images of PATH-DT-MSU dataset.
The developed method makes it possible to classify the areas corresponding to the gastric own mucous glands in the lamina propria and also to distinguish the tubular structures of a highly diferentiated gastric adenocarcinoma with normal glands.
Further improvements of current work will be focused on the enlarging the number of whole slide images in PATH-DT-MSU dataset, adding more detailed annotations and increasing the number of supported types of tissues. The performance of the proposed method is also planed to be improved by pretraining the model with autoencoders on WSI images that are not annotated.
The work was funded by RFBR, CNPq and MOST according to the research project 19-57-80014 (BRICS2019-394). ceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [11] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). [12] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow: A system for large-scale machine learning, in: 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), 2016, pp. 265– 283. [13] S. Melnikov, V. Popov, Slideio: a new python library for reading medical images, 2020–.
URL: https://gitlab.com/bioslide/slideio. [14] S. Gillies, et al., Shapely: manipulation and analysis of geometric objects, 2007–. URL: https://github.com/Toblerity/Shapely.