Colorectal cancer (CRC) is a commonly found and highly fatal carcinoma at the later stage. Colonoscopy is recommended for the early detection and prevention of CRC by finding and removing the leading CRC precursors, the polyps. Polyp segmentation, which helps extract polyps from colonoscopy images, is an essential step for diagnosing and developing an automatic real-time polyp classification system. In the EndoCV2021 challenge, aiming to enhance PraNet [1], we utilized a parallel res2Net-based network with reverse attention and proposed a new post-processing workflow to predict polyp segmentation masks.EndoCV2021 and three more datasets were used to test the performance and generalization ability of the proposed segmentation methodology with seven metrics. Quantitative and qualitative evaluation results show that we develop a generalizable model with excellent real-time eficiency ( ∼ 32fps) and precision / accuracy / sensitivity of 81.14%-88.94% / 94.11%-98.57% / 77.63%-91.48%, respectively. Compared to the original PraNet, our proposed method improves segmentation precision significantly by a level of 10%-29% and is more accurate with an increase of 2%-6%.
Colorectal cancer (CRC) is the third most common cancer and the fourth most common cause of
cancer-related death [
Deep learning is widely used in medical imaging analysis and computer-aided detection
(CAD) systems due to its excellent feature extraction ability [
In 2020, Fan et al. [
To further improve PraNet, our proposed methodology starts with a parallel res2Net-based network with reverse attention to segment polyp from colonoscopy image. In addition, the segmentation results are post-processed to eliminate the uncertain pixels and distinct the segmentation edge. By doing so, we clarify the polyp boundaries and diminish large numbers of false-positive cases, thus enhancing the precision, Jaccard, and Dice value. Four datasets are used to assess the performance of the model. Evaluation results demonstrate that our method ofers the advantages of high generalization capacity, real-time eficacy, precision, and accuracy. Comparing to PraNet, our method is much more precise and accurate. Therefore, the tremendous emotional stress of the patients and additional tests can be avoided in clinical practice.
Diferent from data augmentation, we first split the data randomly into five equal sub-datasets to employ the five cross-validation training strategy. In this strategy, five diferent models were trained each time, using four datasets to find the appropriate model parameters and prevent over-fitting.
The images have a diferent size in the subfolder of the EndoCV2021 dataset [ 10]. We then resized images to the same size with a multi-scale training strategy to feed into the neural network. Using this strategy to scale the input images to diferent sizes, the model can better adapt to images of various sizes and increase the number of training sets.
The model we used is a Res2Net-based network, which extracts multi-level features from polyp images. Figure 1 shows the outline framework. The architecture could be divided into three processes. First, we used the feature extractors to generate five-level features that included two low-level and three high-level features. After that, the high-level features will be up-sampled and paralleled in connection to the partial decoder. We next put the high-level features and output of the partial decoder (PD) to reverse attention (RA) component, which could adjust segmentation to accurate results. Each process will be described as follows.
The Res2Net-based [11] backbone network was utilized to extract multi-level features. As Figure 1 shown, we designed five extractors, and each of them contains several network layers based on a Res2Net-based network. In details, we called the first two extractors low-level extractor and the others high-level extractor, which extract low-level features{1, 2} and high-level features{3, 4, 5} respectively.
The high-level features would occupy fewer computing resources and contribute more to the network, thus are concentrated more than low-level features[12] in our model. Therefore, we apply paralleled connection on high-level features rather than all features. High-level features will be up-sampled and using the Concat function to be connected parallelly to be more specific. To aggregating these deep features, a partial decoder ()[12] component is computed by (3, 4, 5). The previous partial detector generates a global map with the paralleled connection of high-level features. Using the deepest CNN network, high-level extractors will roughly extract the feature information.
Figure 2(a) shows a traditional medical image segmentation framework, which generating map by adopting a full decoder to integrate all level features. However, Wu et al. [12] proved the high-level features would occupy fewer computing resources and contribute more to the network. Besides, they conducted visualization experiments on multi-level feature maps and found that the fifth layer still reserved edge information. Inspired by their work, our model concentrates more on high-level features and uses five-level extractors to elicit features. As Figure 2(b) shown, we applied paralleled connection on high-level features rather than all features. Therefore, high-level features are up-sampled and connected parallelly by the Concat function. To aggregating these deep features, a partial decoder p()[12] component is computed by (3, 4, 5). This partial detector generates a global map . With the paralleled connection of high-level features, high-level extractors will roughly extract the feature information using the deepest CNN network.
To further detail the information, we adopt a reverse attention component for each high-level feature branch. A shallow-forward strategy is applied to ensure that the maps generated by each branch are gradually refined. Specifically, the deep maps {4, 5, } will be delivered to the RA component of the shallow part to compute reverse attention features {3, 4, 5}. Each deep map and reverse attention feature element-wise is multiplied and generates the shallow maps {3, 4, 5}. The map gradually sharpens by sequentially fusing with shallow maps.
Our loss function formula is represented as = (, ) , where the deep feature maps {, = 3, 4, 5, } will be up-sampled to the same size of the ground-truth G, and calculate the loss between each and by function . Function is defined as (1).
= + (1) Among them, and
express the weighted IOU loss and BCE loss, respectively.
Comparing with standard IOU and BCE loss, the weighted loss would concentrate more on hard pixels instead of allocating the same weights for all pixels.
The original prediction, generated by the deep paralleled network with reverse attention modules, could roughly divide into polyp cases and pending cases. We can distinguish two cases easily by the presence or absence of a clear polyp site. The polyp case contains more white pixels and is more concentrated than the pending case, while most of the gray pixels distribute on the edge of white pixels. By contrast, polyps have no presence in the pending case, so their white pixels are sparse, and gray pixels are scattered irregularly. Therefore, we introduced post-processing to the original predictions to clarify the polyp case boundary and eliminate the uncertainty. (a) Uncertain case
(b) Polyp case
Concretely, the smallest polyp size, which is the tiniest white pixel in the dataset’s polyp case, is appointed as the reference value. We subsequently use the number of white pixels to judge the type of case from the original prediction. When the number of white pixels in the original prediction is more than the reference value, we consider it the polyp case because it is suficient to form the smallest polyp. Thereafter, we set a threshold to polarize the gray pixels into black and white pixels for polyp cases—this conversion distinct the boundary and inner area of polyp clearly through polarized the pixels. For a pending case, where white pixels are rare, we consider it a non-polyp image and convert all non-black pixels to the background. The original prediction, post-processed results of each case, and the corresponding ground-truth are shown in Figure 3.
Our model mainly used the EndoCV2021 challenge dataset [10] for endoscopic images for polyp segmentation in this work. The dataset contains 1,449 endoscopic images with ground-truth masks in C1-C5 folders. We split the dataset into 80% for training and 20% for validation. To verify the generalization abilities of our method, we also utilize three publicly available endoscopy datasets, Kvasir-seg [13], CVC-Clinic [14], and EndoScene-CVC300 [15] for testing.
The deep models are implemented based on PyTorch and trained on an NVIDIA GeForce RTX 3090 GPU using Adam optimizer with a learning rate of 1 − 4. The batch size is set to 16, while input images are resized to 512512 with multi-scale training parameters {0.75, 1, 1.25}.
To quantitatively evaluate our model’s performance, seven metrics provided by the challenge for the segmentation task are employed: Jaccard (Jac), Dice, F2-score, Precision (Positive Predictive Value, PPV), Recall (Rec), Accuracy (Acc), and Hausdorf distance (Hdf). A toolbox provided by the challenge organizer at https://github.com/sharibox/EndoCV2021-polyp_det_seg_gen [16, 17] is utilized to calculate scores between each prediction and ground-truth. Also, we add the Frame Per Second (Fps) metric to evaluate the real-time performance of our model.
Figure 4 presents our model’s loss on the training set for varying epochs. When the epoch is less than 50, the graph describes a decreasing trend, while it remains stable after the epoch greater than 50. That is to say, the loss gradually converges downward, which proves that our model is feasible. Moreover, we experiment with diferent input images’ sizes to attempt to obtain the appropriate size on this dataset. We can observe that when the input size is small, the loss value after convergence is relatively large, and vice versa. Among them, the size of 512512 reaches the lowest convergence value in the training phase.
Table 1 provides our model’s segmentation results on EndoCV2021 C1 and CVC-ClinicDB datasets with diferent input image sizes. Consistent with the results in Figure 4, the larger input size performs better because it contains more semantic information, which may be lost after resizing to the smaller input size. Also, we employ the fps metric for real-time performance evaluation. Our model’s real-time inference speed can reach ∼ 30fps even using diferent datasets or input sizes. That is to say, our model not only performs real-time eficiency well but also can be extended to colonoscopy videos.
Quantitative results of model performance, including seven metrics, are presented Table 2. The results demonstrate that our model’s varying component is a crucial contributor to the improvement of the model segmentation ability. The precision, accuracy, and sensitivity are 81%89%, 94%-98%, and 64%-80%, respectively, across diferent source datasets. The post-processing component enhances the Jaccard, Dice, precision, and accuracy metrics. In other words, our model can better distinguish the patients without polyps. Comparing with PraNet, the accuracy increases 2%-6%, and segmentation precision enhances a significant 10%-29%. Besides, Table 2 also shows that the method achieves excellent generability by delivering well segment precision on datasets never train the model, the Kvasir-seg, and EndoScene-CVC300 datasets.
In this manuscript, we introduce a deep model for the EndoCV2021 polyp segmentation task. The model adopts a parallel res2Net-based network with reverse attention for polyp segmentation and concentrates on extracting the high-level features. The results are predicted with a postprocessing component. The experimental results indicate that our model (1). has excellent generalization ability on diferent datasets, (2). delivers real-time eficiency, and (3). ofers better precision and accuracy that can better identify negative polyp individuals. The 512512 input size is a balance between our GPU memory and image resolution on the EndoCV2021 challenge datasets. Future works will focus on the data augmentation of polyps and lighter feature extractors to further improve the segmentation results.
Special thanks are given to the EndoCV2021’s organizing committee, clinical collaborators, program committee, and GPU cloud-based inference admin. This research was partly supported by the National Natural Science Foundation of China (Grant No. 41876098), the National Key R&D Program of China (Grant No. 2020AAA0108303), Shenzhen Science and Technology Project (Grant No. JCYJ20200109143041798), Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School (Grant No. HW2018008). The authors would like to also thank Zhe Xu and Yu Yang in the Intelligent Computing Lab, the Department of Shenzhen International Graduate School, Tsinghua University for their valuable discussions. segmentation, International Conference on Medical image computing and computerassisted intervention (2015) 234–241. doi:10.1007/978-3-319-24574-4_28. [7] P. Brandao, E. Mazomenos, G. Ciuti, R. Caliò, F. Bianchi, A. Menciassi, P. Dario, A. Koulaouzidis, A. Arezzo, D. Stoyanov, Fully convolutional neural networks for polyp segmentation in colonoscopy, Medical Imaging 2017: Computer-Aided Diagnosis 10134 (2017) 101340F. doi:10.1117/12.2254361. [8] B. Murugesan, K. Sarveswaran, S. M. Shankaranarayana, K. Ram, J. Joseph, M. Sivaprakasam, Psi-net: Shape and boundary aware joint multi-task deep network for medical image segmentation, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2019) 7223–7226. doi:10.1109/EMBC.2019.8857339. [9] Y. Fang, C. Chen, Y. Yuan, K.-y. Tong, Selective feature aggregation network with areaboundary constraints for polyp segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention (2019) 302–310. doi:10.1007/ 978-3-030-32239-7_34. [10] S. Ali, D. Jha, N. Ghatwary, S. Realdon, R. Cannizzaro, M. A. Riegler, P. Halvorsen, C. Daul, J. Rittscher, O. E. Salem, D. Lamarque, T. de Lange, J. E. East, Polypgen: A multi-center polyp detection and segmentation dataset for generalisability assessment, arXiv (2021). [11] S. Gao, M. M. Cheng, K. Zhao, X. Y. Zhang, P. Torr, Res2net: A new multi-scale backbone architecture, IEEE Transactions on Pattern Analysis and Machine Intelligence (2019) 1. doi:10.1109/TPAMI.2019.2938758. [12] Z. Wu, L. Su, Q. Huang, Cascaded partial decoder for fast and accurate salient object detection, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 3907–3916. doi:10.1109/CVPR.2019.00403. [13] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, H. D. Johansen, Kvasir-seg: A segmented polyp dataset, International Conference on Multimedia Modeling (2020) 451–462. doi:10.1007/978-3-030-37734-2_37. [14] J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, F. Vilariño, Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians, Comput Med Imaging Graph 43 (2015) 99–111. [15] D. Vázquez, J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, A. M. López, A. Romero, M. Drozdzal, A. Courville, A benchmark for endoluminal scene segmentation of colonoscopy images, Journal of healthcare engineering 2017 (2017). [16] S. Ali, F. Zhou, B. Braden, A. Bailey, S. Yang, G. Cheng, P. Zhang, X. Li, M. Kayser, R. D.
Soberanis-Mukul, et al., An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy, Scientific reports 10 (2020) 1–15. [17] S. Ali, M. Dmitrieva, N. Ghatwary, S. Bano, G. Polat, A. Temizel, A. Krenzer, A. Hekalo, Y. B. Guo, B. Matuszewski, et al., Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy, Medical Image Analysis 70 (2021) 102002.