The detection of polyps is helpful to the diagnosis of early colorectal cancer. With the rapid development of deep learning, more and more researchers apply detection and segmentation technology to assist polyp detection. This work is our solution to the polyp segmentation subtask in the EndoCV2022 challenge. We come up with the idea from the semi-supervised video object segmentation and build on STCN [1] for this challenge. STCN is built for the task when the correct segmentation mask of the first frame of the video is given as input, then the model just tracks the target, no matter what it is. We modify STCN into a sequence polyp segmentation network named improved-STCN, which can not only segment the polyps but also track the polyps. As EndoCV2022 challenge [2] [3] is a sequence challenge, the images in the same sequence are very similar, which will lead to bad performance. Thus, we adopt semi-supervised learning to get more abundant data for training. We also carry out experiments on how to make the segmentation results more credible, that single frame detection and reverse sequence information will help in this part. Finally, on the round-II test, our system achieves a segmentation score of 0.7654 and ranked the second.
Colorectal cancer (CRC) is a common malignant tumor in
the gastrointestinal tract. Its incidence rate and mortality
rate are the second most important in digestive system
cancer, followed by gastric cancer, esophageal cancer
and primary liver cancer. Polyp is considered a sign of
precancerous lesions, thus, finding it at any time during
precancerous lesions and blocking it not only reduce the
mortality of colorectal cancer, but also reduce the
incidence rate. Colorectal lesions are usually diagnosed by Figure 1: Example of EndoCV2022 challenge sequence data
colonoscopy, but unfortunately, it is estimated that about
6-27% of pathological missed diagnosis in colonoscopy
[
Detection in Colonoscopy Videos challenge, more and However, for sequence data in real scenes, this kind of
more datasets and challenges have been launched, which method can not efectively model timing information.
further promote the application of deep learning-based In the field of video object segmentation, the model is
endoscopic vision [
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) EndoCV2022 challenge. The proposed approach is built encoder and a value encoder respectively. The key enon STCN, a semi-supervised video object segmentation coder encodes the images into the key feature space and network. In particular, we modify STCN into a sequence the value encoder encode both the images and mask into polyp segmentation network ,which can not only seg- the value feature space. The key correspond with value ment the polyps but also track the polyps. In short, our one by one will be stored in the memory bank. Then, main contribution for this work are as follows: when a new frame in the video sequence is collected, the frame will be encoded into the key feature space firstly, • We modify STCN into a sequence polyp seg- and then calculate the similarity with the key features mentation network, which will no need the first of the previous frame stored in the memory bank. The frame’s mask to predict like it used to be. And we most similar features will be combined into the feature also practice the experiment on training strategy space of the current frame for model outputs. Here, the to find a stronger model. negative square Euclidean distance is used as similarity • We learn from semi-supervised learning to gener- functions, which is defined as follows: ate more training data, as the image of the same sequence have great similarity, which is not con- = −|| − ||22 (1) ducive to the improvement of network generalization and feature extraction ability. where represents the previous frames’ key features, represents the current frames’ key feature. • We propose an enhanced scheme to make the Then the aggregated readout feature for the current segmentations results more credible. Overall, frame can be computed as a weighted sum of the memory our method is proved to be efective in the En- features with an eficient matrix multiplication: doCV2022 challenge round-I and round-II. = . (2)
which is then passed to the decoder for mask generation
[
Besides, we carried out the semi-supervised learning men2.3. Enhanced scheme tioned in the Subsection 2.2. We also explored the efects Although the model mentioned in the Subsection 2.1 of illumination and size on model’s performance. In part has the ability to segment and track the polyps, we find two, we used the same train strategy as the part one to that train two models to segment and track polyps sepa- train model with all the dataset we have, and tested model rately will get better results. As figure 3 shows, SFSN that with the Endocv2022 challenge unseen dataset. The enchange from STCN is used to segment the polyps in the hanced scheme was adopted to get the more credible ifrst few frames of the sequence data. Meanwhile, STCN segmentation results. will also outputs the segmentation results. The results of the two models will use the same calculation method to obtain confidence, which is defined as the average value
tal of 46 sequence data for all participants. According to
the statistics, the EndoCV2022 challenge Dataset consists
3348 frames sampled in the real-world clinical scenario.
As figure 4 shows, most polyps are around 400 in size
while a few polyps are larger than 800. Due to the
diferent sizes of polyps and images, we need to pay attention
to using some strategies to reduce the sensitivity of the
network to resolution, such as Multi-scale training.
Although polyps have diferent shapes and sizes, the image
of the same sequence data have great similarity, which is
not conducive to the improvement of network’s
generalization and feature extraction ability. Thus, in baseline
experiments, we split the EndoCV2022 challenge Dataset
into 80% for training and 20% for validation in sequence.
To enhance the generalization and feature extraction
ability of our model, we also utilized three well-known
publicly endoscopy sequence datasets, ETIS-Larib Polyp [
and inference were run on the NVIDIA TESLA V100
GPU. Here, we minimized the cross-entropy loss using
Adam optimizer with default momentum 1 = 0.9, 2
= 0.999. The learning rate lr=0.0001 and the batch size
was set to 16. The input image size of the model was
384 × 384 pixels As it was an sequential learning task,
the maximum temporal distance between frames was
set to be [
Table 1 shows the Ablation study result of Endocv2022
validation and CVC-Clinic datasets. Firstly, we see that
when we use semi-supervised learning, the dice
coeficient of the model in the Validation Set (EndoCV2022
validation + CVC-Clinic) has increased by 3%. It proves
3.2. Evaluation Metrics that adding more sequence data for model to learn does
help .Secondly, colonoscopy is a product of a combined
The EndoCV2022’s organizing committee provided par- light source, thus, the collected images are either very
ticipants a toolbox to calculate the scores between the bright or very dark. We set color jitter of (brightness=0.5,
predicted mask and the ground truth mask at github contrast=0.03, saturation=0.03) to simulated light change.
[
Where TP represents true positive "polyp", while FP dataset while the dice coeficient is up to 0.7423.This reand FN represents false positive and false negative re- sult already make us ranked the top5 on the leaderboards. spectively. Similarly, the Dice coeficient is calculated as When we adopt the two methods mentioned in the subfollows: section 2.3, the dice coeficient has increased by 2% and by 3% respectively. From the results, we see that our en = (4) hance scheme mentioned above does help. Unfortunately, 2 * 2 * + +
In this work, we have detailed our solution for the polyp segmentation subtask in the EndoCV2022 challenge. We have proposed improved-STCN network with a semisupervised learning method to improve model’s generalization and an enhanced scheme to make model output more credible results. Limited experimental results show that our method achieves consistently high Dice scores at very low standard deviations, suggesting its suitability for polyp segmentation on endoscopic sequence data.