The detection and segmentation of polyps during colonoscopy can substantially contribute to the prevention of colon cancer. Assisting clinicians using automated systems can mitigate the risk of human error. In this work, we present our polyp segmentation approach, submitted to the EndoCV2022 challenge. Common polyp segmentation methods are based on single-model, single-frame predictions. This work presents a symbiosis of three separate models, each with their own strength, as part of a segmentation pipeline and a post-processing step designed to leverage unique predictions for more temporally coherent results.
Colorectal cancer is one of the most commonly found can- The dataset provided by the EndoCV2022 polyp
segmencer types, ranking second in females and third in males tation sub-challenge[
Our challenge strategy rests on three main pillars: (1) A
data pre-processing step to ensure high data annotation
quality, (2) the network architecture selection and
training step, which yields the segmentation models, and (3) a
post-processing step, which leverages model
heterogeneity and uses structural similarity[
than having a larger number of frames for training. The external datasets described in section 2 underwent the same selection process. Including external data, the resulting training dataset amounted to 4,106 image-mask pairs. we expected more knowledge to be leveraged from the provided high-resolution video sequences. 3.2.1. nnU-Net The nnU-Net is able to automatically determine key decisions to set up the segmentation pipeline for training, irrespective of the dataset. While it has ranked first on many 3D-segmentation challenges1, its self-configuring strategy can also be applied to 2D images. The nnU-Net was expected to provide a solid base prediction. 3.2.2. Hierarchical Multi-Scale Network
vision task, it is possible to use established segmentation models that perform well on complex natural images. The Hierarchical Multi-Scale Attention Network (HMANet) was chosen as it is a state-of-the-art architecture in semantic segmentation on Cityscapes2. The HM-ANet operates on higher resolutions and combines predictions from diferent scales. This was expected to result in a precise polyp segmentation, irrespective of the size of the polyp.
In order to solve the polyp segmentation task, a model
ensemble was designed that consists of parts with com- Most of the current segmentation models operate on
plementary strengths. This was realized by using an a frame-by-frame basis. In order to capture temporal
nnU-Net[
and outperform other ConvNet backbones. One GRU- 4.1. Implementation details layer was added to the bottleneck of the Ef-UNet, to form an Ef-GRUNet. Consecutive images are loaded in batches of size two. They are encoded, pooled, flattened, sequentially fed into the GRU-layer and then reshaped and fed to the decoder. The Ef-UNet is trained separately from the Ef-GRUNet. Variants of both combined form the Ef-UNet ensemble.
The nnU-Net was used as a framework and manual
changes were made to its automatically generated
conifguration. The short edge of the image was resized to
512px with the other being resized according to aspect
ratio. The patch-size was set to 448 x 448. The data was
then heavily augmented with operations such as rotation,
intensity and gamma augment, scaling, mirroring and
blurring.
3.3.1. Combining networks and weighting For the HM-ANet, the data was normalized and
ranSince the HM-ANet operates on high resolutions, it was dom scaling between [0.5,1], random crop to 512x1024,
expected that it performs well on very small polyps, as RGB-shift, and random vertical and horizontal flipping
well as being able to fully capture larger polyps in their was performed. The model was initiated with weights
entirety. During ensembling, the HM-ANet was designed pre-trained on PaddleClas3. The training was conducted
to be weighted higher for the small and large polyps. in three phases: 1) Training the model on original
chalSince there is no standardized definition of polyp sizes, lenge data, 2) fine-tuning the model on challenge and
the thresholds were set empirically by observing refer- external data, and 3) fine-tuning again on challenge data
ence labels of public polyp datasets[
The nnU-Net was trained with external data added to the training set, resulting in a mean CV DSC score of 0.67. Table 1 Training only on the challenge set or the external dataset Cross-Validation scores of all models, including the comporesulted in a worse DSC score of 0.57 and 0.55. nents of the Ef-UNet ensemble. Underscored values indicate The HM-ANet had a mean CV DSC score across all folds selection for Ef-UNet ensemble. Bold values indicate compoof 0.70. During training and inference, predictions of nents of the final heterogeneous ensemble. scales [0.5,1] were combined. Experiments with scales DSC score Fold 0 Fold 1 Fold 2 Fold 3 of [0.5,1,2] resulted in a worse performance of 0.69 with nnU-Net 0.65 0.84 0.70 0.50 more false-positives in empty images. Training in three HM-ANet 0.67 0.82 0.69 0.60 steps as described in sub-subsection 4.1. yielded the best Ef-UNet_480 0.67 0.80 0.69 0.62 result. Other training strategies such as training on a Ef-UNet_256 0.68 0.80 0.71 0.65 combined dataset or pre-training on the external dataset Ef-GRUNet 0.61 0.72 0.58 0.60 and fine-tuning on the oficial dataset resulted in a worse Ef-UNet Ens 0.67 0.80 0.71 0.60 performance. 4-fold cross-validation was used to determine the stopping epochs for all three phases. A final inference model was then trained on the entire dataset. 4.3. Reweighting and ensembling results The three Ef-UNet models were each trained on the combined dataset over four folds, resulting in 12 models. The mean CV DSC scores of the Ef-UNet_480, Ef-UNet_256 and Ef-GRUNet were 0.69, 0.71 and 0.62, respectively. As an alternative experiment, the Ef-UNet_480 was trained with external data for pre-training and challenge data for fine-tuning. This performed worse compared to using the combined dataset, resulting in a mean CV DSC score of 0.65. In order to decrease inference time, two EfUNet_480, one Ef-UNet_256 and one Ef-GRUNet were selected for the ensemble, based on validation score and fold representation. The final prediction was determined by majority vote. The mean CV DSC score of the final ensemble was 0.70.
the proportion of small and big polyps was calculated for the validation splits. For folds 0-3, the ratios were 45%, 28%, 37%, and 65%. For single models, fold 1 had the most medium polyps and highest average CV score. Fold 3 has the most non-medium polyps, and the lowest average CV score. However, the diference in DSC scores between models is small. Since the ratio was highest for fold 3, an experiment is conducted where the three single models were validated on only the small and big polyp images of fold 3 (n = 483 out of 738 frames). The resulting DSC scores are 0.63, 0.66 and 0.70. The simple ensemble receives a score of 0.73 and the ensemble with reweighting of HM-ANet a score of 0.74. Adding postprocessing did not decrease or increase the score for this validation set.
Our investigation showed that the HM-ANet was favorable for small and large polyp cases, which our dedicated weighing strategy takes into account during ensembling. Notably, on a dataset with small and big polyps, it achieves a DSC score of 0.74, improving the best-performing single model HM-ANet by 0.04. The post-processing leverages self-adaptive training as well as temporal and high resolution information by enabling unique predictions of all three heterogeneous components, resulting in less false-negative predictions. The inference time as the sum of the slowest component (nnUNet) and the ensembling step is 0.71 fps.
German Cancer Research Center(DKFZ) and the Robert Bosch Center for Tumor Diseases(RBCT). Part of this work was funded by Helmholtz Imaging(HI), a platform of the Helmholtz Incubator on Information and Data Science.