Automated analysis of endoscopic images is becoming increasingly significant for an early detection of numerous cancers and minimally invasive surgical procedures. The paper briefly describes the methodology adopted for the 2020 Endoscopy Artefact Detection and Segmentation (EAD2020) challenge1. A number of novel variants of the DeepLab V3+ encoder-decoder architecture have been investigated, implemented and tested for the segmentation sub-challenge. Modifications were introduced to improve: selection of image futures, segmentation of small objects, and use of the encoder output information. The proposed methods achieved competitive segmentation score results on both release-I and releaseII test datasets. For the detection sub-challenge three off-theshelf deep detection networks have been optimised and evaluated on the EAD data.
Automated analysis of endoscopic images has obvious
practical clinical importance. For example, colorectal cancer is one
of the leading causes of death worldwide, e.g. in the United
States, it is the third largest cause of cancer deaths; whereas
in Europe, it is the second largest with 243,000 deaths in
2018 [
Automation of the analysis of endoscopic images poses significant technical difficulties. As evident from the EAD challenge, the segmentation task is a very demanding problem, with multiple difficult to define semantic categories, possibly represented within the same/similar image locations and structures of significantly different sizes. Additionally, some of these categories (e.g. “bubbles”) are difficult to discriminate with respect to appearance and spatial distribution. 1It refers to the results submitted by the CVML team.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Segmentation is one of the key enabling technologies in
medical image analysis with a great variety of methods
proposed [
The key architectures used as the baseline for the
segmentation methods, developed for the EAD challenge, are Dilated
ResFCN [
Only the data, which have been made available as part of
the EAD2020 challenge [
The original EAD2020 training images are augmented by rotation, colour jitter and elastic deformations. For the segmentation task, all the images have been scaled to 513 513 pixels in size, with two training data subsets created. The smaller training subset consists of 11,376 images, augmented from the phase-I training dataset. The networks trained on this smaller subset have been validated on the phase-II training dataset and online on the test datasets. This small training subset was predominantly used to quickly verify specific design choices made during the methods’ development. The larger training subset consists of 38,195 images augmented 2//github.comf/ultrlytics/yolov3,/open-mmlab/mmdetection,/hujiefrank/SENetg. from the phase-I and phase-II training datasets. That bigger training set was used to train architectures which have been thought to provide competitive results when trained on the smaller dataset. The networks trained on the larger training subset were only evaluated online on the EAD2020 test datasets.
For the detection sub-problem, the images have been scaled to 667 400 pixels in size. As for the segmentation, two augmented training subsets were created. The smaller subset with images augmented from phase-I training dataset consists of 8800 images, whereas the large subset has 30,372 images augmented from the phase-I and phase-II training datasets.
DeepLab V3+ [
A number of novel network architectures (here
collectively named as DeepEAD), based on the DeepLab V3+, have
been proposed and validated for the EAD2020 segmentation
challenge. In order to segment the overlapping objects, the
original multi-class classifier is replaced with 5 binary
classifiers. Further changes lead to three network architectures:
Network 1: The original DeepLab V3+ main
subnetwork is replaced by the SE-ResNeXt-50 [
Network 2: Based on Network 1, with the global pooling removed from the ASPP and replaced with 3 3 convolutions. The corresponding receptive fields are expected to improve segmentation of the small objects. Furthermore, the number of the convolution kernels at each resolution is selected to emphasise small objects. Fig. 2. The number of valid weights in the dilation kernels shown in Fig.1.
Network 3: Shown in Fig.1, is based on Network 2, with the squeeze and excitation module added behind the ASPP module. This is to introduce attention gating at the output of the original encoder to better utilise information available in the computed feature maps.
Following on the methodology proposed in [
Since the proposed networks don’t have built-in rotation invariance, to improve the segmentation accuracy the image rotation augmentation during test time has been investigated. For this purpose, rotated versions of the test image are presented to the network and the corresponding outputs are averaged to better utilise generalisation properties of the network. The adopted test time augmentation process is explained in Fig.3. The corresponding results, shown in section 4, demonstrate that the test time augmentation does indeed have a significant impact on the segmentation performance.
This section reports on a sample of results obtained for the segmentation and detection methods described above. Table 1 shows a representative sample of the results obtained for the segmentation task on both validation and release-I test datasets. The results obtained on the validation data (phase-II training data) are reported in the second column, with all the networks trained only on the augmented images from the phase-I training dataset. The results on the release-I test dataset are reported in the third column. The symbol “*” indicates that the result has been obtained for the network trained on the large training dataset (i.e. images augmented from the phase-I and phase-II training sets), otherwise results have been obtained for the network trained on the small training dataset (i.e. images augmented from the phase-I training dataset only - see section 2 for more details). It could be concluded that the gradual improvement of the results on the validation data is replicated on the test data. As expected the use of the large training set also improves performance. This can be seen from the results reported for Network 2, with the segmentation score of 0.50 for the network trained on a smaller training set, and score of 0.59 for the exactly the same network but trained on the large dataset. It seems that the segmentation score of 0.5934 (for the Network 2 trained on the larger training set) was a competitive result on the release-I test dataset.
The best results obtained on the release-II test dataset, with all the networks trained on the larger training dataset, are reported in Table 2. As evident from the table, Network 3 provides the best segmentation results with the test time augmentation improving the segmentation score by 0.0434, i.e. about 8%. The effects of the test time augmentation are
DeepLab v3+ Network 1 Network 2 Network 3 shown in Fig.4 demonstrating impact of the augmentation on segmentation of the ”instrument” class.
Various post-processing operations have been also tested, including hole filling and removal of objects from the image black boundary. These, though, had a relatively small, and difficult to predict, effect on the segmentation score. The segmentation score result for the final submission was reported as 0.5916, which was slightly lower than the best result of 0.5922 (see Table 2).
Table 3 shows results obtained for different detection networks tested on the release-II test data. It could be observed that R-CNN networks outperform the Yolo network, with the best detection score achieved by the Faster R-CNN. This is different from the results obtained on the release-I test set (not reported here) where the Yolo network achieved better result. This though could be possibly explained by optimisation of the networks design parameters during the second phase of testing.
The paper describes novel segmentation networks, highlighting the key characteristics of the proposed deep architectures. The proposed methods achieved segmentation scores of 0.5934 on the release-I test data and 0.5922 on the release-II test data, which seem to be competitive. The overall detection performance also seems comparatively reasonable with best detection score of 0.2335 on the release-II test data. However, the statistical significance of these results would need to be investigated. Further improvements could be possible, e.g. with the image aspect ratio augmentation to reflect the input format of the adopted networks, or use of the segmentation network as a pre-selection tool for detection of small objects (e.g. specularity artefacts).