Deep neural network-based methods are becoming popular for disease diagnosis and treatment in Endoscopy. In this paper, we discuss our submission to Endoscopic Artefact Detection Challenge (EAD2020). The competition is part of grand challenges in Biomedical Image Analysis and consists of three sub-tasks1: i) Bounding box-based localisation of artefacts ii) Region-based segmentation of artefacts, and iii) Out of sample generalisation task. For the first sub-task, we modify the Faster R-CNN object detector by integrating a powerful backbone network and a feature pyramidal module. For the second sub-task, we implemented a U-Net based autoencoder with a modified loss function to construct the semantic channels. For the third sub-task, we used ensembling techniques along with a dataaugmentation technique inspired by RandAugment to boost the generalisation performance. We report a Scored of 0:1869 0:1076 for the first task, sscore of 0.5187 with a sstd of 0.2755 for the second task and mAPg of 0.2620 and a devg of 0.0890 for the third task on the test data-set. Our method for the third task, achieves the third position on the leaderboard for the mAPg metric and also surpasses the results obtained by many methods in the previous EAD2019 challenge.
Endoscopy is widely used as a clinical procedure for early detection of numerous cancers (e.g., nasopharyngeal, oesophageal adenocarcinoma, gastric, colorectal cancers, bladder cancer, etc). It is also used for therapeutic procedures and minimally invasive surgeries (e.g. Laparoscopy). During this procedure, an endoscope which is a thin, long and flexible tube with a camera and a light source located at its proximal tip is used which helps to visualise the internal organs and helps for further diagnoses by the clinicians. A major drawback of the video frames obtained from this 1https://ead2020.grand-challenge.org
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). process, is that they are corrupted with multiple artefacts (for e.g. motion blur, pixel saturation, bubbles, fluid, debris, specularity reflections, etc) even though, the videos might be captured at a very high resolution. These artefacts prevent effective diagnoses of pathologies, post-analysis with respect to retrieving frames for report generation and video mosaicking for follow-ups. Thus, it becomes essential to use frame restoration algorithms, which helps to restore the frame to its highest quality. The frame restoration algorithms require accurate detection of the spatial location of multi-class artefacts in the corresponding frames. But present endoscopy workflow supports the restoration of only one type of artefact class, which is insufficient for high-quality frame restoration. So, it becomes essential to build multi-class artefact detectors which can lead to the development of artefact correction and frame restoration algorithms for each specific artefact class.
Endoscopic Artefact Detection challenge 20201, aims to address the key problem inherent in endoscopy. There are three sub-tasks in this particular challenge. They are: i) Bounding box localisation of multi-class artefacts. In this task, we are required to identify the class of the artefact, along with its spatial location by identifying the closest bounding box co-ordinates around the artefact. ii) Semantic segmentation of artefacts, where we are required to identify the class of the artefact along with its accurate region in the frame. Semantic segmentation is more effective than bounding box based localisation, because the region of interest, i.e. the artefact region is accurately marked in this task. iii) Out of sample generalisation task, where we are required to identify the type and the region of the artefacts through bounding boxes, from frames not captured for training purposes.
In our work, we demonstrate the following novelty: We modify the Faster R-CNN [1] module for the object detection task. We use a powerful version of the backbone ResNeXt-101 [2] for effective extraction of aggregated features. Additionally, we apply a feature pyramidal network (FPN) [3] module for multi-scale feature representation.
In-order to improve generalisation, we came up with
an augmentation technique inspired by RandAugment
[4]. RandAugment provided one of the highest boost
in AP on the COCO [
using this technique, we achieved a significant increment in performance for Task 3. Using an ensemble of the improved Faster R-CNN and RetinaNet module along with the augmentation techniques, we achieved the third position in the leaderboard with respect to the mAPg metric. We also demonstrate in Section 4.3 that our model surpasses the results obtained by methods used in the previous EAD2019 challenge. More details regarding the augmentation technique are provided in Section 3.3.
For the second task, we use a U-Net [
For the EAD 2020 challenge, two types of data-sets [
The data-sets for this task were provided in three phases. There were eight classes of artefacts i.e. specularity, bubbles, saturation, contrast, blood, instrument, blur and imaging artefacts. In the first phase, we received 2200 endoscopic frames. In the second phase, we received 99 frames and in the third phase, we received five sets of sequential frames, totally adding to 232 in number. Table 1 provides the classwise split of the total number of artefacts present in the total data provided.
From the table, it is can be inferred that there is dataimbalance between the classes. Specularity has the highest number of instances equal to 11856 whereas blood has the lowest number of instances equal to 528. Also, one more challenge observed with respect to this data-set is the nonuniformity of the image size/aspect ratio across the training
The data-sets for this task were provided in three phases as well, with the first release having 474 samples, followed by the second release of 70 samples, capped off with a final release of 99 samples amounting to a total of 643 instances. With segmenting out the artefacts from images as the goal, each instance had an RGB Image of an arbitrary size paired up with a corresponding five-channel TIF mask file. The five channels from the masks represented Instruments, Specularity, Artefact, Bubbles and Saturation in that order. Each image had the possibility of overlapping masks.
Faster R-CNN module is a two-stage object detector containing a backbone with a feature extractor and a prediction module. Fig 1 shows the different components of the Faster RCNN module. We discuss below the improvement made to the Faster R-CNN module to boost the performance towards the tasks.
Backbone networks are used for the low-dimensional representation of input data. Usually, they are fully convolutional layers. The choice of a backbone network is crucial in determining how well the input data, in our case an image is encoded into a low-dimensional space. Typically, a stronger backbone network extracts effective features from the input image which leads to better accuracy of the output predictions.
In the EAD2020 train data-set (as extension of EAD2019),
as observed, the class objects can be very small in size and
difficult to differentiate from the background [
Given the complications of our data-set, we choose ResNet-101 having a cardinality C of 64, i.e. a ResNeXt101 with a cardinality of 64. Fig 2 shows the building block of the ResNeXt-101 architecture. Inception based modules are based on a similar concept to the ResNeXt modules and achieve very good results. But it has complications with respect to hyper-parameter tuning. The filter size and strides need to be tailored for each stage and it is unclear as to how to adapt the architecture to new data-sets.
FPN constructs an image pyramid by fusing intermediate layers from the backbone network. It is a top-down pathway consisting of lateral connections so the network efficiently constructs a rich, multi-scale feature pyramid from a single resolution input image. Since, the data-set in our case consists of both small and large-sized objects and also objects which are difficult to detect, using an FPN module builds a highlevel semantic representation of the input image at both high and low resolutions, which helps for better predictions. To achieve this, we build an FPN module on top of the ResNeXt101 backbone. We construct a pyramid with levels P3 through P7, where l indicates the pyramid level (Pl has resolution 2l lower than the input). Also, all pyramid levels have 256 output channels in concordance with [3].
The output prediction module consists of two sub-components, the RPN head and the bounding box prediction module. The RPN-head proposes regions of interest from the intermediate feature representations coming from the FPN module. The bounding box prediction module is again a convolutional neural network. We chose to go with the standard modules for the RPN and the bounding box neural network as stated in [1].
The technique being employed for the Semantic
Segmentation task was the U-NET Architecture Autoencoder [
I oU =
T P (T P + F P + F N )
To prepare the data for training, the images and masks are
scaled down to the uniform size of 256x256. Then, split in
the ratio 80:20 of training to test data, followed by the
aforementioned augmentations applied at random to both pools.
We were able to accomplish this by making use of the inbuilt
Keras [
The out-of-sample generalisation task requires us to detect artefacts from images different from what was obtained for training purposes. This requires that the neural network does not get over-fit to the training data while maintaining its performance on the test data.
Data-augmentation is a standard technique which
performs label-preserving transformations to the input data-set.
It is a well-known fact that data-augmentation improves
the generalization capability of the deep neural network.
Choosing the right transformations and adopting an effective
methodology for implementing the transformations is crucial
to achieve a high generalization score. Recently, a lot of
research has gone into identifying effective data-augmentation
methodologies. Cubuk et.al. [
The main idea is to randomly select N transformations
from a total pool of T operations and apply it to the image
sequentially with a magnitude M that can be varied in the range
[1; 10]. This algorithm requires less computational power to
implement and surprisingly achieved similar results as [
To further increase the generalisation capability, we
created an ensemble of the improved Faster R-CNN module from
Section 3.1 along with a RetinaNet [
The results from the trained model on the test data-set are discussed next. The models were trained on NVIDIA GTX 1080Ti and RTX 2070 GPU’s.
The test data-set was provided in two phases. The first phase contained 150 images and the second phase contained totally contained 317 images including the first phase images. As discussed in Section 3.1 we use an improved Faster R-CNN module for this task. To establish a comparison of our backbone with other standard backbones we trained the same Faster R-CNN with different backbones like ResNeXt-50 and ResNet-101. We compared the performance of the backbones by using only the first phase of the test data-set. Table 2 summarises the results. It can be observed that the modified ResNeXt-101 algorithm gives the highest Scored score over other backbones, thus demonstrating its effectiveness over other standard backbones.
Also, to establish a comparison with other state-of-the art object detection techniques, we also train a Cascade R-CNN and a RetinaNet detector with the same configuration of the ResNeXt-101 backbone. The training set images were resized to (1300,800) and the learning rate was set to 0.01. We also applied the augmentation techniques discussed in Section 3.3. We stopped the training after 12 epochs because the models showed signs of over-fitting. Table 3 summarises the obtained results on the final test data-set. It can be observed from the results that Faster R-CNN performed way better than the RetinaNet and Cascade R-CNN. However, Faster R-CNN model showed higher error of 0:1076.
Faster R-CNN RetinaNet Cascade R-CNN 0:1869 0:1725 0:1686 0:1076 0:0989 0:0907
As discussed in Section 3.2, we used the U-NET Architecture
with different backbones to train on our augmented data. All
models were trained for 150 epochs and implemented using
the Segmentation Models [
We used the best performing Model on the validation data-set (U-Net with the ResNext50 Backbone) on the test Dataset which had 162 samples for our submission. The results are shown in Table 5.
The test data-set for this task consisted of 99 frames. As discussed earlier in Section 3.3, we implemented the augmentation technique and produced meaningful label-preserving images. To demonstrate the effectiveness of the augmentation technique, we compare the performance of the improved Faster R-CNN model trained using the augmented images with the model trained without using the augmented data. Both the models were tested on the first phase of the test dataset to establish comparisons. Table 6 summarises the results. It can be clearly observed that the model trained using the augmented images easily surpasses the model trained without the augmented images by achieving a difference of 5.9% on the mAPg metric.
We created an ensemble model of Faster R-CNN and RetinaNet to boost the performance for the generalisation task. To prove the effectiveness of the ensemble model, we provide comparisons with non-ensembled models. As shown in Table 7, the ensembled model achieved a mAPg of 0.2620 and a devg of 0.0890. Even though the Faster R-CNN module achieves the lowest devg score, the mAPg score is higher for Fig. 5. Sample Result obtained from the U-NET Model for the Tool Detection Channel of the Semantic Segmentation Task
With the augmented images Without the augmented images mAPg 0.2583 0.1987 devg the ensemble based model. This score is recorded as the third highest in the leaderboard.
Also, we compare our entire methodology (Augmentation
+ Ensembling) adopted for this task, with methods used in the
previous EAD2019 challenge. Mohammad et.al. [
In this paper, we discuss the methods used for the EAD 2020 challenge and also present the corresponding results obtained. For the first task we used a improved Faster R-CNN module with a powerful backbone and a FPN module. For the second sub-task we used a U-Net architecture by modifying the loss function. For the third sub-task, we designed an augmentation technique inspired by RandAugment [4]. We also used an ensemble of Faster R-CNN and RetinaNet to further boost the results.
We demonstrate that the modified ResNext-101 backbone achieves better results than the standard backbones. For the first task, we achieved a Scored of 0:1869 0:1076. We also compare the results against other state-of-the-art techniques. For the second task, we achieve a sscore of 0.5187 and sstd of 0.2755. For the third task, we first demonstrate that the model trained on the augmented performs better than the model not trained on the augmented images. Then, we demonstrate that the ensembled based model performs better than non-ensembled models. Lastly, we show that our model achieves the third position in the leaderboard with respect to the mAPg metric and that it performs better than a few models used in the EAD2019 challenge. We believe more research can be done towards identifying better semantic segmentation algorithms and better ensembling techniques in the future.
We thank the EndoCV2020 organisers for the opportunity. We would also like to extend our thanks to Shanthanu Chakravarthy, Raghu Menon and Varun Seshadrinathan from Mimyk team for all the support during the competition.
[1] Ren et.al. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS), page 9199, 2015. [2] Saining Xie, Ross Girshick, Piotr Dolla`r, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431, 2016. [3] Tsung-Yi Lin et.al. Feature pyramid networks for object detection, 2016. [4] Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical data augmentation with no separate search. ArXiv, abs/1909.13719, 2019.
Keras.