Endoscopy Artefact Detection and Segmentation (EAD2020) includes 3 sub-tasks: Multi-class artefact detection, Semantic segmentation and Out-of-sample generalisation. This manuscript summarizes our solution. The challenge can be considered as two independent problems: object detection and semantic segmentation. For the detection problem, we use Cascade R-CNN with FPN and Hyper Task Cascade. For the segmentation problem, we use DeepLab v3+ model with bce+dice loss.
Endoscopy is a widely used clinical procedure for the early detection of numerous cancers. However, a major drawback of these video frames is that they are heavily corrupted with multiple artifacts. Thus, accurate detection and even segmentation of artifacts are very helpful to improve the endoscopy tools. This task aims to localise bounding boxes, predict class labels and pixel-wise segmentation of 8 different artifact classes for given frames and clinical endoscopy video clips.
The details of Endoscopy Artifact Detection and
Segmentation Dataset are described well in the original papers [
We combine the two phases of the dataset together. As shown in Table 1, the distribution of different classes is very imbalanced. The counts of ‘blur’, ‘instrument’, and ‘blood’ are significantly smaller than others, which could become hard examples when training models. The counts of ‘specularity’ and ‘artifact’ are very big and the objects of them are very small in size. Based on the condition, we pay attention to the balance of each class when we divide 20% of data as validation set.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
specularity saturation artifact blur
36.2% 4.7% 29.6% 2.5%
contrast bubbles instrument blood
6.1% 17.3% 1.7% 1.8% fg/Pfg 36.39% 10.69% 9.33% 20.40% 23.18% fg/(fg+bg) 4.13% 1.21% 1.06% 2.31% 2.63%
Many ground-truth pixel values are between 0 and 255 in the dataset. After dividing all ground truth pixel values by 255 and using a threshold of 0.5 to classify foreground and background pixels, the results are shown in Table 2. Foreground pixels are significantly fewer than background pixels. The foreground pixels of different classes in ground-truth images are imbalanced as well. As shown in Table 3, the most common size of images is 512 512 and 1349 1079. The others contain different sizes. We shuffle the dataset randomly and use 20% of the data as the validation set.
We use Cascade R-CNN [
In training data, we perform random flip, normalization and resizing. The images are resized to 512 512.
We extract candidate bounding box with RPN (region
proposal network) , and use non-maximum suppression (NMS)
to filter the useful bounding-box. Observing that some small
objects are ignored, NMS threshold is increased from 0.7
to 0.8. It slightly improves the recall rate and mAP.
SoftNMS [
We use SGD optimizer with a momentum of 0.9 and a weight decay of 0.0001. In order to get better results when convergence, we add a warm-up period to make the training rate linearly increase to 0.0025 in the first 500 iterations. The network is trained for 13 epochs totally.
We use DeepLab V3+ network [
The output of the network is activated by sigmoid function
to get the probability map, since there may be overlap among
different channels of the mask. The segmentation problem is
considered as multiple binary segmentation tasks.
3.2.2. Loss
We evaluated different losses, including Binary Cross
Entropy, Dice Loss, Lovsz-Hinge Loss [
ygt) log ypred + 1 (2 P ygt ypred + P ygt + P ypred + ) ( = 10 7, ygt and ypred are flattened tensors)
We apply random brightness and contrast changes, random
horizontal and vertical flip, random shift scale rotation,
Gaussian blurring, resizing and normalization to images of the
training set. All random transformations are applied by a
probability of 0.5 with the default parameters of
Albumentations library [
The images are resized to 512 512 and 1024 1024 during the training phase, see 4.2.
We load the weights pre-trained on the ImageNet for the
backbone network. The network is trained using SGD with a
momentum of 0.9 and a weight decay of 0.0001. We train the
model using mini-batches of size 4. The learning rate is
increased linearly over the warm-up period of 5 epochs, to the
maximum value of 0.01, then adjusted by cosine annealing
with warm restarts [
The results of Table 6,7,8 are provided by the official leaderboard. Table 6 shows the detection scores in the first phase of test data. The Hybrid Task Cascade network performs better in mAP, while getting a lower score in IoU.
Table 7 shows the scores in the final test. We get a higher detection score with the Cascade R-CNN network.
As shown in Table 8, resizing the image to 1024 1024 instead of 512 512 doesn’t give a better score but contributes to generalization performance.
To evaluate the results of different losses, we train a DeepLab
V3+ model with ResNet101 backbone and a modified
UNet [
The threshold to predict foreground pixels is 0.5. Other configurations are the same as 3.2.4. In Table 9 and Table 10, ‘bce is Binary Cross Entropy loss, ‘dice is Dice Loss, ‘bce+dice’ is defined in 3.2.2. All ‘p’ and ‘r’ in the tables stand for precision and recall.
In Table 9, the experiment shows that bce+dice gets the best score in Dice, F2, and IoU score. The precision of bce+dice is pretty close to dice, while not losing much recall. In Table 10, we can see a significant improvement of UNet using bce+dice, showing the effectiveness of this loss.
After 300 epochs, the DeepLabV3+ model using bce+dice got 0.7927 in F1, 0.8386 in F2, 0.6857 in IoU, 0.7422 in precision and 0.887 in recall. The U-Net models don’t get much better scores as they almost converge after 165 epochs.
We also tested Lovsz-Hinge loss. In our test, it is hard to converge if the model is trained from the ground up. Hence, we use Lovsz-Hinge loss to fine-tune the Deeplab model trained with bce+dice for 300 epochs. Table 11 shows the results of the first 20 epochs (Epochs means the training epochs with Lovsz). This model converges after 30 epochs but these results are worse than the model before fine-tuning, so we give up this method.
We choose bce+dice to train the final model.
Table 12 shows another experiment to compare different networks. We find that Xception-based DeepLabV3+ converges significantly slower than ResNet101-based model, and does not get better scores than the U-Net model.
F1 0.4189 0.5823 0.5535
The training parameters are listed in 3.2.4. All the results above are provided by the official leaderboard. In Table 13, Model 1 is trained with 512 512 images and a threshold of 0.5. Model 2 is the same as model 1 except changing the threshold to 0.7. Model 3 is trained with 1024 1024 images and a threshold of 0.7.
We resized the image to 512 512 at first. However, as discussed in Table 3, there are many bigger images. This can be found in the first phase of test images as well. Compared with the models only trained with images resized to 512 512, models trained with 1024 1024 get better scores in the validation set. Some predictions look smoother, as shown in Figure 2,3,4.
We find that adding segmentation data of EAD2019 to the training set also helps a little, although there is potential validation data leakage, making validation metrics unbelievable. However, it does not help in the detection task.
We chose Model 3 to predict the final test data and got scores as Table 14 shows.
In task 1, we compare Cascade R-CNN with Hyper Task Cascade to get a better detection model. FPN and Soft-NMS are used to improve the detection precision due to class imbalance. A proper threshold of NMS is helpful to improve the recall rate of small objects.
In task 2, we select DeepLabV3+ to solve the problem. We select bce+dice as the loss function to balance precision and recall. Image sizes of the dataset is a noticeable part at the training phase. Adjusting the threshold of predicting also contributes to a more balanced model.
[1] Sharib Ali, Felix Zhou, Christian Daul, Barbara Braden, Adam Bailey, Stefano Realdon, James East, Georges Wagnieres, Victor Loschenov, Enrico Grisan, et al. Endoscopy artifact detection (ead 2019) challenge dataset. arXiv preprint arXiv:1905.03209, 2019. [2] Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden, James East, Xin Lu, and Jens Rittscher. A deep learning framework for quality assessment and restoration in video endoscopy. arXiv preprint arXiv:1904.07073, 2019.