Accurate detection of artefacts is a core challenge in a widerange of endoscopic applications addressing multiple different disease areas. Our work aims to localise bounding bboxes and predict class labels of 8 different artefact classes for given frames and clinical endoscopy video clips. To solve the task, we use Cascade R -CNN[1] as network architecture and adopt ImageNet pretrained ResNet101[2] as backbone with Feature Pyramid Network (FPN) [3] structure. To improve the network performance, methods like data augmentation and multi-scale are also be adopted. In the end, we analyze the major challenge of the task.
Endoscopy is a widely used clinical procedure for the early detection of numerous cancers (e.g., nasopharyngeal, oesophageal adenocarcinoma, gastric, colorectal cancers, bladder cancer etc.), therapeutic procedures and minimally invasive surgery (e.g.,laparoscopy). However, video frames captured by an endoscope usually contain multiple artefacts, which not only present difficulty in visualising the underlying tissue during diagnosis but also affect any post analysis methods required for follow-ups. Existing endoscopy workflows are not competent qualified for restoring high-quality endoscopic frames because they can detect only one artefact class in most cases. Generally, the same video frame can be corrupted with multiple artefacts, e.g. motion blur, specular reflections, and low contrast can be present in the same frame. Besides, corruption varies with video frames in artefact types. Therefore, improving detection accuracy is a core challenge in a wide-range of endoscopic applications.
Recently, deep ConvNets have significant improved
image classification and object detection accuracy[
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). network architecture is it achieves state-of-art detection performance.
The 8 artefact classes in the dataset for “Endoscopic
Artefact Detection” include specularity, specularity saturation,
artifact, blur, contrast, bubbles, instrument and blood. The
visualization of ground truth bboxes are shown in Fig 2. The
artefact detection task will be evaluated based on the results of
the test dataset provided from a subset of the data collected for
training. Specifically, the training dataset for detection
consists in total 2200 annotated frames over all 8 artifact classes
and test dataset 100[
The model architecture is shown in Fig 1. We use Cascade
R-CNN[1] as network architecture and adopt ImageNet
pretrained ResNet101[2] as backbone with Feature Pyramid
Network (FPN)[
For data augmentation, each image will be horizontally
flipped with a 50 percent chance. We replace the nms
operation with the sof t-nms[
In the experiment, we find that specularity, artifact and bubbles are hard to classify. A probable reason is these three artefacts have similar appearance (e.g. Some of them all appears as spots of light). To solve this problem, we modify the loss function. In specific, we up-weight loss when model mistakenly classify these three artefacts. The result turns out to be an improvement of AP for these three artefacts but a decline of mAP.
We randomly divide the data provided into 5 subsets and use one of them for validation while others for training. The following metrics are based on the validation set.
We report our results obtained from baseline in Table 1. Our baseline achieves 0:26 mAP. To improve the model performance, we added image resizing operation to the data augmentation pipeline. Specifically, each image will be randomly resized among the range from (512, 512) to (1024, 1024) with the same aspect ratio as the original. Considering the image size varies, we believe this operation will be effective.
The results are shown in Table 2. According to the results, we argue that resizing operation can obvious improve the model performance with an increase in mAP of 0.017. We notice that the improvement is mainly on APsmall. The main reason is that in most cases the resizing operation enlarges image size and thus makes it possible to detect more small objects.
Note that the scales of test images are often larger than train images, i.e. images in testset often with height and width larger than 1000 while images in trainset around 500, so the resizing operation can solve the scale mismatch problem between training images and testing images.
In the experiment, we found that network has some difficulties in distinguishing classes among specularity, artifact and bubbles. To demonstrate the problem clearly, we calculated the confusion matrix, which is shown in Table 4. According to the Table 4, the network has two drawbacks. Firstly, the network tends to confuse specularity, artifact and bubbles in the classification procedure. Secondly, the network has poor performance in detecting blur.
To solve the first problem, we modified the loss function. Specifically, we increased the loss weights to the misclassification of specularity, artifact and bubbles. The result turned out to be an improvement of AP for these three artefacts but a decline for mAP.
To find out what kinds of artefact our model can successfully detect, we show some qualitative results in Fig 3 and Fig 4. The qualitative results indicate a). for artefacts with not so small size, our model tends to generate accurate detections; b). more artefacts in an image lead to more difficulties in detecting; c). our model generates a fair number of true negative blur. We are not sure the reason for problem c) mentioned above is whether the shortcomings of the model itself or the absence of annotation blur, because the corresponding images show blur characters.
to 300. Then we used the model to obtain the testset results and the performance is shown in Table 3.
In our work, we found the major challenge in “Endoscopic Artefact Detection” task is the difficult classification among specularity, artifact and bubbles. One intuitive explanation is that some of them all appears as spots of light, sharing a high degree of similarity. In the future, we intend to train 3 separate classifiers for these 3 artefacts and adopt more advanced feature extraction networks, which may solve this challenge to some extent. Boxes ensemble method was performed in our experiment. However, it seemed this method caused lower mAP.
To sum up, we constructed a Cascade R-CNN based model to solve the “Endoscopic Artefact Detection” task. We adopted several methods to improve the network performance, including data augmentation, modifying loss function and boxes ensemble. We also identified the major challenge in this task.
We added image resizing operation to the data augmentation pipeline and fine-tuned the maximum box number per image [1] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018. [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision