Detecting artefacts in video filmed in endoscopy is an important problem for downstream computer-assisted diagnosis. When tackling this problem, one challenge is that the size of an artefact varies in a wide range. The other challenge is that labeling endoscopic images is labor- extensive and is hard to outsource the labeling task to untrained people without the aid of doctors. In this report, we demonstrate how the performance of a Faster R-CNN model can be improved by scaling an image to the right scale before training and testing. The training method overcomes the issue that a convolution neural network trained on one scale barely works when detecting the same category of objects on a different scale. The method is totally independent of the model and can be easily adapted with other models. Besides, it saves time and memory by focusing on the patches that include objects when training the model. The source code? for this report will be made public upon the publishing of my solution.
Endoscopy is a widely used clinical procedure for the early detection of numerous cancers (e.g., nasopharyngeal, gastric, colorectal cancers, bladder cancer etc.), therapeutic procedures and minimally invasive surgery. Video taken by the camera of an endoscope is usually heavily corrupted with multiple artefacts (e.g., pixel saturations, motion blur, defocus, specular reflections, bubbles, fluid, debris etc.). Accurate detection of artefacts is a core challenge in a wide range of endoscopic applications addressing multiple different disease areas. The importance of precise detection of these artefacts is essential for high-quality endoscopic frame restoration and crucial for realizing reliable computer assisted endoscopy tools for improved patient care.
In the last few years, convolution neural network (CNN) has outperformed previous non-CNN based methods in solving the object detection problem. The dominant CNN-based methods fall into two categories, one-stage approach and twostage approach, with the former method shining in speed and the latter in accuracy. These two methods meet the demand in different fields. For example, in the field of self-driving car, speed is a prerequisite given an acceptable detection performance as a self-driving car has to react instantly. In the case of diagnosis in biomedical engineering, we can bear with slightly more computation time for higher accuracy. Since the advent of R-CNN [1], which is a two stage approach, this region-based detection method has become increasingly mature. Along the line, Fast R-CNN [2] introduced a RoI pooling operation that does forward pass on all the object proposals in an image simultaneously. Faster R-CNN [3] further speeds up R-CNN by training a region proposal network (PRN) using the feature maps generated by the convolution operations at the low level, without introducing much cost. Thus, I chose Faster R-CNN as the base framework in this challenge.
However, two challenges have to be solved when developing the model. One special challenge is that the size of the artefacts varies in a wide range and the other one is a limited number of labeled images (2,192 in total). The scale-related challenge is associated with the architecture of a CNN. The low level feature maps of a CNN capture features like edges and have a small receptive field, whereas the high-level features capture more semantic features, and have a larger receptive field [4, 5, 6]. Thus, the high-level features of small objects (e.g. less than 32 pixels) get mixed with features for background or objects nearby if the features do not disappear due to dimension reduction caused in feature extraction. e.g. For a feature stride of 32, the highest level features were shrunk 32 times compared to the raw image. For very large objects, the deeper layers suffer from extracting high-level semantic features due to failing to integrate low-level features given a limited feature stride.
To alleviate the problem caused by the wide range of
object size, various solutions have been proposed. One category
of solution focused on designing new CNN architectures to
exploit the features at different levels. Under this paradigm,
SSD [
In this study, we demonstrated the successful application of the idea of detecting objects of various size at the right scale in detecting artefacts in endoscopy. The report is organized in such an order: datasets, methods, results, discussion and conclusion.
The training dataset consists of 2,192 endoscopic images (Fig.
1 A), in which seven categories of artefacts were labeled [
The model we built is a Faster R-CNN with a FPN as the
backbone. An FPN consists of mainly two parts, an encoder
and an decoder, which is very similar to a U-Net [
When training the model, we adapted the method
proposed in [
When cutting a patch of the image (Fig. 2 B, step 1), the size of the patch and location of the patch was jittered, which allows us to generate not exactly the same patch every time even though the patch with the same object is selected. Note that the size of the patch is always larger than the object and a larger patch was cut if a smaller object exists in that patch. Otherwise, if the object were always in the center or same location in the patch, the model would not learn to detect objects but learn to localize objects assuming there is always an object, which is not true. The setting for the size of a patch (spatch) is defined by this equation: spatch = r sbbox, where r = 4.5, 2, 1.5, and 1.2, respectively, for the cases, sbbox < 80, 160, 350, and > 350. If no object exists in a random patch, a fixed size of patch was cut from an image.
After cutting a patch from an image, the patch was scaled up or down depending on the size of the object in that patch (Fig. 2 B, step 2). The scaling provides a zoomed-in view for small object objects and a zoomed-out view for large objects. Thus, both high-level and low-level feature maps will exist after an image passes a CNN. The scaling ratio (r) is inversely proportional to the size (s) of the object in the patch: r = 160/s and the size of all the objects are grouped into 6 bins. So the settings used here are (r=4, s < 40), (r=2, s¡80), (r=1, s < 160), (r=0.5, s < 250), (r=0.25, s < 640), and (r=0.13, s > 640). The reason for choosing such settings is that there will be 5 pixels in the last layer of the encoder if a raw input of 160 pixels is fed into the model, which has a feature stride of 32. A patch, which has no objects in it, was scaled with a random ratio on the fly.
After scaling up or down the patch cut from each image in a batch, objects that are too large/small were excluded. The thresholds for too small and too large objects are 32 and 2000, respectively. Choosing 32 as the threshold is because of the feature stride of the model is 32 and choosing 2000 is just because it is large enough.
In each training iteration, multiple patches from multiple images were cut and normalized as a batch by padding the patch with with the channel mean and concatenated, resulting in a batch of images, whose width and height are a multiple of 32 (Fig. 2 B, step 3). For a patch that is already a multiple of the stride size of the encoder, no padding was added. Since the padding is always on the bottom and right side if necessary, the coordinates of the bounding box does not change. Collating multiple samples and unifying them in size can be easily implemented in Pytorch1. For other details like how the bounding boxes were adjusted accordingly when cutting a patch from an image, see our code on Github.
In inference, we tested one image on all the scales used in training (scales: 4, 2, 1, 0.5, 0.25 and 0.13). The coordi1https://pytorch.org bubbles Instrument artifact specularity specularity saturation artifact blur contrast bubbles instrument
C6000 5000 nates of all the detected objects were transformed back to the original scale. To remove redundant bounding boxes, nonmaximum suppression were conducted on all the detected bounding boxes for each category of object. It was observed that many false positive, which were small objects, were detected on the scale of 4, so I finally chose not to include the prediction on that scale.
The performance of our model was evaluated by a hybrid metric which was a weighted score of mean average precision (mAP) and Intersection over Union (IoU): 0.6 mAP + 0.4 IoU. We compared the performance of two ways for training the same model. One way is training the model on the whole image every time and the other is on patches generated following the method described above. The threshold for the probability when determining an object is 0.65. In the former case, the best overall score on the two testing sets was 0.221. For the latter case, we trained the network for 27,105 iterations (batch size is 8 in each iteration) and a significant boost in performance was observed. The score we reached was 0.293, which was among the top 10 teams on the leaderboard.
The training method boosted the performance of Faster RCNN in two ways. First, as we discussed in the Introduction section, it alleviates the scale variation problem by scaling up/down an object to the right scale to detect. Second, randomly cutting a patch which includes an object allows us to generate far more different training images compared to feeding the whole image to the model. Thus, cutting a patch serves as a data augmentation technique. Besides, it offers flexibility to deal with the class imbalance as we can choose which patch to cut from a training image, considering the distribution of the counts of all the classes.
Since the detected bboxes on all the scales were merged, a
bbox was called if it was detected on any of the scales. Such
an integration approach tends to report more false positive.
One failure case we observed is that a false positive object
does look like a true object because the model decides
without considering the context of that object. We run into the
case when an image is scaled up by 4 times. Thus the
context information does matter and a context refinement
probably corrects such kind of errors [
Balanced bounding boxes
True bounding boxes the scaled down images include more context information, we expect the problem to be solved in this way.
To further boost the detection performance, the regular
convolution operation in the FPN backbone can be replaced
by deformable convolution operation will enhance the
transformation modeling capacity of CNNs [
6. REFERENCES [1] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014. [2] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015. [3] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015. [4] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014. [5] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in neural information processing systems, pages 4898– 4906, 2016. [6] Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3578–3587, 2018.