Endoscopy is a widely used clinical procedure for the early detection of numerous diseases. However, the images produced are usually heavily corrupted with multiple artifacts that reduce the visualization of the underlying tissue. Moreover, the localization of actual diseased regions is also a complex problem. For that reason, EndoCV2020 challenges aim to make progress in the state-of-the-art in the detection and segmentation of artifacts and diseases in endoscopy images. In this work, we propose approaches based on U-Net and UNet++ architecture to automate the segmentation task of EndoCV2020. We use the EfficientNet as our encoder to extract powerful features for our decoders. Data augmentation and pre-trained weights are employed to prevent overfilling and improve generalization. Test-time augmentation also helps in improving the results of our models. Our methods performs well in this challenge and achieves a score of 60.20% for the EAD2020 semantic segmentation task and 59.81% for the EDD2020's.
Endoscopy is a widely used clinical procedure for the early
detection of cancers, therapeutic procedures, and minimally
invasive surgery in hollow-organs. Computer-assisted
methods would improve diagnosis and assist in surgical planning.
These endoscopic applications face two main challenges: 1)
Endoscopy video frames are usually heavily corrupted with
multiple artifacts that reduce the visualization of the
underlying tissue and affect post-analysis. Accurate detection of
these artifacts allows corrections of frames and is, therefore,
a core challenge in a wide range of endoscopic applications
[1]. 2) Even with uncorrupted video frames, temporally
consistently localizing and segmenting of disease ROIs is a
challenge due to non-planar geometries, variation in imaging
modalities, and deformations of organs. For these reasons,
after the success of EAD2019 [
Copyright c 2020 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
(a)
(e)
(b)
(f)
(c)
(g)
(d)
(h)
vision challenges on segmentation and detection 2020
(EndoCV 2020) are held to make progress in the state-of-the-art
further. It consists of two sub-challenges: Endoscopy
Artefact Detection and Segmentation (EAD2020), and Endoscopy
Disease Detection and Segmentation (EDD2020) [
In this paper, we introduce our works on these challenges.
First, Sec. 2 presents our observations of the datasets. Then,
we describe our methods in Sec. 3. We participate in all tasks
of both challenges but focus mainly on the two semantic
segmentation ones using fully convolutional networks such as
U-Nets [
The EndoCV2020 challenge consists of two datasets. The
EAD2020 dataset is an extended version of last year’s
EAD2019 challenge [
The ground-truth for these challenges contain some issues. They can be classified into annotator disagreement and systematic error.
Some annotator disagreement was spotted in the EAD2020 detection dataset. For example, an artifact region is identified in one frame but is not in the next frame, or one connected region is marked with two bounding boxes in one frame but only one in the other. There is also misclassified segmentation mask. We consider this type of error as noises in the dataset since we do not have the resource to make adjustments.
We corrected all the systematic errors that we noticed. There are some small anomalies in bounding box ground truths, likely due to rounding when converting from PascalVOC to Yolo format. There is also a line at the bottom of some EAD2020 segmentation ground-truths that do not correspond to any artifact.
We focus mainly on the two segmentation tasks with models in the U-Net family. In the case of disease detection, we will take advantage of our segmentation results. For the artifacts detection tasks, we train a separate YOLOv3 [7] network due to the difference in the number of classes (eight classes for detect and five classes for segmentation), and due to disagreement between EAD2020 segmentation and detection groundtruths (e.g, Fig 1(a) and Fig 1(b)).
3.1.1. Models The state-of-the-art of semantic segmentation are methods based on encoder-decoder architecture such as U-Net, U-Net++. These encoder-decoder architectures use skipconnections to combine low-resolution, semantically-rich at deeper feature maps with shallower, fine-grained ones to recover fine detail of region of interest.
Instead of using the original U-Net encoder, we use
EfficientNet [
We start our experimentations with the U-Net architecture and the EfficientNet B5. Subsequent tests show that a deeper encoder is not needed, so we transit to a larger decoder (UNet++) with smaller encoder (EfficientNet B2 and latter B1). An illustration of our networks could be found in Fig. 2
We speed up the training process with pre-trained weights. Although it was trained on ImageNet, which is a database of natural images, the pre-trained weights do improve the training and local validation scores.
Input images are resized (while keeping their aspect ratio) and
zeros-padded to fit into 512x512 pixels. We randomly apply
these augmentation techniques with 50% probability:
Rotation with random angle, RGB value shift, horizontal or
vertical flipping, random scaling, elastic deformation [
The semantic segmentation tasks are evaluated with four
metrics: F1 (i.e., Dice score), F2, precision, and recall. The
semantic segmentation score is the average of these four metrics
[
We train on 80% of the train set and use the other 20% for validation. We start by fixing the pre-trained encoder and train the decoder part for 80 epochs using Adam optimizer with a learning rate (LR) of 10 3. From the 41st epochs, we train the whole network, starting at LR=10 3 and decrease with a factor of 0.5 if the validation score does not decrease after 40 epochs. We trained for a total of 1000 epochs. The training is early stopped if the training score could not be increased after 88 epochs. We select the weights that maximized the score on the validation set for evaluation of the test set.
Test images are resized and zero-padded to 512x512. We keep the aspect ratio and do not enlarge small images. Since our prediction sometimes contains small holes which rarely appear in the EAD2020 train set, a small hole-filling operation is applied at the end of the pipeline. This hole-filling process did not make a significant improvement in the final score. (a) EfficientNet B1 encoder and extracted intermediate feature maps (Xi) (b) Building blocks (c) U-Net (d) U-Net++
The segmentation results could be further improved with
Testtime augmentation (TTA). This approach has been
demonstrated in the literature (e.g., in semantic segmentation of
brain tumor [
For this task, we will take advantage of our segmentation results. The bounding boxes of connected components (CCs) larger than 0.5% of the image area are presented as our detection results. This approach is not optimal because it cannot handle slit-and-merge of detection bounding boxes, that is the cases where a single CC is marked with more than one box, or a single box marks several CCs.
For reasons stated earlier, we have to train a separate detector
for this task. We choose YOLOv3 because this model’s
effectiveness has been shown in this type of data [1]. It is also
relatively faster than two-states detectors such as RetinaNet
[
We train our standard YOLOv3 on 416x416 inputs. We start by training the last three detection layers for 20 epochs at LR=10 2, then the upscaling part of the network for 40 epochs at LR=10 3. Finally, we train the whole network at LR =10 5 and reduce the LR by a factor of 0.5 if validation loss does not decrease after five epochs.
As presented in Sec. 3.1.3, the semantic segmentation are evaluated by the mean of F1, F2, precision, and recall. The detection tasks are evaluated by a combination of mean average precision (mAP), and intersection over union (IoU). However, how these metrics are weighted was not disclosed until after the challenge ends.
The evaluation was done online at https://endocv.grandchallenge.org/. It is divide into two phases. The first test-phase only evaluate 50% of the test data. The size of EAD2020 and EDD2020 test data is summarized in Tab. 1. The EAD2020 detection task includes an out-of-sample set, which contains images provided exclusively from the training or other test datasets.
The quantitative results of our models are presented in Tab. 2 and Tab. 3. Our segmentation models performed well with double the number of decoder filters, i.e., Dec(x; 2) becomes Dec(2x; 2), (-): information is not available because that method was not submitted to that test phase.
Full Test Set Recall Score Final Score and consistently on both test subset. In Tab. 2, we can observe that the hole-filling operator adds a small boost while TTA improves the final score significantly. Our best approach is U-Net++ with B1 encoder, ran with TTA and post-processed with holes-filling. We could also see that our detection model needs improvements.
In this work, we demonstrate the effectiveness of U-Net and U-Net++ with pre-trained EfficientNet backbone for the segmentation of disease and artifact in endoscopy images. Our experiments also show that TTA can provide better segmentation results compared to only predicting on original images.
For further works, we intend to improve the generalization of our segmentation and detection models by applying more data augmentation techniques and using synthetic data. We are also considering to improve further the decoder of our models with an attention mechanism.
[1] Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden, James East, Xin Lu, and Jens Rittscher. A deep learning framework for quality assessment and restoration in video endoscopy. arXiv preprint arXiv:1904.07073, 2019. [7] Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement.