Endoscopic video frames tend to be corrupted by various artefacts impairing their visibility. Automated detection of these artefacts will foster advances in computer-assisted diagnosis, post-examination procedures and frame restoration software. In this work, we propose an ensemble of deep learning object detectors to automate multi-class artefact detection in video endoscopy. Our approach achieved a mean average precision (mAP) of 0.3087 and an average intersection-over-union (IoU) of 0.3997 on the EAD2019 test set. This resulted in a final score of 0.3451 and the 3rd rank in the EAD 2019 object detection sub-challenge leaderboard.
Tissue characteristics of hollow organs, as well as the different instruments and illumination modes applied in medical endoscopy lead to a high number of artefacts that obstruct visibility. The frames produced during endoscopic interventions can be corrupted by bubbles, instruments and image deficiencies such as specular reflections, strong contrasts, saturated pixels, motion blurs, or other artefacts. These corruptions have a negative effect on both the live diagnosis and postintervention procedures. Successfully detecting the artefacts will thus assist endoscopic experts and will be the cornerstone of successful endoscopic video frame restoration.
Over recent years deep learning techniques have become
the state-of-the-art in medical image analysis and they have
in particular proved successful in related medical endoscopy
computer vision tasks, such as polyp detection [
Our method consists of an ensemble of seven RetinaNet architectures that vary in hyperparameters, backbone networks, transfer learning, data augmentation, and training subset used. The models are combined based on an efficient voting scheme.
The RetinaNet detector consists of a backbone network for
extracting a convolutional feature map and two subnetworks
that perform object classification and bounding box
regression via convolution. The classification loss is given by the
focal loss and the regression loss is given by the smooth L1 loss.
The sum of both these losses constitutes the overall loss that
is minimized during training. It is a one-stage method,
meaning that it does not require a region proposal module. Instead,
anchors at different scales and aspect ratios are densely
distributed across the image and they will all be classified by the
network. In order to construct a multi-scale feature pyramid
from a single resolution input image, the backbone network is
augmented by a feature pyramid network, FPN [
In our experiments, we used both VGGNet [
Focal Loss is an extension of the cross-entropy loss that uses
a weighting factor to prevent one-stage detection methods
being overwhelmed by the large amount of ‘easy’ background
examples. Typically, one-stage methods have around 100k
anchors per image. Most of these are background anchors that
are easy to classify and swamp the classifier, undermining its
ability to focus and learn on the harder, foreground examples.
This imbalance is countered by the addition of a weighting
factor (1 pt), which reduces the weight of easily classified
anchors and thereby shifts the focus onto harder examples. pt
is given in [
if y = 1 1; otherwise (1) where is a tunable hyperparameter that modifies the extent to which the loss function prioritizes hard examples. If = 0, then our loss function is equal to the cross-entropy loss and no priority is given to hard examples. If for instance = 2 and pt = 0:9 for a given anchor, then its contribution to the loss will be 100 times lower than for the standard cross-entropy loss. Our experiments (Table. 3) show that setting = 1:5 yields the best performance.
Besides the focal loss weighting factor, a further
weighting factor was applied. Correctly classified anchors are
weighted by and misclassified ones are weighted by 1
with 2 [0; 1]. t is defined analogously to pt. According
to [
The final -balanced version of the focal loss is given by: F L(pt) = t(1 pt) log(pt) (2)
In order to counter the variance in the network output and increase performance, we implemented an ensemble method. Let M be the number of models used in the ensemble method. Our final method used M = 7 models.
The single trained models were first ordered according to their individual test scores (Table. 5). We then iterated through the M 1 first models and compared for each model mi (i M 1) its bounding box predictions to all subsequent models mj (j > i) in a pairwise manner. For example with three models, m1 would be compared to m2 and m3. In the next iteration step m2 would be compared to m3 (Fig. 1).
Each prediction box from mi forms the root of a stack and boxes from mj can then be assigned to that stack. Whenever a bounding box from mj is assigned to the stack of a box in mi, it will be removed entirely to avoid assigning one box to multiple stacks. Having the most accurate bounding boxes as roots of the stack proved beneficial. Therefore it is important to first order the models according to their test score to achieve optimal results. Boxes are assigned to a stack based on the weighted sum of their combined average objectness confidence score and IoU with the root of that stack. Our overlap score combines the average confidence score and the IoU. Our experiments have shown that the overlap score computed from weighting the average score by 0.7 and IoU by 0.3 respectively yielded the best performance. Each time we compare a model mi to a model mj , we evaluated the overlap score between all of their bounding boxes and assigned boxes to each other in descending order of overlap score. For instance, if bounding box A from mi and bounding box B from mj have an overlap score of 0.92 and that is the highest score between all boxes from these two models, then B is assigned to the stack of A and B is no longer considered in future comparisons. We used a threshold overlap score of 0.46 to assign boxes to each other.
Given all the final stacks we compute their respective summed confidence scores. If this aggregated score exceeds a voting threshold of 1.68 (corresponding to an average score of 0.24 per model for M = 7), a final detection will be yielded from that stack (Fig. 3). The value 1.68 was found to optimize the trade-off between high mAP and high IoU. For each model we only considered detections with a confidence score greater than 0.2. Considering detections with a lower confidence score did not increase the performance but slowed down our ensemble method. The four corner points of the detection was then calculated according to a weighted (by the respective scores) average of all the bounding boxes from that stack. In order to reward the fact that a detection was confirmed by many models, we introduced a frequency factor of 0.03 that is multiplied by the number of boxes in that group and then added to the average score of the final detection. This added a slight improvement to our scores.
By visual inspection of the detection output we noticed that many bounding boxes were drawn around smaller bounding boxes of the same class label. As we deemed the smaller boxes to be superfluous, we added a post-processing step that removed final detections if another detection of the same class label with intersection-over-area (i.e. the ratio of intersection with a given box and its own ratio) > 90% was present. Thus, whenever a bounding was present and its area was more than 90% within another box of the same label, we removed the outer box. This improved our scores.
Another thing we observed is that there are frequently two or more detections of a different class that seem to overlap almost perfectly. In these scenarios we attempted to improve our score by removing the detection that had a lower confidence score, but this did not improve our score.
We found that our test scores were optimal if we used seven models in our ensemble method. These seven models were selected and designed with the aim of achieving high dissimilarity between the models and high individual performance. Specifically we created the seven single models using different data augmentation techniques, different CNN depths and different configurations of the loss function.
Unless otherwise specified, all of these models used the 50-layer version of ResNet or ResNet-50 as the feature extractor. Our ResNet-50 was not trained from scratch but uses weights pre-trained on the MS COCO dataset. Initially we used a version with pre-trained weights from the ImageNet1k dataset, but experiments showed that MS COCO weights yield a better result (Table. 4). The deeper backbone networks, ResNet-101 or ResNet-152 used in two of the seven models in the ensemble, were pre-trained on ImageNet1k. Unless stated otherwise, training batch size of 1 was used. The number of training iterations used differ between models and is generally derived from the validation scores and with the goal of increasing the diversity between the models.
In the following we provide the specifications of the seven different models used in our ensemble method. The models are denoted m1 to m7 in descending order according to their single model performance, Table 5.
m1 was our best performing model, where all
configurations were optimized to the best of our knowledge. Besides
changing the focal loss parameter from 2 to 1.5, thereby
slightly reducing the extent to which the loss function
prioritized hard examples, other parameters were mostly set as
specified in [
Next, we tried to experiment with how the classification and regression loss were combined to yield the overall loss. By either increasing the weight of classification or regression loss we aimed to shift the focus between both losses. Doubling the weight of the classification loss provided a good balance between high performance and dissimilarity and hence this weighting was introduced for m2. This model was trained for 81k iterations.
For the third model m3 we chosen a different in the focal loss. We wanted this model to focus more on harder examples and set = 3:5 to achieve this. If one box was classified with a confidence score of 0.9, this means this box would contribute 100 times less to the loss in m3 than in m1. The model was trained for 58k iterations.
In the dataset we observed that the endoscopic frames were exposed to different illumination modes leading to different colorings of the images. Hence, we added a data augmentation step that randomly adds values to the RGB channels for model m4. This was done in addition to the random geometric transformation applied in training all our models. For each epoch, there is a 1/9 chance for each image that a value between 50 and 200 was added to either of the RGB image channels. Training batch size was set to 4 and conducted for 10k iterations.
Model m5 was trained with a 101-layer ResNet backbone. While this model performed worse than ResNet-50 models, we added it to the ensemble under the assumption that deeper CNNs will discover more advanced features and therefore add to the diversity of the ensemble. The model was trained for 45k iterations.
Analogously to m5, we also added a model m6 with a 152-layer ResNet as the feature extractor. m6 was trained for 69k iterations.
The last model m7 was trained on a subsampled trainset. For this model we added Gaussian noise at a scale of 127.5 (note 8-bit image intensity values of 0-255). Analogous to m4, this augmentation step was added on top of the random geometric transformation and was applied to 1/9 of the images at each epoch. The model was trained using a batch size of 4 for 11k iterations.
Our dataset consists of the 2,193 endoscopic frames that were
released by the EAD2019 challenge [
Models were trained with the Adam optimization
algorithm [
During training of the RetinaNet framework, anchors were considered a true positive based on the correctness of the predicted class and their IoU with the corresponding groundtruth annotation. We experimented with using different IoU thresholds to consider an anchor as true or false, Table. 1. We found that a negative threshold of 0.4 and a positive threshold of 0.5 worked best. This means that anchors with IoU below 0.4 were considered as false, above 0.5 were considered true, and those in between were ignored.
As previously stated, we tried out different CNNs to use as the feature extractor in our RetinaNet framework. Even without pre-training on MS COCO, ResNet50 was the most effective and outperformed deeper ResNet models. 1.00 1.25 1.50 1.75 2.00 2.50
Tuning the focal loss parameters had the greatest effect on our
single model performance. Indeed, increasing or decreasing
enabled us to decide to what extent we wanted the model to
focus on hard examples. This was especially useful in our use
case as the data was both unbalanced and some of the classes
were much easier to detect than others. Using a value of 1.5
yielded the best performance for us. In the original paper [
As previously mentioned, using a ResNet-50 model pretrained on MS COCO improved our performance substantially compared to models pre-trained on Image1kNet.
The single model performance is summarised in Table. 5. We initially used 3 models in our ensemble method. By continuously adding models to the ensemble, our score kept on Model m1 m2 m3 m4 m5 m6 m7 improving until we reached 7 models. Thereafter the score decreased again. Our performance was partly increased by an optimized combination strategy. This was largely thanks to the introduction of the overlap score, which handled the way boxes were assigned to each other, and to the frequency factor used to compute a weighted average of bounding box positions. The post-processing step of removing boxes that encompass boxes of the same class provided an additional performance boost. Finally, through testing and optimizing various parameters of our ensemble method we reached our final, highest score. These parameters include the following: overlap score threshold, the weighting between IoU and average score in the overlap score, frequency factor, score threshold of each individual model, and the overall voting threshold for each detection stack. Table. 6 summarise the describe stepwise improvement in test score. Our proposed ensemble method achieved a final score of 0.3451 on the EAD2019 test set. Our mAP was 0.3087 and IoU was 0.3997. For this submission, mAP on the EAD2019 generalization set was 0.2848 with a deviation score of 0.0696. In a previous submission, with slightly different ensemble parameters and the introduction of class-specific voting thresholds, we scored 33.45 on the test set and a mAP of 0.3508 on the generalization set with a deviation score of 0.0556.
Fig. 4-7 depict outputs from our ensemble method and models m1, m4 and m5, respectively, on the same example image from the EAD2019 test set. The figures illustrate how different RetinaNet models are combined to produce a superior output.
Our approach tackles the novel issue of multi-class artefact detection in endoscopy by proposing the application of the one-stage detection method RetinaNet. RetinaNet matches the speed of other one-stage methods and its focal loss addresses the imbalance between easy and difficult examples. By intelligently combining multiple models that were trained according to the specific nature of endoscopic video frames, our score improved substantially and resulted in a EAD2019 object detection score of 0.3451.
Future exploration could include the implementation of more advanced backbone networks and/or more advanced transfer learning approaches, such as pre-training on medical images.
“Adam: A arXiv preprint