This paper presents the submission of team F-HOST for the Medico: Transparency in Medical Image Segmentation task held at MediaEval 2021. We propose a U-Net-based ensemble model for solving the automatic polyp segmentation task and interpret the predictions using a specific method for obtaining uncertainty. Our predicted segmentation masks show a mean Dice score of 45.01% based on the test data. The corresponding uncertainties show systematic errors towards the training data, which indicates overfitting.
I
Polyps are abnormal growths inside the lining of the colon or
rectum. They can potentially develop into being malignant, leading to
colorectal cancer, and thereby act as a precursor for cancer.
Detecting and removing polyps with colonoscopic polypectomy during or
before further development, will allow for more treatment options
and overall improved prognosis [
Currently, the gold standard of finding and removing polyps
is through a procedure called colonoscopy. This procedure is
dependent upon diferences in skill, experience, and technique of
the endoscopists. However, studies show that up to 28% remain
undetected [
In this section, the approach to the Medico task "Transparency
in Medical Image Segmentation" of the MediaEval 2021 challenge
is presented. All models were trained using the PyTorch
framework [
II.1
There is a total of 1, 362 images in the development dataset [
II.2
We used the U-Net architecture as the base model for the Deep
Ensemble, with a total of five U-Nets. The development data was
resized into 256 × 256 pixels before training, due to memory
constraints and to reduce training time. The training data was split into
batches of 32 images in order to obtain greater training eficiency
as opposed to a larger batch size of, e.g., 64. Data augmentation was
performed on the fly for each training iteration in order to obtain
improved generalization. We use techniques such as blurring, color
jitter, horizontal flip, random rotate 90◦ , and vertical flip. Instead
of using transposed convolution in the decoder part of the network
as proposed in the original U-Net paper [
When performing prediction with the deep ensemble, each individual model is loaded, and each predict on the input image from the test dataset. The element-wise mean is calculated from the output from each of the models in the ensemble. They are later pushed through a Sigmoid activation and thresholded into binary pixel values. The variance provided by the ensemble is used as an approximation for the uncertainty of each prediction mask. This is calculated by taking the squared sum of each probability prediction (Sigmoid output) minus the mean probability prediction from the ensemble. This squared sum is later divided by the number of models in the ensemble, five in this case.
For subtask 2: "Algorithm Eficiency", the time in seconds was calculated for the ensemble to make its overall mean prediction for each of the test images in order to measure the model eficiency of the ensemble. A Docker image is made, and using this image will make a .csv file with the image name and its corresponding prediction time in seconds. The Deep Ensemble will be run on the challenge organizers’ hardware, and they provide us with the frames per second (FPS), which is the average number of masks from the test dataset the ensemble is able to make per second.
For subtask 3: "Transparent Machine Learning Systems", all source code is made publicly available on GitHub1, which also includes the uncertainty images for the prediction masks. III
Table 1 summarizes the results for the Medico subtask 1,
including the mean DC, mean JI and mean Accuracy for the prediction
masks on the validation data and the oficial task test data. These
results show that the Deep Ensemble generalize poorly onto the
test data, with a decrease of approximately 55% in the DC score
and 46% decrease in the mean JI when comparing the results from
the validation data on the test data. There is a high variance of DC
score in the individual images from the test images, some get a DC
as high as 0.8935, whereas some images get as low as 0.0000. Higher
performance can be increased by performing more hyperparameter
tuning, training the Deep Ensemble on more training examples
including similar datasets such as for example the CVC-ClinicDB
dataset [
For the eficiency subtask, a FPS of 82.9496 was obtained. This means that the time of approximately 2.4111 seconds in total was used to generate the masks on the entire test dataset. This result indicates satisfactory model eficiency, but in return the deep ensemble is both memory- and time consuming to train.
A set of three randomly chosen images from the test data and their corresponding prediction masks and uncertainty heatmaps are shown in Figure 1. The brighter areas in the heatmaps illustrate the pixels where the models in the ensemble disagree the most. These results show that the borders of the detected polyps are where they disagree the most. Furthermore, the two uncertainty heatmaps (from the left) shows an outlining of a rectangle in the bottom left corner. Many of the input images in the HyperKvasir dataset show green rectangles located in the same area, this is information important to the medical experts. Thus, it is common to observe several images with green rectangles in the development dataset. However, note that the input images do not contain these green rectangles. These results indicate that the ensemble expected these rectangles, thus showing systematic bias towards the training data. Increasing the number of training examples, as well as performing 1https://github.com/feliciajacobsen/MediaEval2021 corrections to training images where these rectangles appear by, e.g., cropping them out may boost model performance.
In this paper, we presented a method of obtaining the approximate uncertainty values for a set of predicted segmentation masks. The uncertainty masks provide an uncertainty measure of the performance of a U-Net based DL model trained on medical colonoscopy images of polyps.
A mean Dice score of 0.4501 was obtained on the test data, and
compared to the Dice score of 0.8226 from the validation data, this
indicated that the Deep Ensemble model was being overfitted to
the training data, and thus generalizing poorly onto unseen data.
Increasing the number of training examples by including similar
datasets, decreasing the number of training iterations, increasing
the number of models in the ensemble, as well as including
adversarial training may improve generalization. A total average FPS of
82.9496 was obtained on the test data, but came at a high
computational cost when training the Deep Ensemble. In future work, we
will add the aforementioned proposed extensions, as well as
experiment and compare to alternative methods such as Masksembles [