In this paper we describe our approach to gastrointestinal disease classification for the medico task at MediaEval 2018. We propose multiple ways to inpaint problematic areas in the test and training set to help with classification. We discuss the efect that preprocessing does to the input data with respect to removing regions with sparse information. We also discuss how preprocessing afects the training and evaluation of a dataset that is limited in size. We will also compare the diferent inpainting methods with transfer learning using a convolutional neural network.
Medical image diagnosis is a challenging task in the industry of
computer vision. In the last couple of years, as computing power
has increased, machine learning has become a tool in the task of
image detection, segmentation and classification. In this paper we
are looking in depth how to use machine learning to help solve
classification tasks on the data-set from the Medico task [
Similar to other parts of image detection, the Medico dataset
encounter the challenges that the amount of data is too small, or
that the training data does not cover the full distribution of the data
in the test case. The main goal of this task is to classify medical
images. Our proposal is to use unsupervised machine learning for
removal of the green corners that are in the Medico dataset. The
details of the task are described in [
Our approach is divided in to two steps: first preprocessing, then classifying. Our focus is mainly on the preprocessing of the data to remove the green corners in the medical images.
After the preprocessing the dataset we run it through a Convolutional Neural Network (CNN) based on transfer learning. We chose the CNN model based on the top 5 and top 1 accuracy of the pre-trained networks on the Keras documentation pages.
In our approach we use the InceptionResNetV2 [
The medical data has 1 main feature that we focus on during the preprocessing, namely the green square in the bottom left corner. A neural network often struggle with areas with really sparse information. Our hypothesis is that just replacing the green area with a similar black area will not yield a better result.
We have a dataset that we use as a base case. This dataset was not augmented, other than shrinking the size of every image to a ifxed resolution. The other datasets were augmented in a way that would cover up the green square in one way or another.
Our hypothesis it that if we recreate the areas as they would
look like without any sparse areas, the classifier can focus on the
right features for classifications. We propose 4 diferent methods
on how to inpaint the corner area of the medical images.
An autoencoder [
For the autoencoder approach, we created and trained a custom
autoencoder from scratch. Our autoencoder consist of a
encoderdecoder network, with 2D convolutions as well as rectified linear
units as activation functions. In the layer between the encoder and
the decoder we included a 25% dropout. [
To preprocess the medical data we feed the whole image through the encoder-decoder network. We take the loss of the whole reconstructed image, but only keep the inpainted part. Under training, the goal is to minimize the loss: L(x, д(f (x˜))) Where x is an image without a green corner, and x˜ is the same image with an artificial green corner. In theory we can replace any part of the image with this method.
For the context encoder approach, we created a new
encoderdecoder network. Here the encoder has a similar structure to the
autoencoder, but our decoder is only making outputs at the size of
the desired area to inpaint. In addition to the loss generated from
taking a MSE loss[
With the context encoder we feed images without a green corner in to the encoder-decoder network. The output of the network is the same size as the area we want to fill.
For the generative adversarial approach, we create a similar structure as the autoencoder. We have a constant 10% dropout at each layer in the discriminator. As with the autoencoder we have the same size input as output, but we only decide to keep the parts we want to inpaint.
We use the same type of loss as the context encoder, with 15% of the loss coming from a MSE loss, and the remaining 85% coming from the adversarial loss.
The last method was just to crop the images in a way that excluded the green corner. Since every image is scaled down to 256x256 px during preprocessing, the same is done with the clipped version (after the clip the size was reduced to 256x256).
The clipping was done in a way so that we had the most amount of center frame, and minimal amount of the bottom left corner, without sacrificing to much of the image.
We made the augmented datasets before we trained the preprocessing model. This means that the transferlearning model did not augment the images at runtime. We split the data into a 70% train set, and a 30% validation set.
Our results on the test set are tabulated in Table 1. The oficial Results on the test set are tabulated in Table 2. Table 3 shows the confusion matrix from the CC-GAN from the oficial test set.
The results show that the CC-GAN got the highest MCC score with 0.926, and also the most realistic inpaintings. The context encoder had the lowest MCC score with 0.920, and also the worst inpainted areas. The ocfiial result did have the same pattern in Method Autoencoder
CC-GAN Contextencoder
Clipping Non-augmenteted
REC MCC score, though the base case got the best result. In both cases the clipping gave significantly worse result.
As expected, most of the images was classified correctly, but we had some problems distinguishing between esophagitis and normal-z-line. We also had a few cases of instruments where there were none.
In general, when training on a dataset that is homogeneous, the preprocessing is less valuable. We want to remove areas with sparseness, and areas that has nothing to do with the classification. In our example we used 3 diferent methods to do this, and we had no improvements in the results. As we can see from the validation set, we saved under a percent on the best method, and we got a worse score on the oficial results.
We conclude that preprocessing the Medico dataset is not worth the hassle. The efort put in to preprocess the images yields little to no improvement to the result. We recommend that the time is used to find the right network, with the right hyper-parameters instead. A reason to lackluster results might be caused that the training and the test set have the same green squares in the same classes. We suspect that the similarity in the test and train set makes the squares an essential part of the image. We believe that the result would be much better if the test set would be completely without the squares, as they would if they were ”real time” images.
In a future test we would also recommend removing the four black edges too. With the images being round, this might be a challenge, since there are no full-resolution images (without zoom) that captures the edges. With the medico dataset, this method will probably not give a better score, on the basis that every image in the dataset has the same four black corners.