The Medico: Multimedia Task focuses on developing an eficient framework for predicting and classifying abnormalities in endoscopic images of gastrointestinal (GI) tract. We present the HCMUS Team's approach, which employs a combination of Residual Neural Network and Faster R - CNN model to classify endoscopic images. We submit multiple runs with diferent modifications of the parameters in our combined model. Our methods show potential results through experiments.
Medico: Multimedia Task at MediaEval 2018 challenge [
In our approach, we introduce a stacked model consisting of two
deep networks, a Residual Neural Network (Resnet) [
In the field of medical image processing, deep neural networks have
been used in order to solve several problems related to endoscopic
images of the gastrointestinal (GI) tract. Particularly, to localize and
identify polyps within real-time constraint, deep CNNs has recently
shown an impressive potential when achieving up to 96.4% accuracy
- published in 2018 by Urban G et al. [
Dataset Preparation 3.1.1 Disease region localization. In order for the Faster R-CNN model to be trained, objects in the image have to be tagged with bounding boxes and passed to the model as input. We annotate the signal of disease in all images of the following classes: dyedresection-margins, dyed-lifted-polyps, instruments and polyps.
3.1.2 Re-labeling Medico development dataset. After training with the development set, we find some training samples with inappropriate labels according to the priority list. Therefore, in order for our model to learn with the least confusing, we apply the new labels, predicted by the trained model, to these images.
3.1.3 Instruments dataset augmentation. Instruments - the second highest priority class has only 36 images with the limitation of background context in the development set. In order to maintain the balancing between all of the classes and also improve the diversity of the instruments images, we generate more images for the instruments based on the current given development set by placing the instruments on the foreground of other diseases backgrounds.
Among the 36 instruments images, we carefully select 24 of them and crop the instruments along their edges. Then, we randomly select 20% of the images from dyed-lifted-polyps, dyed-resectionmargins, ulcerative-colitis classes, and use them as the background of the cropped instruments. By applying this method, we are able to generate more than 800 images for the instruments class. 3.2
3.2.1 Fine-tuning deep neural network for medical images. In
our approach, both Residual Network with 101 layers and Faster
R-CNN [
3.2.2 First run. Residual network with 101 layers model are ifne-tuned on the original development set provided by the task organizers along with our instruments increased dataset. After passed through Resnet101, output images classified as special classes become the input of Faster R-CNN network, which is trained for detecting instruments in images. • First case: Images predicted as instruments by Resnet101 are double-checked. In case instruments are not detected by Faster R-CNN in those images, they are re-labeled as the class of their second highest score proposed by Resnet101. • Second case: Images predicted as dyed-lifted-polyps, dyedresection-margins, ulcerative colitis by Resnet101 are fed forward through Faster R-CNN network to detect instruments. They are classified as instruments if detected or keep the original prediction otherwise.
3.2.3 Second run. Feeding forward a large number of images in the three classes through Faster R-CNN causes a bottle-neck of inference time, as Faster R-CNN has high time complexity. Therefore, in this second run, we limited the images passed through Faster R-CNN by only performing the first case of the first run.
3.2.4 Third run. The configuration of the third run is as same as the second run. Instead of using the original training set mentioned in the first run, we train our model on the re-labeled development set combined with the augmented instrument set.
3.2.5 Forth run. In this run, we reduce the number of images used for training by selecting randomly 75% images of each class in the same training set as the third run. Other processing steps are also configured in the same way.
3.2.6 Fifth run. Throughout our experiments, normal-z-line and esophagitis are the top most confusing classes not only for Resnet101 but also for human to distinguish them. In the priority list, esophagitis has a higher rank than normal-z-line’s. Thus, after several times evaluating our model on the development dataset, we propose a condition for these two classes when they are predicted by Resnet101. As Resnet101 provides a probability distribution over the 16 classes for each image, whenever the normal-z-line appears to be the highest class, we add a small bias 0.3 to the probability of the esophagitis. Hence, the model is more likely to emit the esophagitis class. This intuitively means that our model prefers esophagitis to normal-z-line when it is confused between these classes.
Table 1: Oficial evaluation result for both sub-tasks (provided by the organizers) and speed (fps) on Tesla K80 GPU RunID Run01 Run02 Run03 Run04 Run05
PREC 94.245 93.959 94.600 93.043 94.508
REC 94.245 93.959 94.600 93.043 94.508
ACC 99.281 99.245 99.325 99.130 99.314 There is a trade-of between speed and accuracy when comparing the result of Run01 and Run02. In Run02, we reduce a large number of images passing through Faster R-CNN for the sake of time, so its performance seems to be relatively worse than Run01’s.
As we mentioned earlier in section 3, data pre-processing takes an important role in building a deep-neural network model. Through our experiments, in the case of less training data, the augmented dataset helps us improve the performance of deep-neural network model. Run03 and Run05 show impressive results comparing to the first two runs. This implies that training on our re-labeled development set provides better models.
On the other hand, using the Residual neural network cannot classify eficiently the two classes esophagitis and normal-z-line. The same problem also occurs between the dyed-resection-margins and dyed-lifted-polyps classes. It can be observed in the confusion matrices of the two pairs (Figure 1). Therefore, these are the two main reasons which mainly bring negative impact to our results.
Additionally, as we mentioned in section 3, the configuration of Run05 intuitively prefers esophagitis to normal-z-line, which may leads to an increasing of the false-positive cases in the result.
By comparison to the others, Run04 has the lowest precision since it uses 75% of training data. Decreasing the amount of training samples of course afects the performance in deep-learning models. Nevertheless, the result is still acceptable when it decreases only a few percentages and its configuration is as same as Run03. This is an evidence that we are even able to reduce up to 50% of data when the less training time is preferred over the accuracy. 5
Medico image classification is a challenging problem because of the ifne-grained images, less training data and require high accuracy. In our current approach, we focus on training a combination of Residual Neural Network and Faster R-CNN with diferent modifications of the training set. Additionally, object detection method is applied to detect small symptoms of diseases, which are useful evidences for the classification task. Accuracy and inference time that we reach is acceptable and appropriate for real-time constraint. However, for future works, we need a more robust approach to exploit the distinction between easy-confused classes, e.g, esophgitis and normal-z-line, or dyed-lifted-polyps and dyed-resection-margins.