This paper presents our approach for the 2017 Multimedia for Medicine Medico Task of the MediaEval 2017 Benchmark. We propose a system based on global features and deep neural networks, and preliminary results comparing the approaches are presented.
Following the initiative to investigate how multimedia can improve medical systems [15], the 2017 Multimedia for Medicine Medico Task [18] addresses the challenge of detecting diseases based on multimedia data collected in hospitals [13], i.e., the task focuses on detecting abnormalities, diseases and anatomical landmarks in images in the gastrointestinal (GI) tract. There do exist some proposals in this area using various approaches [20,21], and in this paper, we describe our solutions, based on both our global-featuresbased and neural-network-based EIR prototypes [12,14,16,17].
The proposed approaches are based on the hypothesis that GI tract diseases and findings can be recognized and classified based on color, shape and texture properties. In this challenge, there is no detailed ground truth ROIs provided for the training dataset, thus, already existing and well performing approaches to objects recognition are not suitable for this particular task. Moreover, a relatively low amount of training data is provided making it difficult to use modern convolutional neural network (CNN) image segmentation and region-based classification approaches. Furthermore, some objects like polyps and resection margins have a compact body and can be easily differentiated from the surrounding tissue, but other findings like ulcerative colitis have only tissue with a slightly different color properties. To address these different detection challenges, we present 17 different approaches that implement our idea of using visual properties of images for performing multi-class classification with the limited training set size. For the final classification step, we use the WEKA machine learning support library [7] which is an open source collection of algorithms for machine learning and data mining. For all the approaches based on global features (GFs), we use Lucene Image Retrieval (LIRE) [10], an open source implementation of global and local features extraction and comparison. For all the deep-learning-based approaches, we use Keras [3], an open Copyright held by the owner/author(s). MediaEval'17, 13-15 September 2017, Dublin, Ireland source high-level neural networks API with Google Tensorflow [1] as a computational back-end.
For the GF-based approaches, we use features that represent the overall image visual properties, they are easy and fast to calculate, and they can be used for image comparison, distance computing and image collection search. Here, we use the indexes of visual features extracted from training image set. A classifier is used to search the index for the image that is most similar to a given input image. The GFs we use are JCD, Tamura, Color Layout, Edge Histogram, Auto Color Correlogram and Pyramid Histogram of Oriented Gradients [10]. We decided for these combinations based on our previous findings and experiments in [14,16]. Multi-class classification is implemented as an additional classification step to determine the final image class based on the the ranked lists of a search-based classifier for each class of findings. We use the random tree (RT), random forest (RF) and logistic model tree (LMT) classifiers [7] from WEKA.
For the deep-features-based approaches, we use a combined method with deep residual networks for image recognition as features extractor and machine-learning classifier with the input of extracted deep-features as a multi-class classifier. We use the Inception v3 [19] and ResNet50 [8] models pre-trained on a set of general images. The models were modified in order to produce numerical probability output for all recognized object classes. Then, we use the class (concept) probabilities (1000 values for both networks) directly in the Concepts runs. For the Features runs, we have used the same pre-trained models without including the fully-connected layer at the top of the network, which give us an output of high-level feature probabilities (16384 values for Inception v3 and 2048 for ResNet50). Finally, we combine the probabilities by simple early fusion in one big vector of floating point numbers and use it as an input for the same classifiers we used in the GF-based approaches.
For the CNN-based approach, we created and trained a custom CNN from scratch. Our CNN consist of six convolution layers. As an activation function, we used the rectified linear unit (ReLU) [6] and maxpooling for pooling. In all the layers, we also included a 0.5 dropout, and the final classification step was performed using two dense layers with first ReLU and then Sigmoid as activation functions. Both networks were trained for 200 epochs using the Adam optimizer [9].
For the transfer-learning-based (TFL) approach, we use the pretrained Inception v3 [19] model and transfer learning technique [2] to train the network on our specific training set. We re-trained the base model and fine-tuned the last layers on the training set following the DeCAF approach [5]. We did not perform complex data augmentation and only relied on transfer learning. We froze all the basic convolutional layers of the network and only retrained the two top dense layers. The dense layers were retrained using the RMSprop [4] optimizer that allows an adaptive learning rate during the training process. After 1,000 epochs, we stopped the retraining of the dense layers and started fine tuning the convolutional layers. For that step, we did the analysis of the Inception v3 model layers structure and decided to apply the fine-tuning on the top two convolutional layers. For this training step, we used a stochastic gradient descent method with a low learning rate to achieve the best effect in terms of speed and accuracy [11].
First, we have performed an initial evaluation of the approaches using the development dataset only randomly splitting it into new training and test sets with the equal number of 2, 000 images in each. We assessed 17 different methods executed in 17 internal runs using the new sets generated. An overview of the conducted internal runs can be found in table 1 where we provide the measured performance metrics [13]. We can see that not all our approaches can perform efficiently on the given dataset. In general, we can conclude that for all the machine-learning-based classification approaches, the LMT classifier is performing the best, the RF classifier is slightly worse, and the RT classifier performs the worst. The 6 Layers CNN and Inception v3 TFL approaches performs with the comparable precision, but Inception v3 TFL have slightly better results. The Inception v3 Concepts and ResNet50 Concepts approaches performs with the comparable precision too, but all the ResNet50 Concepts approaches perform slightly better. The Inception v3 Features approaches perform the worst compared to all other features-based approaches even for the efficient LMT classifier, which can be caused by the huge feature values vector generated by the Inception v3 network. Finally, the best performing approach is the ResNet50 Features approach with the LMT classifier showing the performance of 0.828 for R K and 0.856 for F1 score.
Based on the initial evaluation, we have selected the five different approaches for the official competition submission. The approaches selected (see table 2) are the best performing in the internal runs while keeping as much diversity of the methods as possible. The official evaluation results provided by the organizers is presented in table 2. The best performing approach is again the ResNet50 Features approach with the LMT classifier (run #4) with the R K value of 0.802 and F1 score of 0.826. The confusion matrix of this run is presented in table 3. The often miss-classified classes are Esophagitis and Z-line that is caused by the nature of the used visual features. Both of these classes consist of pictures of Z-Line, but Esophagitis
In this paper, we presented 17 different combined approaches designed for multi-class classification of medical imaging data with the limited training dataset. We presented a novel comparison of the performance of the various visual-features-based methods with traditional custom CNN and Inception v3 with transfer-learningbased approaches. We used modified Inception v3 and ResNet50 networks and the LIRE library for the features extraction, with machine-learning classification algorithms from WEKA. Despite the limited training dataset and a presence of visually similar image classes, we achieved a good multi-class classification performance with the R K value of 0.802 and a classification speed of 46 frames per second. For our future research, we will investigate the combined approach with the fusion of multiple deep-network-based feature extractors for the initial coarse image classification together with the fine-tuned local-feature-based sub-classification for the efficient cross-class detection between visually similar images.
This work is founded by the FRINATEK project "EONS" #231687.