Transfer learning is an approach where a model trained for a given task is used as a starting point on a second task. Many advanced deep learning architectures have been pre-trained on ImageNet and are currently available, which makes this technique very popular. We evaluate 10 pre-trained architectures on the task of nding gastrointestinal diseases and anatomical landmarks in images collected in hospitals. Our analysis considered both processing time and accuracy. We also study if global image features bring advantages to the pre-trained models for the problem of gastrointestinal medical image classi cation. Our best models achieved accuracy and F1-score values of 0.988 and 0.908, respectively. Our fastest model classi es an input instance in 0.037 seconds, and yields accuracy and F1-score of 0.983 and 0.866, respectively.
The Medico Task proposes the challenge of predicting diseases
based on multimedia data collected in hospitals [
To solve the task we created several models based on features extracted from deep convolutional architectures and global image features. The deep architectures were trained on ImageNet. The strategy was to create three kinds of models: (i) those having only features extracted from deep architectures as input, (ii) those that considered only global image features, and (iii) those created with all features available.
The approach of extracting features from pre-trained models is usually referred to as transfer learning. It has become popular because many models are available. We selected 10 architectures trained on ImageNet as extractors, and compared their performance on classifying images for the Medico Task.
The architectures di er in several characteristics, which impacts the time to compute the features. Since e ciency is an important matter on this task, we computed the data processing speed for each test images and made e orts to reach a balance between solution quality and running time.
Transfer learning has been used in several problems in a number of
domains. In 2017, Agrawal et al. [
The development and test dataset contain 5,293 and 8,740 images,
respectively. For each image, visual features were extracted and
provided as feature vectors by the task organizers, namely: JCD,
Tamura, ColorLayout, EdgeHistogram, AutoColorCorrelogram and
PHOG [
We used 10 architectures trained on ImageNet: DenseNet121 [
The input layer has the same number of nodes as the feature vector sizes. That said, in a model that uses only global features the input layer has 1166 nodes. In a model that uses global features and a given architecture as feature extractor, the input layer has the feature vector of that architecture plus 1166.
The best model for most of the input features uses one hidden layer that has 512 nodes and each node uses Relu as activation function. We added a Dropout of 50% in the training stage and l2 regularization to prevent over t. Models with more layers tend to over t very easily with just a few epochs. It would be possible to create simpler models for small feature vectors like VGG architectures, but we decided to report the same network for all input vectors for comparison purposes. The output layer has 16 nodes (one for each class) and uses softmax activation to classify the image in one of the classes. 4
During the training stage we split the development set with 5,293 images into train (3,038 images), validation (1,722 images), and test (531 images) datasets. We used train and validation sets to train and tune the classi er. Test dataset was used only once to generate the results in Table 1 after we were satis ed with the validation scores.
The model that uses only global image features yields an accuracy of 0.813, and F1-score of 0.782, which we consider a baseline result. Table 1 summarizes the results using transfer learning in the test dataset with 531 images. We based our decision about which model was best to submit on the F1-score and accuracy metrics.
Our rst decision was to disregard the models that had transfer learning plus global features because the improvement after adding global features was considered irrelevant. As an example, the architecture DenseNet201 had a small increase of accuracy from 0.923 to 0.925. These were the best models using accuracy metric. If we consider the F1-score, models with global image features became even worse in several cases. Taking into account that these models with global features have 1166 more inputs than their counterparts, we decided to keep the simpler models.
DenseNet201 and Resnet were selected as the two best models considering accuracy and F1-score, respectively. MobileNet was selected because it is amazingly fast, and e ciency is an important matter on this task. Indeed, we consider this model the best tradeo since its 0.889 of accuracy is not far away from the 0.923 accuracy of DenseNet201 (best accuracy model) and runs 6.5 times faster. DenseNet121 was the last model selected because is somewhere in between the best accuracy model (DenseNet201) and the fastest model (Mobilenet).
Selected models were submited and evaluated against the dataset with 8,740 images. In this dataset, our results were much better than we anticipated. Results are shown in Table 2, where we also report the o cial competition ranking indicator Rk . ResNet and DenseNet models achieved accuracy higher than 0.987. Mobilenet yields accuracy of 0.983, which is very close to the top accuracy of 0.988 achieved by DenseNet201 and Resnet.
F1-score shows that Resnet and DenseNet201 are the best models and that MobileNet is somewhat worse than the others. However, we believe MobileNet has best trade-o if we consider that it returns solution in much less than a second. DenseNet121 does not appear a good choice because it is slower than Resnet and present somewhat worse results. Therefore, Resnet should be preferred in any situation. DenseNet201 was the top accuracy and F1-score, but it is so close to Resnet that it is di cult to argue that it is enough compensation for the fact of being twice as slower.
We believe these models can be improved and the confusion matrix shown in Figure 1 provides some insights. We analysed how they performed in each of 16 class and found out that the class “out-ofpatient” is particularly problematic, since it has only 4 instances in the development set and 5 instances in the test set. Furthermore, none of our models was able to classify these 5 instances right, which does not impact accuracy but degrades F1-score. In the future, some data augmentation should be performed to improve this class.
Another class we need to study is “esophagitis”, because 170 instances were classi ed as “normal-z-line”, which accounts for 30.57% of instances in this class. The classes “Stool-plenty” and “colon-clear” have most of the instances, and our models did a good job on classifying them right, which boosted the scores.
We thank CAPES and CNPq (grant 400487/2016-0).