In this paper, we describe our approach for the classification of medical images depicting the human gastrointestinal tract. Searchbased classification is performed in three stages. In the first stage, we extract deep features for each image using pre-trained deeplearning models [1]. In the second stage, we use LIRE [3] to index the generated features, so that we can then, in the final stage, search the index for similar images and make our predictions based on the results. With this approach, we achieved a MCC score of 0, 54 and a accuracy of 0, 94, which shows that deep features combined with search-based classification are a viable option for medical image analysis.
The aim of the 2018 MEDICO task [
The deep-learning features are extracted with a Python script, using
the Keras API [
The models are pre-trained on the ImageNet dataset [
For the other models we are also not using the top layers, but in addition to the GlobalMaxPooling2D layer we also added a Dense layer to get a length of 1024 values for our feature vectors. The generated feature vectors are then stored in comma-separatedvalues (CSV) files. We create one file for each model and it contains the filename followed by the feature vector. Before the indexing of the files we also perform a quantization step to bring the feature vectors from double range to integer range. This step not only increased the accuracy in our tests, but it also made the index smaller and reduced the processing time for indexing and searching. The diferent models are then combined in the indexing stage. To do this we create one entry per model per file in the index, resulting in 8 index entries per file. 3
In order to index the extracted features (see section 2) with LIRE [
Analog to the indexing in section 3, we had to adapt the existing LIRE classes for searching the index to support the new feature classes and work with file paths instead of images. To find most similar images, we used the cosine distance function, which was already implemented in LIRE. 5
For the classification, we took the best nine results of each model
and generated predictions for each model by counting the returned
categories. These intermediate predictions are weighted and
combined to form the final prediction. The weights are manually
generated based on experiments on the development data set [
For our submitted runs we used all the files of the training dataset to create the index. We used following configurations for our four submitted runs: • Run 1: Integer Features with Bitsampling Hashing
The extracted features are quantized, to use integer values instead of double values, and then indexed using bitsampling hashing. • Run 2: Integer Features with Metric Spaces Hashing
The extracted features are quantized, to use integer values instead of double values, and then indexed using metric spaces hashing. • Run 3: Integer Features with Bitsampling and Metric Spaces Hashing
The extracted features are quantized, to use integer values instead of double values. Then they are indexed using M. Steiner, M. Lux, P. Halvorsen Run 1 Run 2 Run 3 Run 4
Recall
As shown in Table 1, we achieved the best results with Run 1. Here, we used the features from the deep learning Models, configured as described in Section 2, and quantized them from the double range to integer range. Run 2 shows, that metric spaces hashing seems to be less suited for this task. It has given better result in predicting Esophargitis correctly, but over all the results are significantly worse. And in the combination of both hashing methods (run 3), we could notice that for example the correct prediction of Esophargitis got slightly better, but overall the worse performance of metric spaces had a negative impact on the prediction. Run 4 shows that the quantization from double to int brings a slight increase in overall performance.
Table 2 shows the confusion matrix of our best results in Run 1.
The main problem areas here are the classification of Esophargitis as
Normal-Z-Line and mixing up Dyed-Resection-Margins with
DyedLifted-Polyps. The results for the classes Out-Of-Patient (OOP) and
Instruments (INS) are very low, this is mostly because of the lack
of example images in the dataset [
In this paper, we showed an approach to utilize deep feature vectors
for search-based classification, by extracting the features with deep
neural networks and indexing those features to make them
searchable. We got an accuracy of 0, 94 and MCC score of 0, 54. This was
achieved by using bitsampling indexing combined with features
quantized to the integer range, which may cause a noise reduction
compared to the double features. Further experiments could be
made, to see if better results can be achieved by training the models
on medical images rather than using the models trained on the
ImageNet dataset[
We would like to thank our colleagues who helped for all the input and discussions. Travel was funded by Alpen-Adria Universität Klagenfurt.