ImageCLEF [1] is an initiative with the aim of advancing the field of image retrieval (IR) as well as enhancing the evaluation of technologies for annotation, indexing and retrieval of visual data. The initiative takes the form of several challenges, and it is especially aware of the changes in the IR field in recent years, which have brought about tasks requiring the use of different types of data such as text, images and other features moving towards multi-modality. ImageCLEF has been running annually since 2003, and since the second version (2004) there are medical images involved in some tasks, such as medical image retrieval. Since those versions, the ImageCLEFmedical challenge group of tasks [2] has integrated new ones involving medical images, with the medical image captioning task taking place since 2017. It consists of two subtasks: concept detection and caption prediction. Although there have been changes in the data used for the newest versions of the challenge, the goal of this task is the same: help physicians reduce the burden of manually translating visual medical information (such as radiology images) into textual descriptions. This year, the concept detection task within the ImageCLEFmedical challenge aims at identifying the presence of relevant biomedical concepts in medical images.
In this document we describe the participation of our team from the HAIVis group 1 within the artificial intelligence laboratory2 at PUC Chile (PUC Chile team) in the concept detection task at ImageCLEFmedical 2021 [2]. Our team reached fourth place in the challenge, and our best submission was a combination of deep learning techniques to visually encode the medical images with a VGG convolutional neural network [3], followed by a KNN similarity search using Perceptual Similarity [4] rather than traditional cosine similarity. The rest of the paper is structured as follows: section 2 briefly describes related work, section 3 our data analysis, and section 4 details of our proposed methods. In section 5 we describe our results, and finally in section 6 we conclude this article.
In previous versions of the competition, the best participants have used a wide variety of techniques, mainly based on Convolutional Neural Networks, Natural Language Processing, K-Nearest Neighbors and Clustering [5]. In the 2020 concept detection challenge, the best F1 score achieved was 0.394 by the AUEB NLP Group. This winning approach consisted of a variation of ChexNet and DenseNet-121 [6] with a feed-forward neural network as the classification head [7]. This team also won the second place in the competition. The second best group was PwC_Healthcare, with a F1 score of 0.3924. They submitted three approaches, using CNN and NLP techniques and clustering, achieving 3rd, 4th and 5th place. Looking at previous years, the best submissions achieved an F1 score of 0.1108 in 2018, 0.2823 in 2019 and 0.3940 in 2020 [5].
The data set provided for this challenge consisted of 2,756 images of varying size for the training set and 500 for the validation set. Since the task is a multi-target classification problem, each sample has one or more labels. Figure 1 shows the distribution of number of labels per image in the training set. Each label corresponds to a UMLS concept [8]. There are 1,315 different concepts in the training set and 817 in the validation set. Furthermore, only 547 concepts are present in both sets. This is worthy of consideration because it indicates an intrinsic skew in the dataset. We can represent this situation using Figure 2: In the training set, the most frequent concept is C0040398 (Tomography and Emission-Computed), with 1,159 images. This corresponds to 42% of the images in the training set. The ten most frequent concepts can be seen in Figure 3.
The data set analysis motivated us to try a naive method as a benchmark that consisted in classifying all inputs as the most frequent concept in the training set, C0040398 (Tomography and Emission-Computed). When submitted, this approach gave us an F1 score of 0.141 over the testing set.
We tried using a KNN algorithm over the images to detect concepts. This is, given a test image, we find the K closest training examples according to a specific distance metric, which is detailed further in the article. Then, the concepts assigned to the test image are the concepts that are present among all selected (K) neighbours. Figure 4 shows an example using K=3 neighbours.
The metric used to calculate the distance between different images was Learned Perceptual Image Path Similarity (LPIPS) [4] 3 , a learned metric based on the similarity between deep features from several layers of a neural network, typically a VGG model [3]. This distance tries to capture the similarity between two images according to human perception. The use of this metric as a good estimator for this dataset arose in the caption prediction task, where a similar approach meant a significant performance gain.
Another method used at an early stage of the study was the ResNet-18 [9] model pre-trained on the ImageNet dataset, from PyTorch4 . The model was fine-tuned with the ImageCLEFmedical 2021 data set. However, this approach strongly overfitted to the training set. Over the validation set, the model almost always predicted the class to be the most frequent concept in the training set, C0040398 (Tomography and Emission-Computed), so the results achieved on the testing set was 0.141 F1 score, same as the naive approach. This is probably due to the imbalanced nature of the dataset, and that no regularization or drop-out layers were used.
Using Perceptual Similarity as the distance metric for the KNN algorithm leads to images representing similar concepts being grouped together in a neighbourhood. Figure 5 shows the performance in F1 score of the KNN approach for different number of neighbours. The best performance of the KNN classifier results when only choosing the closest neighbour.
Two submissions were made for the task, the first one using a naïve approach in order to get insights of the results with a baseline, and a second approach based on K-NN using the perceptual similarity implemented in LPIPS, as shown in Table 1. We improve the score more than twice by K-NN employing LPIPS.
This article describes the participation of the PUC Chile team for the concept detection task in ImageCLEFmedical challenge 2021. Our best submission included a VGG deep neural network for visual encoding integrated with a KNN image search using Perceptual Similarity [4] to select concepts. This metric, as the original article states, provides a robust similarity approach, although more expensive to calculate than traditional approaches such as cosine-based similarity.
This work was partially funded by ANID -Millennium Science Initiative Program -Code ICN17_002 and by ANID, FONDECYT grant 1191791.