The University of Essex participated in the fourth edition of the ImageCLEFcaption task which aims to detect concepts on radiology images as an approach to medical image understanding. In this paper, the University of Essex team presents its participation in the ImageCLEF 2020 caption task based on a retrieval based approach for concept detection. A Densely Connected Convolutional Network is used to encode the images. This paper explores compares several modi cation of the baseline considering several aspects such as the image modality or the selection of concepts among the top retrieved images. The University of Essex was third best team participating in the task achieving a 0.381 mean F1 score, very close to the results obtained by the top two teams. Code and pre-trained models are available at https://github.com/fjpa121197/ImageCLEFmedEssex2020.
This paper describes the participation of the School of Computer Science and
Electronic Engineering (CSEE) at the University of Essex at ImageCLEFcaption
2020 task [
ImageCLEFcaption 2020 task is the forth edition of this successful task. In
previous editions [
The rest of the paper is organised as follows. Section 2 presents collection and the evaluation methodology used in this work. Section 3 explains the techniques proposed in this paper including a detail description of the runs submitted to the ImageCLEFcaption task. The results are presented in Section 4. Finally, the conclusions are given in Section 5. 2
In this work we used the ImageCLEFmed caption 2020 collection [
The images originate from biomedical journal articles extracted from the
PubMed Central R (PMC)3 repository [
The UMLS CUIs from the test set were not distributed and, therefore, not
used to build the model. The ImageCLEFcaption task [
In 2020, the ImageCLEFmed caption collection is classi ed in seven medical image modalities (Angiography, Computer Tomography, Magnetic Resonance, Positron Emissions Tomography, Ultrasound, X-ray and combined modalities in one image). 3
The proposed approach is based in a content-based image retrieval model, where DenseNets are used for feature extraction (see Section 3.1). A similarity comparison is done between the query image and the images in the training and validation test sets (see Section 3.2). Finally, concept selection is performed to predict the medical concepts for the query image (see Section 3.3).
Figure 2 shows an overview of the approach. 2 https://github.com/fjpa121197/ImageCLEFmedEssex2020 3 https://www.ncbi.nlm.nih.gov/pmc/
Image: ROCO2 CLEF 26437
Concepts:
{ C0040398: Tomography, Emission-Computed
{ C0224338: Structure of sternal muscle
{ C0040405: X-Ray Computed Tomography
Following the success of the AUEB NLP Group at ImageCLEFmed Caption
2019 [
DenseNet models are Convolutional Neural Networks (CNN) models where
each layer is connected directly to other layers [
The input image is resized to 64 64 and transformed to an array, then a preprocessing module from DenseNet Keras is used. This module is in charge of transforming the pixel values into a 0-1 range, and also to normalise the values based on the ImageNet dataset. The DenseNet-121 model is then used to encode each image representing it as a vector of 4,096 dimensions excluding the classi cation layer.
Fine-tuning. In this work, a ne-tuning strategy is also explored to transfer learned recognition capabilities to the speci c challenge of concept detection. The ne-tuning has been done speci cally for each image modality, where a fully connected layer has been added to the DenseNet-121 model transforming it into a multi-label classi cation model. The last fully connected layer was trained for 10 rounds.
In particular the following parameters were used: { Optimizer : RMSProp { Learning rate: 0.0001 { Batch size: 32 { Momemtum: 0.0
The model was trained in two phases: { 1st phase: Only training the classi cation layers. { 2st phase: Training a portion of the feature learning layers and the classi cation layer.
Each phase consisted on 10 epochs (each epoch consisted of 100 steps, of which 10 steps were for validation). 3.2 Image retrieval In this work, the image modality is used to improve the system performance. Each image in the test set is compared to all the images in the training or validation sets belonging to the same image modality as the query image. In the case of run 64104 the images were retrieved from all the training set without considering the modality.
In order to perform the comparison, Canberra and Manhattan distances are computed given the encoded features (see Section 3.1). This metrics were chosen based on their accuracy and speed of their computational performance. The 10 most similar images to the given query were selected and their associated concepts extracted. Each of the extracted concept is tagged with a score based on its ranked position or the computed distance value (see next Section 3.3 for more details). 3.3
In order to assign the concepts to the query images in the test set two methodologies were tested: Ranking based selection. Each concept is assigned with a score based on the ranking of the 10 retrieved image which they were associated to. If the concept is associated to more than one image, then the value is added to it. For example, the highest ranked image has all its concepts given a value of 10 and the second highest has all its concepts given a value of 9. If the nal score (after the addition) given to a concept is equal or over the threshold 20, then the concept is considered relevant to the query image and assigned to it. Distance based selection. Each concept is assigned a scored based on the distance value computed of the 10 retrieved image which they were associated to. Similar to the ranked based selection, if the concept is associated to more than one image, then the value is added to it. For each concept nal score (after the addition), the mean or percentile (99 or 95) is set as a threshold to select the concept. If the score was equal or over the threshold, then the concept is considered relevant to the query image and assigned to it. During the experimental set up other thresholds were tested such as percentiles 75 and 98 or a normalisation process, however there were no nally submitted to the challenge since mean and percentile 95 and 99 achieved better F1 score on the validation set. 3.4
This section provides a detailed description of the runs submitted to ImageCLEFcaption 2020 task. The methods used to implement these runs are described in Section 3. Table 1 summarises the techniques used by each run. { Run 64104 - baseline: In this run, DenseNet-121 is used to encode the images. The top 10 images are retrieved from the training set using Canberra distance. Ranking based selection is used to select the relevant concepts from the retrieved images. { Run 67416 : This run is similar to the baseline. In this case the image modality information is used in the retrieval step. The top 10 images from the same modality as the query image are retrieved from the training set. { Run 63804 : This run is similar to the Run 67416. In this cases, the images are retrieved from both training and validation sets. The modality information is also considered. { Run 64394 : This run is similar to the Run 67416. In this run, ne-tuning is applied. { Run 68019 : This run is similar to the Run 64394. For this run, distance based selection is used to select the relevant concepts from the retrieved images using the mean of the scores as a threshold. { Run 68026 : This run is similar to the Run 64394. For this run, distance based selection is used to select the relevant concepts from the retrieved images setting 95th percentile as a threshold. { Run 68025 : This run is similar to the Run 64394. For this run, distance based selection is used to select the relevant concepts from the retrieved images setting 98th percentile as a threshold. { Run 68022 : This run is similar to the Run 64394. For this run, distance based selection is used to select the relevant concepts from the retrieved images setting 99th percentile as a threshold. { Run 68022 : This run is similar to the Run 68027 but using the Manhattan distance in the retrieval step.
This paper describes the participation of CSEE at the University of Essex at ImageCLEFcaption 2020 task. CSEE proposes a retrieval-based approach using a DenseNet-121 model to encode the images in the collection. CSEE compares di erent modi cations in the baseline to study their e ects on the nal performance. Best submitted run used ne-tuning per image modality and Canberra distance in the retrieval step. Concepts were selected based on the top 10 ranked images. CSEE was the third best team at the benchmark achieving a F1 score of 0.381, very close to the results obtained by the top two teams. In 2020, the image modality was provided and future improvements can tackle an initial modality classi cation step as well as training the retrieval step per each modality. Further work is also needed to better understand the e ects of the concept selection step. 13. Zhang, Y., Wang, X., Guo, Z., Li, J.: Imagesem at imageclef 2018 caption task: Image retrieval and transfer learning. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org <http://ceur-ws.org>, Avignon, France (September 10-14 2018)