This paper presents the participation of the Image Semantics group (ImageSem) of the Institute of Medical Information at the ImageCLEF 2018 caption task. We participated in both of the concept detection and the caption prediction tasks, with submitting 15 runs in total. In this study, we applied LIRE, an open source Lucene Image Retrieval, to index 222,314 images in training and 9,938 images in test sets. In concept detection subtask, we retrieved the similar images in the training set and applied Latent Dirichlet Allocation (LDA) for clustering concepts of the similar images. The transfer learning method was integrated to solve muti-label annotation in the concept detection task. In caption prediction, we used image retrieval strategies by tuning the parameters: the top similar images and number of candidate concepts. In the evaluation, ImageSem achieved the best F1 Score of 0.0928 in the concept detection subtask and the Mean BLEU score of 0.2501 in the caption prediction subtask.
The corpus of annotated medical images, interpreting and summarizing the insights of
images, are important for medical image processing and machine learning technology
application [
The concept detection subtask aims to identify the UMLS [
The caption prediction subtask aims to predict and generate natural language caption for a given medical image. We proposed a retrieval-based method using LIRE on the training set and combined with preferred concepts recognized from the preceding subtask. This paper is organized as follows: Section 2 introduces the task data and our data preprocessing method. Section 3 describes our methods for concept detection. Section 4 presents our methods for caption prediction. Section 5 summarizes all the runs submitted by our team. Section 6 makes a brief conclusion.
The training and test datasets contained 222,314 and 9,938 biomedical images
respectively. The images were extracted from scholarly articles in PubMed Central (PMC)
[
We firstly analyzed the annotated concept frequency distribution in order to better
understand the task images. The distribution is important for multi-label training object
selection and similar image measurement. The training data includes 222,314 images
associated with 111,156 CUIs. Table 1 shows the concept distribution. It can be seen
most annotated CUIs (92.19%) were used less than 100 times among 222,314 images.
Thus, it is challenging to train a model to learn the annotation patterns of these 102,480
concepts. For the frequent concepts, there are 1312 concepts with frequency greater
than 1000. Table 2 shows the top ranked concepts, their annotated image number, and
their corresponding UMLS terms. Some general concepts like medical image
(C1704254) and image (C1704922) were highly used but meaningless.
We used LIRE [
Further, after analyzing the training data, we found that a number of CUIs co-occur in almost the same set of medical images. To make use of this characteristic, we clustered the CUIs according to their similar scores based on their co-occurrence. The formulation of calculating the similar scores between CUIs is shown as follows: SIMILAR_SCORE (A, B) =
Where, SIMILAR_SCORE(A,B) denotes the similar score between the CUI A and the CUI B. images_A and images_A separately represents the set of medical images in which the CUI A and the CUI B appears. CUIs with similar score more than 0.8 are clustered into the same group. Accordingly, 1312 CUIs are clustered into 459 groups and the first CUI of each group is selected as the representation CUI. Appearance of the representation CUI is the same as the appearance of all the CUIs in its group. Eventually, just the 459 representation CUIs are fed into the model for multi-label classification.
The medical images which contains at least one of the 1312 CUIs were selected for training. And for each image, we re-built its corresponding set of CUIs, only retaining the CUIs inside the 1312 CUIs and mapping them to the representation CUIs of their corresponding groups. Finally, 208595 medical images with 459 representation CUIs were used to train the transfer learning model for multi-label classification. 3
For the concept detection sub task, we employed three methods to find multiple CUIs for a specific image, including the multi-label classification method, retrieval-based method and the topic modeling method.
In the multi-label classification method, Convolutional Neural Networks (CNNs) was applied to assign one or multiple CUIs from the predefined CUIs label set.
In the retrieval-based method, we used LIRE (Lucene Image Retrieval) to retrieve the most similar images and corresponding CUIs from the training set.
In the topic modeling method, Latent Dirichlet Allocation (LDA) was used to analyze the topic distribution of CUIs from retrieved similar images and their CUIs. 3.1
In recent years, deep neural network such as convolutional neural networks(CNN) and
recurrent neural networks(RNN) have made great success in large-scale image
processing, image content recognition, and image caption generation. Inception-v3, a
convolutional neural network(CNN) model of Google, is an architecture that often achieves
superior performance with low computational cost. The key advantage of Inception-v3
is the factorization of convolution kernel, for example, it can decom-pose a 7x7
convolution kernel into two one-dimensional kernels(a 1x7 kernel and a 7x1 kernel). Through
the factorization of convolution kernels, it can accelerate the training and increase the
depth of the network. In this study, the Inception-v3 model is pre-trained on the
ImageNet datasets with more than 1 million images and 1000 classes[
However, for our concept detection task which is treated as a multi-label classification problem, directly retraining the whole Inception-v3 model based on the training set needs to take at least a few days. Therefore, we used the pre-trained Inception-v3 based transfer learning method to identify the concepts from medical images. Specifically, we froze the parameters of all the previous layers, removed the last softmax layer and added a fully-connected layer and a sigmoid layer. While training, only the last two layers need to be trained to map the medical images to the CUIs. Totally, 208595 medical images with 459 representation CUIs were fed into the model. Eventually, after getting the predicting results of the test set, we extended the results through replacing the representation CUIs with all the CUIs in their corresponding groups according to the clustering result. 3.2
LIRE is an open source Java library that provides a simple way to retrieve images and photos based on color and texture characteristics. We used LIRE to create a Lucene index of image features on the whole training set for content based image retrieval (CBIR). to figure j.
We submitted each query image from the test set to LIRE and selected top 50 visually similar images from the training set. For a given test image, we combined related CUIs of similar images as candidate concepts, then computed a concept score s(c) to determine which concepts to be assigned as semantic labels. In the following concept score equation, α denotes the normalization weight of similar figure j, P(j) denotes the probability of figure j and P(c|j) represents the probability of concept c that is assigned Concept score: s(c) = ∑ =1 ∙ ( ) ∙ ( | ) In which, P(c|j) = Then candidate concepts were ranked according to their concept score s(c). We set a threshold τ and select top K CUIs as final related concepts.
Besides, we also considered the method of applying QuickUMLS or Metamap tools to label CUIs on retrieved similar images captions, but by testing, we found some difference between automatic tagging and original provided CUIs in the training set, which may due to different parameter settings or unknown concept expanding strategies. To avoid this uncontrollable noise, we focus on the analysis of provided CUIs. 3.3
On the basis of retrieved similar images and candidate CUIs, we employed topic modeling method to select more relevant concept for a given test image. Latent Dirichlet Allocation (LDA) is a widely used generative statistical topic model in natural language processing. In this subtask, we assume concepts related to each image are collected into documents, so each document is a mixture of a number of topics and each concept is attributable to one of the document’s topics.
We applied Gensim, a topic modeling Python package to modeling topic distribution on retrieved similar images and candidate CUIs. For a given test image with its retrieved 50 similar images, we collected 50 documents of CUIs as the input of LDA model. According to the topic distribution θ of the current document set, we picked the topic with the highest probability p(z|D) as the candidate topic, and finally selected CUIs from the candidate topic that with probabilities p(c|z) above the threshold φ0 as the final result.
Before submitting the runs we carried out experiments on the training data using highly related concepts detected in CNNs method as a hint for choosing better candidate topics. However, it didn’t provide better results. So we submit the normal runs of topic modeling. 4
We used retrieval-based method in the caption prediction. The basic assumption is that similar images have similar lingual descriptions/captions. 4.1
For each test image, we used LIRE to retrieve similar images from the training set, and combined the captions of similar images as a new caption of the test image. We tuned the parameter, the number of top similar images, to determine the candidate captions for further combination. 4.2
We combined preferred concepts detected in the preceding concept detection subtask. The preferred concepts including CUIs from CNNs’ high score output, as well as CUIs from the output of LDA model. We extracted all the UMLS terms of each CUIs, and combined them with captions generated in the previous section. 5
This section provided a detailed description of our runs submitted to ImageCLEF 2018 caption task.
Run Concept_Run10 Concept_Run2 Concept_Run4 Concept_Run1 Concept_Run6 Concept_Run7 Concept_Run3 5583 5556 5561 5554 5574 5575 5558 Concept_Run1_submission_ID_5554: We exploited the LIRE to retrieve the 50 most similar images to each medical image of the test set. Then, for a given test image, all the CUIs of its corresponding 50 most similar images were taken as the input of LDA model. Further, we pick the topic with the highest probability p(z|D) as the candidate topic. Finally, we select CUIs from the candidate topic that with probabilities p(c|z) above 0.005 as the CUIs of that test image.
Concept_Run2_submission_ID_5556: Multi-label classification using transfer learning model, based on the pre-trained Inception-v3. The batch size was set to 20 and the learning rate was set to 0.003. While, the training steps were set to 15000. Finally, for a test image, top 10 representation CUIs of the predicting results were selected as the preliminary result, and we extended the preliminary result through replacing the representation CUIs with all the CUIs in their corresponding groups according to the clustering result as the final result of that test image.
Concept_Run3_submission_ID_5558: The same as the Concept_Run1_submission_ ID_5554 except that, in the final submission file, all the CUIs were separated by com ma.
Concept_Run4_submission_ID_5561: The same as the Concept_Run1_submission_ ID_5554 except that of all the CUIs of the given test image’s corresponding 50 most s imilar images, only the CUIs which appears in more than 1000 images of the training data were taken as the input of LDA model.
Concept_Run6_submission_ID_5574: The same as the Concept_Run2_submission_ ID_5556 except that the training steps were set to 5000 Concept_Run7_submission_ID_5575: The same as the Concept_Run2_submission_ ID_5556 except that for a test image, top 20 representation CUIs of the predicting resu lts were selected as the preliminary result.
Concept_Run10_submission_ID_5583: The same as the Concept_Run2_submission _ID_5556 except that the training steps were set to 25000. Caption_Run3_submission_ID_5526: We exploited the LIRE to retrieve the 50 most similar images to each medical image of the test set. Then, for a given test image, the captions of its top 2 most similar images were concatenated together as the result of that test image.
Caption_Run4_submission_ID_5527: The same as the Caption_Run3_submission_I D_5526 except that for a given test image, the captions of its top 3 most similar image s were concatenated together as the result of that test image.
Caption_Run6_submission_ID_5528: The same as the Caption_Run3_submission_I D_5526 except that of the top 2 most similar images to the test image, only the captio ns of images with similar distance less than 5 to the test image were concatenated toge ther as the result of that test image.
Caption_Run7_submission_ID_5531: The same as the Caption_Run3_submission_I D_5526 except that, for a given test image, we added the top 1 CUI of the predicting result of that test image from the concept detection task based on transfer learning into the final result.
Caption_Run8_submission_ID_5545: The same as the Caption_Run3_submission_I D_5526 except that, for a given test image, we added the top 2 CUI of the predicting result of that test image from the concept detection task based on transfer learning into the final result.
Caption_Run9_submission_ID_5546: The same as the Caption_Run3_submission_I D_5526 except that, for a given test image, we added the top 3 CUI of the predicting result of that test image from the concept detection task based on transfer learning into the final result.
Caption_Run13_submission_ID_5548: The same as the Caption_Run3_submission _ID_5526 except that, for a given test image, we added the top 1 CUI of the predictin g result of that test image from the concept detection task based on the retrieval metho d and LDA into the final result.
Caption_Run19_submission_ID_5552: The same as the Caption_Run3_submission _ID_5526 except that, for a given test image, both the top 1 CUI of the predicting resu lt of the concept detection task based on the retrieval method and LDA and the top 1 C UI of the predicting result based on transfer learning are added into the final result. 6
This paper presents the participation of the Image Semantics group (ImageSem) at the ImageCLEF 2018 caption task. We submitted 7 runs in the concept detection and 8 runs in the caption prediction tasks. The evaluation results showed that we achieved the best F1 Score of 0.0928 in the concept detection subtask and the Mean BLEU score of 0.2501 in the caption prediction subtask. Our methods mainly relied on image retrieval and transfer learning.
In our experiments, we found the ground truth concept annotations were not exactly represent the semantics of the images. It is difficult for error analysis from either computational view or clinical/biomedical view. In the future work, we would like to contribute the corpus construction together with the ImageCLEF committee. 7
This study was supported by the National Key Research and Development Program of China (Grant No. 2016YFC0901901 and No. 2017YFC0907500), the Key Laboratory of Medical Information Intelligent Technology Chinese Academy of Medical Sciences, the National Population and Health Scientific Data Sharing Program of China, and the Knowledge Centre for Engineering Sciences and Technology (Medical Centre).