This paper presents the work of ImageSem group in the ImageCLEFmed Caption 2021 task. In the concept detection subtask, we employed the transfer learning-based multi-label classification model as our baseline. We also trained multiple fine-grained MLC models based on manually annotated semantic categories, such as Imaging Type, Anatomic Structure, and Findings, which may reveal clinical insights of radiology images. We submitted 9 runs to the concept detection subtask, and achieved the F1 Score of 0.419, which ranked 3rd in the leader board. In the caption prediction subtask, our first method simply combines detected concepts according to the sentence patterns. The second method used a dual path CNN model for matching images and captions. We submitted 4 runs to the caption prediction subtask, and achieved the BLEU score of 0.257, which ranked 6th among the participating teams. Concept detection, caption prediction, multi-label classification, fine-grained semantic Information and Library, Chinese Academy of Medical Sciences, our Image Semantics group (ImageSem)
The medical track of ImageCLEF[
The concept detection subtask aims to identify the UMLS [
The caption prediction subtask asks participants to generate coherent captions for the entirety of an image,
which requires higher accuracy and semantic interpretability of expression. We also employed two
methods for caption prediction. The first method was the pattern-based combination of concepts identified
in the previous task. The second method was based on the dual path CNN model [
2021 Copyright for this paper by its authors. is commonly used in the image-text retrieval field to match images and captions for instance-level retrieval. This paper is organized as follows. Section 2 describes the data set of the ImageCLEFmed Caption 2021 task. Section 3 presents the methods for concept detection and caption prediction. Section 4 lists all of our submitted runs. Section 5 makes a brief summarization.
The ImageCLEFmed Caption 2021 task is in its 5th edition this year. Compared with previous years, the
released images were strictly limited to radiology, and the number of images and associated UMLS
concepts were reduced. There were 222,314 images with 111,156 concepts in 2018 [
The organizers provided UMLS concepts along with their imaging modality information, for training purposes. We observed that most images were assigned with concepts indicating the diagnostic procedure or medical device, and some images were accompanied by concepts indicating the body part, organ or clinical findings. As shown in Table 1, the high-frequency concepts are concentrated in several specific semantic types. For our experiments, we utilized this feature and manually classified three concept categories for building fine-grained multi-label classification models.
This section describes methods we used in two subtasks. Fig. 1 shows the workflow and submissions of ImageSem in ImageCLEFmed Caption 2021.
In the concept detection subtask, for one thing, we employed the transfer learning-based multi-label classification model to identify overall concepts; for another thing, we paid more attention to the distinction of labels with different semantic types, and focus on three major categories of concepts, which may reveal clinical insights of radiology images.
In our previous work, we employed a transfer learning-based multi-label classification model to assign
multiple CUIs to a specific medical image. This is a classic approach under the condition of limited tag
size and high frequency concepts. In our first method, for modeling overall concepts, we applied the
Inception-V3[
During the training process, we collected 1,586 CUIs from both of training set and validation set as our labels. Then we fine-tuned the models on the validation set. For a given test image, concepts of high probabilities above the threshold were selected as the prediction labels. Empirically, we adjusted the threshold gradually from 0.1 to 0.7 on the basis of the validation set.
In this method, according to the UMLS semantic types, we go further to divide ImageCLEF concepts into four semantic categories, namely Imaging Type (IT), Anatomic Structure (AS), Findings (FDs) and others. Based on the official training set and validation set, we reprocessed the images and associated concepts via our medical image annotation platform.
As shown in Figure 2, for a given radiology image, there are three sources of related concepts. The first
one is ImageCLEF concepts annotated by concept extraction tools and medical doctors. These concepts
are semantically related, but often incomplete, since many images having only one concept. The second
source of concepts are automatically annotated from the given image captions, using the Metamap tool
[
We invited graduate students majoring in medical imaging to label images with reference to visual information, caption descriptions and the above three sources of concepts. The labeling protocol is that each radiology image was assigned with at least one IT label, zero or more AS labels, and zero or more FDs labels. Specifically, ImageCLEF concepts that are difficult to be classified to the above categories, can be assigned to the ‘Others’.
Then we build three image-concept sub collections for training fine-grained MLC models. These collections have same training and validation images, but differentiate in related concepts. Table 2 shows the distribution of different concept categories.
We verified our MLC models based on the re-annotated validation set. The experimental results showed that our model performs well on the prediction of Imaging Type labels, with F1 score of 0.9273. However, the predictions for the other two kinds of labels are far from satisfactory. One possible reason is that there are few images but too many labels for training. It is intuitively understandable that images of the same or similar cases would have a similar anatomic structure or medical findings label. Whereas the data characteristics of this subtask are obviously not suitable for specific diseases, which raised the difficulty to predict accurate body part, organ, or findings.
For generating reasonable image captions, the first method was the pattern-based combination of concepts identified in the previous task. We designed a simple sentence pattern based on the characteristic of captions in the training and validation set, see Table 3. Obviously, the accuracy of concept detection results would directly determine the quality of sentence generation. <image> of <body> demonstrate / show /suggest <findings> <image> demonstrate / show / suggest <findings> in/of/within <body>
Sample synpic24243: Sagittal T1-weighted image of the cervical spine demonstrates cord expansion. synpic19193: Lateral radiograph of the skull shows lytic lesions in the temporoparietal region.
In this method, we employed the algorithm commonly used in the image-text retrieval field to match images and captions for instance-level retrieval. It is based on an unsupervised assumption that every image/test group can be viewed as one class, so each category is equivalent to 1+m (1 image vs m descriptions) samples.
We use the model proposed by Zheng[
The training process includes two stages: in the first stage, we use the instance loss to learn fine- grained differences between intra-modal samples with similar semantics. in the second stage, we use the ranking loss to focus on the distance between the two modalities to build the relationship between the image and text. ● Test In this experiment, we use 16,280 sentences from training set and validation set as candidate captions, each sentence is corresponding to its text feature extracted by text CNN. For each test image, we first extract the image feature by image CNN, and then use the cosine distance to evaluate the similarity between the image and candidate sentences.
When we use the model trained on ImageCLEF datasets, we get the almost same top 10 sentences from 16,280 candidate captions, because the features learned by text CNN between each captions is not discriminative. However, when we test it on the model trained by MSCOCO datasets, each query image can get different sentences, but they do not match either.
concepts of Imaging Types achieved the best F1 score of 0.419, indicating the high precision and coverage of this kind of concepts in radiology images. As to the concepts from other types and baseline results, they introduce more unmentioned words and reduce the overall score. However, in view of our experience on manual labeling, we believe that some unmentioned words may also be helpful in interpreting the given image. Figure 3 shows two examples of our method on the validation set. 03ImagingTypes 02Comb_ImagingTypes_Baseline 07Intersect_06_baseline 04Comb_ImagingTypes_AnatomicStructure 05Comb_ImagingTypes_MedicalFindings 06Comb_ImagingTypes_AnatomicStructure_Findings 08AnatomicStructure 09Findings 01baseline
This paper presents the participation of the ImageSem Group at the ImageCLEFmed Caption 2021 task. We tried different strategies for both subtasks. In the concept detection subtask, we used the transfer learning-based MLC model to detect overall 1,586 concepts. We also trained multiple fine-grained MLC models based on manually annotated semantic categories. One of the lessons is that we have become much clearer about which concepts are clinically relevant to radiology images, and in order to obtain better predictions, the semantic labels of images should be more focused and specific. Furthermore, how to generate a readable description based on clear and clinically meaningful concepts, is still worth exploring.
This work has been supported by the National Natural Science Foundation of China (Grant No. 61906214), the Beijing Natural Science Foundation (Grant No. Z200016).