Medical image captioning is an important and challenging task, which covers computer vision and natural language processing. This ImageCLEF 2019 [6] Caption competition is dedicated to research this eld. The purpose of this year challenge is using radiological images to detect the concepts representing the key information. In this paper, we illustrate the proposed method to address the issue, based on multilabel classi cation model and CNN-LSTM architecture with attention mechanism. We also perform a detailed analysis and processing for the overall dataset and demonstrate performance with the baseline in the caption prediction task. In nal evaluation, we completed 9 submissions and ranked second among 12 participants with our best mean F1-score.
Medical images, such as radiological images, are widely used in hospital
diagnosis and disease treatment. The reading and summarization of medical images is
usually performed by experienced medical professionals, and obtaining
information from radiological medical images is a time-consuming and laborious task.
Therefore, it is essential to automatically and e ciently extract vital
information from medical images. ImageCLEF 2019 Caption [
Inspired by the recent successes of convolutional architectures on other
endto-end frameworks [
The paper is organized as follow: Section 2 describes the analysis of the overall data, Section 3 introduces the method for the concept detection task, Section 4 demonstrates the details of the experiments and results, and Section 5 discusses and concludes our work. 2
In the ImageCLEF 2019 Concept Detection Task, the overall dataset contains
70,786 radiology images of several medical imaging modalities. The images are
collected from open access biomedical journal articles (PubMed Central) [
To further understand the datasets, we performed statistical analysis to reveal the overall data distribution. The Top-10 concepts descriptions, and the statistics of the length of concept sequence corresponding to each image and the distribution of concept frequency are shown in Table 1 and Fig. 1, respectively. From Table 1, we can see that among the 10 concepts of high frequency, the C0043299 and C1962945, or the C0040395 and C0040405 have similar meanings. Thus, we conducted correlation analysis for all pairs of concepts. See section 4.1 for details. We counted the length of the concept sequences corresponding to each images in dataset. Only one or two concepts of the sequence account for 17.09% of the overall sample. From Fig. 1, the total number of concepts with a frequency less than 3 is 2,293, accounting for 41.48% of the entire concept dictionary, while there are only 15 concepts with a frequency of more than 3,000 times. We design two distinct methods to address the concept detection issue, one is to transform the issue into a multi-label classi cation problem, the other is to treat it as an image captioning task, using the encoder-decoder network to generate the concepts. Since there is no strong contextual correlation between the concepts of an image, we transform this task into a multi-label classi cation problem. That is, an image has several labels. Let li be the label of i-th image, as follows: li = [ci;1; ci;2; : : : ; ci;j ; : : : ci;n] (1) where n is the total number of labels. If the i-th image has the j-th label, the ci;j is set to 1, else 0.
We utilize the latest deep learning method to solve this problem, which has
achieved great success on the eld of image processing, such as classi cation,
captioning. Empirically the deeper the network is, the richer the features extracted
on di erent levels. While the drawbacks of gradient vanishing and explosion make
it di cult to converge. To overcome this problem, He et al. proposed ResNet [
In our approach, the encoder built upon the pre-trained ResNet-101 is rst
applied to extract visual features from the input images. We resize the input
images normalized by the mean and standard deviation to 224 224 for uniformity,
and then ne-tuned the convolutional blocks on the given medical dataset with
a smaller learning rate. We utilize the visual features captured by the conv 5
convolution block in the ResNet to better describe the local information.
Meanwhile, the model combine soft attention mechanism [
In decoder, We apply a long short-term memory (LSTM) network [
Concept association mining It has been found that some concepts have a certain relevance as they often appear simultaneously in di erent radiological images. Thus, we lter out the high-correlation concept combinations from the high-frequency concepts in training dataset. First, we utilize association rule mining to search for relationships between all concepts de ned as a set of items C = fc1; c2; c3; : : : cM g, and I = fi1; i2; i3; : : : iN g represents a collection of all training samples, where im is the concept sets corresponding to each image. Obviously, iN C. The form of association rule for the concept sets X and Y can be written as: X ! Y , where X C, Y C, and X \ Y = ;. The support is the fraction of training set that contain both X and Y , and the conf idence represents the measure that how often concepts in Y appear in sample sets that contain X. We suppose represents the frequency of occurrence of an item-set. Speci cally,
support(X ! Y ) = conf idence(X ! Y ) = (X [ Y )
N (X [ Y ) (X) (2) (3)
Second, the concept subsets divided with support > 0:02 is a total of 99, and from which we select the combinations contain with the most elements with conf idence > 0:9. Finally, we de ne 9 di erent concept combinations as 9 new concepts, called concept grouping, as shown in Table 2. During the training process, we replace the concepts involved with the 9 new concepts and de ne the dataset changed as Cg, and then map them in predicted results. Data ltering It is obvious that the dataset is extremely unbalanced through the statistics above. The low-frequency concepts would not only not be learned, but bring great bias to the model. Therefore, we lter out the concepts which are indeed rare. Firstly, we pick out all the concepts which only occurs once and get the corresponding images. Then checking all the related concepts on each image one by one, if the frequencies of all related concepts are once either, the image would be moved out of the dataset. The ltered dataset denotes as Df;1 with 163 concepts and 98 images omitted.
At the same time, we also roughly lter out the concepts with frequency less than 3 or 5 times de ned Df;3 and Df;5, to avoid these noises a ecting the overall dataset distribution.
Data redivision Since the pre-divided training dataset provided by organizer dose not contain all the concepts need to be learned, we re-divided all the data as follows:
(a) Picking out the images form validation dataset, as mentioned above, which has the concepts that are not occurred in training dataset.
(b) Changing these images slightly by random transformation, such as mirroring, rotation, etc.
(c) Appending these transformed images to the training dataset, while the original validation dataset keeps unchanged.
Image preprocess and normalization In order to make full use of the provided data, several data augmentation operations are introduced in our experiment. Speci cally, the image is rstly random ipped horizontally or vertically with a probability of 0.5, and then resized to di erent scale ranging from 0.6 to 1.2 with bilinear interpolation. Finally, the transformed image is random cropped into the size of 224 224 before input into the backbone net. 4.2
Multi-label classi cation method Parameters of the last fully connection
layer are initialized by MSRA method [
coder, the model was trained with cross entropy loss for 30 epochs, batch size of 20 and dropout rate of 0.5. In concept generation, we set the dimensions of all hidden states and word embeddings as 512. We used the Adam optimizer and the learning rates for the CNN and the LSTM were 1e 4 and 4e 4 respectively. Early stopping was used to prevent over- tting when performance on a validation dataset started to degrade. The best model saved was used to predict the sequence of concepts in the test images. 4.3
We completed a total of 9 graded submissions before the deadline, the evaluation results for our submitted runs is shown in Table 3 and the speci c method for each run is as follows: Run ID 27103: In this run, we applied multi-label classi cation model introduced in Section 3.1 and the dataset trained was ltered dataset Df;1. We chose the pre-trained ResNet-101 model as backbone in our experiment and performed concept grouping Cg in data preprocessing. Meanwhile, we applied the Adam optimizer to ne-tune the model with an initial learning rate of 1e 3, and the max epoch was 100 in training procedure.
Run ID 27184: This process is similar to ID 27103. We chose the pre-trained ResNet-101 model and the learning rate is 1e 3. The multi-label classi cation model was trained with ltered dataset Df;1 except for the concept grouping strategy and the max epoch was 60 in training procedure.
Run ID 26786: This run we utilized the CNN-RNN architecture with attention mechanism, based on pre-trained ResNet-101 and LSTM. We used the Adam optimizer and the learning rates for the CNN and the LSTM were 1e 4 and 4e 4 respectively. In the training dataset Df;3, concepts occurring less frequently than 3 was ignored. Early stopping was used, and the best model saved was used to predict the sequence of concepts in the test images.
Run ID 27107: We combined the predicted results of ID 27103 and ID 26786, that is, the nal results in test dataset was the union of two methods for each sample.
Run ID 27106: Similarly, the nal results in this run was the union of the predicted results of ID 27184 and ID 26786.
Run ID 27188&26877&27111&27158: These process were based on ID 26786,
the pre-trained ResNet-101 was used for the vision model, and an LSTM was
employed to generate sentences. The dataset trained was ltered dataset Df;5.
Otherwise, we made an attempt to apply reinforcement learning [
Overall, we have completed this challenge in the medical image concept detection task and our group rank second among 12 participants (see Table 4). The method adopted has achieved preliminary results and we will further investigate the medical image captioning task based on higher quality datasets and advanced deep learning algorithms.