1 Introduction
Deep learning methods are being developed to automatically interpret biomedical
images in order to help clinicians who examine large numbers of images daily [
10
]. The
ImageCLEFmed Caption task [
12
] is part of ImageCLEF 2019 [
6
].1 Image CLEF is a
campaign that suggests novel challenges and develops benchmarking resources for the
evaluation of systems operating on images. The ImageCLEFmed Caption Task ran for
the 3rd year in 2019. It included a Concept Detection sub-task, where the goal was to
perform multi-label classification of medical images by automatically selecting medical
concepts that should be assigned to each image. The concepts come from the Unified
Medical Language System (UMLS).2 Selecting the appropriate concepts per image can
be a first step towards automatically generating image captions, longer medical reports,
and can also assist, more generally, in computer-assisted diagnosis [
9
]. In the two
previous years, ImageCLEFmed also included a Caption Prediction (generation) sub-task
[
2, 4
], which was not included this year.
This paper presents the four Concept Detection systems that AUEB’s NLP Group
used to participate in ImageCLEFmed 2019 Caption. The systems were ranked 1st, 2nd,
3rd, and 5th. The system that was ranked 3rd consists of a DenseNet-121 [
5
]
Convolutional Neural Network (CNN) image encoder and a k-Nearest Neighbors (k-NN)
retrieval component that uses the encoding of the image being classified to retrieve similar
training images with known concepts; these are then used to assign concepts to the new
image. The top-ranked system is a re-implementation of CheXNet [
14
], with
modifications for ImageCLEFmed Caption 2019. CheXnet also uses the DenseNet-121 encoder
[
5
], combined with a feed-forward neural network (FFNN) that performs multi-label
classification. The second-best system is an ensemble combining concept probability
scores obtained from the CheXNet-based system and image similarity scores produced
by k-NN retrieval of similar training images. The system ranked 5th uses the VGG-19
image encoder [
15
], which was also used by Jing et al. [
7
], combined with a FFNN for
multi-label classification.
2
Data
The ImageCLEFmed Caption 2019 dataset is a subset of the Radiology Objects in
COntext (ROCO) dataset [
13
]. It consists of medical images extracted from open access
biomedical journal articles of PubMed Central.3 Each image was extracted along with
its caption. The caption was processed using QuickUMLS [
16
] to produce the gold
UMLS concept unique identifiers (CUIs). An image can be associated with multiple
CUIs (Figure 1). Each CUI is accompanied by its corresponding UMLS term.
In ImageCLEFmed Caption 2017 [
2
] and 2018 [
4
], the datasets were noisy. They
included generic and compound images, covering a wide diversity of medical images;
3https://www.ncbi.nlm.nih.gov/pmc/
there was also a large total number of concepts (111,155) and some of them were too
generic and did not appropriately describe the images [
18
]. In the ROCO dataset,
compound and non-radiology images were filtered out using a CNN model. This led to
80,786 radiology images in total, of which 56,629 images were provided as the training
set, 14,157 as the validation set, and the remaining 10,000 images were used for testing.
In ImageCLEFmed Caption 2019, the total number of UMLS concepts was reduced to
5,528, with 6 concepts assigned to each training image on average. The minimum
number of concepts per training image is 1, and the maximum is 72. Table 1 shows the 6
most frequent concepts of the training set and how many training images they were
assigned to, according to the gold annotations. We note that 312 of the 5,528 total
concepts are not assigned to any training image; and 1,530 concepts are assigned to only
one training image.
CUI UMLS term Images
C0441633 diagnostic scanning 6,733
C0043299 x-ray procedure 6,321
C1962945 radiogr 6,318
C0040395 tomogr 6,235
C0034579 pantomogr 6,127
C0817096 thoracics 5,981
We randomly selected 20% of the training images and used them as our development
set (11,326 images along with their gold concepts). The models we used to produce the
submitted results were trained on the entire training set. The validation set was used for
hyper-parameter tuning and early stopping.
3
Methods
This section describes the four methods we developed for ImageCLEFmed Caption
2019.
3.1
System 1: DenseNet-121 Encoder + k-NN Image Retrieval (Ranked 3rd)
In this system, we followed a retrieval approach, extending the 1-NN baseline of our
previous work on biomedical image captioning [
9
]. Given a test image, the previous
1-NN baseline returned the caption of the most similar training image, using a CNN
encoder to map each image to a dense vector. For ImageCLEFmed Caption 2019, we
retrieve the k-most similar training images and use their concepts, as described below.
We use the DenseNet-121 [
5
] image encoder, a CNN with 121 layers, where all
layers are directly connected to each other improving information flow and avoiding
vanishing gradients. We started with DenseNet-121 pre-trained on ImageNet [
1
] and
fine-tuned it on ImageCLEFmed Caption 2019 training images.4 The fine-tuning was
4We used the implementation of https://keras.io/applications/#densenet.
performed as when training DenseNet-121 in System 2, including data augmentation
(Section 3.2). Without fine-tuning, the performance of the pre-trained encoder was
worse. ImageCLEFmed Caption 2019 images were rescaled to 224 224 and
normalized with the mean and standard deviation of ImageNet to match the requirements of
DenseNet-121 and how it was pre-trained on ImageNet. Having fine-tuned
DenseNet121, we used it to obtain dense vector encodings, called image embeddings, of all
training images. The image embeddings are extracted from the last average pooling layer of
DenseNet-121. Given a test image (Fig. 2), we again use the fine-tuned DensNet-121 to
obtain the image’s embedding. We then retrieve the k training images with the highest
cosine similarity (computed on image embeddings) to the test image, and return the r
concepts that are most frequent among the concepts of the k images. We set r to the
average number of concepts per image of the particular k retrieved images. We tuned
the value of k in the range from 1 to 200 using the validation set, which led to k = 199.
Further fine-tuning may improve performance further. This system ranked 3rd.
This system, which is based on CheXNet [
14
], achieved the best results in
ImageCLEFmed Caption 2019. In its original form, CheXNet maps X-rays of the ChestX-ray
14 dataset [
17
] to 14 labels. It uses DenseNet-121 [
5
] to encode images, adding a FFNN
to assign one or more of the 14 labels (classes) to each image.
We re-implemented CheXNet in Keras5 and extended it for the many more labels
(5,528 vs. 14) of ImageCLEFmed Caption 2019. The images of ImageCLEFmed
Caption 2019 were again rescaled to 224 224 and normalized using the mean and standard
deviation values of ImageNet. Also the training images of ImageCLEFmed Caption
2019 were augmented by applying random horizontal flip. Image embeddings are again
extracted from the last average pooling layer of DenseNet-121. In this system, however,
the image embeddings are then passed through a dense layer with 5,528 outputs and
sigmoid activations to produce a probability per label. We trained the model by minimizing
binary cross entropy loss. We used Adam [
8
] with its default hyper-parameters, early
stopping on the validation set, and patience of 3 epochs. We also decayed the learning
rate by a factor of 10 when the validation loss stopped improving.
At test time, we predict the concepts for each test image using their probabilities, as
estimated by the trained model. For each concept (label), we assign it to the test image if
the corresponding predicted probability exceeds a threshold t. We use the same t value
for all 5,528 concepts. We tuned t on the validation set, which led to t = 0:16.
3.3
System 3: Based on Jing et al., VGG-19 Encoder + FFNN (Ranked 5th)
This system is based on the work of Jing et al. [
7
], who presented an encoder-decoder
model to generate tags and medical reports from medical images. Roughly speaking,
the full model of Jing et al. uses a VGG-19 [
15
] image encoder, a multi-label classifier
to produce tags (describing concepts) from the images, and a hierarchical LSTM that
generates texts by attending on both image and tag embeddings; the top level of the
LSTM generates sentence embeddings, and the bottom level generates the words of
each sentence. We implemented in Keras a simplified version of the first part of Jing et
al.’s model, the part that performs multi-label image classification.
Again, we rescale the ImageCLEFmed Caption 2019 images to 224 224 and
normalize them using the mean and standard deviation of ImageNet. We feed the resulting
images to the VGG-19 CNN, which has 19 layers and uses small kernels of size 3 3.
We used VGG-19 pre-trained on ImageNet.6 We feed whole images to VGG-19, unlike
Jing et al. [
7
], who divide each image into regions and encode each region separately.
The output of the last fully connected layer of VGG-19 is then given as input to a dense
layer with a softmax activation to obtain a probability distribution over the concepts.
The model is trained using categorical cross entropy, which is calculated as:
E =
jCj
X ytrue;i log2(ypred;i)
i=1
where C is the set of jCj = 5; 528 concepts, ytrue is the ground truth binary vector
of a training image, and ypred is the predicted softmax probability distribution over the
concepts C for the training image. Categorical cross entropy sums loss terms only for
the gold concepts of the image, which have a value of 1 in ytrue. When using
softmax and categorical cross-entropy, usually ytrue is a one-hot vector and the classes are
mutually exclusive (single-label classification). To use softmax with categorical cross
entropy for multi-label classification, where ytrue is binary but not necessarily one-hot,
the loss is divided by the number of gold labels (true concepts) [
3, 11
]. Jing et al. [
7
]
achieve this by dividing the ground truth binary vector ytrue by its L1 norm, which
equals the number of gold labels. Hence, the categorical cross-entropy loss is computed
as follows:
E =
jCj
X
ytrue;i
i=1 k ytrue k1
log2(ypred;i) =
1 M
X log2(ypred;j )
M j=1
6https://keras.io/applications/#vgg19
(1)
(2)
where M is the number of gold labels (true concepts) of the training image, which is
different per training image. In this model, the loss of Eq. 2 achieved better results on
the development set, compared to binary cross entropy with a sigmoid activation per
concept. We used the Adam optimizer with initial learning rate 1e-5 and early stopping
on the validation set with patience 3 epochs. Given a test image, we return the six
concepts with the highest probability scores, since the average number of gold concepts
per training image is 6.
3.4
System 4: Ensemble, k-NN Image Retrieval + CheXNet (Ranked 2nd)
This method is an ensemble of System 1 (DenseNet-121 + k-NN Image Retrieval) and
System 2 (CheXNet-based), where System 1 is modified to produce a score for each
returned concept.
Given a test image g, we use System 1 (Fig. 2) to retrieve the k most similar training
images g1; : : : ; gk, their gold concepts, and the cosine similarities s(g; g1); : : : ; s(g; gk)
between the test image g and each one of the k retrieved images. Let C be again the set
of jCj = 5; 528 concepts. For each concept cj 2 C, the modified System 1 assigns to
ci the following score:
v1(cj ; g) =
k
X s(g; gi) (cj ; gi)
i=1
(3)
where (cj ; gi) = 1 if cj is a gold concept of the retrieved training image gi, and
(cj ; gi) = 0 otherwise. In other words, the score of each concept cj is the sum of the
cosine similarities of the retrieved documents where cj is a gold concept.
For the same test image g, we also obtain concept probabilities from System 2,
i.e., a vector of 5,528 probabilities. Let v2(cj ; g) be the probability of concept cj being
correct for test image g according to System 2. For each cj 2 C, the ensemble’s score
v(cj ; g) of cj is simply the average of v1(cj ; g) and v2(cj ; g). The ensemble returns the
six concepts with the highest v(cj ; g) scores, as in System 3, on the grounds that the
average number of gold concepts per training image is 6.
4
Results
Systems were evaluated in ImageCLEFmed Caption 2019 by computing their F1 scores
on each test image (in effect comparing the binary ground truth vector ytrue to the
predicted concept probabilities ypred) and then averaging over all test images [
6
]. Table 2
reports the evaluation results of our four systems on the development and test data, as
well as their ranking among the approximately 60 systems that participated in the task.
The ensemble (System 4) had the best results on development data, but the
CheXNetbased system (System 2) had the best results on the test set.
Ranking
We described the four systems that AUEB’s NLP Group used to participate in
ImageCLEFmed 2019 Caption. The four systems were ranked 1st, 2nd, 3rd, and 5th. Our
top system was a re-implementation of CheXNet [
14
], with modifications to handle
the much larger label set of ImageCLEFmed 2019 Caption and data augmentation. The
system that was ranked 3rd used DenseNet [
5
] to encode images and k-NN retrieval to
return the concepts of the most similar training images. Our second-best system was an
ensemble of the previous two (CheXNet-based and k-NN based), indicating that the two
approaches are complementary. Our weakest system, which nevertheless was ranked
5th, was based on the multi-label classification part of the system of Jing et al. [
7
],
which aims to generate draft medical reports using an encoder-decoder approach.
In future work, we aim to experiment with, combine, and improve upon additional
methods and datasets for medical image captioning. Towards that direction, we recently
published a survey on medical image to text methods [
9
], which we also plan to extend.