This article describes PUC Chile team's participation in the Caption Prediction task of ImageCLEFmedical challenge 2021, which resulted in the team winning this task. We first show how a very simple approach based on statistical analysis of captions, without relying on images, results in a competitive baseline score. Then, we describe how to improve the performance of this preliminary submission by encoding the medical images with a ResNet CNN, pre-trained on ImageNet and later fine-tuned with the challenge dataset. Afterwards, we use this visual encoding as the input for a multi-label classification approach for caption prediction. We describe in detail our final approach, and we conclude by discussing some ideas for future work.
ImageCLEF [
In this document we describe the participation of our team from the HAIVis group1 within
the artificial intelligence laboratory 2 at Pontificia Universidad Catolica de Chile (PUC Chile
team) in the image captioning task at MedicalImageCLEF 2021 [
The rest of the paper is structured as follows: section 2 describes our data analysis, while in section 3 we provide details of our proposed methods and experiments for model training and validation. Later, in section 4 we provide details of our results, and finally in section 5 we conclude our article.
The dataset provided for this challenge consists of two sets of 2,756 and 500 image-caption pairs for training and validation, respectively. Each caption consists of a natural language text, which is a highly technical annotation made by physicians about abnormalities and medical objects in the image it corresponds to.
Each caption was processed with the NLTK library3 [
Figure 2 shows the distribution words per number of appearances in the dataset, for example, 28% of words have only one occurrence. Figure 3 shows the distribution of caption lengths.
While addressing the task we tried three main approaches: a pure statistical method, a multi-label classification approach ( MLC) and a perceptual similarity based model (Sim).
Our initial approach to the challenge tried to leverage the statistics related to the composition of each caption. This first model was a naive algorithm that randomly selected a caption length from the training set, created a list of this length with the most popular words in the dataset, and shufled them to get a random order. This simple method obtained a mean BLEU score of 0.357 on the validation set and, when submitted, scored 0.378 points in the test set.
This first approach helped us gain an intuition about how the BLEU score varied and how susceptible it was to diferent components of the caption. Our initial hypothesis was that more relevant than the order of words in the caption, the correctness of them was the most significant element for the metric. To test this assumption, we explore the alternative of a multi-label classification approach that, given an image, predicted the most relevant words in the caption.
In this approach we consider each word as a class, and trained a Convolutional Neural Network
(CNN) to predict the words of a caption given an image. Then, the top classified words are
selected and ordered by a statistical rule to produce a final caption. Figure 5 shows the full
pipeline for caption generation with our approach, and we give more details next5:
5For all our implementations we used PyTorch as our main DL framework: https://pytorch.org/
3.2.1. Preprocessing
We process each image-caption pair to reduce the number of target classes (words), and to
prepare the image to pass it through the network. The following steps were applied:
1. Caption processing: We processed each caption according to the evaluation
methodology described in the previous section, transforming each caption into a list of stemmed
words (labels). The vocabulary is composed by all the words in the training data with
four appearances or more. We did not perform any special handling for words in the
validation set that were not present in the training vocabulary. After filtering, the training
vocabulary size was reduced to 1,075 words (1,189 when using training and validation
set).
2. Image processing: Each image is transformed to have pixel values within a [
In addition, the last layer of the network was replaced with a fully connected layer that
matched the dimensionality of the training vocabulary size. Furthermore, we added a dropout
layer and passed the resulting values by a Parametric ReLU (PReLU) [
In training, we sought to minimize the Binary Cross Entropy loss between the vector predicted by the model and the one-hot encoded ground truth, calculated as:
= ∑︁ − [ · log () + (1 − ) · log(1 − ())]
where is the number of labels to classify. In code, this loss was calculated with BCEWithLogits7
function from pytorch. As optimizer, we used Adam[
3.2.3. Captioning Once the classification output is obtained from our visual model, it needs to be translated into a caption. We define N as the length of the output caption, a hyper-parameter of the model and choose the N highest scoring words. Then, we used a statistical approach to order the words in a logical sentence: for each word, we define its position as the most common one it has across all training captions.
Two output examples from our model are shown next, with good (Fig. 6) and bad (Fig. 7) performance:
Another method that we used and resulted in a fairly good experimental performance was a
similarity-based approach. For each test image, we ranked the most similar images in the training
set using the Learned Perceptual Image Path Similarity (LPIPS) 8[
This approach resulted in a very good test performance and helped us to reach and maintain the top 3 in the leaderboard. Furthermore, we tested this approach for the concept detection task where we also achieved better performance.
To evaluate our model we measured the BLEU score [
Additionally, we measured word recall as a metric for the classification method. Since BLEU is a precision-based metric, including a recall-based metric should help evaluate the performance
more extensively, leading to better captions.
The best result was achieved with the multi-label classification approach, using a ResNet34
[
Four submissions were made to crowdai.org using the methods described, the details and results are shown in Table 2.
Subm1 Subm3 Subm4 Subm6
In this article we have provided details of the participation of the PUC Chile team, winners of the caption prediction task within the ImageCLEFmedical challenge 2021. In the process of building our final submission, we tested several approaches, detailed in this paper. Our final submission was based on a ResNet34 architecture to visually encode the input medical image, followed by predicting captions as a multi-label word classification task, and finally re-ranking the word order based on statistical information from the training dataset. In future work, we plan at testing other CNN architectures, perform further experiments exploiting perceptual similarity, and test other techniques for neural language modeling.
This work was partially funded by ANID - Millennium Science Initiative Program - Code ICN17_002 and by ANID, FONDECYT grant 1191791.