We have participated in the ImageCLEFmedical2024 caption prediction subtask as team “KDE-MED-CAPTION". Although the caption detection subtask is not a direct target of our team, we have taken advantage of the CUI (Concept Unique Identifier) codes attached to each piece of data, provided for the caption detection subtask. In this paper, we propose a method for medical image captioning, using CUI-based classification, followed by feature similarity. Specifically, according to the frequency of concepts (CUIs) of the training data, we perform multi-class classification in the first step, where we use several diferent Deep Neural Networks (DNNs), varying the number of classes. Once the classification is done, we partition images and captions into the specified number of classes. We then extract features from trained DNNs for training, validation, and test datasets provided by organizers this year from ROCOv2-2023 data, where ROCO is an acronym for Radiology Objects in COntext. Given a test image, we apply our pre-trained DNN to predict the most likely class. Finally, we conduct similarity computation between the feature of a test image and the features of the class predicted, ranking the result by sorting in descending order, so that the caption from the top 1 ranked data, which we believe should be the most appropriate caption for the image. We repeat this process for all the test images, varying the DNNs and the number of classes we have adopted, ending up with 5 runs in our team.
appropriate caption for the image. We repeat this process for all the test images, varying the DNNs and the number of classes we have adopted.
Image captioning has been studied almost for decades. It is interpreted as describing the content of a
given image in words ordinarily through natural language processing. Popular benchmark datasets
for image caption include COCO [
On the other hand, image captioning for medical images where gray-scale images are dominant, has shorter history compared with the same task for general color images such as COCO and Flickr30k. As far as the authors can tell, medical image captioning in ImageCLEF has begun sometime around 2017, and has been one of main tasks in ImageCLEFmedical.
It should be noted that medical image dataset such as ROCO (Radiology Objects in COntext) [
Examples of recent approaches to image-text matching are as follows: CLIP or Contrastive LanguageImage Pre-training [12] was proposed for matching images and texts. BLIP or Bootstrapping LanguageImage Pre-training [13] was introduced to outperform CLIP by filtering the noise generated by CLIP. BLIP was also used for image captioning as well as visual question answering. BLIP-2 was proposed [14] to generate a descriptive text given an image. Last year, our team used CLIP-based approach to ImageCLEF2023 medical image captioning [15].
Popular evaluation criteria for medical image captioning include BLEU [16], ROUGE [17], METEOR [18], and CIDEr [19], typically has been used in machine translation.
Since 2023, CLIPScore [20] and BERTScore [21] have been added as evaluation measures. From year 2024, other evaluation criteria are added, including BLEURT [22], RefCLIPScore [20], ClinicalBLEURT [23], and MedBERTScore [23]. Currently BERTScore is the oficial main evaluation measure for caption prediction subtask.
Since the beginning of the Medical Image Caption task in ImageCLEF2017, the number of medical images and their associated captions has increased year by year. At the same time, medical images themselves have become more diverse and multimodal in terms of UMLS (or CUI (Concept Unique Identifier)) concepts. The volume of data itself has also increased to the extent that the total word vocabulary returned by an ordinary tokenizer (e.g., NLTK word tokenizer [24]) exceeds 100,000 even on training data alone. For example, for diversification and multimodality, it is possible to take an image of an organ with diferent devices such as CT, MRI, and ultrasound. If we could pre-classify medical images into diferent classes, it may be possible to reduce the risk of adding “t1" and “t2" in captions for CT and ultrasound images, as these words are specific to medical images taken by MRI devices. We have also experienced that a recent deep learning algorithm sufers from computational time and space, due to the larger amount of data to process in an epoch loop. The idea of our approach is rooted in the above observation.
Based on the above observation, we decided that a DAC (Divide-and-Conquer) approach would be appropriate for handling this amount of data. In implement the DAC approach, we have taken advantage of CUI codes or UMLS concepts, for the subtask of caption detection. First, we took statistics (i.e., frequency) of CUI codes for the training data. The top twelve frequencies of CUI codes are shown below, where the first column corresponds to the frequency number of each CUI code, the second column corresponds to the short description of the code, and the third column is the CUI code itself in square brackets:
A natural DAC approach might be to take the top of several classes according to the ranking as above. We have noticed that the top two (i.e., CT (Computed Tomography) and Plain X-ray) have a large numerical dominance compared to other CUI codes. It should also be noted that each image has one or more CUI codes. Taking this into account, we believe that by appropriately prioritizing the CUI codes, it is possible, to some extent to balance the number of data in each class. For example, a chest x-ray image should naturally be classified in “X-ray group" without fine level priorities. However, if we have “Chest group" and a higher priority than “X-ray group", we believe “Chest group" takes advantage of collecting chest x-ray images before “X-ray group" devours them. Specifically, we have developed an approach to partition the data based on two priorities using the following algorithm: Algorithm 1: Priority-based Partition input , , 1, 2 begin for ∈ .() do /* is an image file name */ − ← [] − ← for ∈ do if ∈ 1 then /* if CUI code is included in priority1 */ [].() − ← if then
continue else − ← [0]/* head element */ if ∈ 2 then /* if CUI code is included in priority2 */ [].()
In efect, for the case of 10 groups of training data, with the first scan by priority1 = [‘C0002978’,‘C0041618’, ‘C0037303’, ‘C0023216’, ‘C1140618’, ‘C0817096’] and the second scan by priority2 = [‘C0024485’, ‘C0040405’, ‘C1306645’], we have the following number: [ C0002978 ] Angiography, 3965 [ C0041618 ] Echo (Ultrasonography), 8333 [ C0037303 ] Bone, 3323 [ C0023216 ] Lower (Extremity), 2331 [ C1140618 ] Upper (Extremity), 1126 [ C0817096 ] Chest, 7411 [ C0024485 ] MRI, 9729 [ C0040405 ] CT, 18686 [ C1306645 ] (Plain) X-ray, 4435 [ other CUIs] Misc, 824
In this way, introducing a priority-based partitioning algorithm makes it possible to decrease the dominance of CT and (plain) X-ray group numbers. Note that Bone, Chest, Upper, and Lower groups’ CUI might be taken from not always the first column in caption concepts data as illustrated in Figure 1.
Apart from partitioning given data, in order to infer a better caption for each unknown image in the test dataset, we have applied simple filtering to remove some of the verbose expressions in the caption ifles in the 2023 data. Our filtering strategy is as follows: • To remove expressions containing “Permission” or “Source” in the last part of the caption • To remove expressions related to annotations from images such as (arrow), (arrowhead), and (circle) • To trim too long captions • To replace number expressions such as Figure 1 by (plain) Figure • To remove the starting sentence’s parentheses such as (1) and (a)
The idea behind this preprocessing comes from the fact that we do not synthesize an image caption, but rather search for the best matching caption from the 2023 caption data. In other words, we think that the retrieved caption should be as simple sentences as possible, so that acknowledgment of the source (item 1), annotations (item 2), excessively long and detailed captions (item 3), figure numbers (item 4), and non-ascii symbols at the beginning of the caption (item 5) should be avoided as much as possible. As a specific example of “excessibly long and detailed caption", in the training caption dataset, ROCOv2_2023_train_046629 has more than 400 words. 4.3. Classification of Each Group by Deep Neural Networks Based on the subdivided groups as described in the previous sections, we have formulated deep neural networks to classify images from ImageCLEF2024 caption prediction data. Specifically, we have adopted EficientNet [ 25], ResNeXt [26], and Vision Transformer [8] of our deep neural networks for classification. The reason for adopting these DNN models lies in our relevant research into the detection of cardiac diseases where these DNNs turned out to perform quite well among several DNNs. The output of each image by these DNNs is a feature vector. EficientNet (or EficientNetB0) has 1,280 dimensions, ResNeXt (or ResNeXt50) has 2,048 dimensions, and Vision Transformer (or ViT32) has 768 dimensions, respectively. The feature vector is extracted from one layer before the last one of the original DNN, pre-trained by ImageNet. It should be noted that we have performed each classification with a specific DNN using training data from ImageCLEF2023 to train the DNN and validation data to evaluate the DNN in PyTorch. Then, we have extracted feature vectors from all the data of each group, as well as from the test data of ImageCLEF2024.
In this stage, we compute the similarity between the feature vectors in the test dataset and feature vectors in each group. Then we sort the result in descending order. We simply select the most similar element and return the caption of the element.
The overall process in this stage is illustrated in Figure 2. Thus, the caption for the unknown input image can be retrieved from the captions belonging to the predicted group.
Our evaluation results (run1 to run5) with the test data received from the organizers are summarized in Table 1. As shown in Table 1, we have tested three diferent numbers of groups; 6, 10, and 11. Initially we thought 10 groups seemed the best among these. Thus, in Runs 1,2, and 3, we keep DNN as EficientNetB0. It turned out that 11 groups seemed a slightly better than the other number of groups. In practice, however, we kept 10 groups while varying DNNs (from EficientNet to ResNeXt and ViT32). As a result, the ViT32 DNN model with 10 groups proved to be the best among our submissions.
Table 2 summarizes the numbers of all the evaluation measures returned by the roganizer.
In this paper, we proposed a method for medical image captioning, using CUI-based classification, followed by feature similarity. Specifically, we took DAC (Divide-and-Conquer) approach, where we first divided the data (training, validation, and test data of ImageCLEF2023) into 5, 10, or 11 groups (classes) in advance. We then used several Deep Neural Network (DNN) models, including EficientNet, ResNeXt, and Vision Transformer, to the training data to obtain a trained DNN model from each DNN. For images in each group, we extracted features from one layer before the last layer of the pretrained DNN. Similarly, we extracted features from the test images. Finally, we computed feature vector similarity between these two to find out the most similar image in each group. We returned the caption corresponding to the most similar image from each group.
Even though we took a simple and no natural language processing approach, the result returned by organizers showed that the BERT Score of our team seemed quite close to other teams.
A part of this research was carried out with the support of the Grant for Toyohashi Heart Center Smart Hospital Joint Research Course and the Grant-in-Aid for Scientific Research (C) (issue numbers 22K12149 and 22K12040). [6] O. Vinyals, A. Toshev, S. Bengio, D. Erhan, Show and tell: A neural image caption generator, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. [7] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: F. Bach, D. Blei (Eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, PMLR, Lille, France, 2015, pp. 2048–2057. URL: https://proceedings.mlr.press/ v37/xuc15.html. [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=YicbFdNTTy. [9] A. Tran, A. Mathews, L. Xie, Transform and tell: Entity-aware news image captioning, in:
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. [10] L. Ke, W. Pei, R. Li, X. Shen, Y.-W. Tai, Reflective decoding network for image captioning, in:
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. [11] W. Lin, Z. Zhao, X. Zhang, C. Wu, Y. Zhang, Y. Wang, W. Xie, PMC-CLIP: Contrastive languageimage pre-training using biomedical documents, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part VIII, Springer-Verlag, Berlin, Heidelberg, 2023, p. 525–536.
URL: https://doi.org/10.1007/978-3-031-43993-3_51. doi:10.1007/978-3-031-43993-3_51. [12] M. V. Conde, K. Turgutlu, Clip-art: Contrastive pre-training for fine-grained art classification, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 3951–3955. doi:10.1109/CVPRW53098.2021.00444. [13] J. Li, D. Li, C. Xiong, S. Hoi, BLIP: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation, in: K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, S. Sabato (Eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, PMLR, 2022, pp. 12888–12900. URL: https://proceedings.mlr.press/v162/li22n.html. [14] Y. Tewel, Y. Shalev, I. Schwartz, L. Wolf, Zerocap: Zero-shot image-to-text generation for visualsemantic arithmetic, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 17918–17928. [15] M. Aono, H. Shinoda, T. Asakawa, K. Shimizu, T. Togawa, T. Komoda, Multi-stage Medical Image Captioning using Classification and CLIP, Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), 2023. URL: https://ceur-ws.org/Vol-3497/paper-113.pdf. [16] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040. doi:10.3115/1073083.1073135. [17] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013. [18] A. Lavie, A. Agarwal, METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments, in: Proceedings of the Second Workshop on Statistical Machine Translation, Association for Computational Linguistics, Prague, Czech Republic, 2007, pp. 228–231. URL: https://aclanthology.org/W07-0734. [19] G. Oliveira dos Santos, E. L. Colombini, S. Avila, CIDEr-R: Robust consensus-based image description evaluation, in: Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), Association for Computational Linguistics, Online, 2021, pp. 351–360. URL: https://aclanthology.org/2021.wnut-1.39. doi:10.18653/v1/2021.wnut-1.39. [20] J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, Y. Choi, CLIPScore: A reference-free evaluation metric for image captioning, in: M.-F. Moens, X. Huang, L. Specia, S. W.-t. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021, pp. 7514–7528. URL: https://aclanthology.org/2021.emnlp-main.595. doi:10.18653/v1/2021.emnlp-main.595. [21] T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, in: International Conference on Learning Representations, 2020. URL: https://openreview. net/forum?id=SkeHuCVFDr. [22] T. Sellam, D. Das, A. Parikh, BLEURT: Learning robust metrics for text generation, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 7881–7892.
URL: https://aclanthology.org/2020.acl-main.704. doi:10.18653/v1/2020.acl-main.704. [23] A. Ben Abacha, W.-w. Yim, G. Michalopoulos, T. Lin, An investigation of evaluation methods in automatic medical note generation, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 2575–2588. URL: https://aclanthology.org/2023.findings-acl.161. doi: 10. 18653/v1/2023.findings-acl.161. [24] S. Bird, E. Klein, E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, O’Reilly, Beijing, 2009. URL: https://www.nltk.org/book. doi:https: //my.safaribooksonline.com/9780596516499. [25] M. Tan, Q. Le, EficientNet: Rethinking model scaling for convolutional neural networks, in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 6105– 6114. URL: https://proceedings.mlr.press/v97/tan19a.html. [26] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, K. He, Aggregated residual transformations for deep neural networks, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, IEEE Computer Society, 2017, pp. 5987–5995. URL: https://doi.org/10.1109/CVPR.2017.634. doi:10.1109/CVPR.2017.634.
s r e z i n a g r o e h t y 3 3 2 2 7 5 9 0 5 0