Introduction

Machine learning, especially Deep Learning, is creating breakthroughs in many different fields, and its impact on biomedicine is remarkable. With the exponential growth of biomedical data, researchers are exploring its potential in biomedical engineering, advanced computing, imaging systems, and biomedical data mining algorithms based on machine learning [1]. One important area is Diagnostic Captioning. Diagnostic Captioning is the process of automatically generating diagnostic text based on a set of medical images collected during a medical examination. It can assist less experienced physicians by minimizing clinical errors and helping experienced physicians generate diagnostic reports faster [2].

ImageCLEF is an annual multimodal machine learning campaign, part of the Cross-Language Evaluation Forum (CLEF), which has been running since 2003. It encourages breakthroughs in research and development of processing systems. Advanced multimedia processing in computer vision, image analysis, classification and retrieval in a multilingual, multimodal context. This year, one of ImageCLEF's four main missions is ImageCLEFMedical, which includes a series of challenges from annotating images to creating synthetic images and answering questions. In ImageCLEF 2024 [3], we took part in the ImageCLEFmedical Caption task [4]. As in previous years, this task comprised two subtasks: concept detection and caption prediction.

Concept detection aims to associate biomedical images with related medical concepts while captioning prediction focuses on automatically generating preliminary diagnostic reports that accurately describe medical conditions and structures and anatomy shown in images. Concept detection also supports diagnostic notes by identifying key concepts that should be included in the preliminary report. Additionally, it can be used to index medical images according to related concepts, facilitating more efficient organization and retrieval.

Captioning prediction, in other words, diagnostic captioning, remains a challenging research problem, designed to support the diagnostic process by providing a preliminary report rather than replacing the physicians and human factors involved [2]. It is designed as a tool to assist in generating an initial diagnostic report of a patient's condition, helping doctors focus on important areas of the image [5] and assisting them in making diagnoses. Guess more accurately quickly [6]. This approach can increase the efficiency of experienced clinicians, allowing them to handle high volumes of daily medical examinations more quickly and efficiently. For less experienced clinicians, automated annotation can help reduce the likelihood of clinical errors [7].

DarkCow Team Contributions

In this paper, we presented the experiments and the systems that were submitted by our DarkCow team in this year's caption prediction task, which helped us secure third place on the leaderboard (see Table 1). Our new approaches build on the rapid development of deep learning techniques, especially the Transformer [8] encoder-decoder architecture and the Query Transformer [9] for Large Language Model [10]. We leveraged the Vision Transformer (ViT) to extract visual features from radiology images. To optimize the use of information, we also used VinVL [11] to extract features of objects in the images. Our first approach is based on encoder-decoder architecture to generate image captions. In the second approach, we leveraged Query Transformer to help LLM understand images. We also conducted experiments with image pre-processing, caption length, and object features to analyze the impact of those aspects.

Background and Related Works

Radiology Techniques

With the continuous advancement of imaging technology, medical imaging diagnosis has evolved from a supplementary examination tool to the most important clinical diagnostic and differential diagnostic method in modern medicine. Radiology techniques are used to scan images within the body, which are then interpreted and reported by radiologists to specialists [12]. With advancements in imaging technology, various imaging diagnostic methods have been developed, each with its own advantages and limitations. For example, X-ray imaging [13] offers non-invasive, quick, and painless imaging, but it involves exposure to ionizing radiation, which increases the risk of developing cancer later in life. On the other hand, MRI imaging [14] provides non-ionizing radiation and high spatial resolution, but it has relatively low sensitivity and longer scanning times, etc.

Former Medical Image Captioning Datasets

Medical imaging diagnosis today plays an incredibly important role in both the healthcare and information technology sectors. It not only aids in diagnosis and increases understanding of diseases but also holds immense potential in improving healthcare delivery and enhancing quality of life. The application of deep learning in medical image captioning in an era where AI is ubiquitous is evident; it automates the annotation process and significantly accelerates image analysis. Several datasets have been created to facilitate the training of medical image captioning tasks such as ROCO [15], PadChest [16], MIMIC-CXR [17], IU X-Ray [18], and MedICaT [19].

Related Work Methods

For the task of medical image captioning, various methods have been developed, with pioneering work in applying the CNN-RNN encoder-decoder approach to generate captions from medical images conducted by Shin et al. [5]. They utilized either the Network-in-Network or GoogLeNet architectures as encoding models, followed by LSTM [20] or GRU [21] as the decoding RNN to translate the encoded images into descriptive captions. In the process of translating images into biomedical text, MDNET [22] made a notable advancement by incorporating an attention mechanism. This model employs RESNET for image encoding, extending its skip connections to mitigate gradient vanishing.

In recent studies by Wang et al. [23], Kougia et al. [24], and Li et al. [25], a fusion of generative models and retrieval systems for Medical Image Captioning (MIC) has been explored. For instance, Wang et al. [23] proposed an approach that alternates between template retrieval and sentence generation for rare abnormal descriptions. This method relies on a contextual relational-topic encoder derived from visual and textual features, facilitating semantic consistency through hybrid knowledge co-reasoning. Additionally, Kougia et al. [24] from AUEB NLP group presented various systems for the Image-CLEFmed 2019 Caption task. One approach utilized a retrieval-based model that leverages visual features to retrieve the most similar images based on cosine similarity, combining their concepts to predict relevant captions. Another system incorporated CheXNet [26] with enhanced classification labels, employing a CNN encoder and a feed-forward neural network (FFNN) for multi-label classification. They also suggested an ensemble model by combining these systems, computing scores for returned concepts and merging them with image similarity scores to select the most relevant concepts.

Large language models (LLMs) have catalyzed significant progress in medical question answering; Med-PaLM [27] was the first model to exceed a "passing" score in US Medical Licensing Examination (USMLE). However, this and other prior work suggested significant room for improvement, especially when models' answers were compared to clinicians' answers. Med-PaLM 2 [28] bridges these gaps by leveraging a combination of base LLM improvements, medical domain finetuning, and prompting strategies including a novel ensemble refinement approach.

Dataset

Thanks to AUEB NLP Group for providing an excellent analysis of the dataset in the study of Kaliosis et al. [29]. When comparing ImageCLEFmedical2023 data with ImageCLEFmedical2024, we found no significant differences in the task of caption prediction. Therefore, we decided to reapply to analyze the dataset in this section. This year's ImageCLEFmedical Caption task provided a dataset that includes 70,108 radiology images in the training set, each annotated with medical concepts using UMLS terms and diagnostic captions. The organizers initially divided the dataset into training and validation subsets [30]. Building on previous campaigns, this year's dataset is an updated and expanded version of the Radiology Objects in Context (ROCO) dataset, which is sourced from a variety of biomedical studies in the PubMed Central OpenAccess (PMC OA) subset. The dataset used for the caption prediction task includes images from different modalities, such as X-ray and Computed Tomography (CT), although specific details about the image types were not provided. The goal of the caption prediction task is to generate open-ended diagnostic texts for the medical images (see Figure 1).

Table 2

The ten most common words and their frequencies in the ImageCLEFmedical2024 train set. Preoperative CT scan. 9

In the Caption prediction sub-task, each image has a diagnostic caption describing the described medical condition. There are a total of 69,743 captions in the training dataset and 9,959 captions in the validation dataset, one for each image. Similar to last year's campaign, the majority of captions (99.47%, or 69,743 out of 70,108) were unique. This is a notable difference from previous versions of the quest, where the uniqueness percentage was much lower. As a result, traditional retrieval methods based on nearest neighbor search are less efficient this year, including variants with a weighting mechanism based on the cosine similarity of the retrieved images. Therefore, more complex methods of creating subtitles are needed. We observed that the maximum number of words in a single caption is 848 (occurred once), while the minimum is 1 (encountered 1 time). The average caption length is 20.84 words. These statistics apply to the entire dataset ( training set and valid set). The five most common captions, as well as the ten most popular words (excluding stopwords), can be found in Tables 3 and 2, respectively. In Figure 2 and Figure 3, we present a distribution caption length of the training and valid sets, both indicating that the majority of captions contain fewer than 100 words.

Image Pre-processing 4.1. Denoising

Denoising is crucial in enhancing image quality by reducing the noise while preserving the important details. Noise in medical images can come from various sources, such as sensor imperfections, poor scan conditions, or inherent patient movements during image acquisition.

The smoothness of images is controlled through the utilization of a Gaussian filter with a fixed kernel size. The Gaussian filter operates by smoothing images using a technique called convolution. It employs a Gaussian kernel -a matrix based on the Gaussian function to adjust pixel values. This kernel is applied over each pixel in the image, averaging the pixel values in its vicinity, weighted by their distance from the central pixel. The standard deviation 𝜎 of the Gaussian determines the amount of blurring: a larger 𝜎 results in more blurring, smoothing out more details and noise. This process helps in reducing noise and is often used as a preparatory step in image processing tasks to enhance image quality without losing critical structural details (see Figure 4).

The 2-D Gaussian function is given by:

𝐺(𝑥, 𝑦) = 1 2𝜋𝜎 2 𝑒 − 𝑥 2 +𝑦 2 2𝜎 2 (1)

Medical image enhancement is one of the most widely used medical image processing techniques in medical domain. Its purpose is to improve the visual effect of the image and facilitate the analysis and understanding of the image by humans or machines. The Laplace transform and the Sobel gradient operator are two common ways of performing edge detection, image sharpening, and enabling the enhancement of the image (see Figure 5).

Step 1 Laplace Transform: Apply the Laplace transform to enhance contrast by emphasizing areas of rapid intensity change in the original image.

Step 2 Sobel Operator: Use the Sobel operator to enhance the edges of the image. This step also helps to smooth out noise, making the edges clearer and more cohesive.

Step 3 Smoothing: Smooth the image processed by the Sobel operator using a 3x3 mean filter. This step increases the contrast of the edges against the background.

Step 4 Dot Product: Intensify the contrast by performing a dot product of the smoothed image with the result from the Laplace transform from step 1.

Step 5 Addition for Final Sharpening: Enhance the sharpness and visibility of detail by adding the result of the dot product back to the original image.

Step 6 Histogram Equalization: Apply histogram equalization to distribute the histogram of the image uniformly, improving the overall contrast and making fine details more visible.

Image Enhancement

Figure 5: The image after a series of processing.

Proposed Method

Encoder-Decoder Approach

Features Embedding

We propose VisionDiagnostor centers around the implementation of Transformer encoder-decoder approach and deployed to evaluate methods having ClinicalT5 [31] and BioBART [32] as encoderdecoder module (see Figure 6). ClinicalT5, based on the T5 [33] architecture, and BioBART, a variant of the BART [34] architecture, have both been pre-trained on large of biomedical text data. These models stand out as the preeminent and potent pre-trained language models for the medical domain, ensuring the efficacy and robustness of our proposed method.

Object features: To extract object features in an image, we used the VinVL model to extract object features 𝑅 = {𝑟 1 , 𝑟 2 , ..., 𝑟 𝑘 } from an image, with each 𝑟 𝑖 being a 2048-dimensional vector. Bounding box coordinates are normalized as

𝑏 𝑖 = [︁ 𝑥 𝑚𝑖𝑛 𝑖 𝑤 , 𝑦 𝑚𝑖𝑛 𝑖 ℎ , 𝑥 𝑚𝑎𝑥 𝑖 𝑤 , 𝑦 𝑚𝑎𝑥 𝑖 ℎ

]︁ , forming 𝐵 obj = {𝑏 1 , 𝑏 2 , ..., 𝑏 𝑘 }. Final object features 𝑉 obj are computed by projecting 𝑅 and 𝐵 obj to the language model dimension and summing the results:

𝑉 obj = 𝑅 ′ + 𝐵 ′ obj (2)

We use ViT for visual feature extraction due to its ability to capture global information through its attention mechanism. By freezing ViT and projecting the last hidden state to match the language model's dimension, we obtain visual features 𝑉 . The input embedding to the encoder-decoder module is:

Input = Concat(𝑉, 𝑉 obj )(3)

Where 𝑉 are the visual features from ViT, and 𝑉 obj are the VinVL region object features. The Concat(•) function concatenates these features.

Encoder-Decoder Module

In this task, we employed the Transformer encoder-decoder architecture, which is used in ClinicalT5 [31] and BioBART [32] for the encoder-decoder module of VisionDiagnostor. The encoder receives the input features and then passes them to the decoder to generate the output sentence. In the decoder, attention mechanisms are employed, directing focus to both the output of the encoder and the input of the decoder.

Encoder

Multi-Head Attention:

Attention (Enc) (𝑄, 𝐾, 𝑉 ) = softmax (︂ 𝑄𝐾 𝑇 √ 𝑑 𝑘 )︂ 𝑉(4)

where 𝑄, 𝐾, and 𝑉 are the query, key, and value matrices, and 𝑑 𝑘 is the dimensionality of the key vectors.

Encoder Feed-Forward Network:

FFN (Enc) (𝑥) = ReLU(𝑥𝑊 (Enc) 1 + 𝑏 (Enc) 1

)𝑊

(Enc) 2 + 𝑏 (Enc) 2(5)

where

𝑊 (Enc) 1 , 𝑊(Enc) 2

, 𝑏

, and 𝑏 (Enc) 2 are learnable parameters.

Encoder Layer Normalization:

LayerNorm (Enc) (𝑥) = LN (Enc) (𝑥 + LayerNorm (Enc) (𝑥))(6)

where LN (Enc) is the layer normalization function.

Decoder

Decoder Self-Attention:

Attention (Dec) (𝑄, 𝐾, 𝑉 ) = softmax (︂ 𝑄𝐾 𝑇 √ 𝑑 𝑘 )︂ 𝑉(7)

where 𝑄, 𝐾, and 𝑉 are the query, key, and value matrices, and 𝑑 𝑘 is the dimensionality of the key vectors.

Decoder-Encoder Cross-Attention:

Attention (Dec) (𝑄, 𝐾, 𝑉 ) = softmax (︂ 𝑄𝐾 𝑇 √ 𝑑 𝑘 )︂ 𝑉 (8)

where 𝑄 comes from the decoder and 𝐾, 𝑉 come from the encoder.

Decoder Feed-Forward Network:

FFN (Dec) (𝑥) = ReLU(𝑥𝑊 (Dec) 1 + 𝑏 (Dec) 1

)𝑊

(Dec) 2 + 𝑏 (Dec) 2(9)

where

𝑊 (Dec) 1 , 𝑊 (Dec) 2 , 𝑏(Dec) 1

, and 𝑏 (Dec) 2

are learnable parameters.

Decoder Layer Normalization:

LayerNorm (Dec) (𝑥) = LN (Dec) (𝑥 + LayerNorm (Dec) (𝑥))(10)

where LN (Dec) is the layer normalization function.

Query Transformer Approach

Inspired by the BLIP2 architecture [35], we leveraged the Query Transformer (Q-Former) module, which serves as the trainable intermediary between a fixed image encoder and a fixed Large Language Model. It extracts a consistent number of output features from the image encoder, irrespective of the input image resolution. Q-Former comprises two transformer submodules that share self-attention layers: an image transformer for visual feature extraction from the fixed image encoder and a text transformer acting as both an encoder and decoder. We initialize a set number of learnable query embeddings as input to the image transformer. These queries engage in self-attention and cross-attention interactions with each other and the frozen image features. Additionally, they can interact with the text through self-attention layers, with different attention masks applied based on the pre-training task.

In our experiments, we employ 64 queries, each with a dimensionality of 768, matching the hidden dimension of Q-Former. We utilize VIT-huge [36] as the frozen image encoder and BioMistral-7B [37] as the frozen LLM for caption generation, and we call it VisionDiagnostor-Q-BioMistral which is depicted in Figure 7. This bottleneck architecture, combined with our pre-training objectives, compels the queries to extract visual information most pertinent to the accompanying text.

Evaluation Metrics

BERTScore

BERTScore is computed as proposed by Zhang et al. [38], where the cosine similarity of each hypothesis token 𝑗 with each token 𝑖 in the reference sentence is calculated using contextualized embeddings. Instead of using a time-consuming best-case matching approach, a greedy matching strategy is employed. The F1 measure is then calculated as follows:

𝑅 BERT = 1 |r| ∑︁ 𝑖∈r max 𝑗∈p cos(𝑖 ⃗ , 𝑗 ⃗ ),(11)𝑃 BERT = 1 |p| ∑︁ 𝑗∈p max 𝑖∈r cos(𝑖 ⃗ , 𝑗 ⃗ ),(12)BERTScore = 𝐹 BERT = 2 • 𝑃 BERT • 𝑅 BERT 𝑃 BERT + 𝑅 BERT .(13)

The BERTScore correlates better with human judgments for the tasks of image captioning and machine translation.

Other Metrics

In addition to BERTScore, the competition also uses many other metrics such as ROUGE [39], BLEU-1 [40], BLEURT [41], METEOR [42], CIDEr [43], CLIPScore [44], RefCLIPScore [44], ClinicBLEURT [45] and MedBERTScore [46]. Applying a variety of these metrics helps us have a more accurate and general view of the model performance of participating teams. Each measure has its own advantages and provides a different perspective on text quality that makes it relevant in a medical context. This multi-dimensional evaluation helps identify outstanding models and gain an objective view of the competition.

Experiment Results

Experimental Configuration

All our proposed methods were trained and fine-tuned using the Adam optimization [47]. We utilized an A100-GPU setup with 80GB of memory to train models, taking 10 hours on average for each method.

We set the learning rate to 3e-05, dropout is set at 0.2, batch size is 32, and the training process is terminated after 3 epochs of not finding any reduction in the valid loss. Table 4 presents a comprehensive of the results on the test set achieved by individual models, showcasing their BERTScore and other metrics. The findings underscore significant disparities in performance among the various models, providing valuable insights into their respective strengths and weaknesses. Notably, within the baseline models, VisionDiagnostor-BioBART stands out as the top performer, showcasing an impressive BERTScore of 0.6267 and almost all other metrics with the smallest size at 227M parameters. Moreover, Table 4 demonstrates that using large-scale pre-trained models in VisionDiagnostor-Q-BioMistral with a very large size (8B) does not result in significant performance improvement in this task.

Main Result

Result Analysis

In this section, we conduct a subjective analysis of the valid set due to the limited number of submissions in the competition. This means that instead of using the test set for objective evaluation, we used the valid set to analyze the results our proposed methods achieved.

Impact of Image Pre-processing Table 5

Results of models with image pre-processing in valid set. △ indicates the increase (↑) or decrease (↓) and compares without pre-processing (*).

Model BERTScore

VisionDiagnostor-Q-BioMistral △ 0.6841* ↓ 0.0101 VisionDiagnostor-ClinicalT5 △ 0.7071* ↓ 0.0166 VisionDiagnostor-BioBART △ 0.7165* ↑ 0.0198 Table 5 presents the results comparing the performance of the models with and without image preprocessing on the validation dataset, evaluated using BERTScore. Specifically, for the VisionDiagnostor-Q-BioMistral model, BERTScore decreased from 0.6841 to 0.6740 after applying pre-processing, corresponding to a decrease of 0.0101. Similarly, VisionDiagnostor-ClinicalT5 also saw a decrease in performance from 0.7071 to 0.6905, a decrease of 0.0166. In contrast, VisionDiagnostor-BioBART is the only model with an improvement with BERTScore increasing from 0.7165 to 0.7363, an increase of 0.0198.

Overall, applying image pre-processing does not appear to yield significant improvement for most models. Even for two of the three models (VisionDiagnostor-Q-BioMistral and VisionDiagnostor-ClinicalT5), image pre-processing degrades performance. The reason may be because of the input images are of good quality and have almost no noise. Some images also have clear instructions, such as arrows pointing to the relevant caption of the image (see Figure 1 in Section 3), making it easy for the model to understand and process the content without additional pre-processing. The details of the test set based on different groups of lengths are in Table 6. Classification is done as follows:

Impact of Caption Length

• Short caption: These are captions shorter than 21 words. Specifically, the two models VisionDiagnostor-ClinicalT5 and VisionDiagnostor-BioBART based on the encoder-decoder method have similar trends, both showing a gradual decrease in BERTScore as the caption length increases. This may indicate a limitation in handling longer captions with this method.

It is worth noting that the VisionDiagnostor-Q-BioMistral model represents a different case, with performance increasing as the caption length increases. This may imply that this model is capable of handling longer captions more efficiently than other models, possibly due to its complexity and magnitude.

Impact of Object Features

According to papers from competing teams in previous years [48], [49], [29], the most popular image feature extraction methods today have two main directions: convolutional neural networks (CNN) and Vision transformers (ViT). Studies and demonstrations have shown that ViT often gives better results than CNN in the task of image captioning. ViT is capable of capturing long-term and global relationships in images more effectively, leading to the creation of richer and more accurate captions. However, to improve the quality of feature extraction further, we used the VinVL model. VinVL takes advantage of the power of the ability to detect and represent objects in images in detail. This allows the model to gain a deeper understanding of the context and elements in the image, thereby creating more accurate captions.

Table 7

Results of models with image pre-processing in valid set. △ indicates the increase (↑) or decrease (↓) and compares with models using object features (*).

Model BERTScore

VisionDiagnostor-ClinicalT5 △ 0.7071* ↓ 0.0239 VisionDiagnostor-BioBART △ 0.7165* ↓ 0.0321 Table 7 presents the results of the models when not using object features in the valid set. The figures show that not using object features significantly reduced the performance of the models.

Specifically, the VisionDiagnostor-ClinicalT5 model has a BERTScore of 0.7071 when using object features. However, when not using object features, the performance of this model drops by 0.0239. Similarly, the VisionDiagnostor-BioBART model also shows a significant decrease when not using object features, with BERTScore decreasing from 0.7165 to 0.6844, corresponding to a decrease of 0.0321.

These results indicate that using object features has an important effect in improving model performance. Object features can provide detailed and characteristic information about objects in images, helping models understand and describe images more accurately. Removing object features results in the loss of important information, reducing the model's ability to produce accurate and detailed captions, which in turn reduces BERTScore significantly.

Conclusion and Future Works

In this paper, we have proposed three different models to solve the task of medical image captioning, in other words medical image diagnosis, including VisionDiagnostor-ClinicalT5 and VisionDiagnostor-BioBART based on encoder-decoder architecture, VisionDiagnostor-Q-BioMistral based on BLIP2 architecture with Query Transformer which leveraging the power of Large Language Models (LLM).

Our results show that the VisionDiagnostor-BioBART model achieved third place on the leaderboard, with the highest BERTScore of 0.6267, despite being the smallest in size with only 227M parameters. Additionally, we performed analysis of the results to gain a deeper understanding of the factors that influence the performance of the models, including image pre-processing, caption length, and object features. These analyses have provided the comprehensive insight needed to shape and improve future methods and models for this task.

In future works, our objective is to delve deeper into the applications of other biomedical large language models (LLMs) BioMedLM [50], BioGPT [51], especially focusing on enhancing their capabilities to generate precise captions that are context-sensitive. This development will be pursued through methods like instruction tuning and better alignment of the models with specific user requirements. In addition, we plan to explore the integration of dense retrieval techniques into the biomedical image captioning process [52]. By adopting frameworks akin to Retrieval Augmented Generation, we intend to supplement the LLMs with an external, non-parametric memory using a FAISS index [53], thereby enriching their reasoning capabilities. Another area of interest will be investigating the interconnections between these approaches. We also anticipate evaluating the qualitative variations in the captions generated through these different methodologies to ascertain their efficacy and practicality in real-world applications.