Introduction

Medical report generation (MRG), the main application of image captioning [1] in the medical domain, stands as a pivotal task in healthcare, aiming to automatically produce precise and coherent reports delineating the impressions and observations derived from medical images [2,3,4]. High-quality cross-modal annotations, comprising meticulously paired medical image-report datasets, are essential for the success of MRG. Despite the emergence of a few open-source datasets fostering research in this domain [5,6,7], their limited scale poses significant challenges for the development of deep and expansive neural architectures.

Recent advancements in vision and language applications have led to the rise of large-scale pretrained models, termed foundation models (FMs), designed for general applicability, and demonstrate versatility across various tasks, owing to prompt engineering or parameter-efficient fine-tuning (PEFT). The scalability of both model and data enables FMs to acquire emergent capabilities, empowering them to tackle tasks previously deemed challenging for smaller models [8,9,10]. In the pursuit of effective MRG, researchers have explored methodologies to harness the capabilities of FMs while addressing the scarcity of labelled medical image-report pairs. One notable approach is the BLIP-2 architecture [11], a state-of-the-art vision-language pre-training methodology that facilitates knowledge transfer from single-modality vision and language foundation models.

In this study, we present a novel approach to address the challenges in medical image captioning and report generation using medical vision-language foundation models (Med-VLFMs). Drawing inspiration from the BLIP-2 and its adaptations in the medical domain [11,12,13], our Med-VLFM incorporates a lightweight query Transformer (Q-Former) to connect three foundation models (FMs): an ensemble of EVA ViT-g [8] and BiomedCLIP [14] serving as the vision encoder, and Pangu-𝛼 [15] as the language decoder. Our proposed Med-VLFM introduces two innovative techniques: vision encoder ensembling (VEE) and modality-aware adaptation (MAA). VEE combines general and medical specialist vision foundation models to create a more comprehensive representation of medical images. MAA calibrates vision and text features using small sets of trainable parameters in LLMs. Through experimentation on the ImageCLEFmedical 2024 caption prediction challenge [16,17], our Med-VLFM ranked 1 𝑠𝑡 on most of the automatic evaluation metrics, including BERTScore, ROUGE-1, BLEU-1, BLEURT, METEOR, CIDEr and RefCLIPScore, demonstrating the effectiveness of VEE and MAA in enhancing medical image captioning and report generation.

Related Works

Medical/Radiology Report Generation Motivated by the rapid development of image captioning [1,18,19,20,21], the field of medical report generation has seen significant research interest in recent years. Different from pure-text scenarios like chatting with patients [22,23,24] and discharge instruction generation [25], medical report generation needs to "translate" medical images into detailed reports. As such, one line of research focus on improving the cross-modal alignment between medical images and reports, which is usually achieved by reinforcement learning [3,26], contrastive learning [27,28], well-designed modules like hierarchical attention [29] and memory [30]. Given that generating accurate reports requires domain expertise, another line of research opts to provide models with effective priors through retrieval [3,31,32] or augment models with knowledge [28,31,33]. However, the language models used in these medical report generation work are typically shallow and may lack the capacity to capture the nuances of context and execute complex reasoning.

Medical Vision-Language Foundation Models

Recent advancements in conversational AI have shown promise in aiding biomedical practitioners. LLaVA-Med [34] proposes an efficient approach to training a vision-language conversational assistant for answering biomedical image research questions. Med-PaLM [35] provides high-quality answers to medical inquiries, while R2GenGPT [36] enhances Radiology Report Generation by aligning visual features with language model embeddings. Additionally, XrayGPT [37] introduces a conversational medical vision-language model for analyzing chest radiographs. While these models signify significant progress in multimodal conversational AI for the medical domain, their efficacy relies heavily on the quality and quantity of paired training samples. As such, Med-MLLM [38] learns radiology representations from unlabelled data to quickly deploy tools for rapid response to rare diseases. Despite the above efforts, the potential benefits of incorporating multiple vision models have yet to be explored.

Parameter-Efficient Fine-Tuning

With the proliferation of foundation models (FMs) [9,10], efficiently adapting FMs to a specific task becomes a research hotspot. One effective technique is prompt engineering [39], which aims to affect the behaviors of language FMs by providing them with a textual template filled with task-related priors [40,41], demonstrations of several examples [42,43], or a chain of thoughts [44,45,46]. Alongside prompt engineering, parameter-efficient fine-tuning (PEFT) has also emerged as a popular technique to influence the intermediate hidden states and final responses of FMs. In implementation, PEFT either introduces lightweight components, e.g., Adapter [47], continuous prompts [41,48], and LoRA [49], vectors that scale the inner activations [50], into FMs, or adapts a small portion of inherent weights of FMs [51]. Recent practices utilizing these two techniques have demonstrated the effectiveness of adapting general-purpose FMs to the medical domain [52,53,54]. Distinct from these practices, our work focuses on medical report generation.

Approach

Overview As shown in Figure 1, our proposed Med-VLFM for medical/radiology report generation comprises four major components: vision encoders parameterized by Θ 𝑣 = {𝜃1 𝑣 , 𝜃 2 𝑣 }, a query Transformer (Q-Former) parameterized by 𝜃 𝑞 , a LLM parameterized by 𝜃 𝑡 , and modality-aware adaptation parameterized by ∆ = {𝛿 𝑣 , 𝛿 𝑡 }. Given the medical image 𝐼 and the target caption/report y = {𝑦 1 , 𝑦 2 , . . . , 𝑦 𝑁 }, our model minimizes the following negative log-likelihood:

ℒ = − 𝑁 ∑︁ 𝑛=1 log 𝑝(𝑦 𝑛 |y <n , 𝐼; {Θ 𝑣 , 𝜃 𝑞 , 𝜃 𝑡 , ∆}).(1)

Next, we elaborate on each component of our model.

Vision Encoder Ensembling

We consider general-purpose and specialist vision encoders to produce comprehensive visual representations and thus adopt EVA-ViT-g [8] pre-trained on 29.6 million natural images and BioMedCLIP [14] pre-trained on 15 million medical images crawled from the scientific publications in PubMed Central 1 . We use the fine-tuned weights from our previous work [12] for EVA-ViT-g and the official weights for BioMedCLIP. Following BLIP-2 [11], we remove the last layer of these encoders and use the second last layer's output features. To stabilize training, a layer normalization layer [55] is added to the end of each vision encoder. As different vision encoders may have their distinct image processing pipelines, we adhere to their original settings and feed images of resolution 364 × 364 and 224 × 224 into EVA-ViT-g and BioMedCLIP, resulting in features

𝑅 1 ∈ R 𝐾 1 ×𝑑 1 and 𝑅 2 ∈ R 𝐾 2 ×𝑑 2 ,

respectively. For fusion, we apply the non-parametric bicubic interpolation operation on 𝑅 2 to obtain 𝑅 ′ 2 ∈ R 𝐾 1 ×𝑑 2 first and then concatenate 𝑅 1 and 𝑅 ′ 2 along the channel dimension to attain the final visual features

𝑅 ∈ R 𝐾 1 ×(𝑑 1 +𝑑 2 ) . In our case, 𝐾 1 = 676, 𝐾 2 = 196, 𝑑 1 = 1408, 𝑑 2 = 768.

Query Transformer (Q-Former) Directly feeding 𝑅 into the subsequent language modeling process will introduce redundancy and lead to high memory and computational costs due to the quadratic nature of the standard self-attention mechanism [56]. Thus, we adopt a Q-former with 𝐿 (𝐿 ≪ 𝐾 1 ) learnable query tokens to aggregate 𝑅 into visual tokens 𝑉 ∈ R 𝐿×𝑑 llm . In practice, we follow the setting of [13]: Q-Former is a randomly initialized BERT-like encoder [57] and has 𝐿 = 32 query tokens, 6 Transformer blocks with hidden size 𝑑 𝑞 = 768 and cross-attention layers inserted at a frequency of 2, and a linear projection layer that maps 𝑑 𝑞 to 𝑑 llm .

Large Language Model (LLM)

We use Pangu-𝛼 [58] for language modeling according to our computation budget. In particular, we replace its vocabulary with that of OPT-2.7B [59], since Pangu-𝛼 was trained mainly with Chinese words. To ensure the alignment between randomly initialized token embeddings and the other weights of Pangu-𝛼, we train token embeddings while keeping the rest of the LLM parameters frozen. As Pangu-𝛼 is a decoder-only Transformer, we prefix its original input sequence, i.e., text tokens 𝑇 ∈ R 𝑁 ×𝑑 llm , with 𝑉 following common practices [36,60] to achieve vision-grounded medical report generation. The resulting LLM's input sequence is denoted as 𝐻 = [𝑉 ; 𝑇 ] ∈ R (𝐿+𝑁 )×𝑑 llm . We note that using more advanced or medical-related LLMs [61,62,63] may yield better performance. We leave this exploration to our future study.

Modality-Aware Adaptation Let's assume the LLM has 𝐽 blocks, each of which is parameterized by

𝜃 (𝑗) 𝑡 (𝑗 ∈ [1, 𝐽]

), and denote 𝐻 (𝑗) = [𝑉 (𝑗) ; 𝑇 (𝑗) ] as the original output of the 𝑗-th block, we can compute 𝐻 (𝑗) as follows:

𝐻 (𝑗) = LLMBlock(𝐻 (𝑗−1) ; 𝜃 (𝑗) 𝑡 ),(2)

where 𝐻 (0) = 𝐻. There are two incoming problems: (1) 𝜃

(𝑗)

𝑡 is optimized for the text-only modality, which can be unsuitable for the mixed-modal input due to the modality gap [64] and (2) using the same set of parameters to learn multimodal representations may limit the collaboration of vision and text modalities [65]. Therefore, we propose modality-aware adaptation (MAA) to mitigate the above two problems. In particular, MAA introduces light-weight adaptation modules independent of blocks and modalities, i.e., the newly added parameters can be defined as

∆ (𝑗) = {𝛿 (𝑗) 𝑣 , 𝛿 (𝑗)

𝑡 }. In this work, we consider a simple implementation of MAA: putting adaptation modules right after each block to calibrate 𝐻 (𝑗) . So Eq. ( 2) is modified as follows:

𝐻 (𝑗) = LLMBlock(𝐻 (𝑗−1) ; 𝜃 (𝑗) 𝑡 ), 𝐻 (𝑗) = Adaptation(𝐻 (𝑗) ; ∆ (𝑗) ) = [𝑉 (𝑖) + Adapter(𝑉 (𝑖) ; 𝛿 (𝑗) 𝑣 ); 𝑇 (𝑖) + Adapter(𝑇 (𝑖) ; 𝛿 (𝑗) 𝑡 )],(3)

where 𝐻 (0) = 𝐻, pre-trained weights 𝜃 (𝑗)

𝑡 are kept frozen, [; ] denotes concatenation along the sequence dimension, and Adapter(•) is a bottle-neck MLP as in [47], i.e., Adapter(𝑥) = 𝑊 up (𝜎𝑊 down (𝑥)). In the implementation, we set the non-linearity activation function 𝜎 as GELU [66] and the bottleneck size of adapters to 64. Moreover, inspired by the zero-initialization technique [49], we randomly initialize 𝑊 up but initialize 𝑊 down with all zeros, so that the intermediate and output features of LLM can be adapted smoothly. [67] to split captions into tokens. Roughly 99% captions contain less than 128 tokens.

Experiments

Experimental Setups

Dataset The development dataset of the ImageCLEFmedical 2024 caption prediction challenge [17] is ROCOv2 [7], which is an updated and extended version of the Radiology Objects in COntext (ROCO) dataset [68]. ROCOv2 provides 70,108, 9,972, and 17,237 radiology images for training, validation, and testing respectively. Images originating from biomedical articles are annotated with one medical caption each. In Figure 2 (a) and (b), we visualize the word clouds of ground-truth reports from the training and validation sets. As we can observe, there are many computed tomography (CT) and X-ray images in the dataset. Besides, some common expressions like "black arrow" indicate that humans have marked a large portion of images.

Metrics

As noted in the guidelines of the competition2 , the major and secondary metrics used for the challenge are BERTScore [69] and ROUGE-1 [70]. Specifically, BERTScore is a model-based metric that calculates the semantic similarity of two sentences. ROUGE-1 measures the number of matching unigrams between a model-generated text and a reference. Besides, we also report BLEU [71], METEOR [72], CIDEr [73], BLEURT [74], CLIPScore and RefCLIPScore [75] to comprehensively evaluate the effectiveness of our proposals. For all metrics, higher is better.

Comparing Model We treat our last year's solution [12] (abbreviated as PCLmed-23) as a baseline. It adopts EVA-ViT-g [8] as the vision encoder, v1.0 ChatGLM-6B3 [76] as the decoder, and adapts ChatGLM-6B with P-Tuning [48] (please see the original paper for more details). We use PCLmed-23's original hyper-parameters and directly evaluate PCLmed-23 on the validation set of ROCOv2. Note that although we do not train PCLmed-23 on ROCOv2 (this year's data), there is a significant overlap between last year's and this year's training data.

Image and Text Processing

During training, we process images with random resized cropping with ratios falling into [0.9, 1.0]. The cropped images are resized to the maximum resolution required by vision encoders, i.e., 364 × 364 in this work. We apply bilinear downsampling on the same cropped images for vision encoders with lower-resolution images as inputs. Unlike PCLmed-23, we keep all symbols in texts this year and truncate them into a maximum length of 128 based on the histograms shown in Fig. 3.

Model Settings Most model details have been introduced in Section 3. Unlike PCLmed-23, we instruct the LLM with an empty text prompt since we found it is slightly better than the text prompt like "generate a medical report for the input image" in our preliminary experiments. Besides, we insert the adaptation modules (i.e., Adapter) into the LLM at a frequency of 2.

Hyper-Parameters

During training, we use AdamW [77] and L2 weight decay of 0.05 to train models with 32 samples per batch for 10 epochs. The learning rate (𝑙𝑟) is increased to 1e-4 in 1,000 warm-up steps and follows a cosine annealing scheduler. We train token embeddings with 0.1 × 𝑙𝑟 to stabilize training. During evaluation, we use the beam search decoding algorithm with a beam size of 3 to generate medical captions and force the model to generate at least 8 tokens and up to 64 tokens. We set the repetition penalty to 2.5 to avoid duplication and the length penalty to 2.0 to encourage generating longer captions.

Quantitative Results

We submit three runs to the ImageCLEFmedical 2024 caption prediction challenge, which correspond to #{2, 3, 5} in Tables 1 and 2. In addition, we also run some ablations locally to highlight the effectiveness of our proposals. Based on quantitative results in Tables 1 and 2, We have the following observations.

• Comparing #{1, 2, 3}, we can see that ensembling BioMedCLIP and EVA-ViT-G boosts performance on all metrics except CLIPScore. This is because CLIPScore is a reference-free metric, meaning that the metric score could be biased by the pre-trained knowledge of CLIP [78]. Instead, RefCLIPScore considers the semantic similarity between references and predictions and #3 outperforms #{1, 2}. From another perspective, the overall performance improvements suggest that different pre-trained vision models may have learned complementary visual features and our adopted ensembling/fusion strategy, i.e., concatenating visual features along the channel dimension, helps produce the holistic representations of medical images. • Comparing #{3, 4}, we can observe that fine-tuning LLM with adapters regardless of vision and text modalities may suffer from performance degradation. For example, #4 performs worse than #3 on 6 out of 11 metrics listed in Table 2. • Comparing #{4, 5}, we can see consistent improvements for all metrics if we adapt vision and text features in LLM independently. This suggests that we should allocate modality-specific parameters within LLM to alleviate modality interference and boost modality collaboration.

In short, the above observations verify the effectiveness of our proposed vision encoder ensembling and modality-aware adaptation. Our final model (#5) surpasses PCLmed-23 and generally boosts performance by a large margin compared with #{1, 2}. Nonetheless, there still exist many potential improvement directions of our proposals, e.g., (1) designing a more advanced ensembling/fusion strategy to unify the intelligence of different vision foundation models for medical tasks and (2) exploring a more reasonable way to insert/allocate modality-specific parameters within LLM.

Qualitative Analysis

In Figure 4, we visualize three qualitative examples, where ground-truth captions and captions generated by different models are presented. We have the following observations.

• All models can precisely identify different imaging modalities, i.e., X-Ray in (a), Computed Tomography in (b), and Ultrasonography in (c). • In Figure 4 (b), we can see that all models can predict the existence of "pericardial effusion", "pleural effusion", and blue and red arrows. However, PCLmed-23 and #{1, 3} fail to ground the abnormalities to the images, i.e., they mistakenly relate "pericardial effusion" to blue arrows and "pleural effusion" to red arrows. This suggests the importance of improving Med-VLFMs' grounding abilities.

• Benefited from the proposed vision encoder ensembling and modality-aware adaptation, our model (#5) captures the abnormality in medical images more accurately, e.g., "bilateral hilar lymphadenopathy" in (a) and "internal jugular vein thrombosis" in (c). • Although our model (#5) generally performs the best, it still makes simple mistakes, i.e., "right" in (c). There are several possible solutions for this problem, e.g., (1) enlarging the image resolution [79] and (2) incorporating positional embeddings into the Q-Former-like connection module [62].

Conclusion

In this study, we propose a parameter-efficient training pipeline for medical image captioning and report generation with single-modality pre-trained vision FMs and LLMs. By introducing vision encoder ensembling and modality-aware adaptation, our method leverages the merits of both general and medical vision models and calibrates vision and text features in the LLM. Our Med-VLFM achieved 1st place in the ImageCLEFmedical 2024 caption prediction challenge, excelling across multiple evaluation metrics, including BERTScore, ROUGE-1, BLEU-1, BLEURT, METEOR, CIDEr, and RefCLIPScore. This success highlights the potential of our approach to significantly enhance medical AI solutions and support radiologists in their work. Source codes with model weights are available at OpenMedIA4 [80].