PCLmed: Champion Solution for ImageCLEFmedical 2024 Caption Prediction Challenge via Medical Vision-Language Foundation Models Bang Yang1,2 , Yue Yu1 , Yuexian Zou1,2 and Tong Zhang1,* 1 Peng Cheng Laboratory, Shenzhen 518055, China 2 ADSPLAB, School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China Abstract Automatically generating captions and reports for medical images has become increasingly important due to the growing workload of radiologists in hospitals. To tackle this challenging task with limited annotation data, there is a rising interest in developing medical vision-language foundation models (Med-VLFMs). These models leverage the capabilities of vision foundation models and large language models (LLMs) and often utilize the parameter-efficient fine-tuning (PEFT) technique. However, current Med-VLFMs face two critical issues: (1) relying on a single vision model to represent the semantics of medical images, and (2) adapting LLMs with PEFT without considering the interference between vision and text modalities. This work presents a novel Med-VLFM with vision encoder ensembling (VEE) and modality-aware adaptation (MAA) to address these limitations. VEE combines the strengths of general and medical specialist vision foundation models to produce a more holistic representation of medical images. MAA introduces two small sets of trainable parameters into LLMs to calibrate vision and text features, respectively. Our proposed Med-VLFM ranked 1๐‘ ๐‘ก on most of the automatic evaluation metrics, including BERTScore, ROUGE-1, BLEU-1, BLEURT, METEOR, CIDEr and RefCLIPScore, in the ImageCLEFmedical 2024 caption prediction challenge. Our code and models are available at https://openi.pcl.ac.cn/OpenMedIA/PCLmed24. Keywords Medical Image Captioning, Vision-Language Models, Medical Vision-Langauge Foundation Models, Parameter- Efficient Fine-Tuning, Vision Encoder Ensembling, Modality-Aware Adaptation 1. Introduction Medical report generation (MRG), the main application of image captioning [1] in the medical domain, stands as a pivotal task in healthcare, aiming to automatically produce precise and coherent reports delineating the impressions and observations derived from medical images [2, 3, 4]. High-quality cross-modal annotations, comprising meticulously paired medical image-report datasets, are essential for the success of MRG. Despite the emergence of a few open-source datasets fostering research in this domain [5, 6, 7], their limited scale poses significant challenges for the development of deep and expansive neural architectures. Recent advancements in vision and language applications have led to the rise of large-scale pre- trained models, termed foundation models (FMs), designed for general applicability, and demonstrate versatility across various tasks, owing to prompt engineering or parameter-efficient fine-tuning (PEFT). The scalability of both model and data enables FMs to acquire emergent capabilities, empowering them to tackle tasks previously deemed challenging for smaller models [8, 9, 10]. In the pursuit of effective MRG, researchers have explored methodologies to harness the capabilities of FMs while addressing the scarcity of labelled medical image-report pairs. One notable approach is the BLIP-2 architecture [11], a state-of-the-art vision-language pre-training methodology that facilitates knowledge transfer from single-modality vision and language foundation models. CLEF 2024: Conference and Labs of the Evaluation Forum, September 09โ€“12, 2024, Grenoble, France * Corresponding author. $ yangbang@pku.edu.cn (B. Yang); yuy@pcl.ac.cn (Y. Yu); zouyx@pku.edu.cn (Y. Zou); zhangt02@pcl.ac.cn (T. Zhang) ย€ https://web.pkusz.edu.cn/adsp/ (Y. Zou)  0000-0003-2019-0377 (B. Yang); 0000-0002-8838-4963 (T. Zhang) ยฉ 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings In this study, we present a novel approach to address the challenges in medical image captioning and report generation using medical vision-language foundation models (Med-VLFMs). Drawing inspiration from the BLIP-2 and its adaptations in the medical domain [11, 12, 13], our Med-VLFM incorporates a lightweight query Transformer (Q-Former) to connect three foundation models (FMs): an ensemble of EVA ViT-g [8] and BiomedCLIP [14] serving as the vision encoder, and Pangu-๐›ผ [15] as the language decoder. Our proposed Med-VLFM introduces two innovative techniques: vision encoder ensembling (VEE) and modality-aware adaptation (MAA). VEE combines general and medical specialist vision foundation models to create a more comprehensive representation of medical images. MAA calibrates vision and text features using small sets of trainable parameters in LLMs. Through experimentation on the ImageCLEFmedical 2024 caption prediction challenge [16, 17], our Med-VLFM ranked 1๐‘ ๐‘ก on most of the automatic evaluation metrics, including BERTScore, ROUGE-1, BLEU-1, BLEURT, METEOR, CIDEr and RefCLIPScore, demonstrating the effectiveness of VEE and MAA in enhancing medical image captioning and report generation. 2. Related Works Medical/Radiology Report Generation Motivated by the rapid development of image captioning [1, 18, 19, 20, 21], the field of medical report generation has seen significant research interest in recent years. Different from pure-text scenarios like chatting with patients [22, 23, 24] and discharge instruction generation [25], medical report generation needs to โ€œtranslateโ€ medical images into detailed reports. As such, one line of research focus on improving the cross-modal alignment between medical images and reports, which is usually achieved by reinforcement learning [3, 26], contrastive learning [27, 28], well-designed modules like hierarchical attention [29] and memory [30]. Given that generating accurate reports requires domain expertise, another line of research opts to provide models with effective priors through retrieval [3, 31, 32] or augment models with knowledge [28, 31, 33]. However, the language models used in these medical report generation work are typically shallow and may lack the capacity to capture the nuances of context and execute complex reasoning. Medical Vision-Language Foundation Models Recent advancements in conversational AI have shown promise in aiding biomedical practitioners. LLaVA-Med [34] proposes an efficient approach to training a vision-language conversational assistant for answering biomedical image research questions. Med-PaLM [35] provides high-quality answers to medical inquiries, while R2GenGPT [36] enhances Radiology Report Generation by aligning visual features with language model embeddings. Addition- ally, XrayGPT [37] introduces a conversational medical vision-language model for analyzing chest radiographs. While these models signify significant progress in multimodal conversational AI for the medical domain, their efficacy relies heavily on the quality and quantity of paired training samples. As such, Med-MLLM [38] learns radiology representations from unlabelled data to quickly deploy tools for rapid response to rare diseases. Despite the above efforts, the potential benefits of incorporating multiple vision models have yet to be explored. Parameter-Efficient Fine-Tuning With the proliferation of foundation models (FMs) [9, 10], effi- ciently adapting FMs to a specific task becomes a research hotspot. One effective technique is prompt engineering [39], which aims to affect the behaviors of language FMs by providing them with a textual template filled with task-related priors [40, 41], demonstrations of several examples [42, 43], or a chain of thoughts [44, 45, 46]. Alongside prompt engineering, parameter-efficient fine-tuning (PEFT) has also emerged as a popular technique to influence the intermediate hidden states and final responses of FMs. In implementation, PEFT either introduces lightweight components, e.g., Adapter [47], continuous prompts [41, 48], and LoRA [49], vectors that scale the inner activations [50], into FMs, or adapts a small portion of inherent weights of FMs [51]. Recent practices utilizing these two techniques have demonstrated the effectiveness of adapting general-purpose FMs to the medical domain [52, 53, 54]. Distinct from these practices, our work focuses on medical report generation. ็ŽฐไปฃไฟกๅทไธŽๆ•ฐๆฎๅค„็†ๅฎž้ชŒๅฎค Advanced Data & Signal Processing Laboratory Framework Overview Query Tokens Input Text Q-Former tied LLM Embedding Vision LN Encoder 1 Visual Tokens Text Tokens Input Self-Attention Feed Forward Network Vision V-Adapter T-Adapter Encoder 2 LN LLM with Modality-Aware Adaptation Frozen Trainable LLM Head Concat Vision Encoder Ensembling Output Text Head CT demonstrating left parotiditis. Figure 1: Overview of our proposed Med-VLFM for medical/radiology report generation. 2024/6/1 ๆฑ‡ๆŠฅไบบ๏ผšๆจ้‚ฆ PKUSZ-SECE 3. Approach Overview As shown in Figure 1, our proposed Med-VLFM for medical/radiology report genera- tion comprises four major components: vision encoders parameterized by ฮ˜๐‘ฃ = {๐œƒ๐‘ฃ1 , ๐œƒ๐‘ฃ2 }, a query Transformer (Q-Former) parameterized by ๐œƒ๐‘ž , a LLM parameterized by ๐œƒ๐‘ก , and modality-aware adap- tation parameterized by โˆ† = {๐›ฟ๐‘ฃ , ๐›ฟ๐‘ก }. Given the medical image ๐ผ and the target caption/report y = {๐‘ฆ1 , ๐‘ฆ2 , . . . , ๐‘ฆ๐‘ }, our model minimizes the following negative log-likelihood: ๐‘ โˆ‘๏ธ โ„’=โˆ’ log ๐‘(๐‘ฆ๐‘› |y