The second edition of the Medical Visual Question Answering for the Gastrointestinal Tract (MedVQA-GI) challenge at ImageCLEF [1] introduces a new goal that focuses on the use of generative models of text-to-image in medical diagnosis. This combines natural language processing and image generation to potentially improve diagnostic processes in healthcare by providing more comprehensive datasets that can be used for training machine learning models. In contrast to last year's focus on a Visual Question Answering (VQA) task that required retrieving images or masks from user questions, this year's overall goal was to use generative models to create synthetic medical images from textual inputs. Participants were tasked with generating synthetic images using existing generative models developed using a dataset derived from last year's MedVQA-GI challenge [2].
Machine learning has been a common method used to identify lesions in gastrointestinal (GI) images [3,4,5,6,7,8,9,10,11,12,13]. Traditionally, the emphasis in GI analysis has been on disease detection from images or videos, focusing mostly on polyp detection [14,15,16,17,18,19,20]. Several challenges have demonstrated consistent advancements in this field, including some challenges we have organized in the past [21,22,23,24,25]. However, there has been a growing interest in extending the capabilities of image analysis GI through the generation of synthetic images [26,27]. This new focus aims to develop models that generate realistic GI images that can be used in-place of real data. Such synthetic images can be used to train medical professionals, refine diagnostic algorithms without the privacy concerns of real patient data, and improve the interpretability and reliability of AI systems in clinical settings. To this end, this year's MedVQA-GI focuses on synthetic GI image generation. The dataset and the scripts used to verify and evaluate submissions are available in our public GitHub repository 1 .
The remainder of this paper is organized as follows. First, we start with an explanation of the creation of the dataset, looking at how the data was collected and organized. Then, we discuss the specific sub-tasks involved in the MedVQA-GI challenge and the evaluation methods used. Finally, we present statistics on the participants and the results of the submitted runs.
Generate an image containing a polyp.
Generate an image from a colonoscopy procedure.
Generate an image containing text.
Generate an image containing oesophagitis.
Generate an image from a gastroscopy procedure.
Generate an image containing metal clip.
Generate an image containing biopsy forceps.
Generate an image containing the z-line.
Generate an image with 2 findings.
Generate an image containing tube.
Generate an image with a polyp of size < 5mm. Generate a polyp of type paris iia.
Examples from the development dataset that was provided by the challenge organizers. The samples represent different types of images contained within the dataset. Under each image is a prompt that is associated with an image. Note that there can be multiple prompts that match to the same image.
The dataset used for this challenge is based on data developed for last year's challenge, which was based on the HyperKvasir dataset [28] and the Kvasir-Instrument dataset [29] datasets. Participants were provided with a dataset consisting of 2, 000 image and text pairs, which was organized in a directory containing the images and CSV files with prompts and connections to the image filenames. Example images and corresponding prompts can be seen in Figure 1.
This year, participants could participate in two sub-tasks: Image Synthesis and Optimal Prompt Generation. Participants could submit to either sub-task and were not limited to the number of submissions.
The first sub-task, Image Synthesis, involves using text-to-image generative models to construct a comprehensive dataset of medical images from textual descriptions. This sub-task requires participants to create accurate visual representations of various medical conditions described solely in text. For example, with a description such as "An early-stage colorectal polyp," participants must generate an image that precisely reflects the given text. Participants could use the development dataset, described in Section 2, to develop their models.
For the submission, each participant received a list of 5, 000 prompts. They were required to create synthetic images based on these prompts and submit them to the organizers by email. Each submitted image file was named according to the prompt's index number from the list. The quality of the synthetic images was assessed using two metrics: the Inception Score (IS) [30] and the Fréchet Inception Distance (FID) [31]. These metrics evaluated how the synthetic images were compared with three distinct testing datasets. The first data set consisted of images from the previous year's MedVQA-GI challenge. The second dataset was GastroVision [32], which is a newly released open-source collection that includes 8, 000 images obtained from various medical centers. The final data set used for the evaluation was a combination of the first two datasets.
The second sub-task, Optimal Prompt Generation, focuses on participants creating their own prompts to generate images that meet specific medical imaging requirements. This sub-task asked participants to tailor their prompt generation skills to produce images that accurately match a set of predefined categories. These categories are designed to test the model's ability to generate precise and clinically relevant images based on the prompts. Participants had to devise prompts for:
• A prompt that generates an image containing n polyps.
• A prompt that generates a polyp in a specific region of the image.
• A prompt that generates a polyp of a specific type and size.
• A prompt that generates an image containing no findings from either the esophagus or large bowel. • A prompt that generates an image containing one of the following instruments: biopsy forceps, metal clip, and tube. • A prompt that generates an image containing one of the following anatomical landmarks: Z-line, Pylorus, Cecum.
For evaluation, the effectiveness of each prompt was evaluated not only on the accuracy of the image it produced but also on the conciseness of the prompt itself. Shorter and more precise prompts were preferred, as they are more beneficial in clinical settings where clarity and efficiency are necessary. Additionally, the generated images were subjected to the same quantitative evaluation metrics as in sub-task 1, IS and FID, to ensure consistency in assessing the quality of images across different tasks. This dual approach, which combined qualitative assessment of prompt effectiveness with quantitative image quality metrics, provided a comprehensive assessment of participants' proficiency in generating both relevant prompts and high-quality synthetic images.
This section provides an overview of participation in the challenge and discusses the results submitted by those who completed it.
In total, 22 teams signed up for the task, 2 teams submitted runs, and 2 teams submitted working notes papers [33,34]. Table 1 shows an overview of the participants and the number of submissions to each sub-task alongside the number of participants from last year's challenge. This year experienced a noticeable decline in participation compared to last year's challenge. One reason for this may be the complexity of the task and the hardware and model requirements. Furthermore, future editions could benefit from enhanced outreach and support mechanisms, such as tutorials or "getting started" scripts, to broaden participant engagement and lower the entry barriers.
A total of six runs were submitted to sub-task 1, and no runs were submitted to sub-task 2. This section gives an overview of the results of each run and briefly discusses the approach submitted by each participant. The results can be seen in Table 2.
Team MMCP's approach was based on two methods: fine-tuning Kandinsky models and implementing a Medical Synthesis with Diffusion Model (MSDM). They fine-tune pre-trained Kandinsky models to generate images from text prompts. In addition, they experimented with MSDM, showing improved results over Kandinsky-based models. Example images of each sumbission can be seen in Figure 2. For more information on their approach, please read their working notes paper [33].
Generate an image not containing text.
Generate an image containing the z-line.
Generate an image containing tube.
Generate an image containing a polyp.
Generate an image from a colonoscopy.
Generate an image from a gastroscopy.
Team 2 used a different approach for each submission. The first approach did not generate synthetic images, rather it retrieved images that closely related to the input prompt. To do this, they used a Connecting text and images (CLIP) model. The second submission used a fine-tuned stable diffusion model that generated synthetic images. The third submission used a fine-tuned Low-Rank Adaptation of Large Language Models (LoRA) model to generate images. This method uses LoRA to modify preexisting stable diffusion model to enable the production of high-quality images that closely align with the input specifications. Example images of each submission can be seen in Figure 3.
Results for Task 1. Each submission is evaluated using the FID and the Inception Score (IS). The FID scores is calculated against the MedVQA testing datasert (Single), GastroVision (Multi), and a combination of the two (Both). The IS socre is calculated on a 10-way split of the synthetic images, where we display the mean (avg), standard deviation (sd), and median (med). Generate an image from a gastroscopy procedure.
Generate an image from a colonoscopy procedure.
The challenge results highlight several important insights and areas for further exploration. Firstly, the performance across the two teams and runs varied. This variability underscores the complexity of creating high-quality medical images. However, we found that the quality of the images did not always correspond to the scores provided by the quantitative metrics, suggesting that we need more robust synthetic image quality metrics specifically for medical images and their applications. Another notable finding was that there was some confusion surrounding generation of synthetic images. One team submitted a run that retrieved "real" images that corresponded to the submitted prompt. This deviated from the intended goal, as the main point was to generate synthetic images. This highlights the need for clearer communication of the challenge requirements.
Furthermore, reduced participation compared to last year indicates possible entry barriers that may include the complexity of tasks or a lack of foundational resources for newcomers. Addressing these barriers could involve providing more comprehensive datasets, detailed examples of successful implementations, and potentially simplifying the challenge structure to attract a broader range of participants.
This paper discussed the second edition of the MedVQA-GI challenge, which took place at ImageCLEF in 2024. The challenge consisted of two sub-tasks centered on the generation of synthetic images in the gastrointestinal tract. In the future, we plan on making a more robust task with more resources to get started. Furthermore, we also want to merge the tasks from the first year with this year's challenge to keep the task more consistent.