This paper provides an overview of the Medical Visual Question Answering for Gastrointestinal Tract (MedVQA-GI) challenge held at ImageCLEF 2023, a new challenge that combines visual-text question answering with colonoscopy analysis. The challenge is divided into three tasks, each tackling a diferent aspect of visual-text question answering. The first task focuses on answer generation based on an image and question, the second task focuses on question generation based on a given set of images and questions, and the last task is segmentation mask generation based on a given image and question. The paper includes details on the data collection and description, task specifics, evaluation methods, participation, and challenge results.
Identifying lesions within gastrointestinal (GI) images is a popular application of machine
learning with much research behind it [
The challenge is divided into three tasks, each with its own unique requirements. First,
the VQA task asks participants to combine image data with textual questions to generate an
answer to a question. This requires understanding the image and the textual data, requiring
a multimodal approach to produce accurate results. Second, the Visual Question Generation
(VQG) task demands participants to generate text-based questions derived from a given image
and an associated answer. This task requires an understanding of the image and the answer to
produce appropriate questions. Last, the Visual Location Question Answering (VLQA) provides
participants with an image and a question about the location of a certain object, like a polyp,
which should then be segmented and returned by the system. We see this challenge as a good
opportunity to have the medical computer vision community contribute to a relatively new and
novel use-case in medical image analysis, and see this as a perfect fit for ImageCLEF 2023 [
The rest of this paper is organized as follows. We start with an explanation of the dataset creation, it gives insights into how data was collected, validated, and organized. Then, we discuss the specific tasks involved in the MedVQA challenge and the evaluation methods used. In terms of participation, the document outlines key statistics that reveal participants’ geographic and institutional diversity. Finally, the paper presents the results submitted by the participants.
The data used for this challenge is based the HyperKvasir dataset [
For Task 1 (VQA) and Task 2 (VQG), we provided a set of 2, 000 image samples for the development set and 1, 949 for the testing dataset. It is important to note that not all questions directly correspond to the image’s content, meaning that some questions only have relevant answers for some of the images. In such instances, the submissions should be able to handle cases where there is no correct or relevant answer. For Task 3, segmentation masks are included for segmentation training and evaluation. These segmentation masks highlight specific regions of interest within the image. The masks only apply to certain parts of the whole dataset, namely those containing polyps and surgical equipment. An overview of the questions are shown in Table 1, which includes the associated question ID and the expected answer type. Samples from the dataset can be seen in Figure 1, including the ground truth format included in the development dataset.
As mentioned before, the visual parts of the dataset are taken from the HyperKvasir [
{ annotations. It is worth noting that, due to lack of time, not all samples were validated by the medical experts. A more complete version of the challenge dataset will be released in the future, which will contain more samples and complete verification by the domain experts.
This section details the three tasks that are part of this challenge: VQA, VQG, and VLQA. Each task is designed to assess diferent aspects handling both textual and visual data. The scripts used to verify and evaluate the submissions were provided in our public GitHub repository1.
The VQA task challenges participants to generate accurate and descriptive text-based answers in response to given text questions and corresponding images. An example scenario might involve an image portraying a colon polyp, accompanied by the question, "Where in the image is the polyp located?" In response, participants should provide a textual description specifying the polyp’s location within the image. Such a description could refer to spatial locations like "upper-left" or "center". This task gauges the participants’ proficiency in interpreting medical images and translating that interpretation into clear, spatially-referenced, text-based answers.
For submission to this task, participants were instructed to generate a javaScript object notation (JSON) file that encapsulates their task responses corresponding to each image in the designated test dataset. Each entry within this JSON file should contain an entry for each image provided in the testing dataset, where each entry contains 20 question-answer pairs. Here, the questions are pre-defined, while the corresponding answers are by the participant’s system. A visualization of the submission format for Task 1 is shown in Figure 2.
Submissions were evaluated by first preforming a set of prepossessing steps on the answers 1https://github.com/ImageCLEF/2023_ImageCLEFmed_VQA and ground truth, then comparing them directly and calculating the metrics. We used standard accuracy as the primary metrics, which was provided on a global-level, question-level, and image-level.
The VQG task inverts the first task’s approach by asking participants to generate text questions based on a given text answer and an image pair. For instance, if the provided answer is "The image contains a polyp", and the accompanying image indeed contains a polyp, the participant should generate a question such as "Does the image contain an abnormality?". The complexity of this task lies in its requirement for a deep understanding of the image content and the ability to formulate relevant questions based on that understanding.
Submissions for this task were open, and participants were allowed to submit whatever they pleased. This could include, for example, software, source code, or system-specific documentation. Evaluations for this task were subjective, and performed by the challenge organizers, with no objective score tied to it.
The VLQA task extends beyond text-based responses, asking participants to create segmented parts of an image based on a given text question and image pair. This task diverges from the previous two tasks as it necessitates a visual output—a segmentation mask—rather than a textual one. Consider a scenario where the question posed is "Where is the abnormality?" and the provided image contains a polyp. The expected output is a segmentation mask outlining the polyp’s location in the image. The VLQA task, therefore, assesses the participants’ competence in identifying and visually demarcating areas of interest within the medical imagery, based on text-based inquiries.
Submissions to this task is quite similar to the first but instead of a textual answer, the answer is a segmentation mask corresponding to the posed question. Here, participants were asked to submit a JSON file in the format as described in Task 1, answering the questions "Where exactly in the image is the polyp?" and "Where exactly in the image is the instrument?", where the answers would be the name of the mask file that was also included in the submission.
Team Name wsq4747 BITM SSNSheerinKavitha SSN_KDC utk VisionQAries DLNU_CCSE UIT-Saviors Total
This task was evaluated by comparing the participants’ outputs with the ground truth segmentation masks, using standard segmentation metrics like precision, recall, Dice, and Intersection over Union (IoU).
This section provides an overview of the participation in the challenge, and discusses the results submitted by those who completed it. An overview of the submissions to each task is shown in Table 2.
In total, 26 teams signed up for the task, 8 teams submitted runs to solutions, 6 teams wrote and
submitted a working notes paper. Although a drop from 26 down to 6 may seem like a lot, our
experience with previous challenges has shown that about 1/4 of registrations end up finishing
the challenge [
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
VisionQAries
BITM we will develop a separate ground truth and include more strict task requirements. Looking at Figure 3, we see that most teams achieve reasonably good results, with two teams reaching above 80% accuracy. Looking at Figure 4, it looks like "What color is the abnormality?" (Q6) and "Where in the image is the abnormality?" (Q11) were the most dificult questions. We believe this is mostly due to the subjective nature of these questions. For example, the color of the abnormality can vary based on the user making the annotations.
More details about the results of the specific teams can be found in their corresponding
working notes paper [
This paper presented the MedVQA-GI challenge, which was held for the first time at ImageCLEF 2023. The challenge presented three tasks related to visual-text question answering and had eight participants submit results to at least on of the three available tasks. We believe that this is a promising start for the MedVQA-GI challenge, with several quality submissions. In the future, we plan on expanding the dataset to cover diverse conditions and instruments, refining evaluation metrics, and adding a larger and more diverse question set. Furthermore, we would like to expand on Task 2 to be more robust and include stricter participation criteria. on Multimedia Systems Conference (MMSYS), 2017, pp. 112–123. doi:https://doi.org/ 10.1145/3193740. [6] J. Silva, A. Histace, O. Romain, X. Dray, B. Granado, Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer, International Journal of Computer Assisted Radiology and Surgery 9 (2014) 283–293. doi:https://doi.org/10. 1007/s11548-013-0926-3. [7] V. L. Thambawita, D. Jha, H. L. Hammer, H. D. Johansen, D. Johansen, P. Halvorsen, M. Riegler, An extensive study on cross-dataset bias and evaluation metrics interpretation for machine learning applied to gastrointestinal tract abnormality classification, ACM Transactions on Computing for Healthcare (2020). [8] D. Jha, M. Riegler, D. Johansen, P. Halvorsen, H. Johansen, Doubleu-net: A deep convolutional neural network for medical image segmentation, in: Proceeding of the International Symposium on Computer Based Medical Systems (CBMS), 2020. [9] Q. Angermann, J. Bernal, C. Sánchez-Montes, M. Hammami, G. Fernández-Esparrach, X. Dray, O. Romain, F. J. Sánchez, A. Histace, Towards real-time polyp detection in colonoscopy videos: Adapting still frame-based methodologies for video sequences analysis, in: Proceedings of Computer Assisted and Robotic Endoscopy and Clinical Image-Based Procedures (CARE CLIP), volume 10550, Springer, 2017, pp. 29–41. [10] K. Pogorelov, M. Riegler, P. Halvorsen, P. T. Schmidt, C. Griwodz, D. Johansen, S. L.
Eskeland, T. de Lange, Gpu-accelerated real-time gastrointestinal diseases detection, in: Proceedings of the International Symposium on Computer-Based Medical Systems (CBMS)„ IEEE, 2016, pp. 185–190. doi:https://doi.org/10.1109/CBMS.2016.63. [11] M. Riegler, K. Pogorelov, P. Halvorsen, T. de Lange, C. Griwodz, P. T. Schmidt, S. L. Eskeland, D. Johansen, EIR - eficient computer aided diagnosis framework for gastrointestinal endoscopies, in: Proceedings of the IEEE International Workshop on Content-Based Multimedia Indexing (CBMI), 2016, pp. 1–6. doi:https://doi.org/10.1109/CBMI.2016. 7500257. [12] Y. Wang, W. Tavanapong, J. Wong, J. H. Oh, P. C. De Groen, Polyp-alert: Near real-time feedback during colonoscopy, Computer Methods and Programs in Biomedicine 120 (2015) 164–179. doi:https://doi.org/10.1016/j.cmpb.2015.04.002. [13] D. Jha, P. H. Smedsrud, M. A. Riegler, D. Johansen, T. De Lange, P. Halvorsen, H. D.
Johansen, Resunet++: An advanced architecture for medical image segmentation, in: Proceedings of the International Symposium on Multimedia (ISM), 2019, pp. 225–230. doi:https://doi.org/10.1109/ISM46123.2019.00049. [14] J. Bernal, A. Histace, M. Masana, Q. Angermann, C. Sánchez-Montes, C. Rodriguez, M. Hammami, A. Garcia-Rodriguez, H. Córdova, O. Romain, G. Fernández-Esparrach, X. Dray, J. Sanchez, Polyp detection benchmark in colonoscopy videos using gtcreator: A novel fully configurable tool for easy and fast annotation of image databases, in: Proceedings of Computer Assisted Radiology and Surgery (CARS), 2018. doi:https: //hal.archives-ouvertes.fr/hal-01846141. [15] Y. Guo, J. Bernal, B. J Matuszewski, Polyp segmentation with fully convolutional deep neural networks—extended evaluation study, Journal of Imaging 6 (2020) 69. [16] M. Min, S. Su, W. He, Y. Bi, Z. Ma, Y. Liu, Computer-aided diagnosis of colorectal polyps using linked color imaging colonoscopy to predict histology, Scientific reports 9 (2019) answering, in: CLEF2023 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece, 2023. [40] S. Hicks, D. Jha, V. Thambawita, P. Halvorsen, H. Hammer, M. Riegler, An Overview of the EndoTect Challenge at ICPR 2020, in: ICPR2020, 2020. doi:10.1007/ 978-3-030-68793-9_18. [41] S. Hicks, V. Thabawita, H. L. Hammer, T. B. Haugen, P. Halvorsen, M. Riegler, ACM MM BioMedia 2020 Grand Challenge Overview, in: ACMMM2020, ACM MM ’20, Association for Computing Machinery, New York, NY, USA, 2020.