The Medico Multimedia Task focuses on providing multimedia researchers with the opportunity to contribute to diferent areas of medicine using multimedia data to solve several subtasks. This year, the task focuses on transparency within machine learning-based medical segmentation systems, where the use case is gastrointestinal endoscopy. In this paper, we motivate the organization of this task, describe the development and test dataset, and present the evaluation process used to assess the participants' submissions.
Finding and removing colon polyps is an essential step in preventing colorectal cancer. Current procedures have a high miss rate, whereas computer-aided diagnosis systems can reduce the probability that diagnosticians overlook a polyp during a colonoscopy. As machine learning becomes more common in high-risk fields like medicine, the need for transparent systems becomes more critical. In this case, transparency is defined as giving as much detail as possible on the diferent parts that make up a machine learning pipeline, including everything from data collection to final prediction. This task focuses on high-performing, eficient, and transparent algorithms for polyp segmentation.
The Medico Multimedia Task is held for the fifth time at the
MediaEval benchmark. We continue the tradition of using medical
data to develop machine learning models that solve real-world
issues in medicine [
The provided dataset is based on HyperKvasir [
The 2021 edition of the Medico Multimedia Task provides three diferent subtasks, namely the polyp segmentation task, the eficient segmentation task, and the transparent machine learning systems task. The polyp segmentation subtask is the only subtask that is required, the other two are optional. Each task allows for a total of ifve submissions each.
The polyp segmentation task targets high-performing polyp segmentation systems. Using the provided development dataset, participants are asked to develop models that automatically segment the presence of colon polyps in a given image. Submission to this task should be a zip file containing a predicted segmentation mask using the .png file format for each image in the testing dataset. Each predicted mask should use the same resolution as the input image and have the same filename.
Submissions will be evaluated based on the correctness of the predicted masks using various segmentation metrics like pixel accuracy, precision, recall, Sørensen–Dice coeficient (Dice), and Intersection over Union (IoU). The primary metric used to rank the submissions will be IoU. The participants will receive a .csv file containing the evaluation metrics for each run.
The eficient segmentation task aims for eficient segmentation systems while still obtaining a satisfactory prediction accuracy. Model eficiency is measured in the number of frames that a model can process per second. The motivation behind this is the need for real-time detection systems used during live endoscopy procedures. For the system to be considered real-time, it should be able to process at least 30 frames per second. To participate in this subtask, participants must use the development dataset to train a polyp segmentation model. Furthermore, this task also requires the participants to submit a Docker image of their implementation to be evaluated on the organizers’ hardware. The Docker submission should generate a .csv submission file that contains the name of the segmented image and the time (in seconds) used to perform the segmentation. A detailed description of the preparation and submission requirements of the Docker image is available on the oficial GitHub repository 2.
Models will be evaluated based on the performance metrics used to evaluate the polyp segmentation task and the number of frames that can be segmented per second. Submission will be ranked based on a balanced metric between predictive performance and speed. All submissions are evaluated on what can be considered consumergrade hardware, that is, a computer running Arch Linux with an Intel Core i9-10900K processor, an Nvidia GeForce RTX 3090 graphics processing unit (GPU), and 32 gigabytes of RAM. 3.3
The goal of the transparent machine learning system task is to
promote more transparency in medical applications of machine
learning. The motivation behind this task is rooted in a general lack of
transparency in medical machine learning research [
Submissions to this task will be evaluated by a committee comprised of at least three computer scientists and expert gastroenterologists that are familiar with AI. The committee will evaluate the submissions from diferent perspectives. For example, the medical doctors will look at the system from a clinic point of view, assessing transparency based on how it can be used in the clinic. The computer scientists will look at the technical transparency of the submissions, like source code descriptions and the clarity of the implementation. Each team that submits to this task will receive a report on the level of transparency determined by the evaluation committee.
Automatic segmentation of polyps in the gastrointestinal tract is a problem that is highly requested by medical doctors working in the field. Being more transparent about the work that goes into developing these methods would not only help doctors make more informed decisions on what systems should be used, but can also aid in further development by future researchers. We hope that this task will encourage the multimedia community to aid in the development of computer-assisted finding segmentation, and further motivate the use of transparent and open implementations of machine learning systems in medicine.