The Medico Multimedia Task returns for its seventh iteration as a segment of MediaEval 2023. The challenge comprises three main tasks: sperm tracking, sperm detection, and sperm motility predictions. Additionally, this year, we have broadened our focus by incorporating new graph data derived from sperm-bounding boxes and unique identifiers taken from manually annotated data. We invite participants to employ innovative methods, diverging from traditional ones, to study sperm using machine learning. The dataset includes video recordings of spermatozoa, complemented with annotations and graph data.
The 2023 Medico task builds on the previous Medico edition about transparent tracking of
spermatozoa in videos [
The goal of the 2023 Medico task is to encourage the participants to track individual spermatozoa in real-time and combine diferent data sources to predict common measurements used for sperm quality assessment, specifically the motility of the spermatozoa. Solving this task successfully might pave the way for developing improved computer-aided systems to assist medical experts in the fertility clinic.
Annotated videos from the VISEM-Tracking dataset [
Subtask 1: The goal of this subtask is localization and tracking of sperm cells in a given semen video. Specifically, the subtask focuses on examining microscopic videos of sperm, where experts have manually annotated spermatozoa. Participants are tasked with detecting individual sperm cells by providing bounding box coordinates and tracking them by assigning unique IDs. The required format for the bounding box coordinates should align with the structure used in the development datasets.
Subtask 2: This subtask requires the participants to further refine the methodologies employed in Subtask 1, emphasizing not only high prediction accuracy but also computational eficiency and inference time. Participants must provide reports on the average Frames Per Second (FPS) and Floating-Point Operations Per Second (FLOPS) while conducting inference with a batch size of 1.
Subtask 3: The goal of this task is to anticipate sperm motility1 in terms of the percentage of progressive and non-progressive spermatozoa. The prediction needs to be performed patientwise, resulting in a singular value for each patient pertaining to the predicted attribute. To address this subtask, sperm tracking or bounding boxes obtained from Subtasks 1 and/or 2 are indispensable. Participants are strongly encouraged to consider the temporal dimension of the videos, since temporal information propagated from previous frames are crucial for extrapolating properties in subsequent frames (i.e., sperm motility). This is important due to the fact that an analysis based solely on individual frames will be insuficient to capture the movement or motility of sperm, which contains vital information necessary for accurate predictions.
Subtask 4: This task is experimental in nature and asks the participants to generate graphs representing the predicted tracks for the spermatozoa in order to assess the sperm motility. The participants are asked to employ graph data structures as input to a model that predicts the level of motility in sperm samples. The construction of graph structures can be facilitated using the predicted bounding boxes. Graphs for training models to predict sperm motility are provided, while the graphs required for testing the final models must be generated from the prediction models in Subtasks 1 and/or 2.
As in the 2022 edition [
For each patient, we include a video of live sperm (video and extracted frames), manually
annotated bounding box details for each spermatozoon (sample frames are presented in Figure 1),
a set of measurements from a standard semen analysis for the whole sample, a sperm fatty acid
profile, the fatty acid composition of serum phospholipids, study participants-related data, and
WHO analysis data. The bounding box coordinates are provided in two separate folders: one
1Motility is the ability to move independently, where a progressive spermatozoon is able to "move forward" and a
non-progressive would move for example in circles without any forward progression.
folder has bounding box coordinates in YOLO format [
• videos: overview of which video file belongs to what participant.
Regarding Subtask 4, which is optional, graph data2 is provided. The structure of the graphs is depicted in Figure 2. The graphs represent spatial and temporal relationships between sperm in a video. Spatial edges () connect sperms within the same frame, while temporal edges () connect sperms across diferent frames. The graphs have been generated with varying spatial threshold ( ) values, where each threshold value determines the maximum distance between two nodes for them to be connected in the graph. The following threshold values are used: 0.1, 0.2, 0.3, 0.4, and 0.5. The graph data contains separate folders with graphs generated using each spatial threshold, i.e., there are five folders in total. In each threshold folder, there is a subfolder including separate graphs for individual frames in the video and a GraphML file containing the graph for the complete video.
In order to evaluate the proposed solutions thoroughly, several evaluation metrics are computed for each subtask.
Subtask 1: We use the widely recognized COCO evaluation metrics to evaluate the
performance of sperm detection methods. This comprehensive assessment involves precision,
recall, mAP@50 (mean average precision @50), and mAP@50-95, providing a multidimensional
perspective on the accuracy of the methods employed in this critical domain. Furthermore,
we leverage Jonathan Luiten’s TrackEval library [
Subtask 2: We utilize the aforementioned evaluation metrics as in Subtask 1. However, the performance evaluation considers the inference speed of the methods as a weighted factor. This integration adds a new dimension to the evaluation, weighing the performance against the eficiency and agility of inference, thus illuminating the intricate interplay between accuracy and speed in sperm detection and tracking.
Subtask 3 and Subtask 4: Evaluation of regression sperm motility prediction performance involves the use of Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE). These metrics provide insights into the accuracy of the predictions. 2https://huggingface.co/datasets/SimulaMet-HOST/visem-tracking-graphs (b) (e) (h) et (c) (f) (i) N3
N2 es Tr
N0 C Frame t+1
N1 (a) (d) (g) N3
N2 es Tr
N0 C Frame t
N1
This year, we introduced supplementary graph data derived from the details of bounding boxes and feature IDs. These IDs facilitate the identification of the same sperm cell across multiple frames of a video. We believe that participants can explore innovative approaches that diverge from conventional methods in the realm of sperm analysis through machine learning. Alongside experimental Subtask 4, we retained the foundational tasks associated with sperm tracking and the prediction of sperm motility, denoted as Subtasks 1, 2, and 3.