The Medico: Multimedia for Medicine Task is running for the third time as part of MediaEval 2019. This year, we have changed the task from anomaly detection in images of the gastrointestinal tract to focus on the automatic prediction of human semen quality based on videos. The purpose of this task is to aid in the assessment of male reproductive health by providing a quick and consisted method of analyzing human semen. In this paper, we describe the task in detail, give a brief description of the provided dataset, and discuss the evaluation process and the metrics used to rank the submissions of the participants.
The 2019 Medico: Multimedia for Medicine Task tackles the
challenge of predicting certain quality measurements of sperm using
a multimodal dataset consisting of microscopic video recordings
of human semen, associated sensor data, and participant-related
data. Male infertility accounts for approximately 60% of cases
involving couples having problems conceiving a child [
For the 2019 edition of Medico, we present three diferent tasks. The first two tasks aim to predict key indicators of good semen quality, specifically the motility and morphology of sperm. Motility and morphology are two of the most common metrics used to evaluate spermatozoa, and to predict these measurements gives the participants a good opportunity to utilize both the video and sensor data available in the dataset. The third task relates to looking at individual sperm to figure out which one moves the fastest. For all tasks, we require participants to measure and report the data processing performance in terms of time spent on each frame being analyzed. As we aim for real-time applications, the processing speed is an important factor, especially for the individual spermatozoon selection task during the in vitro fertilization procedure (a procedure where a sperm is injected into an egg outside of the body). 2
The VISEM dataset [
For the analysis data, sensor data, and participant-related data, the following CSV-files are provided: • semen_analysis_data: The results of standard semen analysis. • fatty_acids_spermatozoa: The levels of several fatty acids in the spermatozoa of the participants. • fatty_acids_serum: The serum levels of the fatty acids of the phospholipids (measured from the blood of the participant). • sex_hormones: The serum levels of sex hormones measured in the blood of the participants. • study_participant_related_data: General information about the participants such as age, abstinence time and Body Mass Index (BMI). • videos: Overview of which video-file belongs to what participant.
VISEM is publicly available 1 without any restriction. Each study participant agreed to donate their data for research and provided the necessary consent to distribute it. It is also important to point out that all data is fully anonymized and follows the state of the art with respect to the privacy of medical information. 3
Medico 2019 presents three diferent tasks, each meant to target a specific use-case within male fertility assessment. The three tasks for this year’s Medico is the prediction of morphology task, the prediction of motility task, and the unsupervised sperm tracking task. Of these three tasks, the first two are mandatory in order to participate in the challenge. The third task is optional, but highly recommended, as it is an important problem within assisted reproductive technology. For the performance evaluation, the same dataset will be used for development and testing, but for testing, we ask the participants to perform three-fold cross-validation. For the processing speed evaluation, we will use minimum, average and maximum frame processing times in seconds measured by the participants as a time interval from the moment when an image has been completely loaded into memory to the moment of the ifnal decision has been made by the corresponding task analysis algorithm. In the following, we will give a more detailed description of each task and discuss how each will be evaluated and ranked.
Motility is the ability of an organism to move independently. In semen analysis, this is commonly split into three distinct categories, namely progressive, non-progressive, and immotile spermatozoa. A progressive spermatozoon is one that is able to move forward at a slow or fast pace, a non-progressive spermatozoon is a sperm that moves without forward progression, and immotile sperm do not move at all. The purpose of this task is to automatically find the percentage of progressive, non-progressive, and immotile spermatozoa for a given semen sample. We urge participants to perform multi-frame analysis over single-frame analysis. This is important due to the fact that single-frame analysis loses most of the temporal information needed to accurately assess the movement of the spermatozoa.
In order to participate in this task, task participants will submit a ".csv" file containing one prediction for each of the 85 study participants. Each line should be made up of five comma-separated values; ID of the study participant, percentage of progressive sperm, percentage of non-progressive sperm, percentage of immotile sperm, average frame processing time. To evaluate this task we will use root mean squared error and mean absolute error to evaluate the submissions. Submissions will be ranked based on the achieved mean absolute error as it shows the improvement of the automatic prediction compared to that of the manual assessment.
Evaluating the morphology of sperm involves looking for any abnormalities that may be present in the three parts that make up a spermatozoon, namely the head, midpiece, and tail. This task should predict the percentage of sperm with head defects, midpiece defects, and tail defects for a single semen sample.
As with the prediction of motility task, participants have to submit a ".csv" file containing one prediction of morphology assessment for each of the 85 study participants. Each line should be made up of the following five comma-separated values; ID of the study participant, percentage of sperm with tail defects, percentage of sperm with midpiece defects, percentage of sperm with head defects, average frame processing time. Similar to the Prediction of motility task, we will use mean squared error and mean absolute error to evaluate the submissions. Submissions will be ranked based on the achieved mean absolute error.
Hicks et al.
This task is about finding the spermatozoon that moves fastest compared to all others, and requires that task participants track individual spermatozoon in order to evaluate the speed at a given point in time. Within this task, we defined the "fastest" spermatozoon in two diferent ways: (1) Fastest average speed: The one that moves the longest distance during a video defined by the total distance divided by the length of the video. (2) Highest top speed: the one that has the highest intermediate speed.
One specific challenge of this third task is that the video also changes due to the sample being moved while under the microscope to observe the sample in its entirety. We suggest that the participants track each sperm per viewpoint rather than per video. To evaluate this task, we will use manual evaluation with the help from three diferent experts within human reproduction.
We believe the area of automatic semen analysis is an important, yet overlooked, area of research which has the potential of being very beneficial for those working to improve reproductive health. From a computer science point of view, the task is compelling as it features more than just simple image analysis as it requires the participants to analyze the temporal information present from one frame to another in order to make quality predictions. We hope that this task will encourage the multimedia community to aid in the development of computer-assisted reproductive health, and discover new and clever ways of analyzing multimodal datasets.