This paper presents the approach proposed by the organizer team (SimulaMet) for MediaEval 2019 Multimedia for Medicine: The Medico Task. The approach uses a data preparation method which is based on global features extracted from multiple frames within each video and then combines this with information about the patient in order to create a compressed representation of each video. The goal is to create a less hardware expensive data representation that still retains the temporal information of the video and related patient data. Overall, the results need some improvement before being a viable option for clinical use.
In this paper, we detail the approach of the Medico Task
organization team (SimulaMet) as part of MediaEval 2019. The Medico Task
explores the challenge of using multimedia data to make the daily
work of medical doctors more eficient [
The proposed approach is based on handcrafted features extracted form multiple frames within each video and associated patient/sensor data which is fed into a deep learning model for the prediction. This approach is used to solve the prediction of motility task and the prediction of morphology task, which are the tasks required in order to participate in the competition. The prediction of motiltiy task involves predicting three quality metrics (progressive, non-progressive, and immotile) tied to the movement of the sperm in a given semen sample. The prediction of morphology task focuses on predicting more visual features of the sperm, namely, predicting defects which may be present in the head, tail, or midpiece.
As the organization team, our main goal for this year’s Medico Task was to present a baseline approach which utilize all the available data in the dataset to make a prediction on sperm motility and morphology (as the tasks require). Furthermore, we wanted our approach to be computationally eficient so to not require expensive hardware or a complex setup procedure. Therefore, we decided to base our contribution on handcrafted features which were extracted from multiple frames within each video in the provided dataset. These handcrafted features are combined with each category of associated patient data in order to train a deep learning model to make a prediction on each of the sperm quality features. In the following few section, we will describe the approach in more detail. This includes a description on how the data was prepared, information about the model architecture, and how each model was trained. 2.1
To prepare the data for training and evaluation, we start by
extracting the first two frames of each second for 60 seconds for each video.
The result is a sub-sample of each video containing 120 frames from
the first minute of each of the 85 videos in the dataset. From this
step, we extract four diferent types of global features from each of
the frames, namely, Edge Histogram, Tamura, Luminance Layout
and Simple Color Histogram [
The motivation behind this approach is that each column contains the visual features of a single frame, while the temporal information is represented through the change of feature values across the diferent columns. To add the information about the patient, we simply concatenate the values to each frame column so that each patient and sensor value will be the same for each frame in a given training sample. We create these video representations using each of the five provided categories of patient information values in the dataset, namely, sex hormones, patient-related data (age, body mass index (BMI), and days of sexual abstinence), fatty acids in serum, fatty acids in spermatozoa, and sperm analysis data. Each of the five video categories were used to train two deep neural networks, one for predicting morphology and one for predicting motility. Note that for the sperm analysis data, we removed the data regarding motility and morphology when used to predict these values in each respective task. This is important as to not feed the ground truth values to the network during training. 2.2
As explained in the previous section, the data format used to train our deep neural networks is a matrix of extracted image features from 120 frames within each video and associated patient and sensor data. The exact input shape varies depending on the category of patient information used in the representation (number of variables vary depending on the category), but rest of the network is kept the same. We use a novel convolutional neural network architecture to perform our experiments.
The architecture itself is modeled to analyze the video representations using an ensemble of similar networks (network architecture shown in Figure 1). The network consists of an inital convolution block, after which the output gets passed through multiple residual modules. The output of each network is then concatenated before being passed through two fully-connected layers and then making the prediction. The network is repeated multiple times in order to create an ensemble of networks which each analyze the same input. The number of networks used is a parameter which may be tuned, from which we decided to use three based on some internal testing.
As previously mentioned, the same model architecture was used
for both the motility tasks and the morphology task. The model was
trained with a learning rate of 0.00001 using Nadam [
Looking at Table 1 and Table 2, we see the results for the Prediction of Morphology Task the Prediction of Motility Task. Overall, the results show that the deep neural networks for both tasks are able to learn something from the data, but the performance is overall quite poor when compared to the ZeroR baseline. For future work, we aim to use features extracted from a deep neural network. 4
In this paper, we described the approach submitted by the 2019
Medico organization team (SimulaMet). The presented method
used handcrafted features and associated patient data to train a
deep learning model to predict sperm quality in terms of motility
and morphology. Based on the results, we see that the future of
automatic sperm quality prediction is promising, but requires more
work before being used in any real-world scenario. The way of
representing the video into a single image also allows for future
experiments where we want to use grad cam methods, e.g. [