In this paper, we present the approach of team Jmag to solve this year's Medico Multimedia Task as part of the MediaEval 2019 Benchmark. This year, the task focuses on automatically determining quality characteristics of human sperm through the analysis of microscopic videos of human semen and associated patient data. Our approach is based on deep convolutional neural networks (CNNs) of varying sizes and dimensions. Here, we aim to analyze both the spatial and temporal information present in the videos. The results show that the method holds promise for predicting the motility of sperm, but predicting morphology appears to be more dificult.
In an efort to explore how medical multimedia can be used to create
performant and eficient prediction algorithms, the 2019
Multimedia for Medicine Task [
Motility and morphology are properties of sperm which appear diferently in the videos human semen. Motility may be dificult to assess looking only at the spatial dimension, as it is heavily dependent on the temporal information present in a video. By contrast, morphology is highly dependent on the visual features of the sperm and not necessarily their movement, although there may be some correlation between the the movement and the morphology, i.e., a sperm with a tail defect may move slower. Consequently, predicting these two aspects of semen require diferent approaches. To preserve the temporal and the spatial information in a video when predicting motility, we use 3D convolutional neural networks (CNNs). When predicting morphology, we discard the temporal information and make a prediction based on a single frame using a
We present two methods for predicting the motility. First, we use
a simple 3D CNN to see how well a model using just a few layers
performs on this task. Second, we present a deeper and more
complex 3D CNN to see how this improves over the simpler model. The
simple model uses a very shallow network architecture consisting
of only two convolutional layers. Each convolutional layer extracts
32 filters using a kernel size of 4 × 4 × 4 and 5 × 5 × 5 respectively,
which the output is then passed to a fully-connected layer before
making the prediction. The complex model consists of three
consecutive convolutional blocks, where each block is made up of three
convolutional layers and a pooling layer to reduce the spatial and
temporal dimensions. Following the conventions of Li et al. [
Due of the limited amount of data, we perform data augmentation during training. First, we extract 20 random samples from a single data point for which we perform several augmentation techniques including random crops, noise injection, and vertical/horizontal lfips. To decrease training time, we first downsample the resolution of each sample to 128 × 171 pixels for both the training and the validation dataset, then we randomly select samples of consecutive 15 frame intervals and randomly crop the image to 128 × 128 pixels. The frame samples are then randomly flipped both horizontally and vertically with a probability of 0.5 each. Finally, we add some noise injecting each pixels with some random values selected from a uniform distribution in the interval [−0.01, 0.01]. For validation, we split each video into blocks consisting of 15 consecutive frames and discard the frames that remain. Each frame block is then cropped into a 128×128 at the upper left edge of the frame. We then calculate the average prediction score over all blocks of a video for each video, and take the average of these averages to get our final prediction score. We do this to avoid weighing longer videos more than shorter ones in our final score and rather weigh each video the same.
To predict morphology, we use a relatively deep 2D CNN while
avoiding making it deep in order to avoid vanishing gradients [
Method Fold have 32, 128, 128, 512, and 512 filters each respectively. After the ifnal convolutional layer, we pass the output through three fullyconnected layers; one with 1024 nodes, one with 512 nodes and the output layer with 3 nodes. Each layer in the network uses the activation function ReLU, except for the output layer which uses a linear activation.
For both training and validation, data was prepared similarly to that of the motility experiments, the only diference being that we used a single frame to make predictions at a resolution of 240 × 320 and did not perform any cropping. We still, however, performed noise injection with noise retrieved from the same distribution, and random flips with using the same probabilities.
All models were trained for a maximum of 200 epochs, only
interrupting the training if the evaluation loss did not improve over the
last 10 epochs. The models were trained using the deep learning
library Keras [
Looking at the motility experiments (Table 1), we see that the complex model achieves much better results than the simple model. It is clear that our complex model is able to extract more crucial information from the data to make better predictions. Comparing the complex model to the ZeroR baseline, we see a mean absolute error (MAE) improvement of 0.0436 which shows that the deep learning at the very least is able to learn to associate some movement of
32 4 × 4 × 4 Conv 32 5 × 5 × 5 Conv
(a) An illustration of the simple model architecture.
m n × n × n /i × j × k m n × n × n Conv m n × n × n Conv m n × n × n Conv i × j × k MaxPool
(b) An illustration of the convolutional block used in the complex model.
32 4 × 4 × 4 Conv
32 5 × 5 × 5 /2 × 4 × 4
128 5 × 5 × 5 /2 × 4 × 4
256 4 × 4 × 4 /2 × 2 × 2
(c) An illustration of the
compelx model
architecture.
the sperms with the related motility values. For the morphology
experiments (Table 2), we see that our model fails to beat predicting
the mean value of the labels (ZeroR). It fails to learn the individual
shape of each sperm and collectively predict total of each category.
For future work, we will increase the size of the network to make it
more adaptable, which may bring other challenges such as making
the network harder to train due to the increased risk of vanishing
gradients [
In this paper, we presented the work done as part of the Medico Multimedia Task where we participated in two of the three available subtasks. We used deep CNNs for both tasks, where we achieved an average MAE of 0.0883 for the motility task and an average MAE of 0.0613 for the morphology task.