Extracting Temporal Features into a Spatial Domain Using Autoencoders for Sperm Video Analysis

Abstract

In this paper, we present a two-step deep learning method that is used to predict sperm motility and morphology based on video recordings of human spermatozoa. First, we use an autoencoder to extract temporal features from a given semen video and plot these into image-space, which we call feature-images. Second, these feature-images are used to perform transfer learning to predict the motility and morphology values of human sperm. The presented method shows it's capability to extract temporal information into spatial domain feature-images which can be used with traditional convolutional neural networks. Furthermore, the accuracy of the predicted motility of a given semen sample shows that a deep learning-based model can capture the temporal information of microscopic recordings of human semen.

INTRODUCTION

The 2019 Medico task [ 7 ] focuses on automatically predicting semen quality based on video recordings of human spermatozoa. This is change from previous years which have mainly focused on image classification of images taken from the gastrointestinal tract [ 10, 11 ]. For this year’s task, we look at predicting the morphology and motility of a given semen sample. Motility is defined by three variables, namely, the percentage of progressive, nonprogressive, and immotile sperm. Morphology is determined by the percentage of sperm with tail defects, midpiece defects, and head defects. The organizers have provided a dataset consisting of 85 videos of diferent semen samples and a preliminary analysis of each, which is used as the ground truth. For this competition, the organizers have provided a predefined three-fold split of the VISEM dataset [ 5 ], which contains 85 videos from diferent participants and a preliminary analysis of each semen sample. In the dataset paper, the authors presented baseline mean absolute error (MAE) values for motility and morphology. Furthermore, the importance of computer-aided sperm analysis can be identified from the previous works which have been done over the last few decades [ 3, 9, 12 ].

To solve this year’s task, we propose a deep learning-based method consisting of two steps - (i) unsupervised feature extraction using an autoencoder [ 1 ] and (ii) video regression using a standard convolutional neural networks (CNN) and transfer learning. The autoencoder we use is diferent from the state-of-the-art autoencoders used to extract video features [ 2, 13 ] as they use autoencoders to extract feature vectors which are used with long-short memory models or multi-layer perceptron (MLP)s. In contrast, we use autoencoders to extract feature-images for use in CNNs.

APPROACH

Our method can primarily be split into two distinct steps. First, we use an autoencoder to extract temporal features from multiple frames of a video into a feature-image. Second, we pass the extracted feature-image into a standard pre-trained CNN to predict the motility and morphology of the spermatozoa in a given video. In this paper, we present the preliminary results for four experiments based on four diferent input types. The first input type (I1) uses a single raw frame. Input type two (I2) is a stack of identical frames copied across the channel-dimension. The third (I3) and fourth (I4) input type stack 9 and 18 consecutive frames from a video respectively.

The first two experiments (using I1 and I2) were performed as baseline experiments. The two other experiments (using I3 and I4) were performed to see how the temporal information afects the prediction performance of the approach. For all input types, we split the extracted datasets into three folds based on the folds provided by the organizers. Then, three-fold cross-validation was conducted to evaluate our four experiments. An overview of all experiments is shown in Figure 1. 2.1

Step 1 - Unsupervised temporal feature extraction

In step 1, we trained an autoencoder that takes an input frame or frames (I1, I2, I3 or I4) from the sperm videos as depicted in Figure 1. Then, the encoder of the autoencoder extracted feature-images and passed them through the decoder architecture to reconstruct the input frame or frames back (R1, R2, R3, and R4). These extracted feature-images are diferent from traditional feature extractions of autoencoders because the traditional autoencoders extract feature vectors instead of feature-images. In this autoencoder, the mean square error (MSE) loss function is used to calculate the diference between input data and reconstructed data. Then, this error value is backpropagated to train the autoencoder. After training 2,000 epochs, we use the encoder architecture of the autoencoder model to step 2. 2.2

Step 2 - CNN regression model

We have selected the pre-trained ResNet-34 [ 6 ] as our basic CNN to predict the values of motility and morphology of the sperm videos. However, any pre-trained CNN could be chosen for this step and in future work we will test and compare diferent ones in more detail. Firstly, we take an input frame or frames (I1, I2, I3 or I4) and pass through the pre-trained encoder model (only the encoder section of the autoencoder model) which was trained also from the same data inputs in an unsupervised way. Then, the outputs of the encoder model were passed through the CNN model which has a modified last layer to output three prediction values for motility or morphology. Thambawita et al.

RESULTS AND ANALYSIS

According to the average MAE values shown in Table 1, the average motility values of input I3 and I4 shows the best results among other motility values of input I1 and I2. These performance improvements imply that our model is able to learn temporal features into a spatial feature image representation. Furthermore, input I4 which uses 18 stacked frames shows the best motility average values compared to input I3. This performance gain shows that to predict the sperm motility in sperm videos, it is better to analyze more frames at the same time. This might be due to the fact that the behaviour of sperm is something that needs to be observed over time and not in single frames. Moreover, the predictions for our base case inputs I1 and I2 show the same average values. This shows that our model learns temporal information from diferent sperm video frames. Otherwise, it would be shown diferent average values for our two base case inputs I1 and I2.

When we consider the predicted morphology average in Table 1, it shows values that are almost equal to each other. This is expected because the morphology of a sperm is something that can be observed using a single frame. In contrast to predicting accurate morphology, the predicted morphology values support the prove that our model has the capability to learn temporal data from multiple frames because motility predictions show an improvement when we increase the number of frames analyzed simultaneously. 4

CONCLUSION AND FUTURE WORKS

In this paper, we proposed a novel method to extract temporal features from videos to create feature-images, which can be used to train traditional CNN models. Furthermore, we show that the feature-images capture temporal present in a sequence of frames, which can be used to predict the motility of the sperm videos.

This method can be improved by using diferent error functions to force the model to learn more temporal data. For example, researchers can experiment with variational autoencoders [ 8 ] and generative adversarial learning methods [ 4 ] to improve this technique. Additionally, it may be beneficial to embed long short-term memory units to investigate how our feature-images compare to actual extracted temporal features.

References

[1] Pierre Baldi . 2012 . Autoencoders, unsupervised learning, and deep architectures . In Proceedings of ICML workshop on unsupervised and transfer learning . 37 - 49 .

[2] Yong Shean Chong and Yong Haur Tay. 2017 . Abnormal event detection in videos using spatiotemporal autoencoder . In Proceedings of the International Symposium on Neural Networks . Springer, 189 - 196 .

[3] Karan Dewan , Tathagato Rai Dastidar, and Maroof Ahmad . 2018 . Estimation of Sperm Concentration and Total Motility From Microscopic Videos of Human Semen Samples . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4] Ian Goodfellow , Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio . 2014 . Generative adversarial nets . In Proceedings of the Advances in neural information processing systems (NIPS) . 2672 - 2680 .

[5] Trine B. Haugen , Steven A. Hicks , Jorunn M. Andersen , Oliwia Witczak, Hugo L. Hammer, Rune Borgli, Pål Halvorsen, and Michael A. Riegler . 2019 . VISEM: A Multimodal Video Dataset of Human Spermatozoa . In Proceedings of the 10th ACM on Multimedia Systems Conference (MMSys) (MMSys'19) . ACM, New York, NY, USA. https: //doi.org/10.1145/3304109.3325814

[6] Kaiming He , Xiangyu Zhang, Shaoqing Ren, and Jian Sun . 2016 . Deep residual learning for image recognition . In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) . 770 - 778 .

[7] Steven Hicks , Pål Halvorsen, Trine B Haugen, Jorunn M Andersen , Oliwia Witczak , Konstantin Pogorelov, Hugo L Hammer, Duc-Tien Dang-Nguyen, Mathias Lux , and Michael Riegler . 2019 . Medico Multimedia Task at MediaEval 2019 . In Proceedings of the CEUR Workshop on Multimedia Benchmark Workshop (MediaEval).

[8] Diederik P Kingma and Max Welling . 2013 . Auto-encoding variational bayes . arXiv preprint arXiv:1312.6114 ( 2013 ).

[9] Sharon T Mortimer , Gerhard van der Horst, and David Mortimer . 2015 . The future of computer-aided sperm analysis . Asian journal of andrology 17 , 4 ( 2015 ), 545 .

[10] Konstantin Pogorelov , Michael Riegler, Pål Halvorsen, Steven Alexander Hicks, Kristin Ranheim Randel, Duc-Tien Dang-Nguyen, Mathias Lux, Olga Ostroukhova, and Thomas de Lange. 2018 . Medico Multimedia Task at MediaEval 2018 .. In Proceedings of the CEUR Workshop on Multimedia Benchmark Workshop (MediaEval).

[11] Michael Riegler , Konstantin Pogorelov, Pål Halvorsen, Carsten Griwodz, Thomas Lange, Kristin Randel, Sigrun Eskeland, Dang Nguyen, Duc Tien, Mathias Lux, and others. 2017 . Multimedia for medicine: the medico task at MediaEval 2017 . ( 2017 ).

[12] L. F. Urbano , P. Masson, M. VerMilyea, and M. Kam . 2017 . Automatic Tracking and Motility Analysis of Human Sperm in Time-Lapse Images . IEEE Transactions on Medical Imaging (T-MI) 36, 3 (March 2017 ), 792 - 801 . https://doi.org/10.1109/TMI. 2016 .2630720

[13] Huan Yang , Baoyuan Wang , Stephen Lin , David Wipf, Minyi Guo , and Baining Guo . 2015 . Unsupervised extraction of video highlights via robust recurrent auto-encoders . In Proceedings of the IEEE international conference on computer vision (ICCV) . 4633 - 4641 .