Stacked Dense Optical Flows and Dropout Layers to Predict Sperm Motility and Morphology Vajira Thambawita1,2 , Pål Halvorsen1,2 , Hugo Hammer1,2 , Michael Riegler1,3 , Trine B. Haugen2 1 SimulaMet, Norway 2 Oslo Metropolitan University, Norway 3 Kristiania University College, Norway Contact:vajira@simula.no ABSTRACT In this paper, we analyse two deep learning methods to predict sperm motility and sperm morphology from sperm videos. We use two different inputs: stacked pure frames of videos and dense optical flows of video frames. To solve this regression task of predicting motility and morphology, stacked dense optical flows and extracted original frames from sperm videos were used with the modified state of the art convolution neural networks. For modifications of (a) Original frame (b) Dense optical flow - (c) Dense optical flow - stride 1 stride 10 the selected models, we have introduced an additional multi-layer perceptron to overcome the problem of over-fitting. The method Figure 1: Sample images used to construct input image which had an additional multi-layer perceptron with dropout layers, stacks into the models shows the best results when the inputs consist of both dense optical flows and an original frame of videos. to present our two types of input data and two types of methods used in our experiments. Then, the results collected from these 1 INTRODUCTION experiments will be discussed in Section 3. Finally, the paper ends Our main goal of this task is to predict the sperm motility and sperm up with conclusions and future work in Section 4. morphology from videos of sperm samples. In the 2019 Medico task [8], a video dataset was provided with ground truth values of sperm 2 APPROACH motility such as progressive motility, non-progressive motility, and We have selected the pre-trained ResNet-34 [7] to do some basic immotility, and sperm morphology such as head defects, tail de- experiments of predicting sperm motility and sperm morphology us- fects, and midpiece and neck defects. This task was introduced as ing stacked normal raw video frames and a combination of stacked completely new this year, and therefore, we could not find any dense optical flows and raw frames of videos. In this paper, we previous work in previous mediaeval Medico task competitions obtain experimental results using two different types of inputs and [14, 15]. In this competition, the VISEM dataset [6] which contains from two different types of models. sperm videos recorded from 85 participants is used. In the dataset paper, the authors presented baseline mean absolute error values for 2.1 Preprocessing data motility and morphology. Moreover, the importance of computer- aided sperm analysis can be identified from the research works To find estimates for the sperm motility and sperm morphology, which have been done to develop automatic sperm analysis method we first preprocessed the input videos to generate two types of in last few decades [3, 13, 19]. input. In the first type (dataset - D1), we stacked nine consecutive Video analysis is a hot research topic in the field of deep learning. frames from a video to make a single input data point. A sample of Some researchers are experimenting with video classification [2], a raw frame of a video is given in Figure 1a. Before stacking raw detection [1], segmentation [5], and generations [12, 18] for various video frames, we converted the RGB format frames of the video into type of video datasets. Yue-Hei Ng et al. [20] experimented with grayscale images and resized them into 256x256. These nine frames video classification problem using well knows datasets such as represent nine different consecutive frames of a video. Moreover, sports-1M [10] and UCF101 [16]. In these experiments, they have we collected 250 stacked data points (chunks) from 250 locations in generated dense optical flow images and row frames of videos time from a video as described above. to classify 120 seconds long videos. In this paper, we use very For the second type of input (dataset - D2), we generated a short video segments such as nine frames compared to these long tensor with nine channels, which consists of a three-channels (RGB) segments such as 120s X 30 frames/s. original video frame (Figure 1a), a three-channels dense optical flow To solve this new regression problem of predicting morphology image of stride 1 (Figure 1b), and a three-channels image of dense and motility from videos of sperm samples, this paper presents optical flow of stride 10 (Figure 1c). The dense optical flow image two deep learning methods where we used extracted dense optical of stride 1 was generated from two consecutive video frames from flows and raw frames from the videos. In Section 2, we are going a selected location of a video. Then, we generated the stride-10 dense optical flow image using two frames; the first frame of the Copyright 2019 for this paper by its authors. Use video chuck and the 10t h frame of a selected video chunk. To permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). generate dense optical flows [4] of two different frames of a video, MediaEval’19, 27-29 October 2019, Sophia Antipolis, France the OpenCV library [9] was used with its inbuilt functions. Github: https://github.com/vlbthambawita/ MedicoTask_2019_paper_1 MediaEval’19, 27-29 October 2019, Sophia Antipolis, France Github: https://github.com/vlbthambawita/MedicoTask_2019_paper_1 Thambawita et al. Table 1: MAE values collected from the proposed methods: D1-stacked gray-scale nine consecutive frames, D2-stacked an original frame + a dense optical flow image from two con- secutive frames + a dense optical flow from two frames with stride=10; M1 - the basic model of Resnet-34 with modifi- cations of number of input channels and outputs, M2 - the modified model with an additional MLP with dropout layers Motility Morphology Input Method Fold MAE Average MAE Average Fold 1 9.562 5.626 Figure 2: Big picture of our deep learning model: M1 - the M1 Fold 2 8.959 9.200 5.749 5.649 base model of Resnet-34 with a three output last layer, M2 Fold 3 9.079 5.573 D1 - the modified version of Resnet-34 with an additional MLP, Fold 1 9.585 5.424 D1 and D2 represent the two different types of input used in M2 Fold 2 9.28 9.185 5.382 5.394 our experiments. Fold 3 8.689 5.375 Fold 1 9.044 5.933 For both input types, we split the datasets into three folds based M1 Fold 2 8.062 9.372 5.394 5.525 on the folds given in the video dataset provided by organizers. Then, Fold 3 11.01 5.248 a three-fold cross-validation was performed to evaluate our deep D2 learning models which will be introduced in the later sections. Fold 1 8.612 5.549 M2 Fold 2 7.873 8.825 5.463 5.293 Fold 3 9.991 4.868 2.2 Deep learning model implementation For implementation of our deep learning models, we selected Resnet- 34 which is larger than the smallest, Resnet-18, and smaller than 3 RESULTS AND ANALYSIS other large scales Resnet models like Resnet-50, Resnet-101, and According to the average MAE values shown in Table 1, the M2 Resnet-152. The selections of this intermediate Resnet-34 was done method with the input type 2 (D2) shows best results among other based on expandability of the model by adding additional multi- methods and other input types. This method shows the best MAE layer perceptron (MLP) within the available hardware resources value of 8.825 for the sperm motility and 5.293 for the sperm mor- (considering memory limitations of the available graphics process- phology. This improvement of error values can be seen as results ing units). In addition to that, the pre-experiments were done to of accumulated benefits of showing pre-processed temporal infor- identify over-fitting problems of strong models for simpler predic- mation such as dense optical flows to the model and the additional tions and computation time required to finish training. Furthermore, MLP to overcome the problem of over-fitting. Moreover, the added expandability of the number of input channels of the model within MLP in M2 gives better results with both input types (D1 and D2) the available GPU memory was examined. for both predictions: sperm motility and sperm morphology. We For method 1 (M1), we modified the input layer of the selected achieved this performance as a result of the pre-processed input pre-trained Resnet-34 to take nine channel inputs and modified data with dense optical flows and the MLP introduced to overcome the last layer of the model to output only three values which are the over-fitting problem. representing either three values of sperm motility or three values of sperm morphology. We used this method as our base model with 4 CONCLUSION AND FUTURE WORK the two different datasets (D1 and D2) as introduced in Section 2.1 The input with a raw frame and dense optical flows of two difference and recorded results collected from this experiment in D1-M1 and stride values show better results compared to the stacked normal D2-M1 rows in Table 1. frames of videos. Moreover, the modified Resnet-34 model with an In method 2 (M2), to avoid over-fitting problems of this task, MLP which consists of dropout layers with high probabilities did we have embedded additional MLP to the end of the network with achieve better results than the base model in the both cases because dropout layers [17]. The full structure of this additional MLP is it helped to overcome the problem of over-fitting in the training depicted in Figure 2 using a green colour. The dropout values of stage. Finally, the combination of the input with dense optical flows this MLP were selected using pre-experiments, and it is a hyper- and the modified Resnet-34 with an additional MLP shows the best parameter for this model. The collected results of this method are overall performance. tabulated in rows D1-M2 and D2-M2 of Table 1. In future work, it is worth to try CNN models with long short- In the training process of all the above methods, the Adam opti- term memory units to capture temporal features of video frames. mizer [11] with a learning rate 0.001 was used. The mean square Moreover, a 3D CNN can be a promising approach for this kind of error (MSE) was used as the loss function for back-propagating task because 3D CNN models have capabilities to capture temporal error, and mean absolute error (MAE) was used for calculating the information of videos. actual loss of predictions based on ground truth values of motility and morphology. MediaEval’19, 27-29 October 2019, Sophia Antipolis, France 2019 Medico Medical Multimedia Github: https://github.com/vlbthambawita/MedicoTask_2019_paper_1 REFERENCES [11] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic [1] Alan C Bovik. 2010. Handbook of image and video processing. Academic optimization. arXiv preprint arXiv:1412.6980 (2014). press. [12] Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and [2] Darin Brezeale and Diane J Cook. 2008. Automatic video classification: Lawrence Carin. 2018. Video generation from text. In Thirty-Second A survey of the literature. IEEE Transactions on Systems, Man, and AAAI Conference on Artificial Intelligence. Cybernetics, Part C (Applications and Reviews) 38, 3 (2008), 416–430. [13] Sharon T Mortimer, Gerhard van der Horst, and David Mortimer. [3] Karan Dewan, Tathagato Rai Dastidar, and Maroof Ahmad. 2018. Esti- 2015. The future of computer-aided sperm analysis. Asian journal of mation of Sperm Concentration and Total Motility From Microscopic andrology 17, 4 (2015), 545. Videos of Human Semen Samples. In Proceedings of the IEEE Conference [14] Konstantin Pogorelov, Michael Riegler, Pål Halvorsen, Steven Alexan- on Computer Vision and Pattern Recognition (CVPR) Workshops. der Hicks, Kristin Ranheim Randel, Duc-Tien Dang-Nguyen, Mathias [4] Gunnar Farnebäck. 2003. Two-frame motion estimation based on Lux, Olga Ostroukhova, and Thomas de Lange. 2018. Medico Multi- polynomial expansion. In Proceedings of the Scandinavian conference media Task at MediaEval 2018.. In Proceedings of the CEUR Workshop on Image analysis. Springer, 363–370. on Multimedia Benchmark Workshop (MediaEval). [5] Arun Hampapur, Terry Weymouth, and Ramesh Jain. 1994. Digital [15] Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Carsten Gri- video segmentation. In Proceedings of the second ACM international wodz, Thomas Lange, Kristin Randel, Sigrun Eskeland, Dang Nguyen, conference on Multimedia. ACM, 357–364. Duc Tien, Mathias Lux, and others. 2017. Multimedia for medicine: [6] Trine B. Haugen, Steven A. Hicks, Jorunn M. Andersen, Oliwia the medico task at MediaEval 2017. (2017). Witczak, Hugo L. Hammer, Rune Borgli, Pål Halvorsen, and Michael A. [16] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Riegler. 2019. VISEM: A Multimodal Video Dataset of Human Sperma- UCF101: A dataset of 101 human actions classes from videos in the tozoa. In Proceedings of the 10th ACM on Multimedia Systems Confer- wild. arXiv preprint arXiv:1212.0402 (2012). ence (MMSys’19). ACM, New York, NY, USA. https://doi.org/10.1145/ [17] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, 3304109.3325814 and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Neural Networks from Overfitting. Journal of Machine Learning Re- residual learning for image recognition. In Proceedings of the IEEE search 15 (2014), 1929–1958. http://jmlr.org/papers/v15/srivastava14a. conference on computer vision and pattern recognition. 770–778. html [8] Steven Hicks, Pål Halvorsen, Trine B Haugen, Jorunn M Andersen, [18] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018. Oliwia Witczak, Konstantin Pogorelov, Hugo L Hammer, Duc-Tien Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern Dang-Nguyen, Mathias Lux, and Michael Riegler. 2019. Medico Mul- recognition. 1526–1535. timedia Task at MediaEval 2019. In CEUR Workshop Proceedings - [19] L. F. Urbano, P. Masson, M. VerMilyea, and M. Kam. 2017. Auto- Multimedia Benchmark Workshop (MediaEval). matic Tracking and Motility Analysis of Human Sperm in Time-Lapse [9] Itseez 2014. The OpenCV Reference Manual (2.4.9.0 ed.). Itseez. Images. IEEE Transactions on Medical Imaging 36, 3 (March 2017), [10] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, 792–801. https://doi.org/10.1109/TMI.2016.2630720 Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classifica- [20] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, tion with convolutional neural networks. In Proceedings of the IEEE Oriol Vinyals, Rajat Monga, and George Toderici. 2015. Beyond short conference on Computer Vision and Pattern Recognition. 1725–1732. snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4694–4702.