=Paper=
{{Paper
|id=Vol-2670/MediaEval_19_paper_61
|storemode=property
|title=Stacked
Dense Optical Flows and Dropout Layers to Predict Sperm Motility and Morphology
|pdfUrl=https://ceur-ws.org/Vol-2670/MediaEval_19_paper_61.pdf
|volume=Vol-2670
|authors=Vajira Thambawita,Pål Halvorsen,Hugo Hammer,Michael Riegler,Trine B. Haugen
|dblpUrl=https://dblp.org/rec/conf/mediaeval/ThambawitaHHRH19
}}
==Stacked
Dense Optical Flows and Dropout Layers to Predict Sperm Motility and Morphology==
https://ceur-ws.org/Vol-2670/MediaEval_19_paper_61.pdf
Stacked Dense Optical Flows and Dropout Layers to Predict
Sperm Motility and Morphology
Vajira Thambawita1,2 , Pål Halvorsen1,2 , Hugo Hammer1,2 , Michael Riegler1,3 , Trine B. Haugen2
1 SimulaMet, Norway 2 Oslo Metropolitan University, Norway 3 Kristiania University College, Norway
Contact:vajira@simula.no
ABSTRACT
In this paper, we analyse two deep learning methods to predict
sperm motility and sperm morphology from sperm videos. We use
two different inputs: stacked pure frames of videos and dense optical
flows of video frames. To solve this regression task of predicting
motility and morphology, stacked dense optical flows and extracted
original frames from sperm videos were used with the modified
state of the art convolution neural networks. For modifications of (a) Original frame (b) Dense optical flow - (c) Dense optical flow -
stride 1 stride 10
the selected models, we have introduced an additional multi-layer
perceptron to overcome the problem of over-fitting. The method Figure 1: Sample images used to construct input image
which had an additional multi-layer perceptron with dropout layers, stacks into the models
shows the best results when the inputs consist of both dense optical
flows and an original frame of videos. to present our two types of input data and two types of methods
used in our experiments. Then, the results collected from these
1 INTRODUCTION experiments will be discussed in Section 3. Finally, the paper ends
Our main goal of this task is to predict the sperm motility and sperm up with conclusions and future work in Section 4.
morphology from videos of sperm samples. In the 2019 Medico task
[8], a video dataset was provided with ground truth values of sperm 2 APPROACH
motility such as progressive motility, non-progressive motility, and We have selected the pre-trained ResNet-34 [7] to do some basic
immotility, and sperm morphology such as head defects, tail de- experiments of predicting sperm motility and sperm morphology us-
fects, and midpiece and neck defects. This task was introduced as ing stacked normal raw video frames and a combination of stacked
completely new this year, and therefore, we could not find any dense optical flows and raw frames of videos. In this paper, we
previous work in previous mediaeval Medico task competitions obtain experimental results using two different types of inputs and
[14, 15]. In this competition, the VISEM dataset [6] which contains from two different types of models.
sperm videos recorded from 85 participants is used. In the dataset
paper, the authors presented baseline mean absolute error values for 2.1 Preprocessing data
motility and morphology. Moreover, the importance of computer-
aided sperm analysis can be identified from the research works To find estimates for the sperm motility and sperm morphology,
which have been done to develop automatic sperm analysis method we first preprocessed the input videos to generate two types of
in last few decades [3, 13, 19]. input. In the first type (dataset - D1), we stacked nine consecutive
Video analysis is a hot research topic in the field of deep learning. frames from a video to make a single input data point. A sample of
Some researchers are experimenting with video classification [2], a raw frame of a video is given in Figure 1a. Before stacking raw
detection [1], segmentation [5], and generations [12, 18] for various video frames, we converted the RGB format frames of the video into
type of video datasets. Yue-Hei Ng et al. [20] experimented with grayscale images and resized them into 256x256. These nine frames
video classification problem using well knows datasets such as represent nine different consecutive frames of a video. Moreover,
sports-1M [10] and UCF101 [16]. In these experiments, they have we collected 250 stacked data points (chunks) from 250 locations in
generated dense optical flow images and row frames of videos time from a video as described above.
to classify 120 seconds long videos. In this paper, we use very For the second type of input (dataset - D2), we generated a
short video segments such as nine frames compared to these long tensor with nine channels, which consists of a three-channels (RGB)
segments such as 120s X 30 frames/s. original video frame (Figure 1a), a three-channels dense optical flow
To solve this new regression problem of predicting morphology image of stride 1 (Figure 1b), and a three-channels image of dense
and motility from videos of sperm samples, this paper presents optical flow of stride 10 (Figure 1c). The dense optical flow image
two deep learning methods where we used extracted dense optical of stride 1 was generated from two consecutive video frames from
flows and raw frames from the videos. In Section 2, we are going a selected location of a video. Then, we generated the stride-10
dense optical flow image using two frames; the first frame of the
Copyright 2019 for this paper by its authors. Use video chuck and the 10t h frame of a selected video chunk. To
permitted under Creative Commons License Attribution
4.0 International (CC BY 4.0). generate dense optical flows [4] of two different frames of a video,
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France the OpenCV library [9] was used with its inbuilt functions.
Github: https://github.com/vlbthambawita/
MedicoTask_2019_paper_1
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
Github: https://github.com/vlbthambawita/MedicoTask_2019_paper_1 Thambawita et al.
Table 1: MAE values collected from the proposed methods:
D1-stacked gray-scale nine consecutive frames, D2-stacked
an original frame + a dense optical flow image from two con-
secutive frames + a dense optical flow from two frames with
stride=10; M1 - the basic model of Resnet-34 with modifi-
cations of number of input channels and outputs, M2 - the
modified model with an additional MLP with dropout layers
Motility Morphology
Input Method Fold MAE Average MAE Average
Fold 1 9.562 5.626
Figure 2: Big picture of our deep learning model: M1 - the M1 Fold 2 8.959 9.200 5.749 5.649
base model of Resnet-34 with a three output last layer, M2 Fold 3 9.079 5.573
D1
- the modified version of Resnet-34 with an additional MLP, Fold 1 9.585 5.424
D1 and D2 represent the two different types of input used in M2 Fold 2 9.28 9.185 5.382 5.394
our experiments. Fold 3 8.689 5.375
Fold 1 9.044 5.933
For both input types, we split the datasets into three folds based M1 Fold 2 8.062 9.372 5.394 5.525
on the folds given in the video dataset provided by organizers. Then, Fold 3 11.01 5.248
a three-fold cross-validation was performed to evaluate our deep D2
learning models which will be introduced in the later sections. Fold 1 8.612 5.549
M2 Fold 2 7.873 8.825 5.463 5.293
Fold 3 9.991 4.868
2.2 Deep learning model implementation
For implementation of our deep learning models, we selected Resnet-
34 which is larger than the smallest, Resnet-18, and smaller than 3 RESULTS AND ANALYSIS
other large scales Resnet models like Resnet-50, Resnet-101, and According to the average MAE values shown in Table 1, the M2
Resnet-152. The selections of this intermediate Resnet-34 was done method with the input type 2 (D2) shows best results among other
based on expandability of the model by adding additional multi- methods and other input types. This method shows the best MAE
layer perceptron (MLP) within the available hardware resources value of 8.825 for the sperm motility and 5.293 for the sperm mor-
(considering memory limitations of the available graphics process- phology. This improvement of error values can be seen as results
ing units). In addition to that, the pre-experiments were done to of accumulated benefits of showing pre-processed temporal infor-
identify over-fitting problems of strong models for simpler predic- mation such as dense optical flows to the model and the additional
tions and computation time required to finish training. Furthermore, MLP to overcome the problem of over-fitting. Moreover, the added
expandability of the number of input channels of the model within MLP in M2 gives better results with both input types (D1 and D2)
the available GPU memory was examined. for both predictions: sperm motility and sperm morphology. We
For method 1 (M1), we modified the input layer of the selected achieved this performance as a result of the pre-processed input
pre-trained Resnet-34 to take nine channel inputs and modified data with dense optical flows and the MLP introduced to overcome
the last layer of the model to output only three values which are the over-fitting problem.
representing either three values of sperm motility or three values
of sperm morphology. We used this method as our base model with 4 CONCLUSION AND FUTURE WORK
the two different datasets (D1 and D2) as introduced in Section 2.1 The input with a raw frame and dense optical flows of two difference
and recorded results collected from this experiment in D1-M1 and stride values show better results compared to the stacked normal
D2-M1 rows in Table 1. frames of videos. Moreover, the modified Resnet-34 model with an
In method 2 (M2), to avoid over-fitting problems of this task, MLP which consists of dropout layers with high probabilities did
we have embedded additional MLP to the end of the network with achieve better results than the base model in the both cases because
dropout layers [17]. The full structure of this additional MLP is it helped to overcome the problem of over-fitting in the training
depicted in Figure 2 using a green colour. The dropout values of stage. Finally, the combination of the input with dense optical flows
this MLP were selected using pre-experiments, and it is a hyper- and the modified Resnet-34 with an additional MLP shows the best
parameter for this model. The collected results of this method are overall performance.
tabulated in rows D1-M2 and D2-M2 of Table 1. In future work, it is worth to try CNN models with long short-
In the training process of all the above methods, the Adam opti- term memory units to capture temporal features of video frames.
mizer [11] with a learning rate 0.001 was used. The mean square Moreover, a 3D CNN can be a promising approach for this kind of
error (MSE) was used as the loss function for back-propagating task because 3D CNN models have capabilities to capture temporal
error, and mean absolute error (MAE) was used for calculating the information of videos.
actual loss of predictions based on ground truth values of motility
and morphology.
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
2019 Medico Medical Multimedia Github: https://github.com/vlbthambawita/MedicoTask_2019_paper_1
REFERENCES [11] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic
[1] Alan C Bovik. 2010. Handbook of image and video processing. Academic optimization. arXiv preprint arXiv:1412.6980 (2014).
press. [12] Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and
[2] Darin Brezeale and Diane J Cook. 2008. Automatic video classification: Lawrence Carin. 2018. Video generation from text. In Thirty-Second
A survey of the literature. IEEE Transactions on Systems, Man, and AAAI Conference on Artificial Intelligence.
Cybernetics, Part C (Applications and Reviews) 38, 3 (2008), 416–430. [13] Sharon T Mortimer, Gerhard van der Horst, and David Mortimer.
[3] Karan Dewan, Tathagato Rai Dastidar, and Maroof Ahmad. 2018. Esti- 2015. The future of computer-aided sperm analysis. Asian journal of
mation of Sperm Concentration and Total Motility From Microscopic andrology 17, 4 (2015), 545.
Videos of Human Semen Samples. In Proceedings of the IEEE Conference [14] Konstantin Pogorelov, Michael Riegler, Pål Halvorsen, Steven Alexan-
on Computer Vision and Pattern Recognition (CVPR) Workshops. der Hicks, Kristin Ranheim Randel, Duc-Tien Dang-Nguyen, Mathias
[4] Gunnar Farnebäck. 2003. Two-frame motion estimation based on Lux, Olga Ostroukhova, and Thomas de Lange. 2018. Medico Multi-
polynomial expansion. In Proceedings of the Scandinavian conference media Task at MediaEval 2018.. In Proceedings of the CEUR Workshop
on Image analysis. Springer, 363–370. on Multimedia Benchmark Workshop (MediaEval).
[5] Arun Hampapur, Terry Weymouth, and Ramesh Jain. 1994. Digital [15] Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Carsten Gri-
video segmentation. In Proceedings of the second ACM international wodz, Thomas Lange, Kristin Randel, Sigrun Eskeland, Dang Nguyen,
conference on Multimedia. ACM, 357–364. Duc Tien, Mathias Lux, and others. 2017. Multimedia for medicine:
[6] Trine B. Haugen, Steven A. Hicks, Jorunn M. Andersen, Oliwia the medico task at MediaEval 2017. (2017).
Witczak, Hugo L. Hammer, Rune Borgli, Pål Halvorsen, and Michael A. [16] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012.
Riegler. 2019. VISEM: A Multimodal Video Dataset of Human Sperma- UCF101: A dataset of 101 human actions classes from videos in the
tozoa. In Proceedings of the 10th ACM on Multimedia Systems Confer- wild. arXiv preprint arXiv:1212.0402 (2012).
ence (MMSys’19). ACM, New York, NY, USA. https://doi.org/10.1145/ [17] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever,
3304109.3325814 and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Neural Networks from Overfitting. Journal of Machine Learning Re-
residual learning for image recognition. In Proceedings of the IEEE search 15 (2014), 1929–1958. http://jmlr.org/papers/v15/srivastava14a.
conference on computer vision and pattern recognition. 770–778. html
[8] Steven Hicks, Pål Halvorsen, Trine B Haugen, Jorunn M Andersen, [18] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018.
Oliwia Witczak, Konstantin Pogorelov, Hugo L Hammer, Duc-Tien Mocogan: Decomposing motion and content for video generation.
In Proceedings of the IEEE conference on computer vision and pattern
Dang-Nguyen, Mathias Lux, and Michael Riegler. 2019. Medico Mul-
recognition. 1526–1535.
timedia Task at MediaEval 2019. In CEUR Workshop Proceedings -
[19] L. F. Urbano, P. Masson, M. VerMilyea, and M. Kam. 2017. Auto-
Multimedia Benchmark Workshop (MediaEval).
matic Tracking and Motility Analysis of Human Sperm in Time-Lapse
[9] Itseez 2014. The OpenCV Reference Manual (2.4.9.0 ed.). Itseez.
Images. IEEE Transactions on Medical Imaging 36, 3 (March 2017),
[10] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung,
792–801. https://doi.org/10.1109/TMI.2016.2630720
Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classifica-
[20] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan,
tion with convolutional neural networks. In Proceedings of the IEEE
Oriol Vinyals, Rajat Monga, and George Toderici. 2015. Beyond short
conference on Computer Vision and Pattern Recognition. 1725–1732.
snippets: Deep networks for video classification. In Proceedings of the
IEEE conference on computer vision and pattern recognition. 4694–4702.