In the Medico Task 2019, the main focus is to predict sperm quality based on videos and other related data. In this paper, we present the approach of team LesCats which is based on deep convolution neural networks, where we experiment with diferent data preprocessing methods to predict the morphology and motility of human sperm. The achieved results show that deep learning is a promising method for human sperm analysis. Out best method achieves a mean absolute error of 8.962 for the motility task and a mean absolute error of 5.303 for the morphology task.
In an efort to explore how medical multimedia can be used to create
high performing and eficient prediction algorithms, the Multimedia
for Medicine (Medico) Task presents diferent use-cases [
Our approach is based on deep learning using deep convolutional neural networks (CNNs) to predict sperm motility and sperm morphology. All experiments aim to utilize the information in the videos to their fullest, yet still keeping the computational complexity low. The experiments can primarily be split into four distinct groups. Firstly (i), we combine multiple frames channel-wise using diferent stride values (distance between selected frames) and feed this directly into the deep neural network. Secondly (ii), we vary the number of frames used in each sample to see how this may efect the algorithms prediction performance. Thirdly (iii), we threshold the colors of each frame in an attempt to separate the spermatozoa bright color from the darker background, and use this information for prediction. Lastly (iv), we add the patient data to the video analysis to see how this may help in the prediction. Because morphology focuses more on the visual appearance of sperm than the movement, we opted to perform the threshold experiments only on the motility experiments. Internally, we experimented with a wide variety of configurations, but only submitted the best results as the oficial runs. In the following few sections, we will give a brief explanation of our experimental setup (common training configuration between each model and data preparation), and a more detailed description of each approach. 2.1
For each experiment, we use the Inception V3 [
For the methods which used diferent stride lengths to perform prediction on sperm quality, we performed a total of 10 diferent experiments. Stride in this context refers to the distance between two extracted frames within a clip. For example, using a stride length of 5 would select every fifth frame within a given frame sequence. The purpose of this experiment is to exaggerate the change between two frames by increasing the distance of where the two frames were sampled. Each experiment used a clip length of three frames which are greyscaled, resized to 224 × 224 pixels and combined channel-wise. The result is that each clip has a shape of 224 × 3, making it possible to use pre-trained networks. We take advantage of this attribute and train two models for each stride value tested, i.e., one transferring the weights of an ImageNetbased model and one trained from scratch. As previously stated, we performed a total of ten diferent experiments, of which five diferent stride values were used; 1, 5, 10, 30, and 50. 2.3
For the methods which used diferent clip lengths to predict sperm quality, we performed a total of 5 experiments. Each experiment increases the number of frames in a clip by 10, starting at 10 and ending at 50. Each video is captured at 50 frames-per-second, which means that the clips which contain 50 frames represent a whole second of a given video. In contrast to the previous method, each clip included in these experiments have a stride of 1, meaning each frame in a sequence is used for prediction. Similar to the previous method, each frame resized to 224 × 224 and greyscaled before being combined channel-wise. The shape of each clip is then 224 × 224 ×C, where C is the length of the clip.
For the threshold approach, we greyscale each extracted frame and threshold the color at 220, meaning all color values below 220 are set to 0. The spermatozoa in the provided videos have a strong bright coloring which diferentiates it from the darker background. By thresholding the color values, we aim to separate the spermatozoa from the background in order to better emphasize the movement across frames. However, by doing this, we lose some of the visual information present in each sperm, that is why we chose not to apply this method to predict morphology. We organize these experiments in a similar manner as those done for the stride experiments, meaning we stack three frames channel-wise using ifve diferent stride values; 1, 5, 10, 25, and 50.
Each method was evaluated using three-fold cross-validation (as required by the task), and we report the mean absolute error (MAE) and mean absolute error (RMSE) for each experiment. The results for the motility experiments are shown in Table 1, and the results for the morphology experiments are shown in Table 2.
As we can see the prediction of motility results (Table 1), using larger strides between the selected frames in combination with transfer learning works best. The experiments which used a lot of frames per clip seem to have an issue handling the amount if information per sample. Thresholding the color-space seems to preform marginally better than the extended clip length experiments, but are still not as as the experiments using longer strides. Despite the poor results of the threholding approach, all methods beat the ZeroR baseline method. Although the results may not be good enough to be deployed into a clinical setting, it shows that deep neural networks are a promising tool within the field of automatic semen analysis.
Looking at the table for the prediction of morphology results (Table 2), we see that pretty much all experiments lie around the ZeroR baseline. Most, however, beat the baseline by a small margin. It hard to make any strong conclusions about which methods work best, but it seems like using transfer learning for the stride experiments achieves better results than those trained from scratch. As for using diferent clip lengths, all methods seem to achieve a similar results. Overall, the results show that a more specific approach to predicting sperm morphology is needed, for example, analyzing individual spermatozoon using higher image resolutions.
In this paper, we presented the work done as part of the Medico Multimedia Task where we participated in two of the three available tasks. Overall, the results are promising and shows that neural Method Fold 1 Fold 2 Fold 3
MAE RMSE MAE RMSE MAE RMSE
Stride Experiments Stride 1 10.436 14.769 11.079 15.155 11.581 14.404 11.032 14.776 Stride 5 8.563 11.856 9.843 13.754 12.172 15.510 10.192 13.707 Stride 10 9.358 12.711 9.477 15.524 11.892 15.065 10.242 14.433 Stride 25 9.490 13.530 7.149 9.579 10.871 14.532 9.170 12.547 Stride 50 10.005 13.961 9.804 14.468 10.691 13.593 10.167 14.007 TF Stride 1 9.874 13.408 8.450 11.638 10.257 13.972 9.527 13.006 TF Stride 5 10.937 14.699 7.903 10.544 10.322 13.217 9.721 12.820 TF Stride 10 8.714 11.955 8.256 11.153 9.917 13.029 8.962 12.046 TF Stride 25 8.505 11.211 8.818 11.889 10.480 13.919 9.268 12.340 TF Stride 50 9.021 11.505 9.604 11.943 11.338 14.818 9.988 12.755
Clip Length Experiments Clip Length 10 12.400 17.822 11.045 14.110 12.635 16.066 12.027 15.999 Clip Length 20 11.605 16.674 12.867 16.361 11.712 14.778 12.061 15.938 Clip Length 30 10.757 14.871 12.116 21.117 16.435 22.337 13.102 19.442 Clip Length 40 11.225 14.897 9.725 12.866 11.736 15.135 10.895 14.299 Clip Length 50 10.763 14.640 9.843 14.154 11.051 13.728 10.552 14.174
Threshold Experiments Stride 1 9.846 14.397 9.575 13.183 11.371 14.784 10.264 14.121 Stride 5 10.424 14.452 9.991 13.368 9.912 12.942 10.109 13.587 Stride 10 9.544 13.549 11.381 15.570 10.113 13.176 10.346 14.098 Stride 25 9.378 13.536 10.055 13.480 11.062 14.481 10.165 13.832 Stride 50 9.621 13.270 9.331 12.240 11.917 15.083 10.290 13.531
Baseline
ZeroR 13.880 18.680 13.590 16.980 12.090 14.680 13.190 16.860