The automatic annotation of coral images is important for researching the underwater ecosystem, which is the focus of the ImageCLEFcoral task. We participated by re ning our approaches from the last years challenge for localization and classi cation of corals within images of sea oor. Underwater images bear multiple di culties which we tackle with applying image enhancement algorithms. To locate and classify the corals we applied multiple deep learning approaches and also revisioned our two-staged algorithm. The results show that deep learning approaches are the most convincing. Still, the localization of corals is the most challenging part for us, but we managed to increase our models performance signi cantly.
Monitoring coral reefs and their health is an important component to understand
e ects of the climate change on maritime life [
In this paper we present the improvements of our approaches which are based
on the last years ImageCLEFcoral [
Overall we increased the performance of our approaches and can provide more insights, which we present in the following sections.
The research of last year coral task can be divided into classical feature
engineering and deep learning approaches. Caridade and Marcal [
For the purpose of the task [9][
The substrate representatives have a highly imbalanced distribution as shown in table 1. A random split of the data can lead to an even more disadvantageous class distribution, since it can exclude the representatives of the rare classes in one of the sets or increase the impact of frequent classes on the set. We give an example with a split into train and validation subsets with a ratio of 80 : 20 which distribution can be seen in table 2. To prevent the tendency of high imbalance, we propose the following procedure similar to [22].
May P (D) be the relative distribution of classes of the complete image dataset D. Since both splits of the dataset should have the same distribution of classes, P (D) is our target distribution. May A and B, with A \ B = ;, be the two sets after splitting D with the relative distributions P (A) and P (B). The key idea is to swap images between the two initial random splits A and B to make P (A); P (B) and P (D) as similar as possible. Let a; b with a 2 A and b 2 B be the two images to swap between A and B. We de ne A = (A n fag) [ fbg and respectively B = (B n fbg) [ fag. If the similarity between P (A ) and P (D) is smaller than the similarity between P (A) and P (D), we swap the items, so that A = A and B = B . Since the similarity between P (B ) and P (D) decreases when the similarity between P (A ) and P (D) decreases, this approach works w.l.o.g. The loop over the images is running until there are no more swaps, i.e. no swap increases the similarity. To measure this similarity we use JensenShannon divergence [11]. Since this procedure only converges to a local, but not global optimum, not every random split ends up in the same balanced split but consequently the optimal split is not found every time. Further research is needed to evaluate this approach, in regards of optimizations and metrics. The results of the balancing algorithm are shown in the table 3.
The Jensen-Shannon divergence between the train and validation set before swapping is 0.040313, whereas after swapping its divergence is 0.0061. This is an improvement by a factor of almost 8. The "Coral reef image annotation and localisation task" can be divided into two tasks. Segmentation of various coral objects from images of sea ground and classi cation of those with their speci c type of one of the 13 known types of
Due to multiple di culties that underwater images bear, strategies to
enhance the image quality which should help to nd better features are used. We
applied two of the state-of-the-art deep learning approaches and additionally
combined these with an improvement of our own development [
Underwater images inherit problems, like the attenuation of light or the suspension of particles re ecting the light. Those conditions distort colors and visibility, which a ect the performance of machine learning algorithms. Therefore we applied and evaluated multiple enhancement algorithms, that are specialized on underwater images.
Ancuti et al. [
The evaluation [18] of the dataset of 2019 showed an improvement in image quality using the two enhancement algorithms, as seen in table 4 as well as in the subsections 4.4 and 5. Example images are shown in gure 1. Neural networks are still state of the art in segmentation and classi cation tasks. Last year we used Yolov3 [20] as our main neural network approach. Especially areas with a particularly large denseness of smaller corals were challenging. The reason for this is on one hand Yolo's native ROI restraints which sets a natural limit on the regions considered within a certain area. On the other hand we were limited by the input data size with the largest resolution we could use with GPU of 608 608 pixels. Regarding the original resolution of images of 4032 3024, we preserve a scaling factor of at least of 5, i.e. each pixel represents an area of more than 25 pixels in the original image. Considering that the smallest corals consists of 12 12 pixels, the information loss is signi cant. We split the images into overlapping subimages of 608 608 pixels and trained the network on unscaled input. Consequently we got a large amount of images which also meant that the training time of our network was almost one month. Unfortunately, there was a mistake in the training data, therefore a revision of the results of the last years challenge was just recently announced. This left us with not enough time to retrain our working setup. 4.3
For our second neural network approch we chose RetinaNet. The network showed impressive results on the COCO dataset and outperformed Yolov2 with a 17% higher AP0.5 value. RetinaNet is well suited, since it is able to produce more predictions and is capable to work with less balanced data. At its core, the architecture consists of the following components: a feature pyramid network [12] (based on Resnet), a regressor for bounding box prediction and a classi er. Basically, it is a one-stage detector. The particular advantage of RetinaNet is the focal loss [14]. In case of end-to-end object detection, background predictions oftentimes dominate. The optimizer rates the prediction as correct and the loss of the positive background prediction forms the complete return loss. This mostly leads to an optimizer return value of zero for the background areas in case of cross entropy and thus reduce the loss. Focal loss weights the positive samples higher and ensures that the network performs better on unbalanced data.
Right suited anchor boxes are the key to quality of object detection for any architecture that works with a "regions of interest". If the anchors are not properly prepared, the network has in many cases no chance of nding particularly small, neither large objects. In our dataset, we experience a wide variety of sizes of corals. Starting from a box size of 18 9 to a size of 3966 2662 pixels, the standard deviation of the areas is 629 assuming a square size. That means that irregular or peripherally sized objects present a special challenge. To tackle this problem, we chose a solution that was originally used on medical data [25]. In our opinion, the potential for improvement can be easily transferred to coral context, since tumors and nodules as smaller objects are comparable to coral objects of small size. 4.4
Besides the deep learning we also increased the performance of our classic feature based approaches. We evaluated the use of principal component analysis (PCA) [19] to select the best features, which increased the performance slightly. Apart from the features the choice of the classi er is important. Last year we used k-NN, which is depended from the parameter k. To overcome the search for the right parameter we evaluated the use of nave bayes [21] for locating and classifying substrates.
When classifying the coral areas and non-coral areas, the features along with
the approach are the same as in [
Table 5 holds the evaluation of the coral/non-coral grid that is compared with a grid representing the ground truth. The results show that the nave bayes classi er has increased the accuracy, as well as the precision but greatly decreased the recall. PCA on the other hand did not have a big impact.
It could be shown that image enhancement increases the performance, even if just slightly. Table 6 showed that connected components works much better with the nave bayes classi er that k-NN. The nave bayes classi er has an insigni cantly increased, whereas k-NN has signi cantly worse performance. We believe that this is caused by k-NN having to much false-positives which results in big boxes, that cover rather more than less area. Another indication for this assumption is the decrease of precision but increase of recall (compare gure 2b).
We also used the same two classi ers for classifying the bounding boxes.
While using the same set of features, k-NN outperformed nave bayes by far.
Image enhancement shows its contribution again with tripling the accuracy of the
nave bayes classier, as seen in table 7. This substantiate that the enhancement
proposed by Ghani and Isa [
The evaluation was processed on test data that consists of images from four di erent geographical regions than the training set: { same location { similar location { geographically distinct but ecologically connected { geographically and ecologically distinct
In total the test dataset has 400 images, made by 100 images per subset. The
results are examined in more detail below, while the interesting and informative
values are discussed in the text. For the complete list of results, the reader
is referred to the task overview working note [
The classi cation of the substrate types based on the classic features alone fails largely with an MAP0 value of around 27.4. and an MAP0.5 of 1. Although we improved the MAP0.5 value compared to last year by 300 percent, the results are still not really useful due to the low absolute values. This applies to both experiments, for the classi cation by means of k-NN based on the chosen features boosted by PCA and to statistical label assignment as well.
Our neuronal Network based approaches show a signi cantly better performance.
Since we had a limitation in the number of submissions, we chose none linear composition of available options. Our pool of options consisted alongside to classical features of RetinaNet and Yolov3, which both were trained on the unpreprocessed data and also on enhanced images by RD and Fusion. In addition, we worked with a variation of threshold 2 f0:001; 0:1; 0:2; 0:5g, which limits the MAP.
(a) Grund truth boxes.
(b) Inside (white) and outside tiles (black)
using k-NN.
(c) Inside (white) and outside tiles (black) (d) Inside (white) and outside tiles (black)
using PCA and k-NN. with PCA and naive Bayes.
Fig. 2: Visualized process of localization corals with our approach. The raw
picture is taken from the ImageCLEFcoral dataset [
We achieve our best result with an ensemble of RetinaNet and Yolov3. Whereby the predictions of RetinaNet were extended by the predictions of Yolov3. Both Networks were trained on RD-preprocessed images, with a of 0.1. The combination of both systems results in a MAP0.5 of 39.2 % and MAP0 of 80.6 %. It is noticeable that we predicted very few bounding boxes for a MAP0.5 both through Yolov3 and through RetinaNet, additionally the found predictions were far from being present in all of the images.
In case of reduction of the accepted overlap, we get signi cantly more bounding boxes which also leads to a higher chance of hitting the right coral within the test images on cost of our overall accuracy. However, if we increase the accuracy, the MAP value drops. The following are signi cant examples: RetinaNet ( = 0.01) combined with Yolov3 ( = 0.01) has an MAP0.5 of 0.303 and an MAP0 of 0.727, but only an overall accuracy of 7 %. In contrast to it RetinaNet alone produces signi cantly fewer boxes with ( = 0.2), but achieves the best overall accuracy of 14.2 % with an MAP0.5 of only 30.3 and a MAP0 of 66.3 % while the accuracy is 10 % higher than in our best run.
A possible explanation is probably that RetinaNet has not nished training in 35 epochs. The large amount of predictions with a low overlap value could probably be boosted by non-maximum suppression. However, we were not aware of this problem until the results were published.
When provided with such a variety of possibilities, it can not be clearly determined which enhancement variant is the best choice. Certainly enhanced images improve the predictions, while we suspect that RD is superior to Fusion for coral images. 5.1
One major question is the transferability of the results within the test dataset, considering that results, that form the average measure, vary widely. For our best run the MAP0.5 for same location increases by 6 percent to 45.7 % and by 1 % to 81.5 % the MAP0. For similar location, however, it drops to 28.3 % for MAP0.5, but reaches a value of 86.4 % for MAP0 and thus has the highest score among all presented submissions by all participants. Our best approach performs very well on geographically similar data. The MAP0.5 is 42.6 % and the MAP0 81.2 %. Overall, this part of the test dataset seems to be less complex, which is shown by the fact that the performance of almost all of our approaches have an increased performance on it.
It is also worth mentioning that we perform signi cantly worse on geographically distinct data with just an MAP0.5 0.125 and an MAP0 0.362. This tendency is also evident in the other approaches we used. The data seems to di er significantly, we either generalize less or the classes that are harder to recognize are more present. Some additional, yet unknown substrate types may be more dominant in geographically and ecologically distinct rocky reef and lead to distorted results. A further investigation of the dataset is required. 6
Overall, our approaches show signi cantly better results than last year. A comparison of our best approaches between the two years shows that we have improved the MAP0.5 by 13.4 %. The classi cation of the tiles into in- and outside boxes could also be improved by a Bayesian classi er, but is still far from being accurate.
Image enhancement techniques on the last years data were con rmed by the evaluation of the current test dataset, which leads us to the conclusion that the correction of blurry images in terms of contrast and sharpness is necessary. The RD algorithm makes the greatest contribution to improving the quality of the images. Less noteworthy results are made with the classical feature engineering approach. A deeper examination of the features and their information value is needed.
It can be assumed that the fair partitioning of images according to our balancing strategy, also has signi cantly contributed to the improved results. Deep learning strategies generalize quiet well and are superior when using for this task. Especially the performance of RetinaNet, since it is not only better on the coco dataset than the state of the art. So far, the complexity of the images can hardly be handled by a single approach. We still see the most potential in an ensemble of several architectures. The combination of advantages of di erent approaches is the key to a stable solution. Since the amount of data has grown, while remaining relatively small, we still cannot exclude the potential of classic machine learning. Usually neural networks show a much better performance with an increased amount of data. For this reason, the images should be split into overlapping images and thus increase the number of training samples.
For future approaches, we recommend the usage of more speci c features
that are suitable for corals, such as used in [
Information theory 37(1), 145{151 (1991) 12. Lin, T., Dollar, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. CoRR abs/1612.03144 (2016), http://arxiv.org/abs/1612.03144 13. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980{2988 (2017) 14. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV) (Oct 2017). https://doi.org/10.1109/iccv.2017.324, http://dx.doi.org/10.1109/ICCV.2017.324 15. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2), 91{110 (2004) 16. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing 21(12), 4695{4708 (2012) 17. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a \completely blind" image quality analyzer. IEEE Signal processing letters 20(3), 209{212 (2012) 18. Nguyen, H.C.: Comparison of methods for underwater image improvement (2020) 19. Pearson, K.: Liii. on lines and planes of closest t to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2(11), 559{572 (1901) 20. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018) 21. Rish, I., et al.: An empirical study of the naive bayes classi er. In: IJCAI 2001 workshop on empirical methods in arti cial intelligence. vol. 3, pp. 41{46 (2001) 22. Ste ens, A., de A. Campello, A.C., Ravenscroft, J., Clark, A., Hagras, H.: Deep segmentation: using deep convolutional networks for coral reef pixel-wise parsing.
In: CLEF (2019) 23. Venkatanath, N., Praneeth, D., Bh, M.C., Channappayya, S.S., Medasani, S.S.: Blind image quality evaluation using perception based features. In: 2015 Twenty First National Conference on Communications (NCC). pp. 1{6. IEEE (2015) 24. Xydeas, C., , Petrovic, V.: Objective image fusion performance measure. Electronics letters 36(4), 308{309 (2000) 25. Zlocha, M., Dou, Q., Glocker, B.: Improving retinanet for ct lesion detection with dense masks from weak recist labels. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 402{410. Springer (2019)