This paper presents an overview of the ImageCLEFcoral 2020 task that was organised as part of the Conference and Labs of the Evaluation Forum - CLEF Labs 2020. The task addresses the problem of automatically segmenting and labelling a collection of underwater images that can be used in combination to create 3D models for the monitoring of coral reefs. The data set comprises 440 human-annotated training images, with 12,082 hand-annotated substrates, from a single geographical region. The test set comprises a further 400 test images, with 8,640 substrates annotated, from four geographical regions ranging in geographical similarity and ecological connectedness to the training data (100 images per subset). 15 teams registered, of which 4 teams submitted 53 runs. The majority of submissions used deep neural networks, generally convolutional ones. Participants' entries showed that some level of automatically annotating corals and benthic substrates was possible, despite this being a di cult task due to the variation of colour, texture and morphology between and within classi cation types.
Coral reef systems are delicate natural environments, formed of highly complex
non-uniform structures that support the biodiversity found in tropical coral reefs.
Coral reefs also form a vital source of income and food for over 500 million
people, providing ecological goods and services such as food, coastal protection,
new biochemical compounds, and recreation with an estimated value of around
$352,000 ha-1 y-1 [
However, there has been a steady decline in coral reefs in recent years [
Coral reef community composition is an essential element for monitoring reef
health and the importance of automated data collection, 3D analysis and
largescale data processing are increasingly being recognised [
The composition of marine life on a coral reef varies globally. Within the
Coral Triangle, a region that encloses more than 86,500km2 of coral reef area
and includes the world's highest marine biodiversity, there are over 76% of all
coral species and more than 3,000 sh species [
Geographically distinct regions can contain the same species or genera with entirely di erent morphological features and traits. The variety in both environmental conditions and competitive niche lling can lead to changes in phenotypic expression, which makes the task of identifying them di cult without an extensive training image set.
As part of ImageCLEF 2019 [
Participants' entries showed that some level of automatically annotating corals and benthic substrates was possible, despite this being a di cult task due to the variation of colour, texture and morphology between and within classi cation types.
This year, as part of ImageCLEF 2020 [
The annotation task is di erent from other image classi cation and marine substrate classi cation tasks [12{14]. Firstly, the images are collected using low-cost action cameras (approx. $200 per camera) with a xed lens and ring on timelapse or extracted as stills from video. The e ect of this on the imagery is that there is some blurring, the colour balance is not always correct (as the camera adjusts the white balance automatically based on changing environmental variables) and nal image quality is lower than what could be achieved using high-end action cameras or DSLRs. However, the images can be used for reconstructing a 3D model and therefore have useful information in the pipeline. Low cost cameras were used to show this approach could be replicated a ordably for future projects.
Following the success of the rst edition of the ImageCLEFcoral task [
The ground truth annotations of the training and test sets were made by
a combination of marine biology MSc students at the University of Essex and
experienced marine researchers. All annotations were double checked by an
experienced coral reef researcher. The annotations were performed using a web-based
tool, initially developed in a collaborative project with London-based company
Filament Ltd and subsequently extended by one of the organisers. This tool was
designed to be simple to learn, quick to use and allows many people to work
concurrently (full details are presented in the ImageCLEFcoral 2019 overview
[
The overall task comprises two subtasks: { Subtask 1 : Coral reef image annotation and localisation; { Subtask 2 : Coral reef image pixel-wise parsing.
In the \coral reef image annotation and localisation" subtask, the annotation is a bounding box, with sides parallel to the edges of the image, around identi ed features. In the \coral reef image pixel-wise parsing" subtask, participants submit a series of boundary image coordinates which form a single polygon around each identi ed feature (these polygons should not have self-intersections). Participants were invited to make submissions for either or both tasks.
As in the rst edition, algorithmic performance is evaluated on the unseen test data using the popular intersection over union metric from the PASCAL VOC4 exercise. This computes the area of intersection of the output of an algorithm and the corresponding ground truth, normalizing that by the area of their union to ensure its maximum value is bounded. 3 https://essexnlip.uk/marine-technology-research-unit/ 4 http://host.robots.ox.ac.uk/pascal/VOC/
The data set comprises 440 human-annotated training images, with 12,082 substrates, from the Wakatobi Marine Reserve, Indonesia; this is the complete training and test sets as used in the ImageCLEFcoral 2019 task. The test set comprises a further 400 test images (see Figure 1), with 8,640 substrates annotated, from four geographical regions, 100 images per subset: 1. Wakatobi Marine Reserve, Indonesia { the same location as the training images; 2. Spermonde archipelago, Indonesia { geographically similar location to the training set; 3. Seychelles, Indian Ocean { geographically distinct but ecologically connected coral reef; 4. Dominica, Caribbean { geographically and ecologically distinct rocky reef.
The images are part of a monitoring collection and therefore many have a tape measure running through a portion of the image. As in 2019, the data set comprises an area of underwater terrain. Many images contain the same ground features captured from di erent viewpoints. Each image contains some of the same thirteen types of benthic substrates as in 2019, namely hard coral | branching, submassive, boulder, encrusting, table, foliose, mushroom; soft coral; gorgonian sea fan (soft coral); sponge; barrel sponge; re coral (millepora); algae (macro or leaves).
The test set from the same area as the training set will give an indication as to how well a submitted algorithm can localise and classify marine substrate, i.e., the maximum performance. We hypothesise that performance will deteriorate with other test subsets as the composition, morphology and identifying features of the substrate change and exhibit less similarity with the training data. 3.1
Collection Analysis An important consideration when testing across the datasets is that the benthic composition will be di erent in the di erent locations, in addition to di erent species and morphologies being present and the total coverage of benthic fauna (represented by the total coverage of pixels in an image).
Analysis shows that the community distribution is similar in the same
location test dataset to the training dataset, both in terms of structure and cover.
The similar location test dataset shows a much higher distribution of hard corals
and lower distribution of soft corals and sponge, with considerably higher
coverage, indicative of a healthy coral reef. The geographically distinct but
ecologically connected test set had a high distribution of hard corals in composition
and similar coverage, indicative of a recovering coral reef. The geographically
and ecologically distinct had a higher distribution of sponge and algae,
commonly found in Caribbean reefs that su er human and environmental impacts,
and higher coverage indicative of a phase shift away from hard coral towards a
sponge/algae dominated reef (see Table 1).
The task was evaluated using the methodology of previous ImageCLEF
annotation tasks [
In addition, to further analyse the results per types of benthic substrate, the
measure accuracy per class was used [
# true positives # false positives + # false negatives + # true positives In 2020, 15 teams registered for the second edition of the ImageCLEFcoral task. Four individual teams submitted 53 runs. Table 2 gives an overview of all participants and their runs. There was a limit of at most 10 runs per team and subtask. 5.1
Subtask 1: Coral Reef Image Annotation and Localisation performed well across multiple classes. The highest IoU score (0.512) was for the soft coral class from FAV ZCU PiVa.
Table 5 presents the pixel accuracy per location, per team, across classes for Subtask 1. No individual team performed best across all classes. The highest pixel accuracy scores were 0.5925 in the hard coral branching class from FHD and 0.5116 in the soft coral class by FAV ZCU PiVa. Overall performance is best with the same location test subset; however, the accuracy of hard coral branching in the ecologically similar region was very good. 68213 68212 68205 68202 68201 68198 68182 68183 68197 68196 68188 68187 68186 68185 68184 68181 68179 68178 68146 68145 68143 68138 68094 68093 67919 67914 67863 67862 67858 67857 67558 67539 scores were 0.545 for the soft coral class and 0.505 for the hard coral mushroom class from FHD.
Table 8 presents the pixel accuracy per location, per team, across classes for Subtask 2. FHD performed highest in all classes except hard coral submassive. The highest pixel accuracy scores were 0.718 in the hard coral branching class, 0.562 for the sponge barrel class, 0.547 for the hard coral boulder class and 0.556 for the soft coral class from FHD. Overall performance was best with the same location test subset, with the exception of the hard coral branching class which was identi ed considerably more accurately within the ecologically similar test set. This is a good indication that transfer learning may at least be possible in some classes of substrate. formance per class of all runs submitted by the participant.
s e v a e lr o o r c a m e a g l a a r o p e l l i m la r o c e r r e d l u o b la r o c d r a h g n i h c n a r b la r o c d r a h g n i t s u r c n e la r o c d r a h e s o i l o f la r o c d r a h m o o r h s u m la r o c d r a h e v i s s a m b u s la r o c d r a h e l b a t la r o c d r a h Dataset
Team FHD performed well in the pixel accuracy but not as well when considering the MAP scores and this may be indicative of their approach identifying large polygons well but missing many of the smaller polygon objects. 6
FAV ZCU CV [
They partitioned the training data into distinct training and validation sets containing rough 85% and 15% of the total number of training images. As some types of coral were relatively rare in the training set, there were as few as 16 instances for training and 3 for validation. To train neural networks, more data are clearly needed, so they augmented the images with horizontal and vertical ips, resizing and Gaussian blurring. They also noted that some of the image had a blueish tint while others featured a greenish one and simulated these e ects too.
For training SSD, all training images were resized to 512 512, while for Mask R-CNN they were reduced to 1024 1024. It was found that Mask R-CNN detects many more bounding boxes than SDD, most of which are false positives: of the regions detected, 44.7% were true positives with the former, while 71.3% was achieved with the latter. In terms of average precision, gures as high as 62.17% were achieved (SSD for barrel sponges) but ve coral classes were not found by either.
Interestingly, both trained models performed better on the unseen test imagery than on the images they had retained for validation, and by a fairly large margin. The best mean average precision obtained was 49%, for localization using SSD. This phenomenon is particularly surprising given that the test set contains imagery from ocean regions not present in the training set: the designers of the dataset did not anticipate that this would be the case. It is a particularly promising result given that the ultimate aim of the research is to equip marine biologists and ecologists with a recognition system that can be taken anywhere in the world and expected to work.
FAV ZCU PiVa [
In forming their validation set, this team selected every eleventh image and
substituted some of them so that the training and validation sets had similar
distributions. As with other teams, they augmented the provided training set,
using similar transformations as [
The underlying approach was transfer learning, and several `backbone' networks were examined, including ResNet-50 and Inception-ResNet-V2. One renement employed was a `pseudo-labelling' approach inspired by [26], using a trained network to label untrained test data with weak labels. `Accumulated gradient normalisation' [27] is credited as providing a considerable improvement in performance. An ensemble approach was ultimately used, in which multiple networks classi ed the same input and their majority vote yielded the nal classi cation.
HHUD [
The team used 80% of the identi ed regions for training and 20% for validation, again swapping individual regions between the two sets until they exhibited similar distributions. The di culties inherent in underwater photography due to the severe attenuation of the red end of the spectrum were considered and RD [29] was ultimately demonstrated to be the more e ective.
The team used a version of Yolo [30], though they su ered from some di culties in the training data as initially released which meant that the annotations were inconsistent; there was not enough time to re-train after these were identied and corrected. The constraints on image size with their GPU-based implementation is also thought to have an e ect. RetinaNet was also used, comprising a feature pyramid network based on ResNet [31], a regressor and a classi er.
The authors also explored more classical approaches. In 2019, a k-NN classi er was used; this year, it was enhanced with PCA was used to identify the best features and a nave Bayes approach for locating and classifying substrates. It was found that the combination of PCA and nave Bayes classi er improved performance { though despite this, the neural approaches still out-performed classical ones.
The authors' best performance was achieved using a ensemble of RetinaNet
and Yolo v3, using RD-enhanced images for training. The authors' paper [
One of the key aspects the dataset creators were keen to explore was whether the training dataset, which was acquired from a single coral reef, made it possible for trained classi ers to perform well on data sourced from geographically distinct reefs. In this case, this `geographic generalization' was not found, though the number of test images from the di erent geographical regions was quite small.
FHD [21] went to some lengths to counter the attenuation of red illumination in the images and the blurriness of some of them, achieving impressive visual improvements in some cases. Further improvement was obtained by enhancement in HSV space based on the notion of Rayleigh scattering.
The classi cation architecture was again based around Mask R-CNN, implemented using Keras and TensorFlow and with Resnet 101 pre-trained on the COCO dataset [32], with the training images reduced to 1536 1536 pixels. The training data were augmented using similar transformations to the other groups. As expected, data augmentation reduced over- tting. Colour correction led to poorer mean average precision values but better average accuracy. It was observed that the models do not detect objects as well as some other groups' submissions but those that are detected are classi ed very well.
Interestingly, the authors found their algorithms' performance on subtask 1 (bounding boxes) could be improved simply by re-de ning their bounding boxes. This is really an indication that bounding boxes are a poor way of describing the output of processing that involves both segmentation and classi cation, exacerbated by the extended nature of some types of coral. This suggests that bounding boxes should not form part of ImageCLEFcoral in future years.
The analysis of the results in this paper explores the interplay between the performance measures used and the relative rankings of results. It is not known of course whether these apparent performance di erences are statistically signi cant but this is an area that the designers of the imageCLEFcoral task will explore in future releases.
The approach taken by [
The M AP 0:5 IoU score from FAV of 0.582 over the entire test set is excellent, bearing in mind both the di culty of the problem and that the problem involved 13 classes, some of which are sparsely represented. There is a signi cant peroformance margin before the best run from the second-placed team, FAV ZCU CV, and the other teams' best submissions, which are closely spaced. FAV also made the best-ranked submission for M AP 0 IoU but the other teams' best-scoring submissions are much closer to this. However, the best-scoring submission for R 0:5 IoU does not yield the highest accuracy of all the submissions. Clearly then, there is some inconsistency in the evaluation measures employed | and this is more of an indication that the performance evaluation measures in widespread use in the vision research community are imperfect.
It is interesting to review the scores obtained from the four categories of test data. For the geographic regions which are similar in nature performance is generally similar. However, performance drops o for other regions, showing that the di erences present in the imagery a ect the ability to classify the substrates. This shows how di cult it will be to develop a system for marine biologists to automatically classify substrate without signi cant training resources (i.e., labelled datasets) from that area.
For the pixel-wise parsing task, the M AP 0:5 IoU score of the best-placed team, FAV, is actually higher than for the bounding box task, showing that their approach is able to identify the boundaries of the image features somewhat better than those of the other teams. This makes the performance gap between rst- and second-placed teams somewhat larger than for the rst task. Again, the best-scoring run in terms of M AP 0:5 IoU is not the best in terms of accuracy. 7
The results of the 2020 coral exercise demonstrate how e ective modern deep neural networks are at a range of problems: a performance approaching 70% for a 13-class problem is excellent. The results show that the best pixel-wise parsing technique out-performed the best bounding box one, suggesting that future exercises should concentrate on pixel-wise parsing. There are always di culties with overlapping bounding boxes and other types of feature in the background of bounding boxes which together reduce the value of that type of annotation.
It is clear that there are genuine performance di erences between the four geographical categories of test images described above. This is an important practical problem for coral annotation, as well as for vision systems in general. We anticipate future coral annotation tasks will explore ways to overcome this difculty. Close examination of the ground truth annotations for the pixel-parsing task shows that annotators tend to place the bounding polygons just outside the boundaries of the features being annotated. We are considering producing other annotations that lie within feature boundaries and encourage teams in a future exercise to train the same architecture with both, then see which works best. That would give us the opportunity to learn something about how annotations should be produced.
The fact that di erent measures rank-order the di erent runs di erently does not come as a surprise but does show how di cult it is to devise a simple measure that encapsulates performance well. There is clearly research to be done in this regard. Although there are performance di erences between the runs, there is no indication as to whether they are statistically signi cant or not. This analysis shall be explored in future work. Bearing in mind the point made about performance measures in the previous paragraph, it will be especially interesting to ascertain whether di erent performance measures yield statistically-signi cant but inconsistent results.
The authors would like to thank those teams who have expended substantial amounts of time and e ort in developing solutions to this task. The images used in this task were able to be gathered thanks to funding from the University of Essex and the ESRC Impact Acceleration Account, as well as logistical support from Operation Wallacea. We would also like to thank the MSc Tropical Marine Biology students who participated in the annotation of the test set and Dr Van Der Ven and Dr McKew for facilitating their internship. 21. Arendt, M., kert, J.R., ngel, R.B., Brumann, C., Friedrich, C.M.: The e ects of colour enhancement and iou optimisation on object detection and segmentation of coral reef structures. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, Thessaloniki, Greece, CEUR-WS.org <http://ceur-ws.org> (September 22-25 2020) 22. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., Berg, A.: SSD: Single shot multibox detector. In: Proceedings of the European Conference on Computer Vision, Springer (2016) 21{27 23. He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: Proceedings of the International Conference on Computer Vision, IEEE (2017) 2961{2969 24. Chollet, F.e.: Keras. https://keras.io (2015) 25. Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman, A.: The pascal visual object classes challenge 2007 (voc2007) results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html (2007) 26. Arazo, E., Ortego, D., Albert, P., O'Connor, N., McGuinness, K.: Pseudolabeling and con rmation bias in deep semi-supervised learning. arXiv preprint arXiv:1908.02983 (2019) 27. Hermans, J., Spanakis, G., Mo ckel, R.: Accumulated gradient normalization.
arXiv preprint arXiv:1710.02368 (2017) 28. Lin, T., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: Proceedings of the International Conference on Computer Vision, https://doi.org/10.1109/iccv.2017.324, http://dx.doi.org/10.1109/ICCV.2017.324 (October 2017) 29. Ghani, A., Isa, N.: Underwater image quality enhancement through composition of dual-intensity images and Rayleigh-stretching. SpringerPlus 3(1) (2014) 757 30. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018) 31. Lin, T., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. http://arxiv.org/abs/1612.03144 (2016) 32. Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.: Microsoft COCO: Common objects in context. In: Proceedings of the European Conference on Computer Vision, Springer (2014) 740{755