Understanding the composition of species in ecosystems on a large scale is key to developing e ective solutions for marine conservation, hence there is a need to classify imagery automatically and rapidly. In 2019, ImageCLEF proposed for the rst time the ImageCLEFcoral task. The task requires participants to automatically annotate and localize benthic substrate (such as hard coral, soft coral, algae and sponge) in a collection of images originating from a growing, large-scale dataset from coral reefs around the world as part of monitoring programmes. In its rst edition, ve groups participated submitting 20 runs using a variety of machine learning and deep learning approaches. Best runs achieved 0.24 in the annotation and localisation subtask and 0.04 on the pixelwise parsing subtask in terms of MAP 0.5 IoU scores which measures the Mean Average Precision (MAP) when using the performance measure of Intersection over Union (IoU) bigger to 0.5 of the ground truth.
The ImageCLEFcoral task described in this paper is part of the ImageCLEF3 benchmarking campaign [1]. ImageCLEF is part of CLEF4 (Cross Language Evaluation Forum) and it provides a framework where researchers can share their expertise and compare their methods based on the exact same data and evaluation methodology in an annual rhythm. In 2019, there were four tasks in ImageCLEF: ImageCLEFlifelog; ImageCLEFmedical; ImageCLEFcoral; and ImageCLEFsecurity.
In its rst edition, ImageCLEFcoral follows the successful ImageCLEF annotation tasks (2012-2016) [2{6] and requires participants to automatically annotate and localize a collection of images with types of benthic substrate, such as hard coral and sponge. 3 http://www.imageclef.org/ 4 http://www.clef-campaign.org/
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2019, 9-12 September 2019, Lugano, Switzerland.
The genesis of the ImageCLEFcoral task is from the ever-increasing amount of data being collected as part of marine conservation e orts. With the rise in popularity both of Scuba diving and underwater photography, high quality equipment to archive the underwater world has become inexpensive for marine biologists and conservation enthusiasts. The performance of so-called \action cameras" such as GoPros can yield high quality imagery with very little photographic expertise from the user. Such imagery can be used to assess the health of a coral reef, for measurements of individual species and wider benthic coverage of assemblages indicating phase shifts in the marine ecosystem [7]. However, this creates a volume of data that is too large to be annotated by human labellers, and this problem is substantially more di cult than the segmentation of other image types due to the complex morphologies of the objects [8].
A typical computer vision application involves image capture, then segmenting regions of interest from the surroundings, and nally classifying them. There has been a long-standing use of machine learning in the last of these stages in particular, because alternative strategies such as rule-based classi cation are generally less e ective. Classifying man-made objects (such as in previous ImageCLEF annotations tasks), which may have simple, well-de ned shapes, is more straightforward than classifying biological objects because the shape of the latter varies from one instance to another; correct classi cation will often involve a combination of shape and texture.
In the same way, segmenting a natural shape from its surroundings is not necessarily straightforward. For this task for example, colour might be enough to segment some parts of a single coral from a rock lying behind it but other parts of the same coral might lie in front of a similarly-coloured sponge or sh | this makes the segmenting task much more di cult.
The di culty in both the segmentation and classi cation stages explains why there are two sub-tasks within the coral reef exercise. The rst is to identify the type of substrate present within a region without having to identify its outline precisely, while the second involves segmenting the coral correctly as well as classifying it.
The rest of the paper is organised as follows: Section 2 presents the 2019 ImageCLEFcoral task; Section 3 provides an overview and analysis of the collection; Section 4 details the evaluation methodology; Sections 5 and 6 present and discuss the results of the participants; and nally, Section 7 concludes the paper indicating possible new directions for the challenge. 2
In its rst edition, the ImageCLEFcoral task follows a similar format to 2015 and 2016 ImageCLEF annotation tasks [2, 3] and includes two subtasks: Coral reef image annotation and localisation subtask For each image, participants produce a set of bounding boxes, predicting the benthic substrate for each bounding box in the images.
Coral reef image pixel-wise parsing subtask For each image, participants produce a set of polygons bounding each benthic substrate and predict the benthic substrate for each polygon in the image. This task aims to provide a more detailed segmentation of the substrate in the image. 3
The annotated dataset comprises several sets of overlapping images, each set taken in an area of underwater terrain (see Section 3.1). Figure 1 shows an example of an image in the collection. Each image was labelled by experts (see Section 3.2). The training set contains contains 240 images with 6670 substrate areas annotated and the test set contains 200 images with 5370 substrate areas annotated.
Some types of benthic substrates are easy to detect with obvious features, others are much more di cult. For example, hard corals are typi ed by having a rigid skeleton unlike soft corals which often have a soft, u y appearance. The classi cation and descriptions of organic benthic substrate are as follows: Hard Coral - Branching: Morphologies grow like a tree, with branches coming out from a centre point and continually branching (secondary branching). Hard Coral - Sub-Massive: Unlike branching colonies, these are columnar morphologies which do not have secondary branching. All branches come from the rst column and typically they will only have columns. Hard Coral - Boulder: Coral morphologies that grow in spherical shapes and are often called massive corals or brain corals (on some species polyp walls form ridges that look like a human brain).
Hard Coral - Encrusting: Morphologies that form a layer over hard surfaces, and they follow the contours of the surface. These are more di cult to see than more 3D corals, but some of those genera that form 3D morphologies also form encrusting morphologies.
Hard Coral - Table: Corals morphologies that look like a table, typically with a stalk connecting it with the substratum. Many of these colonies begin as branching colonies, but their branches form tight networks to create a at surface with a table-like appearance.
Hard Coral - Foliose: These morphologies are named after their leaf-like structure. They are sometimes referred to as cabbage corals.
Hard Coral - Mushroom: These are single polyp corals. Their skeleton looks like that of an up-turned mushroom, with the ridges mimicking the \gills" of the underside of a mushroom. These can be found all over a reef, and are from only one family, the Fungidae. These single polyps are much larger than the polyps you nd in colonies.
Soft coral: Soft corals cover a wide variety of species that are distinguished by their apparent exibility, texture and colour. They may have clearly distinguishable open polyps. This category covers all soft corals, except Gorgonian sea fans (see below).
Gorgonian: Gorgonian sea fans are distinctive soft corals that grow as large branching planar structures that face into the prevailing current. They can resemble tree-like structures with a thick trunk and branches, as well as more uniformly branching structures that do not form complete plates. Sponge: These can have a varied morphology and can also be highly cryptic.
They di er from corals in that they do not have polyps but tiny holes, giving them a pitted appearance. This category covers all sponges, except barrel sponges (see below).
Barrel sponge: A sponge with a highly distinctive, dark orange/brown barrel shape with a wide opening at the top. Barrel sponges were classi ed morphologically, not taxonomically, so any with this growth form were grouped. Fire coral (Millepora): This is not actually a coral but rather a colony of tiny animals called hydroids. They grow in a variety of shapes but typically a branching staghorn structure. They can also encrust rocks and other organic substrate such as gorgonian sea fans.
Algae: There are several classes of algae but we are interested only in the large-leaved foliose macro-algae and not turf algae, crustose coraline algae, encrusting algae or maerl which are di cult to classify from imagery. This type of algae can vary from large, vivid green leaves to paler, u y bushes. 3.1
The images for the ImageCLEFcoral task are a subset from a growing, largescale collection of images taken of coral reefs around the world as part of a coral reef monitoring project with the Marine Technology Research Unit (MTRU) at the University of Essex. The subset used was collected from several locations in the Wakatobi Marine Reserve in Sulawesi, Indonesia in July 2018. The data was collected using SJCAM5000 Elite action cameras in underwater housings with a red lter attached, held at an oblique angle to the reef. Most images have a tape measure running through a portion of the image because they are part of a monitoring collection. 3.2
The images were manually annotated using a custom online polygon drawing tool (see Figure 2 for a screenshot of the tool).
Each image was hand-annotated by postgraduate coral biology students to identify benthic substrate and validated by an administrator. Using the online tool, areas of organic benthic substrate were identi ed. First a polygon area was created by clicking points on the image (see Figure 3) and then a substrate type was chosen and the label completed (see Figure 4).
Fig. 4: A cropped screenshot of the annotation interface showing a completed polygon classi ed by colour.
Annotators were instructed to ensure the following rules: { Click points must be in sequence around the object to avoid overlapping bounding boxes; { Click points and eventual bounding box should be inside the annotated object; { One bounding box can be used for multiple individuals (substrates) so long as they are all the same benthic substrate type; { Bounding boxes must not overlap.
As a quality control to rank and train annotators, they were provided with ve training images with a rich diversity of substrates, annotated previously to pixel level (see Figure 5). These training images are considered as \golden annotations" and used for the per-class agreement of the users and the groundtruth.
The agreement per class has been calculated, on a pixel level, with the intersection over union (IoU) metric: agreement =
# of true positives # of false positives + # of false negatives + # of true positives :
After discarding the annotators with an agreement level smaller than 15% for quality control, the average annotation agreement and standard deviation was calculated per category of the benthic substrate type. 3.3
For the training set, the proportion of \background" pixels (i.e., pixels not annotated with any of substrates) was 76:44%. Excluding the background, the benthic substrate type pixel-distribution can be found in Figure 6. The gure shows a large imbalance towards soft-corals, whereas seven benthic substrate types have been underrepresented in less than 3% of the total non-background pixels (\Hard Coral - Submassive", \Hard Coral - Table", \Hard Coral - Foliose", \Hard Coral - Mushroom", \Soft Coral - Gorgonian", \Fire Coral - Millepora", \Algae - Macro or Leaves"). The task was evaluated using the methodology of previous ImageCLEF annotation tasks [2, 3],which follows a PASCAL style metric of IoU. We used the following three measures: M AP 0:5 IoU : the localised Mean Average Precision (MAP) for each submitted method using the performance measure of IoU >=0.5 of the ground truth; R 0:5 IoU : the localised mean recall for each submitted method using the performance measure of IoU >=0.5 of the ground truth; M AP 0 IoU : the image annotation average for each method in which the concept is detected in the image without any localisation.
In addition, to further analyse the results per types of benthic substrate, the measure accuracy per substrate was used, in which the segmentation accuracy for a substrate was assessed using the number of correctly labelled pixels of that substrate, divided by the number of pixels labelled with that class (in either the ground truth labelling or the inferred labelling). 5
In 2019, 13 teams registered for the rst edition of the ImageCLEFcoral task. Five individual teams submitted 20 runs. Table 1 gives an overview of all participants and their runs. There was a limit of at most 10 runs per team and subtask. ISEC [9] Coimbra Institute of Engineering, Portugal VIT [10] Vellore Institute of Technology, India HHUD [11] Heinrich-Heine-Universitat Duesseldorf,
Germany SOTON University of Southampton, UK MTRU [12] Marine Technology Research Unit, University of
Essex, UK 1 5 9 0 0 0 0 1 3 1 # Runs T1 # Runs T2 5.1
The HHUD team [11] achieved the best results in terms of M AP 0:5 IoU by applying a state-of-the-art deep learning approach, YOLO. Unlike the regional convolutional neural network (R-CNN) approach adopted by [10], YOLO works on the whole image at the same time, by dividing the entire image into square cells which are predicted to contain bounding boxes of substrate. This should mean that there are fewer background errors compared to R-CNNs because more context is taken into account. In addition, the authors devised an approach of their own, rst locating and then classifying, and then utilising machine learning. The strategy for locating possible substrate is that they di er from background regions, so the authors partition the image into small \tiles" and construct a feature vector containing colour, shape and texture measures: normalized histograms, grey-level co-occurrence matrices and Hu moments, respectively. From these features, a binary classi er is trained to distinguish coral and non-coral tiles. For classi cation of the substrate types, both k-nearest neighbour and convolutional neural network (CNN) approaches were examined, the latter contrasting shallow and deep networks and utilised the popular pre-trained VGG19 with transfer learning.
The contribution from the ISEC [9] team took a traditional computer vision approach. The authors established that colour alone is not a suitable measure for classi cation and established a feature vector that encapsulated colour and texture information: the mean, standard deviation, entropy of a grey-scale version of the original colour image, plus a hue ratio which measures colour content. All of these were calculated in a 5 5 region around each pixel. Using this feature vector, the authors explored a variety of machine learning algorithms (e.g., nearest neighbour, decision trees, discriminant analysis and support vector machines). They found that the most e ective machine learning algorithm was random forests.
The VIT submission [10] approaches the annotation task using CNNs, which have been applied to a wide range of image classi cation tasks in recent years. A standard CNN would fare badly on this task because it is tuned to the entire image containing a single instance of the object to be classi ed, so they have used a variant known as Faster R-CNN. It overcomes the issue by extracting \region proposals" from the image, then using an algorithm to combine them into those passed on to the CNN. They have explored layering this on top of three existing CNNs: NASnet, Inception V2, and Resnet101. All these models are provided with the Tensor ow object detection API and are provided pre-trained using the COCO dataset, which contains 300,000 images in 80 categories of natural and man-made objects. Using the ImageCLEF training images, the authors ne-tuned the ability of the various networks to perform the classi cation task. They also explored the e ect of augmenting the networks' training dataset by adding noise, changing the brightness and contrast, and performing geometrical distortions such as shears, shifts, rotations and mirror imaging. CNNs currently represent the state-of-the-art in machine learning for image classi cation, and this use of well-known techniques provides a good benchmark.
age pixel-wise parsing subtask. Five runs were submitted in this subtask by three teams. In this subtask, a team comprising of researchers at Filament working with the University of Essex [12] achieved best results in terms of M AP 0:5 I oU . They developed a classi cation system based around Deeplab V3, a deep CNN that reduces the amount of post-processing necessary to deliver a nal, semantic segmentation and classi cation. Their post-processing involved connectedcomponent labelling, morphological opening and closing to delete small regions and polygon approximation, all reasonably conventional image processing functions.
The SOTON team used a Keras implementation of Deeplab V3+, pre-trained on the well-known Pascal VOC and ne-tuned for each class using a one-versusall, pixel-wise classi er which was trained on the ImageCLEF dataset, with the loss weighted by the ratio of pixels belonging to the class versus not. As is common with deep networks, the training data were augmented with rotation, ipping, shearing and elastic distortions. The result was then passed through a conditional random eld (CRF), which allows groups of similar entities to be assigned the same label.
The HHUD [11] team also participated in this subtask and used a similar approach from subtask 1 (see Section 5.1).
HHUD and SOTON submitted self-intersecting polygons, like the one in Figure 9, which were excluded from the evaluation.
The ultimate goal of this ImageCLEF task is the reconstruction of 3D models of coral reefs from images and using measurements of complexity, surface area and volume from the reconstructed models for advances in marine conservation monitoring. The 3D reconstruction process is known as visual structure from motion (SfM) and requires only a set of uncalibrated images of the object to work from. This has been used for a wide variety of reconstructions ranging from small arch ological nds to large-scale environments with some success. In the context of this work, reconstructions range from individual coral specimens (from about 100 images) up to entire reefs, the latter using an innovative multi-camera capture system. To assess the health of a coral reef, marine biologists need to measure individual specimens and it has been established that reconstructions are accurate enough for this to be possible. However, this cannot be automated because there is no easy way to establish which reconstructed points belong to which substrate type; indeed, this problem is substantially more di cult than the segmentation of images. Hence, we have conjectured that, given annotated image regions in 2D, it should be possible to carry the assigned class labels forward through the SfM processing pipeline, resulting in a 3D model in which individual substrates are labelled [13]. The 2D labelling task is what this ImageCLEF task addresses.
As such the task is di erent from similar image classi cation and marine substrate classi cation tasks [14{16]. Firstly, the images were collected using low-cost action cameras (approx. $200 per camera) with a xed lens and ring on a three second lapse. The e ect of this on the imagery is that there is some blurring (in some images this is quite bad), the colour balance is not always correct (as the camera adjusts the white balance automatically based on changing environmental variables) and nal image quality is lower than what could be achieved using high-end action cameras or DSLRs which are more typically used in this type of research. However, all of the images used in the task are used for building the 3D model and therefore have useful information in the pipeline. Low cost cameras were used to show this approach could be replicated a ordably for marine conservation projects.
Additionally, the distance and angle the camera was facing the reef was unpredictable due to how they were placed on the multicamera array. This meant that some images were close-range and downward facing whilst other images were oblique across the reef. This has a big impact on the number of objects in the eld of view and the ability of annotators to label them.
Tables 3 and 5 shows how the accuracy of the proposed approach varied between benthic substrate types. In fact, a marine dataset is di cult to annotate due to the wide variety of growth forms of benthic organisms that causes errors in image hand-annotation. Sponges are widely varied in their morphology and can often look like both hard and soft coral species, particularly encrusting varieties. Branching and submassive hard coral growth forms can be easily mistaken for each other due to their similar morphologies and the di culty of noting secondary branching from images alone. Branching hard coral and lobed soft coral are also di cult to distinguish. However, some morphologies are easier to detect, such as gorgonian sea fans and barrel sponges, hence these were annotated as separate categories.
Despite the di culties, the participants applied a variety of machine learning and deep learning approaches with promising results. We noticed that in the coral reef image pixel-wise parsing subtask many self-intersecting polygons were submitted and the evaluation approach excluded this type of polygon which could cause the low performance in the subtask. 7
The rst edition of the ImageCLEFcoral task required participants to automatically annotate and localize benthic substrate (such as hard coral, soft coral, algae and sponge) in a collection of images used for marine conservation monitoring. Five groups participated in the task with a variety of machine learning and deep learning approaches.
We hope that future editions of this task will include images from di erent geographical areas, meaning that the visual features of the substrate classes will be di erent; however, it may be possible to employ a cross-learning technique from training data from other regions. Additionally, we hope to develop methods for evaluating the subtasks based on insitu evaluation and photogrammetric evaluation, giving participants a richer set of metadata to use within their computer vision approaches.
The work of Antonio Campello was supported by Innovate UK, Knowledge Transfer Partnership project KTP010993, and hosted at Filament Consultancy Group Limited. The data collection was funded by a University of Essex Impact Acceleration Account grant ES/M500537/1 with support from Professor David Smith and Operation Wallacea. We would also like to thank the annotators Edward Longford, Ekin Yagis, Nida Sae Yong, Abigail Wink, Gareth Naylor, Hollie Hubbard, Nicholas Adamson, Laura Macrina, James Burford, Duncan O'Brien, Deanna Atkins, Hollie Sams and Olivia Beatty. Multilinguality, Multimodality, and Interaction. Proceedings of the 10th International Conference of the CLEF Association (CLEF 2019), Lugano, Switzerland, LNCS Lecture Notes in Computer Science, Springer (September 9-12 2019) 2. Gilbert, A., Piras, L., Wang, J., Yan, F., Ramisa, A., Dellandrea, E., Gaizauskas, R.J., Villegas, M., Mikolajczyk, K.: Overview of the ImageCLEF 2016 scalable concept image annotation task. In: CLEF Working Notes. (2016) 254{278 3. Gilbert, A., Piras, L., Wang, J., Yan, F., Dellandrea, E., Gaizauskas, R.J., Villegas, M., Mikolajczyk, K.: Overview of the ImageCLEF 2015 scalable image annotation, localization and sentence generation task. In: CLEF Working Notes. (2015) 4. Villegas, M., Paredes, R.: Overview of the ImageCLEF 2014 scalable concept image annotation task. In: CLEF Working Notes, Citeseer (2014) 308{328 5. Villegas, M., Paredes, R., Thomee, B.: Overview of the ImageCLEF 2013 scalable concept image annotation subtask. In: CLEF Working Notes. (2012) 6. Villegas, M., Paredes, R.: Overview of the ImageCLEF 2012 scalable web image annotation task. In: CLEF Working Notes. (2012) 7. Young, G.C., Dey, S., Rogers, A.D., Exton, D.: Cost and time-e ective method for multi-scale measures of rugosity, fractal dimension, and vector dispersion from coral reef 3d models. PLOS ONE 12(4) (04 2017) 1{18 8. Howell, K.L., Bullimore, R.D., Foster, N.L.: Quality assurance in the identi cation of deep-sea taxa from video and image analysis: response to henry and roberts.
ICES Journal of Marine Science: Journal du Conseil 71(4) (2014) 899{906
9. Caridade, C.M.R., Marcal, A.R.S.: Automatic classi cation of coral images
using color and textures. Volume 2380., Lugano, Switzerland, CEUR-WS.org
<http://ceur-ws.org> (September 9-12 2019)
10. Jaisakthi, S.M., Mirunalini, P., Aravindan, C.: Coral reef annotation and
localization using faster r-cnns. Volume 2380., Lugano, Switzerland, CEUR-WS.org
<http://ceur-ws.org> (September 9-12 2019)
11. Bogomasov, K., Grawe, P., Conrad, S.: A two-stage approach for localization and
classi cation of coral reef structures. Volume 2380., Lugano, Switzerland,
CEURWS.org <http://ceur-ws.org> (September 9-12 2019)
12. Ste ens, A., Campello, A., Ravenscroft, J., Clark, A., Hagras, H.: Deep
segmentation: Using deep convolutional networks for coral reef pixel-wise parsing. Volume
2380., Lugano, Switzerland, CEUR-WS.org <http://ceur-ws.org> (September
912 2019)
13. Stathopoulou, E., Remondino, F.: Semantic photogrammetry boosting
imagebased 3d reconstruction with semantic labeling. ISPRS - International Archives
of the Photogrammetry, Remote Sensing and Spatial Information Sciences
XLII2/W9 (0