The purpose of this work is to address the imageCLEF 2019 coral challenge - to develop a system for the detection and identi cation of substrates in coral images. Initially a revision of the 13 classes was carried out by identifying a number of sub-classes for some substrates. Four features were considered { 3 related to greyscale intensity (1) and texture (2), and 1 related to the colour content. The Breiman's Random forest algorithm was used to classify the corals in one of 13 classes de ned. A classi cation accuracy of about 49% was obtained.
Coral reefs are large underwater structures composed of the skeletons of colonial
marine invertebrates called coral. The coral species that build reefs are known
as hermatypic, or "hard," corals because they extract calcium carbonate from
seawater to create a hard, durable exoskeleton that protects their soft, sac-like
bodies. Other species of corals that are not involved in reef building are known
as \soft" corals. These types of corals are exible organisms often resembling
plants and trees and include species such as sea fans and sea whips, according to
the Coral Reef Alliance (CORAL), a non-pro t environmental organization [
Coral reefs support immense biodiversity and provide important ecosystem
services to many millions of people, yet they are degrading rapidly in response
to numerous anthropogenic drivers [
In this paper, we propose to solve this challenge using a fully automatic process to identify coral substrates in digital images. The method developed uses colour and texture to identify regions of interest and, using the Breiman's Random forest algorithm [6] to classify a coral in one of 13 classes. The digital images used in this study are 3024 4032 pixels, in RGB (Red, Green and Blue) format. 2
The data for the ImageCLEF2019 Coral task originates from a growing,
largescale collection of images taken from coral reefs around the world as part of a
coral reef monitoring project with the Marine Technology Research Unit at the
University of Essex [
The training set contains 240 images with 6430 substrates annotated. Two les are provided with ground truth annotations: { one based on bounding boxes
"imageCLEFcoral2019 annotations training task 1" { and a more detailed annotation based on bounding polygon
"imageCLEFcoral2019 annotations training task 2".
The test set contains 200 images [
The methodology proposed to detected and identify substrates in coral images is presented schematically in Figure 1. In a rst phase, the training images are processed, and the regions that de ne each coral substrates are identi ed (Coral substrates). Then, the features of each of these substrates (Coral features) are used to train the classi er (Classi cation). In a second phase, the classi er is applied to the test images, thus obtaining images of classi ed corals. These images are post-processed and their corals (Connected components) are identi ed, a text le with the relevant information is produced.
The 13 types of substrates were identi ed in the 240 training images. A total of 6430 substrates annotations are availably, as listed in Table 1 and illustrated in Figure 2. A colour is assigned for each substrate, presented in the second column of Table 1. The number of occurrences of each substrate in the training images is presented in the third column of Table 1.
Figure 2 shows two training images and the number and type of substrates identi ed in these images. On the image on the left (2018 0714 11244 018), the following substrates were identi ed: "hard coral branching" (4 times), "hard coral boulder" (2 times), "hard coral encrusting" (3 times), "soft coral" (8 times), "sponge" (7 times), "sponge barrel" (1 time) and "algae macro or leaves" (3 times). For image 2018 0729 112525 048 (right): "hard coral branching" (4 times), "hard coral boulder" (4 times), "hard coral encrusting" (2 times), "soft coral" (11 times) and "sponge barrel" (1 time). 3.1
By visual observation of several training images, it was veri ed that within the same substrate there are di erent types of corals, both in shape, colour and texture. Therefore, within each substrate di erent types (sub-classes) were identi ed. Another di culty in the analysis of these images is the overlapping of the substrates identi ed in the training image. For example, in the left image of Figure 2, the substrate "sponge" (number 11 in green) is inside the region de ned by the substrate "hard coral boulder" (number 0 in pink). When we look at the coral images we nd that they have di erent colours from green, blue, red, orange, brown and white. However, these colours are not unique identi ers of a substrate. The textures are also present on the substrates. Some corals are harder, rugged, sharper, and others smoother and softer. Hue is also a relevant characteristic of substrates.
With the replicates identi ed, the 4 most relevant features (by empirical way) were calculated, in a 5x5 neighbourhood of each pixel: mean (M), standard deviation (STD), entropy (E) of grayscale image and hue ratio (HR). The mean and the standard deviation of the neighbourhood identify the spatial arrangement of intensities in a selected region of an image and are represented in Equation 1 and Equation 2 respectively. This is
M =
P p
N ST D = r (p
N
M )2 1
; E =
X p log2(p); where p is the intensity of the pixels and N is the neighbourhood size. The Entropy is a statistical measure of randomness that can be used to characterize the image texture. Entropy (Equation 3) is de ned as where p is the intensity of the pixels.
The RGB image is converted to the HSV (Hue, Saturation, Value) colour model. Only values of Hue (H) below 0.34 or above 0.73 are considered for this feature. This threshold values were obtained by visual inspection of the training images. After that, the Hue ratio (HR) in equation 4 is calculated by dividing the number of pixels belonging to the range (H < 0:34 or H > 0:73) over the total number of pixels in the neighbourhood region. (1) (2) (3) (4) HR = #pixels(H < 0:34 or H > 0:73) #pixels : Using Classi cation learner app available in MATLAB [7] environment, it is possible to classify the training coral image using various algorithms and compare the results in the same environment. After training multiple models, they were compared, based on validation errors. The classi cation models available in this app are: decision trees, discriminant analysis, Support Vector Machines (SVM), logistic regression, nearest neighbors, and ensemble classi cation. With the data described previously and the 4 features used, the best model was Random Forest which obtained a classi cation accuracy score of about 49%.
Random Forest consists of a collection of classi ers based on decision trees in which each tree gives rise to a vote for the output forecast. Random Forest produces a model consisting of n decision trees (ensemble), where each tree is based on a number of randomly selected instances of the training set. Each node of each tree is constructed from a random subset of the attributes. Upon receiving a test instance each tree will decide (vote) on which class it belongs to. The most voted class will be the class provided by the model. The most widely reported Random Forest algorithm is the Breiman algorithm [6].
The application of Random Forest pixel to pixel classi er in a test image with 3024 x 4032 pixels has a large computational weight since 4 matrices of 3024 x 4032 are constructed for the 4 features of each image pixel and another matrix with same size with con dence values. On the other hand, as it was necessary to classify 200 test images, the initial images were reduced by a factor of 4, so that the processing became faster. 3.4
After classi cation it is necessary to nd connected components of pixels belonging to the same substrate (area). As the areas of the substrates already identi ed in the training image are greater than 43,865 pixels, then in the classi cation of the test images, only areas with more than 500 (20 percent of one sixteenth of the average size of the substrates - 20%(43; 865=16) 500) pixels will be validated as substrate areas if the con dence valuer is greater than or equal to 0.5. With the regions identi ed in the images it is necessary to collect information about the image and class to which they belong, the degree of con dence of this classi cation and the position in the image of the bounding box that surrounds it (x minimum, y minimum, width and height). Finally, this information is placed in a text le as follows: 4
The 200 test images were processed according to the methodology described in section 3. Figures 4 and 5 show the results obtained for two di erent test images.
In Figure 4 the results for the image 2018 0712 073252 024 and in Figure 5 for the image 2018 0712 073534 067. The original image (a) and the classied image (b) and (c) where the pixels are identi ed within a substrate. The black pixels are not classi ed one of the following situations: substrate areas or con dence valuer (see section 3.4). The coral identi ed (d) with only the coral substrates presents in the test image. The colour shown in the images correspond to the colours identi ed for each substrate in the table 1. In the case of (a) (c) (b) (d) Fig. 5. : Original image (a), classi ed image in gray (b) and color (c) representation and the coral identi ed (c). The methodology developed allows to identify substrates of di erent classes in digital images of corals. The preliminary results are not very favourable, but there are many potential improvements that can be implemented. The most promising lines of work forward would be to focus on a better identi cation of sub-classes, and the use of additional features related to both colour and texture. Luca Piras, Michael Riegler, Minh-Triet Tran, Mathias Lux, Cathal Gurrin, Obioma Pelka, Christoph M. Friedrich, Alba Garc a Seco de Herrera, Narciso Garcia, Ergina Kavallieratou, Carlos Roberto del Blanco, Carlos Cuevas Rodr guez, Nikos Vasillopoulos, Konstantinos Karampidis, Jon Chamberlain, Adrian Clark and Antonio Campello, ImageCLEF 2019: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 10th International Conference of the CLEF Association (CLEF 2019), Lugano, Switzerland, LNCS Lecture Notes in Computer Science, Springer (September 09-12 2019). 6. Breiman, L.: Random forests, Machine Learning, 45(1), 5-32 (2001). 7. MATLAB R2017a, The MathWorks, Inc., Natick, Massachusetts, United States.