In this paper we present the approaches that achieved the rst place in this years ImageCLEFcoral challenge. The task of the challenge was the localization and classi cation of corals within images of sea ground. Therefore we had to extract bounding boxes for each coral and labeling them with the speci c type of substrate. We applied a state-of-the-art deep learning approach (YOLO) and also developed a two-staged approach, using a grid along with two classi ers. One that classi es the tiles of the grid, the other that classi es the found boxes. We had moderate results using YOLO and discovered that locating the corals is the most challenging part. Furthermore class imbalance and intersecting boxes, made the problem even harder.
Climate change is one of the major problems of the 21st century. Its impact
is growing every year and therefore researched a lot. Since corals are a signi
cant part of the maritime environment they are a ected by the climate change
in many ways [
ImageCLEFcoral dataset [
The training set, which we de ne as dataset (A), contains 240 images with 6670
annotated substrates. Generally there is a di erentiation between 13 substrate
types. Which are: \Hard Coral { Branching, Hard Coral { Submassive, Hard
Coral { Boulder, Hard Coral { Encrusting, Hard Coral { Table, Hard Coral {
Foliose, Hard Coral { Mushroom, Soft Coral, Soft Coral { Gorgonian, Sponge,
Sponge { Barrel, Fire Coral { Millepora and Algae - Macro or Leaves" [
When investigating the dataset, the problem of overlapping boxes appeared to
us. Many of these bounding boxes fully contained or intersected with other boxes.
To be more speci c, only 2672 of 6670 bounding boxes do neither overlap or are
contained in a bigger one. For this reason, we had started to investigate whether
the substrates di er from each other at all, why we searched for meaningful
features. These features were extracted from extracted bounding boxes. We
applied a classical approach using SIFT [9]. Furthermore we calculated structure[
The amount of given data is remarkably low. Usually even a pre-trained neural
network requires a larger data set, why we decided to use data augmentation.
To generate new data, we used the following methods: noise and blur[
The images vary in quality and many of them are out of focus or blurry. To counter this and to create an improved dataset (C), we increased the contrast of entire images and highlighted the details. For this purpose, each pixel value was replaced by the weighted average of its 3 3 neighborhood. The following matrix shows the lter:
The challenge of the annotation and localization task is to nd corals within images of sea ground, de ne bounding boxes for each coral and label them with their speci c type of substrate.
We applied one state-of-the-art deep learning approach and additionally developed an own one. These two are presented in the following subsections, whereas the focus lies on explaining our own approach. 3.1
In contrast to comparable neural networks, like "fast R-CNN" [
Limitations However, there are some limitations. On the one hand each cell of the grid predicts only two boxes, which share the same class label. This is an algorithmic limitation on the number of objects with di erent labels, if the objects are close to each other. On the other hand the authors of YOLO mention that they treat errors in small bounding boxes the same way they treat large bounding box errors. Because of that, errors in small boxes have a larger impact on IoU, which leads to incorrect localization. 3.2
We developed a two-staged approach that rst locates and then labels the substrates. Both of these steps make use of machine learning, to be more precise classi cation algorithms. This leaves room to improve the classi cation task, e.g. by evaluating di erent classi cation algorithms.
Resize image to
t grid Classify tiles
Find connectedcomponents
Delete small areas (<10 tiles)
Extract boxes
Raw image
Classify
area Box coordinates
Substrate label
One advantage of this two-staged approach is, that the two stages are independent from each other, which makes it possible to combine di erent algorithms and approaches together. The algorithm of our approach is shown in gure 1. Locating Substrates The main idea behind locating substrates is based on the assumption, that the coral images have coral and non-coral areas. Such non-coral areas should look relatively similar for all coral types. This is quite di erent for images showing objects like cars or birds. Following our assumption, we segmented an image in coral and non-coral areas. First the image is divided in a grid, small enough to predict all boxes. To classify these areas we used a grid and then extract features from the tiles of this grid. We used a square of a xed size for the tiles, which is based on the size of the smallest boxes in the training set, i.e. the integer average of the smallest width and height. Based on the training set, we recommend to use a tile size of 12 12, so that the smallest box can be located completely without background. To ensure that all tiles of an image have the same size, i.e. image size is the whole multiple of the tile size, the image is scaled to t.
Next we extracted features for every tile of each of the training set images.
For this purpose we used concatenated feature vectors consisting of features
describing the color, texture and shape. For color, normalized histograms are
used which describe the characteristics regarding the color [10]. The textural
features of the tiles are modeled by Haralick texture features [
After each tile of the grid was classi ed, we got an black and white image with 12 12 pixel large tiles. There are multiple strategies to extract boxes out of the resulting picture. We used a relatively naive approach with the application of connected-component labeling. Since we discovered a large amount of single, not connected tiles, we only kept components, that consisted of more than ten tiles. This counters a less bene cial performance of our classi er. Each unique component is now bordered with a bounding box, that borders the outside tiles of the component. Figure 2 is showing the di erent stages of the location process (b - d), as well as the ground truth (a).
Labeling Found Boxes The found bounding boxes were classi ed on previously mentioned features 2.1 using a k-Nearest Neighbors Classi er. In addition to this already presented classi cation approaches, we studied whether the features can also be classi ed using a convolutional neural network. For the research, we subdivided the training data into a training and validation set in a ratio of 80:20 as previously. For comparability of the results we scaled the input data to the size of our grid. In consideration of the low amount of image data, we begun our work with a correspondingly small CNN, which we call baseline. The given CNN consists of one convolutional layer with maxpooling and recti ed linear activation. We use dropout to prevent over tting. The deactivation of neurons happens with a 20% probability. Subsequently, the data is handed to a attening layer which serves as connection between convolutional and following dense layers. The result rst enters a dense layer with RELU as the activation function and is then passed on to a density layer with softmax as activation function. This leads us to a con dence for each bounding box to belong to one of our 13 classes.
Considering that such a simple architecture may not be able to "remember" all relevant features of coral images, we extended our baseline architecture. Therefore we enlarged the existing architecture with two additional convolutional hid(a) Grund truth boxes.
(b) Inside (white) and outside tiles (black). (c) Re ned inside and outside areas.
(d) Bounding boxes of connected components. den layers.
Finally we looked for an extra deep architecture for comparison. All networks were trained with a batch size of 100 and with up to 1000 epochs. We decided to use VGG19 [12] and trained it on our data via transfer learning, since it has been proven to be gold standard in recent years. 4
The following section discusses the submitted runs at ImageCLEF 2019. For a better understanding of the results of the approaches, we evaluated the localization and labeling separately. The results show that YOLO is considered state-of-the-art for a reason.
Besides presenting our results of the submissions, we also discuss the limitations and potentials of our approach as well.
In table 2 we present the results of our submissions. Our own approach is marked with I, and YOLO based submissions with II. MAP 0:5 stands for the localised mean average precision for each submitted method with an IoU 0.5 of the ground truth and R 0:5 for the recall value, respectively MAP 0 represents the image annotation average without any localization. The results for I show, that CNNs and k-NN deliver comparable results. Sharpening has not led to better results, perhaps because it accentuates noise. YOLO combined with statistical probability distribution provides best results with an precision of 0.243 and a recall of 0.131.
Table 2 shows that we worked only on data set (A) and (C). We did not use data set (B) for our run submissions, since it did not lead to any kind of improvement.
All approaches we used have some limitations and therefore leave space for improvement. Some of which we will describe in the following.
YOLO The weakness of YOLO is evident on rather smaller bounding boxes. Predictions on the validation data set showed that small coral substrates are either not found or subsequently labeled incorrectly. This results in the low recall value of 0.131. Corals that are found however, are mostly labeled as "c soft coral". Nevertheless, even on larger corals, YOLO shows rather moderate results. In a quarter of images it did not nd boxes at all, that is why we used the found boxes from our other approach I to complete the results. Our Approach Not only the performance (see Table 2) shows aws in our approach, but also some obvious conclusions do. Since we got an accuracy of 0:534 on labeling boxes, which was evaluated on a 80 : 20 split of the training set, we assume that our approach fails to locate corals correctly. We also tested using SIFT features which had an accuracy of 0:4744.
One problem of the two-staged approach is the assumption, features of coral and non-coral tiles are distinct enough. This leaves room for further evaluation and research, regarding the choice of features and labels. It might be bene cial to use more than two labels, i.e. more than just coral and non-coral. This could e.g. be water in the background, because we discovered that water in the background is often "false positive" classi ed, i.e. as coral area. An additional label would also need an additional annotation.
The question arises, whether 14 (13 substrate classes + background) labels could be used. This approach would only need one, instead of two classi ers. Regarding the tile classi cation, the usage of CNNs to classify the tiles sounds promising because of the high number of tiles.
Another issue with our approach is the size of the tiles. Big tiles prevent small boxes from getting found and increase the chance of two corals in one tile. From a design perspective, boxes should be as small as possible to be as precise as possible. But if tiles are chosen relatively small, not only does the computational time extend, but features contain less information. This could lead e.g. to forms not getting recognized. We encountered the problem of an enormous computational time, because of that we increased the size to 24 24. Also we reduced the training set of tiles by 80%, which decreases the performance not signi cantly as seen in table 3. An approach of using a sliding window should also be considered in future work. Lastly using connected components as the method to extract boxes from the tile images, could be not sophisticated enough. Firstly with a perfect labeling of coral and non-coral tiles, it would not be able to recognize inlying boxes. And secondly it only considers two labels as features. The use of density-based clustering, working on more than just the predicted labels could lead to better results. 5
Overall, our approaches show moderate results. The idea to use neural networks proved to be promising. However, afterwards we can assert that YOLO was not the best choice. It completely fails to nd smaller bounding boxes.
The concept of using feature engineering and searching for features or feature constellations, which are able to describe and represent di erent types of benthic substrate, still seems to be useful regarding the small amount of given data. But there is a lot of room for improvement.
Beside of that there are multiple images in the data set that show the same sea ground and contain for the most part the same corals. This kind of information can be used locally to improve the bounding boxes of corals, since their position can be tracked.
With regard to our approach 2b of labeling coral and non-coral areas, we can make the conclusion that the chosen features are not working properly. Probably we need a kind of back propagation to mark wrong labeled areas and process images multiple times. Additionally we could investigate the set of our features for a more performant subset using boosting. We would also stick to the deep learning approach and try another, maybe more time consuming but also more precise neural network, like an R-CNN.
Finally, the concept of combining deep learning and classic feature engineering is where we see the most potential.
Besides that, another point of potential improvement is the correction and balancing of the data set itself. Currently, seven of 13 coral type classes have a relative ratio of less than two percent, six out of them even less than one percent. The quality of the pictures is very variable too. Some of the images do not even seem to be completely annotated.
For future approaches, we would recommend publishing a larger and more balanced data set, in which each class has almost the same number of representatives.
To address the initial question whether an automatic localization and annotation of corals is feasible, we see good chances for future research. 9. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91{110 (Nov 2004) 10. Pass, G., Zabih, R., Miller, J.: Comparing images using color coherence vectors.
In: ACM multimedia. vol. 96, pp. 65{73. Citeseer (1996) 11. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Uni ed, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779{788 (2016) 12. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)