This paper describes the LifeCLEF 2014 sh task, which aimed at benchmarking automatic sh detection and recognition methods by processing underwater visual data. The task consisted of videobased subtasks for sh detection and sh species recognition in videos and one image-based task for sh species classi cation in still images. Our underwater visual datasets consisted of about 2,000 videos taken from the Fish4Knowledge video repository and more than 200,000 annotations automatically obtained and manually validated. About fty teams registered to the sh task, but only two teams submitted runs: the I3S team for subtask 3 and LSIS/DYNI team for subtask 4. The results achieved by both teams are satisfactory.
Underwater video and imaging systems are used increasingly in a range of monitoring or exploratory applications, in particular for biological (e.g. benthic community structure, habitat classi cation), sheries (e.g. stock assessment, species richness), geological (e.g. seabed type, mineral deposits) and physical surveys (e.g. pipelines, cables, oil industry infrastructure). Their usage has bene tted from the increasing miniaturisation and cost-e ectiveness of submersible ROVs (remotely operated vehicles) and advances in underwater digital cameras. These technologies have revolutionised our ability to capture high-resolution images in challenging aquatic environments and are also greatly improving our ability to e ectively manage natural resources, increasing our competitiveness and reducing operational risks in industries that operate in both marine and freshwater systems. Despite these advances in data collection technologies, the analysis of video data usually requires very time-consuming and expensive input by human observers. This is particularly true for ecological and shery video data, which often requires laborious visual analysis. This analytical \bottleneck" greatly restricts the use of these otherwise powerful video technologies and demands effective methods for automatic content analysis to enable proactive provision of analytical information.
The development of automatic video analysis tools is, however, particularly
challenging because of the complexities of underwater video recordings in terms
of the variability of scenarios and factors that may degrade the video quality such
as water clarity and/or depth. A recent advance in this direction was developed
as part of the Fish4Knowledge (F4K)3 project (funded by The European Union
7th Framework Programme), where computer vision and machine learning
techniques were developed to extract information about sh density and richness
from videos taken by underwater cameras installed at coral reefs in Taiwan [
However, the underwater is a rather complex environment because of several factors that make the study of it particularly complex and challenging. In particular, dynamic or multi-modal backgrounds, abrupt lighting changes (due also to water caustics), and radical and instant water turbidity changes a ect the ability to perform visual tasks also for humans. To complicate even more the situation, the underwater environment shows two almost exclusive characteristics with respect to other domains: three degrees of freedom and erratic and extremely fast movements of objects (i.e. sh). This makes sh less predictable than people or vehicles, as sh may move in all three directions changing suddenly their size and their shape in the video. As a consequence of this, although in the F4K project reliable approaches for video-based sh detection and species identi cation were devised, the problem of automatic analysis of underwater visual data remains still open.
In this paper we describe the \Fish Task" organised as part of the LifeCLEF 2014, where video and image based approaches for sh detection and recognition were tested on underwater video/image datasets achieving very good results.
The remainder of the paper is as follows: Sect. 2 provides an overview of the task as well as the underwater video and image datasets used, Sect. 3 presents the participants to the sh task and their approaches whose results, compared to our baselines, are given in Sect. 4. Sect. 5 concludes this report. 2
The LifeCLEF 2014 Fish task aimed at benchmarking automatic sh detection and recognition methods by processing underwater visual data. It basically consisted of three video-based subtasks and one image-based task. The video-based subtasks were:
Subtask 1 { detecting moving objects in videos by either background modeling or object detection methods; Subtask 2 { detecting sh instances in video frames, thus discriminating sh instances from non- sh ones; Subtask 3 { detecting sh instances in video frames and recognising their species;
The image-based subtask, instead, had the goal: 3 www. sh4knowledge.eu Subtask 4 { to identify sh species using only still images containing only one sh instance.
The participants had to submit at most three runs for a subtask. The run le must have had the same format as the ground truth xml le (see below), i.e. it must contain the frame where the sh was detected together with the bounding box (for all the subtasks 1, 2, and 3), contours (only task 1) and species name (for subtasks 3 and 4) of the sh. 2.1
Dataset
The underwater video dataset was derived from the Fish4Knowledge video
repository, which contains about 700,000 10-minute video clips that were taken in the
past ve years to monitor Taiwan coral reefs. The Taiwan area is particularly
interesting for studying the marine ecosystem, as it holds one of the largest
sh biodiversities of the world with more than 3,000 di erent sh species whose
taxonomy is available at http:// shdb.sinica.edu.tw. The dataset contains videos
recorded from sunrise to sunset showing several phenomena, e.g. murky water,
algae on camera lens, etc., which makes the sh identi cation task more
complex. Each video has a resolution of either 320x240 or 640x480 with 5 to 8 fps.
As the LifeCLEF 2014 sh task included four subtasks, we employed di erent
datasets for each subtask. In particular, for subtask 1 we used eight videos (four
for training and four for testing) fully labeled (each single sh instance was
annotated) using the tool in [
The D20M dataset represents a randomly-selected fraction of the data collected within the Fish4Knowledge project, amounting to more than a billion sh images. The D35K dataset is a subset of the D20M one, containing only (but not all) sh images belonging to the 10 most common species. Image annotation was carried out manually and validated by expert marine biologists. Figure 1 shows sample pictures of the chosen 10 species.
The D1M dataset is also a subset of D20M, obtained by the automatic
annotation technique described in [
The datasets used in the LifeCLEF 2014 sh task (subtasks 2, 3 and 4) were generated from the D1M dataset. In particular for subtask 2, which was meant to identify only some sh instances (unlike the subtask 1 which asked for identifying all sh in a video) in video frames, we used 112,078 annotations taken from 957 videos for the training set and 15,245 annotations from 89 videos for the test set.
For subtasks 3 and 4, we, instead, used 24,441 (belonging to 285 videos) annotated sh (and their species) in the training set and 6,956 annotations (belonging to 116 videos) in the test set. Details about the species distribution in the training and test datasets are shown in Table 1.
It is important to note that some species were more common than others; although we tried to make the dataset as uniformly distributed across species as possible, for some of them (most evidently, Lutjanus fulvus ) it was quite di cult to nd a large number of adequate images, which resulted in a lower presence in the dataset.
The datasets (both the training and the test ones) were provided as zipped folders containing: { A ground truth folder where the ground truth for all subtasks were given as
XML les (Fig. 2): { A video folder where all the videos of the entire dataset were contained { A image folder which contains the images corresponding to the bounding boxes given in the XML les. These images were provided only for subtasks 2, 3, 4 in the training phase and only for subtask 4 in the test phase. About 50 teams registered to the sh task, but only two teams submitted runs for the sh task: one, the I3S team, for subtask 3 and one, the LSIS/DYNI team for subtask 4.
The strategy employed by the I3S team for sh identi cation and recognition (subtask 3) consisted of, rst, applying a background modeling approach based on Mixture of Gaussian for moving object segmentation. SVM learning using keyframes of species as positive entries and background of current video as negative entries was used for sh species classi cation.
The LSIS/DYNI team submitted three runs for subtask 4. Each run followed
the strategy proposed in [
Performance evaluation was carried out on the released test sets by computing:
average precision and recall, and precision and recall for each sh species for
subtask 3; average recall and recall for each sh species for subtask 4. The
baselines for subtasks 3 and 4 were, respectively:
{ Subtask 3: The ViBe [
The results achieved by the I3S team for subtask 3 are reported in Fig. 3 and compared to our baseline. When computing the performance in terms of sh detection, we relaxed the constraint on the PASCAL score as the I3S team often detected correctly a sh but the bounding box' size was much bigger than the one provided in our dataset (see Fig. 4).
While the average recall obtained by the I3S team was lower than the baseline's recall, the precision was improved (see Fig. 5), thus implying that their sh species classi cation approach was reliable more than the sh detection approach. Furthermore, we allowed participants to provide a ranked list (top three species) of sh species for each sh instance in the test set and we treated a recognition as a true positive if the correct species was in the top three species. Considering only the most probable class provided in the results submitted by the I3S team makes the performance drop considerably. The reason behind this may be found in the size of the bounding boxes extracted by the I3S team. In fact bigger bounding boxes may contain also background objects and/or other sh instances, thus a ecting the performance of the sh species classi cation approach.
The results obtained by the LSIS/DYNI team for subtask 4 are shown in Fig. 6. In this case, the performance were computed by taking into account only the most probable sh species (i.e. the rst class in the provided ranked list).
Please note the image-based recognition task (subtask 4) was easier than subtask 3 since it does need any sh identi cation module (which is the most complex part in video-based sh identi cation) and we had only ten sh species with very distinctive features. However, LSIS/DYNI team did a great job outperforming our baseline. 5
In this report, we described the LifeCLEF 2014 sh task, which aimed at benchmarking machine learning and computer vision methods for sh detection and recognition in underwater \real-life" video footage. Although the fty teams registered for the sh task, only two teams submitted runs: the I3S team for subtask 3 and LSIS/DYNI team for subtask 4. The main challenge of our underwater dataset was to identify sh correctly (as moving objects) by processing the entire video sequences. This appears evident by looking the results achieved by the I3S team (Fig. 3 and 5): for instance, the I3S team was not able to detect any of the Lutjanus fulvus instances, as it is a rather tiny sh which often tends to hide behind rocks. The subtask 4 was easier (as shown by the results achieved by our baseline) as the sh were already segmented from the background and the considered sh species have very distinctive features that make their identi cation in the feature space almost trivial. Nevertheless, the results achieved by the LSIS/DYNI team are excellent, outperforming our baseline for almost all species.
We have just completed the labelling of other 25 species and we are working on removing the near duplicates from the dataset to make the sh identi cation and recognition task more challenging.
We would like to thank to all teams who participated to the task and the ImageCLEF and LifeCLEF organisation who made the rst edition of sh task possible.