For this rst edition of LifeCLEF sh identi cation, we have built a processing chain based on background extraction, selection and description of keypoints with an adaptive scale and learning of each species by a binary linear SVM classi er. From the foreground segmentation, we have extracted several groups of blobs, representing a sh each. We have submitted three runs for di erent blob sizes and associations.
Our processing chain starts with a background-foreground segmentation from motion detection (see gure 1). 2.1
For this rst step, an adaptive background mixture model [
The XML metadata provided with the training set de ne the bounding boxes of annotated shes, so we extract keypoints in these bounding boxes ( gure 2). Speci cally, these keypoints are located at the center of the given bounding box and several scales are set for the keypoint descriptor so that the resulting set of keypoints describe the whole sh. The initial scale is set to t the size of the bounding box, the other scales are sequentially incremented from the initial center. To detect a sh with a part of it, the head or the tail, we also slightly shift the central initial keypoint to the left and the right of the bounding box in a spatial-pyramid-like technique. We adopt this strategy of large keypoint descriptors owing to the low resolution of the videos.
The OpponentSift schema [
Since the training distribution must be the same as the test distribution in a supervised learning process, we decide to train a SVM classi er with some OpponentSift descriptors from the video background as negative training samples. The SIFT detector did not give us enough keypoints owing to the low resolution. Thus we choose to densely extract keypoints with approximately same scales as the scales of the training keypoints. Speci cally, we have extracted keypoints with xed scales from 30 to 110 pixels of diameter with respect to the size of the video.
To ensure the de nition of the background, these keypoints were located in the still areas of the motion mask (black areas on gure 1).
For most of the training bounding boxes, part of the background is inside. Thus, before training, we lter the keypoints inside the bounding boxes (which should then be considered as positive training samples of the associated sh): For each one of these keypoints we look for its 10 nearest neighbors and remove any keypoint from the bounding box with more negative neighbors (keypoints from the background not in bounding boxes from the same video) than positive ones (keypoints from other bounding boxes on the same video).
Then, for each species, we train a linear SVM classi er with the species keypoints descriptors as positive samples and the aforementioned ltered background descriptors as negative examples.
We have hence done the rst row of our processing chain during the query video classi cation. First of all, we segment the motion mask to di erentiate each blob. Then, for each blob, the centered and shifted keypoints are extracted with several scales ( xed scales from 30 to 110) and their scores are computed by each species SVM classi er. This score represents the distance between the current descriptor and the SVM decision boundary with the sign of the selected class. Only positively classi ed points with a distance larger than 0.5 are considered. For a blob, we sum up the score over each keypoint associated with each species in order to obtain a global score per species. Then, the list of potential species is given by decreasing scores. For our bounding box construction, we frame every keypoint with respect to its scale and associate the species classi ed with the highest score ( rst species from the list).
Finally, we submitted three runs: 1. The rst one with the blobs computed from the mask segmentation. 2. For the second run, we wanted to detect if there were several shes in connected blobs. Thus we have computed a lighter dilation (than in the rst run) on the initial motion mask and detected the dominant color in each blob. Small blobs with same dominant color were merged into one blob. Then each blob was dilated to obtain the maximum numbers of keypoints on the shes. You can see on gure 5 that even if the sh is separated in several blobs, these ones are then merged together. Finally the same processing is applied to each group of blobs as it is in run 1. 3. The third run has computed groups of blobs as in the run 2 except that we have provided as many answers as little connected blobs regardless their dominant color. 3
At the beginning, we tried to track shes in order to obtain more descriptors with some standard tracking methods as CamShift and segmentation of optical ow for instance. But the tracking was not e ective enough owing to the resolution. We also encountered di culties with moving seaweed and changing brightness as warned by the organizers since the dataset contains videos recorded from sunrise to sunset. We have built up our processing chain to deal with these di culties.
As scoring functions, the organizers of LifeCLEF have computed both
average precision vs recall and precision vs recall for each sh species for the subtask3
( gure 6 and 7). Only bounding boxes matching with a bounding box from test
set with a PASCAL score above a certain threshold are considered. The
organizers have computed a baseline program with the ViBe background modeling
approach [
Compared to the baseline, we have a better precision and worse recall. It is worth noting that the performance in terms of species recognition (based only on the correctly detected bounding boxes) was comparable to the baseline, but our bounding boxes were often too large. This competition was a real challenge since the video data set was di cult regarding video resolution, natural phenomena (e.g. murky water, algae on camera lens, ...) and the huge amount of data to be processed in a limited time. Our results are so pretty encouraging.
To improve our processing chain, a good tracking from the annotated shes would allow us obtaining more positive descriptors for each species. Moreover, some metadata could be used as the GPS coordinates. Finally, with the tracking, we could study the movement of each species to see if this is speci c to species and we could try analyzing the interactions between species.