This paper summarizes the proposal made by the SIANI team for the LifeCLEF 2015 Fish task. The approach makes use of standard detection techniques, applying a multiclass SVM based classi er on large enough Regions Of Interest (ROIs) automatically extracted from the provided video frames. The selection of the detection and classi cation modules is based on the best performance achieved for the validation dataset consisting of 20 annotated videos. For that dataset, the best classi cation achieved for an ideal detection module, reaches an accuracy around 40%.
There are di erent scenarios of application where underwater monitoring is a required ability such as biological, sheries, geological and physical surveys. The everyday larger availability of media captured in this environment poses the challenge to extract useful data automatically. This is indeed a hard scenario where e ective techniques are needed to reduce costs and human exposition.
With this aim, CLEF presented in 2014 for the rst time LifeCLEF: the Labs
dedicated to multimedia life species identi cation [
This year the Labs [
This paper describes the approach adopted by the SIANI team. The following sections detail the di erent elements integrated that basically perform initially a detection to later identify the sh species of the cropped image.
As succinctly mentioned above, the sh identi cation task has been decomposed into two phases: detection and classi cation. 2.1
The goal of the detection phase is to reduce the searching area extracting candidate ROIs from the video stream. Three di erent foreground detection approaches have been tested: fast, histogram backprojection and Gaussian Mixture Modeling (GMM). (1) (2) Fast. This approach makes use of a simple and fast background model computed from the video frames, that is robust enough for the detection and extraction problem in some scenarios. This background modeling solution takes advantage of the static camera con guration in this particular scenario.
To de ne the scene background model, bg, we have used a similar method
to commonly-used techniques like mean lter or median lter [
Once the background model is available, simple and fast background subtraction techniques may be applied in each RGB component of the input image, I. The foreground is computed based on a de ned threshold applied to the sum of squares of RGB components of the subtracted image (DR; DB; DG) pixel value
S(i; j) = DR(i; j)2 + DG(i; j)2 + DB(i; j)2 where Dx = Ix bgx for every RGB component (x = R; G; B).
For a pixel in a given image, I(i; j), its corresponding pixel in the foreground image, f g, is computed as
Histogram Backprojection (BackProj). The second detection method
evaluated is inspired in the idea proposed in [
l=k s(hx(l)) 3. Normalize the resulting histogram: hx(k) = cx(k)=max(cx) 4. Backproject the histogram on every image: Px(i; j) = hx(Ix(i; j)) 5. Sum the squares of values of each component of the pixels: S(i; j) = PR(i; j)2+
PG(i; j)2 + PB(i; j)2 6. Use a threshold to separate the foreground of the background:
GMM Based Background Modeling (GMM). The third background
subtraction method analyzed is the one proposed by Zivkovic and van der
Heijden [
B X ^mN (x; ^ m; ^m2Id) m=1 where XT = fx(t); : : : ; x(t T )g is the training set, for the time period T , ^ 1; : : : ; ^ B are the estimates of the means, ^1; : : : ; ^B are the estimates of the variances, and Id is the identity matrix. B is the number of components weighted by ^m.
An optimization process was launched over the training videos to try to nd a suitable con guration for the GMM foreground detection algorithm, including the number of distributions, the background ratio and the number of training frames and learning rates. 2.2
Detected ROIs are fed to the detection phase to identify the sh species. The classi cation phase has been designed based on local descriptors, that are currently well known techniques in di erent Computer Vision (CV) problems.
In texture analysis, an image is described in terms of a local descriptor codes
making use of a histogram, hi, where the bins contain the number of occurrences
of the di erent descriptor codes present in the image. This approach follows a
Bag of Words scheme [
Once de ned the grid setup, for a particular descriptor, d, the resulting feature vector, xId, contains the concatenation of cx cy cell histograms, i.e. the feature vector is de ned as xId = fh1; h2; :::; hcx cyg, where hi is the descriptor histogram for cell i.
In this particular task, we have evaluated di erent descriptors and grid
congurations. In this sense, we have considered the following 8 descriptors:
{ Histogram of Oriented Gradients (HOG) [
{ Local Oriented Statistics Information Booster (LOSIB) [
This section describes the results obtained for the di erent sh identi cation
task phases, highlighting those con gurations that were submitted to the 2015
Lab focused on this particular problem [
The rst dataset, that we call the training dataset, is a collection of cropped images of the di erent sh species. The second collection contains annotated videos, including media that may present a similar scenario to the test data. We called this collection the validation dataset.
This validation dataset is used in the following subsections to analyze the di erent detection and classi cation alternatives, providing a cue to decide the nal system setup chosen for the Fish task submission. In fact, we used both training and validation datasets to select the classi cation approach, and the validation dataset, to select the detection technique and tune its parameters.
Brie y, the training set contains samples of the 15 di erent sh species, i.e. classes. The number of samples per species is indicated in Table 1. The reader will observe that the di erent species are not equally represented through the dataset, circumstance that also is present in the validation and test sets. The average dimension in the training samples is 88 38 102 49 pixels.
The validation dataset contains 9162 samples distributed per class according to the last column of Table 1. The average dimension of those samples is 52 37 56 39 pixels. 3.2
As mentioned above, the annotated validation dataset videos were used to
analyze the performance of di erent detection algorithms. The detection rates for
the three implementations are shown in Table 2, being computed as the total
number of correct or positive detections divided by the number of annotations.
The false detection rate presented is also the ratio between the number of
unmatched or false detections and the number of annotations. This was done to
have a clear evidence of the number of false detections in relation to the number
of annotations. False detections do not necessarily mean a failure in the
detection module, but that there is not annotation for that particular frame and ROI.
Indeed, the annotations were done only when the sh species was clearly
identi able [
A positive detection is considered when there is a signi cant intersection between a given detection container, B and an annotation container, A. As condence measure, we employed the Jaccard Index, J I. This index relates the intersection of both containers with their union, J I = A\B , providing a value A[B between 0 and 1, larger values meaning better matching. For the analysis summarized in Table 2, we have considered 0:4 and 0:5 threshold values.
The high variability of the video segments made extremely di cult to obtain a good tuning of the algorithm parameters. As a consequence, simple approaches yielded better results both in execution time and detection. Indeed, among the techniques analyzed, both Fast and BackProj algorithms provided not brilliant but acceptable detection rates. Fast was chosen with di erent tuning parameters to setup run1, while BackProj was used for the other two submitted runs (run2 and run3 ). The detection approach is later combined with the classi er providing the best performance in the validation dataset classi cation. 3.3
The detection rates achieved, described in the previous section, allowed us to explore our model based approach on the dataset. Certainly, a model approach is not a priori the best solution for the unbalanced classi cation task, but being newcomers, we were interested in applying our experience in other CV problems to evaluate local descriptors in this scenario.
The analysis described in this section presents results in two steps. Firstly, the study evaluates di erent descriptors with the training set, i.e. the collection of cropped images, see Table 1. Secondly, the best descriptors are later evaluated with the validation dataset, to adopt the most promising con guration for the test set.
Table 3 summarizes the results for di erent local descriptors in a 5-fold cross
validation experiment de ned on the provided training dataset, considering a
single multi-class SVM based classi er. This kind of approach has already been
applied for the task [
With the exception of WLD, the whole collection of descriptors reported a high accuracy at least for a particular grid setup. However, this was not the case in the following analysis on the annotated ROIs extracted from the validation dataset, as summarized in Table 4. It seems, that the grid con guration is important for some descriptors. A larger number of cells is preferred for HOG, LBPu2, LGP and LOSIB, while other descriptors such as LBP, LTP and LPQ provide better results with a lower number of cells. Again, WLD is not providing useful classi cation results.
Considering the vital importance of combining several descriptors [
Table 5 summarizes the results achieved for di erent fusion alternatives. The selection of descriptors and grids are based on the single descriptors results achieved for the validation dataset, see again Table 4. This table does not include results with 4 4 grids, as they were not available in time for the deadline.
The rst three alternatives combine the best descriptors for a given grid resolution; the higher the resolution, the better the accuracy. However, there is no real restriction to make use of a unique grid resolution. For that reason we also evaluated the fusion of the best descriptors with di erent grid resolutions, achieving the best overall accuracy with a RBF kernel using the rst 90 PCA components.
For each combination, the selected descriptors and grid setups are indicated, reporting the results using SVM based classi ers, including linear and RBF kernels, with and without a previous dimensionality reduction by means of a Principal Component Analysis (PCA).
The best performing classi er, the fourth combination using RBF kernel with a PCA based features, is used in combination with the selected detection approaches. 3.4
As mentioned above, our team submitted three runs. They all made use of an identical second phase based on a two stages classi er. The selected descriptors combination is the one highlighted in Table 5. This score fusion selection contains six single descriptor classi ers in the rst stage: LTP1 1, LPQ1 1, HOG3 3, LBP3u2 3, LGP3 3 and LOSIB3 3. The second stage makes use of the classi ers scores, that are projected into a PCA space.
Each run di ers in its detection phase. Our rst run made use of the Fast detection algorithm, while the other two integrate the BackProj detector with di erent parameters setup.
The normalized counting scores of the referred runs in the overall Lab analysis are reported in Figure 3. Two teams are over 50%, followed at a remarkable distance by the best runs of other two teams, including our run3, achieving over 30%. Our main focus was on the classi cation phase, that has provided unbalanced results for di erent classes, likely due to the non homogeneous number of training samples per class. A focus based exclusively on the fusion of local descriptors seems not to be reliable enough for the problem. However, the detection phase requires further attention as a larger number of proper detections would improve the overall score. 4
This document describes the model based approach submitted to the LifeCLEF 2015 Fish task by the SIANI team. The proposal explores the use of local descriptors for this problem. We employed standard detection techniques to later apply an ensemble of SVM multiclass classi ers.
Three runs were submitted with identical classi cation stage. One is based on the Fast detection algorithm, while the other two are based on the BackProj algorithm.
The best accuracy achieved for the ideal annotated containers reaches 40%, suggesting that the approach is still far from being reliable in this scenario. In the close future, our aim is on the one side at improving detection, that might be combined with tracking. On the other side, once we have observed the problems originated in the multiclass classi cation of an unbalanced dataset, and apart from computing more dense grids, we should explore the combination with other techniques to leverage the classi cation stage.
Acknowledgments. Work partially funded by the Institute of Intelligent Systems and Numerical Applications in Engineering and the Computer Science Department at ULPGC.