Fishery video data often require laborious visual analysis, therefore a video-based fish identification challenge is announced in the LifeCLEF campaign for automatic fish categorization and enumeration. We have elaborated a complex system to detect, classify and track objects (fishes) in underwater video by examining each image frame of it. For the detection process we used background subtraction and morphologic methods, and then our solution calculated bounding boxes based on object contours. We used Kalman filter to track the moving objects, but an additional matching method was required to pair the objects in consecutive time periods because of many fishes. We used Hungarian method for this matching problem. We categorized the detected fishes with C-SVC classifier, as an advanced SVM (Support Vector Machine) classifier. The classifier used high level descriptors, which are based on the extracted SURF vectors in each object. For optimization the C-SVC classifier we conducted a preliminary test, and we used the best parameters for teaching the classifier. We predicted the fish species in the official test video set, and our predictions were evaluated officially by NCS (Normalized Counting Score).
The analysis of video data usually requires very time-consuming and
expensive input by human observers, and this is true for underwater videos as well,
although the statistics of data collection would be very useful for exploratory
applications, in particular for fisheries and biological areas. This analytical
"bottleneck" greatly restricts the use of the powerful video technologies and
demands effective methods for automatic content analysis to enable
proactive provision of analytical information; and in order to solve this problem a
challenge is announced in the LifeCLEF [
Object classification and tracking are different tasks, but both of them based on object detection (fish detection) in images of videos. We have implemented fish detection in OpenCV in such way that the bounding boxes of detected fishes are stored as small images with corresponding information (actual timestamp, identifier, etc). The common subtask in both of the problems is mapping, i.e. interconnection of bounding box images and fish identifiers (these identifiers are generated for only distinguishable aim), because the results of the mapping can be used for classification and tracking as well. Thus the bounding box images are input for classification method, which estimates the species of fishes. The consecutive images with common fish identifiers can be classified into different species; therefore the final decision of classification in our solution is based on majority voting. 2.1
The one of the challenges in the fish detection was the observation of many
objects in an image. For object detection at first we have used background
subtraction [
After the detection in our solution Kalman filter [
Initialization: at the first frame, where any detection was, the Kalman filter was initialized; and for every detected object an identity number and a confidence value (CFV) were attached.
Prediction: In this step a prediction was made by Kalman filter on each detected object (using the calculated object centre) to forecast the future position of the investigated fish.
Correction: After prediction the new detections (in next frame of the video)
give the measurements (which are used in the comparison of the
measurements with predictions). These measurements were used for correct the
Kalman filter objects. In order to reach the best tracked-measured coupling
we applied the Hungarian method [
Let vi be a tracked object, where i = 0...k and let wj be a newly detected object where j = 0...l. Before our solution used the Hungarian method a Mkxl matrix was calculated, where M[i,j] denotes the Euclidean distance (measured in pixels in the image) of vi and wj objects. The rows and columns with higher elements than a given threshold were removed to prevent false matching. If a row corresponding to vi was removed from M, then we lowered the confidence value (CFV) of that particular tracking. On the other side removing wj means that we should track this object, i.e. probably this is a new object (a new Kalman filter has been created for this object), because this is far from all others.
At this point the Hungarian method were performed on M which resulted the optimal object tracking (v,w) pairs, i.e. minimal sum of distances. The Fig. 1. illustrates the mechanism of the applied Hungarian method, where the vi objects are representing by a set of black and grey vertices (T) and wj object by red and pink ones (N). The thick edges connect the matched (v,w) pairs, whereas grey and pink colour denote those nodes that have no pair. After the Hungarian method the matched object pairs will be the input of the correction phase of Kalman filter (black vertices are tracked objects and the red ones are the measurements). 2.3 2.3.1
The first part of the classification process is the representation of each image based on their visual content. This consists of three steps: (i) feature detection, (ii) feature description, (iii) image description as usual phases in computer vision.
Feature detection and description: Lots of different feature types can be
detected in an image, e.g. corners, edges, ridges, as “interesting” part of an
image. In our solution we have used Fast-Hessian Detector [
Image description: The next step of creating the representation is the
completion of high-level representation of each image. We have applied BoW
(bag-of-words) model [
Let ( ) ( ) ( ) be the initial histogram of the rth image, where denotes the size of the codebook (each element represent a codeword in H).
We performed 1NN (1-nearest neighbour) algorithm for each SURF descriptor to find the closest codeword (based on Euclidean distance), then the corresponding element of H was incremented by 1, where i cycles through the descriptors created for the rth image 2.3.2
For the classification task we have divided the labelled image set into two
subsets: training and test set. We used the first one to create the codebook,
and train the classifiers, and the latter for preliminary testing. The
histograms were created for each training image, then we performed a variation
of SVM (Support Vector Machine), the C-SVC (C-support vector classification)
[
For the maximization the goodness of classification part we have conducted a preliminary test, as validation phase of the learning process. The labelled image set was divided into train set (11221 images) and preliminary test set (11220 images). The training phase was executed by different parameters according to the number of codewords sizes (5000 and 10000), the number of dimensions of SURF vectors (64 and 128), and the number of SURF vectors in an image (80, 200, and 500). Besides the accuracy we have measured the speed of the algorithm as well, and the results can be seen in Table 1 and 2. After the preliminary testing at 5000 codewords the case of 10000 codewords was testing with only 200 and 500 SURF vectors, because 80 vectors has resulted in very poor accuracy. The best result of the preliminary testing is the 0.663 at case of 128 dimensions, 500 vectors and 5000 codewords. The run time of the training phase was long, but the largest part (92-94%) of this was the SVM teaching, which was important for good accuracy, while the other parts are fast (creating codewords: 3-6%, histogram calculation: less, than 1%); furthermore the test phase was also quick (10 minutes). The run time values were measured on a PC (with Intel Core i7-4770K processor, 16 GB RAM and SSD).
For the prediction of official test we have chosen the SVM model with the best parameters (128 dimensions, 500 vectors and 5000 codewords), and we have submitted 2 prediction files (as results of 2 runs). The used method of the runs was the same (as described above), only small difference was between the two submissions. The first run (BME TMIT RUN1) contains false detections, because of the glances (blinks) and the low level threshold in detection (higher level threshold could avoid these); while at the second run (BME TMIT RUN2) these detections were filtered out, and corresponding predictions in the crucial videos were deleted. 3 3.1
In the official evaluation the normalised counting score is measured (instead of accuracy as in our preliminary testing). The counting score (CS) is defined as can be seen in Equation (1), where d is the difference between the number of occurrences in the run (per species) and the number of occurrences in the ground truth (Ngt). The precision (Pr) is defined as Pr= TP/(TP+FP) with TP and FP being, respectively, the true positives and the false positives. The normalised counting score (NCS) is defined as NCS = CS x Pr. 3.2
Our final official results for each fish species can be seen in Table 3., where the occurrences of fish species in the test set and the NCS (Normalized Counting Score) results are presented.
The aggregated official results of different fish species can be seen in Table 4. (the last column is the product of previous ones), and we can conclude that the second run of our submissions was better. We have elaborated a complex system to detect and track objects (fish) in underwater video by examining each image frame of it. For the detection process we used background subtraction and morphologic methods, and then our solution calculated bounding boxes based on object contours. The first time we detect an object, we assign a Kalman filter to it, which is able to predict the possible location of that object in the next frame. We likely detect that particular object (the same fish) in the next frame also, but we should not apply a new Kalman filter. To deal with this, we used Hungarian method to pair the existing Kalman filters with the new “candidate” ones. Then we erased the candidate Kalman filters with matching pair, and kept the single ones. This way we were able to track the detected objects. We also categorized the detected fishes with C-SVC classifier; however this required representing the objects based on visual information. For this purpose, our system calculated SURF descriptors for each object, and then clustered them with K-means algorithm. Our solution built histograms for each fish based on the resulting cluster centres (according to BoW model), and these histograms were the input of the C-SVC. For optimization the C-SVC classifier we conducted a preliminary test, and we used the best parameters for teaching the classifier. We predicted the fish species in the official test video set based on our implemented model, and the number of occurrences of each fish species was enumerated. At the official evaluation (by the second submission) we reached 0.34 value of NCS (Normalized Counting Score).