The image-based plant identification challenge was focused on tree, herbs and ferns species identification based on different types of images. The aim of the task was to produce relevant species for each observation of a plant of the test dataset. We have elaborated a viewpoints combined classification method for this challenge. We have applied dense SIFT for feature detection and description; and Gaussian Mixture Model based Fisher vector was calculated to represent an image with high-level descriptor. The chosen classifier was the C-support vector classification algorithm with RBF (Radial Basis Function) kernel, and we have optimized two hyperparameters (C from C-SVC and γ from RBF kernel) by a grid search with two-dimensional grid. We have constructed a combined classifier using the weighted average of reliability values of classifier at each viewpoint. The results show that our combined method exceeds our best classifier among the list of classifiers constructed for different viewpoints.
Accurate knowledge of the identity, statistics and uses of plants is essential in the
agricultural development. Identifying plant species is usually a very difficult task, even for
professionals (such as farmers or wood exploiters) or for the botanists themselves.
Using image retrieval technologies is nowadays considered by botanists as a promising
direction in this problem, and in order to solve it a challenge is announced in the
LifeCLEF campaign [
The image-based plant identification task [
The aim of the task was to produce a list of relevant species for each observation of a plant of the test dataset, i.e. one or a set of several pictures related to a same event: one same person photographing several detailed views on various organs the same day with the same device with the same lightening conditions observing one same plant. So the task was observation-centered (not image-centered).
The task was based on the Pl@ntView dataset focusing plants on France (some plants observations came from neighbouring countries). It contains more than 60000 pictures belonging each to one of the 7 types of view reported into the meta-data, in an xml file (one per image) with explicit tags, like ObservationId, species names, date, etc. The task was evaluated as a plant species retrieval task based on multi-image plant observations queries. The goal was to retrieve the correct plant species among the top results of a ranked list of species returned by the evaluated system. An observation may contain 1 to 5 images depicting the same individual plant observed by the same person the same day. Each image of a query observation is associated with a single view type (entire plant, branch, leaf, fruit, flower, stem or leaf scan) and with contextual metadata (data, location, and author). Each participating group was allowed to submit up to 4 runs built from different methods.
User rating information (pictures with the average of the user ratings on image quality) was also available, but we have not used this additional information. 2 2.1
The first part of the classification is the accomplishment of representation of each image
based on the visual content. This consists of three steps: (i) feature detection, (ii) feature
description, (iii) image description as usual phases in computer vision.
Feature detection: Lots of different feature types can be detected in an image, e.g.
corners, edges, ridges, as “interesting” part of an image. Furthermore many possible
feature extraction methods are available for images, but we have chosen SIFT
(Scale-Invariant Feature Transform) algorithm [
Image description: The final step of the representation creating is the completion of
high level representation of each image. We have applied BoW (bag-of-words) model
[
For the classification task we have divided the labelled image set into three subsets:
training, validation and test set (the last one is used for preliminary testing). The
validation image set was used for calibration of the trained model during the validation
phase of the training procedure. To train the classifier (classification model) based on
training image set, a variation of SVM (Support Vector Machine) was used, the C-SVC
(C-support vector classification) [
After the training, the codebook was already available and only Fisher vector of each image should be computed. At the preliminary testing we have selected only 50 species (classes) for training and testing as well. RBF based kernel matrix was built from the Fisher vectors of the test and training images. Each C-SVC classifier was parametered with this matrix and the hyperparameters were the same as in the final classification models. Since the classifiers are assigned to species, the generated model for a classifier is responsible to separate the designated class from the other ones. Thus a classifier is able to provide a confidence value showing a certainty of the class in a given image. We have trained 7 classifiers for each viewpoint and we have evaluated as preliminary testing based on precision and computer run time. The results of the preliminary testing can be seen in Table 1.
The decision about the observation could be based on majority voting of image decisions, but we have used continuous information instead of discrete one. C-SVC classifier calculates continuous reliability value for each class at each image, and we have constructed a combined classifier using the weighted average of reliability values. Our precision 0.341 0.583 0.965 0.492 0.512 0.314 0.482 testing time (per image) [sec] 1.82 1.59 0.95 1.39 1.61 1.44 1.56 combined classifier has applied a formula (as can be seen in Equation 1.) for the aggregated reliability value that an image belongs to class c (species c). (1) NVP is the number of viewpoints, which equals to seven in this challenge wi is the weight parameter of viewpoint i rn(c) is reliability value for class c coming from C-SVC classifier Ni,p is the number of images in viewpoint i taken from the p-th plant observed Based on R(c) values the final decision is always the species that possesses the largest R(c) value. In the challenge the order of predicted species should have been submitted, and we have constructed the order based on R(c) values as well.
At the estimation of weight parameters we have taken the goodness of different viewpoint classifiers into the consideration. As can be seen in the results of the preliminary testing (at Table 1), the LeafScan has the best precision. So the LeafScan has got the largest weight parameter, and on an empirical way we have chosen the following weight parameters: LeafScan: 7.5, Leaf: 2.5, Flower: 1.5, Fruit: 1.5, Stem: 1.5, Branch: 1.5, Entire: 1.5. 3 3.1
In the official evaluation instead of precision (as used in our preliminary testing) a new evaluation metric was defined for measurement of goodness of the observation classification. This metric (S score) is defined as follows. U : number of users (who have at least one image in the test data) Pu : number of individual plants observed by the u-th user Nu,p : number of pictures taken from the p-th plant observed by the u-th user Su,p : score between 1 and 0 equals to the inverse of the rank of the correct species (for the p-th plant observed by the u-th user) Although the goal was to classify the observations containing more images, an additional metric was defined for the image classification as can be seen in Equation 3.
1 U 1 Pu 1 Nu,p Simage U u1 Pu p1 Nu, p n1 Su, p,n (3) U : number of users (who have at least one image in the test data) Pu : number of individual plants observed by the u-th user Nu,p : number of pictures taken from the p-th plant observed by the u-th user Su,p,n : score between 1 and 0 equals to the inverse of the rank of the correct species (for the n-th picture taken from the p-th plant observed by the u-th user) 3.2
Simage score can be calculated for each viewpoint, and these scores can be compared. Our final official results for each viewpoint and the observation can be seen in Table 2., and it can be shown that S score of observation exceeds the best S score of all viewpoints. Our final official observation results (BME TMIT) compared with other participants can be seen in Fig. 1. We have elaborated a viewpoints combined classification method for image-based plant identification task. We have applied dense SIFT for feature detection and description; and Gaussian Mixture Model based Fisher vector was calculated to represent an image with high-level descriptor. The chosen classifier was the C-support vector classification algorithm with RBF (Radial Basis Function) kernel, and we have optimized two hyperparameters (C from C-SVC and γ from RBF kernel) by a grid search with two-dimensional grid. We have constructed a combined classifier using the weighted average of reliability values of classifier at each viewpoint. The weight parameters of the combined classifier were based on our preliminary testing results. Our observation result of the combined method exceeds our best score of all viewpoints. At the official evaluation our solution has reached 0.255 score value.
The publication was supported by the TÁMOP-4.2.2.C-11/1/KONV-2012-0001 project. The project has been supported by the European Union, co-financed by the European Social Fund.