We describe our system at the ImageCLEF 2013 plant identification task. Plant identification is extremely challenging because target classes are often visually quite similar. To distinguish them, we need to extract highly informative visual features. We believe that the key to achieving this is to enhance the discriminative power of local descriptors. We employed multiple local features with our polynomial embedding technique to boost the performance. Further, they were encoded into the sophisticated Fisher Vector representation which enables accurate classification with linear classifiers. Our system achieved promising performance, and got the first place in NaturalBackground and the third place in SheetAsBackground tasks, respectively.
In this report, we describe our contributions submitted to the ImageCLEF 2013
plant identification task [
To distinguish very similar categories, we need to extract highly informative
visual features. We believe that the key to achieving this is to enhance the
discriminative power of local descriptors. Our method can efficiently improve
the discriminative performance of arbitrary local descriptors for bag-of-words
[
We implemented our method with standard object recognition pipelines using
the state-of-the-art Fisher Vector coding [
Descriptor 1 (e.g. SIFT) Descriptor 2 Descriptor K polynomial vectors p(0x,y) = upperV ec v(x,y)v(Tx,y) , 1 We use at most the second-order polynomials in this paper considering the computational cost, although our framework supports higher-order ones. where, upperV ec() is the flattened vector of the components in the upper triangular part of a symmetric matrix.
Moreover, we can efficiently exploit local spatial information by taking the polynomials between neighboring descriptors. When considering two neighbors (left side and right side), (2) (3) (4) (5) (6) v(x,y)
p(2x,y) = upperV ec v(x,y)v(Tx,y) , V ec v(x,y)v(Tx−δ,y)
V ec v(x,y)v(Tx+δ,y) where, V ec() is the flattened vector of the components of a matrix, and δ is an offset parameter for defining neighbors.
In this work, we considered at most two neighbors, although our previous
work suggests that using more neighbors could have resulted in better
performance [
We apply canonical correlation analysis (CCA) [
CCA finds the linear projections s = AT p and t = BT l that maximize the correlation between the projected vectors s and t. We randomly sample {p(x,y), l(x,y)} pairs from the entire training dataset, and let C = Cpp Cpl Clp Cll denote their covariance matrices. Namely,
Cpp = Cll = Cpl = 1 X (p − p¯)(p − p¯)T , N 1 X (l − ¯l)(l − ¯l)T , N 1 X (p − p¯)(l − ¯l)T ,
N
Clp = CpTl, 2 The dimension of the vector is the number of categories. If the image belongs to the category wi, the i-th element is one; otherwise, it is zero. 3 Obviously, this is a rather rough approach, since not all local features within an image are actually related to the image-level labels. Nevertheless, we note that this assumption is justified somewhat for FGVC problems, since objects are often closely targeted by users. where, N is the number of sampled pairs, and p¯ and ¯l are their means. The solution of CCA can be obtained by solving the following eigenvalue problem.
CplCl−l1ClpA = CppAΛ2 (AT CppA = Im), ClpCp−p1CplB = CllBΛ2 (BT CllB = Im), (7) (8) where Λ is the diagonal matrix of the first m canonical correlations, and m is the dimension of the canonical elements. The parameter m corresponds to the dimension of the embedded descriptor, and needs to be tuned manually. One problem is that m can be at most the dimension of the label vector because of the rank problem. If we need more features, we can project p into the orthogonal subspace and iteratively apply CCA to further extract discriminative components.
Using the projections obtained by CCA, we get a compact vector s that embeds a high-dimensional augmented vector, which we call the latent descriptor. s = AT p. (9) Once the latent descriptor is computed, it can be used in the exact same manner as widely-used raw descriptors such as SIFT.
We encode the latent descriptors into a global feature vector using the Fisher
Vector framework [
Plant Identification Task
The goal of the challenge is to identify 250 species of plants from their photos.
There are two main subtasks: SheetAsBackground and NaturalBackground (Fig.
2). While the objective of the former is to recognize leaves spread on white
background, the latter targets more organs and generic background. Therefore,
NaturalBackground task has more generic nature like typical FGVC problems
and thought to be challenging. The performance is evaluated in terms of the rank
of the correct species in the list of retrieved species. The score is normalized by
the numbers of content owners, individual plants, and pictures taken from the
same plant. For more information, refer to [
We used several standard local descriptors in our system, such as SIFT [
D1
D2 SheetAsBackground Scan Scan-like
D3 NaturalBackground
Fruit Leaf
Flower
Stem
Enre D4
D5
D6
D7
D8
angle bins). All these local features are extracted in a dense sampling approach
without rotation invariance [
We used the feature extraction software provided by the authors of [
We submitted three runs under different configurations. Each run consists of some classifiers independently trained for a certain domain as in Fig. 2. Information for identifying the domain of testing samples can be drawn from the corresponding xml files 4.
– Run 1 (D1): A multi-class classifier is trained on the whole dataset of 250 classes without distinguishing SheetAsBackground and NaturalBackground categories (and its sub-categories). – Run 2 (D2, D3): Two classifiers are trained independently for SheetAsBackground and NaturalBackground categories, respectively. We do not distinguish their sub-categories. – Run 3 (D2, D4-8): Classifiers are trained independently for SheetAsBackground and NaturalBackground categories. The former is the same one used in Run 2 (D2). For the latter, we train classifiers independently for each of five sub-categories (D4-8).
To tune our system, we take roughly 10% of the individual plants in the provided training dataset for validation. Table 1 summarizes the number of samples for each domain. For simplicity, we evaluate the classification accuracy on the validation dataset without distinguishing individual plants and owners. After optimizing the parameters, classifiers are trained again on the original training dataset and applied to testing data. 3.3
For various domain and feature combinations, we tuned our system and validated their effectiveness. Table 2 shows the results. “PCA64” denotes the Fisher Vector using 64-dimensional descriptors compressed via PCA 5. This is a typical implementation of Fisher Vector coding and serves as the baseline. For most of the trials, PE reasonably improves the performance of the original descriptors. Also, the relative improvement seems more significant in NaturalBackground domains.
Based on the results, we selected the features for final submissions. In SheetAsBackground (D2) domain, we used only gray SIFT because we found color descriptors were not effective. As for NaturalBackground domain, we chose color SIFTs + SSIM combination considering its good performance in all subcategories. 3.4
Based on the validation results, we submitted three runs. Figure 3 summarizes the performance of submitted runs from all participants. Not surprisingly, Run 4 Note that this is not interpreted as a manual intervention in this challenge. 5 We use the raw SSIM descriptor without applying PCA.
SheetAsBackground 0 ickSaaa1nnbnuRO liIttraae2PnnnuRN liIttraae3PnnunRN liIttraae1PnunnRN liIttraae4PnunnRN lykaT1bunooRUN lykaT3bunooRUN lykaT2bunooRUN iirsLSee2nuRRV iirsLSee1nuRRV ica3nuRM IS2nuRBD IS4unRBD IL1PunRA I4unCRAU IS3unRBD IS1unRBD Sg2PPunRRA SS3PunCRGU I1unCRAU I2unCRAU I3unCRAU Sg1PPunRRA Sg3PPunRRA SS1PunCRGU SS2PunCRGU IS31unR IS32unR SS4PunCRGU ica2unRM ica1nuRM itcce1hnuoRVm itcce2hnuoRVm 0 lykaT3bnuooRUN liIttraae2PnnnuRN lykaT2bnuooRUN liIttraae1PnunnRN lykaT1bunooRUN liIttraae3PnunnRN liIttraae4PnunnRN ickSaaa1nnubnRO IS2unRBD IS3unRBD IS4unRBD I4unCRAU IS1unRBD I1unCRAU I2unCRAU iirsLSee2unRRV iirsLSee1unRRV I3unCRAU itcce1hunoRVm itcce2uhnoRVm IL1PunRA ica2unRM ica3unRM SS3PunCRGU IS31unR IS32unR SS1PunCRGU SS2PunCRGU ica1unRM SS4PunCRGU Sg2PPunRRA Sg1PPnuRRA Sg3PPnuRRA 3 achieved the best in three runs on NaturalBackground task since it consists of multiple classifiers tuned for each sub-category. However, interestingly, the difference in performance is not large compared to the result of Run 2. Moreover, D1 classifier got better performance than D2 classifier on SheetAsBackground task. We noticed that some plants share similar appearance in different subcategories (e.g. ’Flower’ and ’Entire’). In such a case, universal classifier might result in better performance than specific ones for each sub-category.
During this challenge, we bet on implementing powerful image features, rather than classification algorithms and systems. We employed multiple local features with our polynomial embedding technique to boost the performance. They are further encoded into the powerful Fisher Vector representation. Our system achieved promising performance, and got the first place in NaturalBackground and the third place in SheetAsBackground tasks. On the other hand, our learning and classification algorithms are very simple and could be improved. Although some individual plants have multiple images of different organs, our system treats them independently and loses co-occurrence information. It would be interesting to develop classification methods utilizing them in an integrated manner.
This work is partially supported by the Nakajima Foundation.