In this paper, we present the system and learning strategies that were applied by the IBM Research team to the plant identi cation task of LifeCLEF 2014. Plant identi cation is one of the most popular ne-grained categorization tasks. To ensure high classi cation accuracy, we have utilised strong visual features together with fusion of robust machine learning techniques. Our proposed system involves automatic delineation of the region of interest (e.g. plant's leaf, ower, etc.) in the given image, followed by extracting multiple complementary low level features. The features have been then encoded into the sophisticated Fisher Vector representation which enables accurate classi cation with linear classi ers. We have also applied the recent development of deep learning. More importantly our system combines multiple source of information, i.e. integrates organ annotation with image data, and adopts fusion of classi ers which has led to great results. The extensive experiments demonstrate the e ectiveness of the proposed system, where three (out of four) of our submissions outperforms all submissions by other teams, therefore the team achieves the rst place in LifeCLEF 2014 Plant task.
The ne-grained classi cation, i.e. classi cation among categories which are both
visually and semantically very similar, is a very di cult task. It is even
challenging for humans without careful training, and is critical for establishing a more
detailed understanding of the visual world [
This paper presents the system and learning strategies that were applied by the IBM Research team to the plant identi cation task of LifeCLEF 2014. The rest of the paper is organised as follows: Section 2 describes the main components of the system, involving segmentation, feature extraction and learning, and methodologies applied at each module. Section 3 discusses the details of experiments and obtained results on validation and test sets. Section 4 concludes the paper. 2
In this section we rst brie y describe our submission to the LifeCLEF 2014 competition. Then various components of the system, including segmentation (delineation of the region of interest), feature coding and deep convolutional nerual network are explained. 2.1
We have submitted four di erent runs to LifeCLEF [
In most images, using a region of interest (ROI) which encloses the main object is su cient to determine their class label. In fact, segmentation of the main objects and extracting the ROI often results in removing the irrelevant background which could introduce noise to a supervised classier. In categories of ower, fruit, leaf, leafScan and stem, we apply di erent segmentation methods to remove the background. For the two categories of wholeTree and branch, however, we have observed that the whole image contains useful information. Therefore, we choose not to extract any ROI from these views, instead the original image is used in the next modules of the system, i.e. feature extraction and learning.
For the ower and fruit categories, we apply a similar segmentation method to
extract the ROI that contains ower or a fruit, as follows: assuming that a ower
or fruit is usually \more red" compared to leaves, we locate the regions of which
their red channel is larger than the green channel. Each color image is converted
to a gray-value image, and the localized \red" regions are then employed as
initial masks in segmenting the ower or fruit in the gray-value image using a
active contour method [
In images of the leaf category, we observed that a leaf is usually located in the
center area of the picture and there are typically some margin between the leaf
boundary and the picture border. We therefore extract the object that is located
in the center of the picture. In pre-processing, we convert the color picture to
a gray-value image and apply a Gaussian lter on the image with = 3. We
then extract an rectangular ROI in the image as an initial ROI that contain
the leaf. The ROI is de ned by its left top and bottom right coordination in
the image. Let (r; l) is the number of rows and columns in the image, the left
top coordination is ( 6r ; 6l ) and the right bottom coordination is ( 56r ; 56l ). We then
apply active contour on the gray-value image using the rectangular ROI as the
initial mask. The boundary box of the segmented leaf is the nal ROI and some
examples are shown in Figure 4.
In the stem category, the stem in usually located in the center of the image.
For segmenting the stem, we convert the color image to a gray-value image
and creating a central mask on the image by cropping %25 from left and %25
from right. Active contour is then applied on the gray-value image using the
mask. The bounding box of the resulting area is the ROI for the stem image.
This method can e ectively remove most of the background for vertical stems
without branches. Some examples are shown in Figure 5.
Feature coding is the generalization of the popular BoW model [
Suppose we have a probability density function u (x) which models a generative process in feature space. Let X = fx1; ; xN g be the set of N local features And the Fisher vector (FV) for X is denoted as (X) = fGX1 ; GX1 ; where G k and G k is de ned as: ; GXK ; GXK g ik =
kuk(xi) PK
k=1 kuk(xi)
N GXk = X
i=1 GXk = XN 1 i=1 N p
1 N p k 2 k ik xi ik[
k (xi k ; 2 k k)2 extracted from an image. Then the image can be described by the gradient vector of log likelihood with respect to the model parameters :
GX = 1
N r log u (X):
A natural kernel on these gradients is K(X; Y ) = GX0 F 1GY where F is the Fisher information matrix of u : F = Ex u [r log u (x)r log u (x)0]. As F is symmetric and positive de nite, it can be decomposed as F = L0 L , and the kernel K(X; Y ) can be expressed as a dot-product between normalized vectors GX = L GX called Fisher vectors. We stress that learning a kernel classi er using the Fisher kernel is equivalent to learning a linear classier on the Fisher vectors GX . As been recognized widely, linear classi ers o er signi cant advantages in terms of e ciency both for training and testing.
Fisher Vector encoding utilizes a Gaussian mixture model (GMM), u (x) = PK
k=1 kuk(x) trained on local features of a large image set using Maximum Likelihood (ML) estimation. The parameters of the trained GMM are denoted as = f k; k; k; k = 1; ; Kg, where f ; ; g are the prior probability, mean vector and diagonal covariance matrix of the Gaussian mixture respectively. This GMM is used for description of low level features. Then for a set of low level features X = fx1; ; xN g extracted from an image y, the soft assignments of the descriptor xi to the kth Gaussian component ik is computed by: (1) (2) (3) (4) Where k is the square root of the diagonal values of k. The FV has several good properties: (a) Fisher Vector encoding is not limited to computing visual word occurrences. It also encodes the distribution information of the feature points, which will perform more stable when encoding a single feature point. (b) it can naturally separate the image speci c information from the noisy local features. (c) we can use a linear model for this representation.
as the number of Gaussians increases, Fisher vectors become sparser, and the distribution of features in a given dimension becomes more peaky around zero. This issue is addressed by a combination of power normalization and l2 normalization for each Fisher vector descriptor. Suppose z is one dimension of the sher vector , the power normalization is de ned as f (z) = sign(z)jzj where 0 1 is a parameter of the normalization and we choose experiments. Subsequently, the Fisher vectors are l2 normalized. 2.4
We also applied a deep convolutional nerual network (CNN) for the plant
identi cation task. We use the training data from the plant task only, which is
relative small scale data for this deep model. We also tried a pretrained CNN
using ImageNet [
Our CNN has around 60 million parameters. it consists of ve
convolutional layers, some of which are followed by max-pooling layers, and three
fullyconnected layers with a nal softmax layer. To make training faster, we have used
non-saturating neurons and a very e cient GPU implementation of the
convolution operation. To reduce over tting in the fully-connected layers we employed a
recently-developed regularization method called dropout that proved to be very
e ective. We also use the data augmentation as in [
Until now we have presented the sub models of the system, we then fuse the k results at the late stage. Each sub model k provides a con dence score si;j for each observationID/imagei and each category j. We optimize to get the nal con dent score as the weighted sum: Si;j = Pk wk sik;j . The optimization is performed on the validation set. Each time, we select on sub model with best accuracy and tune the weight to get the best fused accuracy. The same set of the weight parameters have been used to get the con dence score on the test set. 3
In this section we discusses the details of experiments and obtained results on validation and test sets. 3.1
Dataset The PlantCLEF dataset focuses on 500 herb, tree and fern species centered on France (some plants observations are from neighboring countries). It contains more than 60000 pictures belonging to one of the seven types of view reported into the meta-data, in a xml le (one per image) with explicit tags. A part of the dataset has been provided as training data and the remaining part will be used later as test data. Test observation will be chosen by randomly sampling 1/3 of the observations of each species.
The training data nally results in 47815 images, including 1987 of \Branch",
6356 photographs of \Entire", 13164 of \Flower", 3753 \Fruit", 7754 of \Leaf",
3466 \Stem" and 11335 scans and scan-like pictures of \leaf". The test data
contains 8163 plant-observation-queries. These queries are based on 13146 images;
731 of \Branch", 2983 photographs of \Entire", 4559 of \Flower", 1184 \Fruit",
2058 of \Leaf", 935 \Stem" and 696 scans and scan-like pictures of \leaf".
Validation Set In order to verify the e ective of each components in our
system, we split the \training data" provided by PlantCLEF into two parts: a train
set and a validation set. The validation set is roughly 1/5 of the total training
data. We split the training data according to the observation id which is critical
since the nal evaluation is based on observation id. In the following section, we
will report the results on both the validation set and the testing set.
Implementation details As previously mentioned, in the feature coding
approach, we extract two types of local features: dense SIFT feature and Color
moment feature. The dimension of each local feature has been reduced to 64
using PCA [
For deep convolutional neural network, we follow the pipeline of Alex [
We use open source libraries, e.g. SIFT from VL-feat [
For proper assessment of each components of our system, we perform diagnosis evaluation on the validation set. The evaluation results are shown in Table 1 based on two metrics as proposed by PlantCLEF organiser, i.e. the Accuracy w.r.t. image (Acc image) and Accuracy w.r.t. observation id (Acc observ).
Based on result shown in Table 1, we can observe the following: 1. The feature coding pipeline (by combing two type of features, i.e. SIFT and color moment) achieves stable results. As reported in the literature, the Fisher Kernel encoding performs best among many of the encoding methods in visual recognition. 2. Segmentation (ROI extraction) improves the pipeline of feature coding and increase the classi cation accuracy. 3. The CNN method results in lower classi cation accuracy, compared to feature coding. We believe the e ect is the due to limited number of training data. Deep learning has been demonstrated great success when using large scale dataset. The PlantCLEF dataset is a middle scale dataset and the deep model can be easily over tted. 3.3
In this subsection, we present our nal results submitted to LifeCLEF2014 plant identi cation task. Results of each run is shown in Table 2, based on two metrics: the Accuracy w.r.t. image (Acc image) and Accuracy w.r.t. observation id (Acc observ). { RUN1 is the result of using deep convolutional nerual network. It obtains reasonable result but inferior to other runs. The main problem is that the deep model is very easy to over tting on the training set with so many parameters. { RUN2 is the raw result with feature coding pipeline which gives quite good result. It demonstrates that the feature coding pipeline is very mature and easily adaptable to many tasks in visual recognition. { RUN3 is the result from combing RUN2 and RUN1. It shows that the CNN framework is complementary to the traditional feature coding framework. { RUN4 is the result from combing RUN3 and feature coding pipeline applied on segmented images. It shows the correct delineation of ROI improves the classi cation results.
Further evaluation results: 1. Overall performance: Our submissions achieve the top three results when comparing with other teams as shown in Figure 7 in terms of both metrics, i.e. the Accuracy w.r.t. image (Acc image) and Accuracy w.r.t. observation id (Acc observ). Our classi cation results obtains 15-20% improvement over other teams. 2. Results for each organ (view): Figure 8 shows the result for each organ (view). Our classi cation results again leads other teams in most of cases. It is worth noting that our system's performance on LeafScan and Flower is better than 50%. Considering such ne-grained categroization task, we believe this result has practical value for real system. 3. Observation-based evaluation: We observe a notable improvement from image-based evaluation to observation-based evaluation on the validation set, which shows the ability of our system in using multiple source of image data and perform a more accurate classi cation. Unfortunately, the trend is not that obvious on the test set. This is caused by dissimilarity between the training and testing set in terms of average number of images per observation id, i.e. there is around 4.5 images per observation id in the training set, while we only have around 1.6 images per observation id in the test set. In this paper, we described the system and learning methodology applied by the IBM Australia Research team to the plant identi cation task of LifeCLEF 2014. We utilized the advanced feature coding method with automatic ROI extraction. We also applied the recent development of deep learning and achieved great result on the validation set. The most important contribution of this work is in e ective fusion of various learning schemes and proper use of multiple source of information (annotation data and image data). The extensive experiments demonstrated the e ectiveness of the proposed system and the nal submitted run achieved the rst place in LifeCLEF 2014 Plant task.