This paper presents the participation of the LSIS/DYNI team for the ImageCLEF 2012 plant identi cation challenge. ImageCLEF's plant identi cation task provides a testbed for the system-oriented evaluation of tree species identi cation based on leaf images. The goal is to investigate image retrieval approaches in the context of crowd sourced images of leaves collected in a collaborative manner. The LSIS/DYNI team submitted three runs to this task and obtained the best evaluation scores (S = 0:32) for the "photograph" image category with an automatic method. Our approach is based on a modern computer vision framework involving local, highly discriminative visual descriptors, sophisticated visual-patches encoder and large-scale supervised classi cation. The paper presents the three procedures employed, and provides an analysis of the obtained evaluation results.
This paper presents the contribution of the LSIS/DYNI group for the plant identi cation task that was organized within ImageCLEF 20124 for the systemoriented evaluation of visual based plant identi cation. Similar to the ImageCLEF 2011 challenge, this second year pilot task was also precisely focused on tree species identi cation based on leaf images. This year, the challenge was organized as a classi cation task over 126 tree species with visual content being the main available information. Three types of image content were considered: leaf "scans", leaf photographs with a white uniform background (referred to as "scan-like" pictures) and unconstrained leaf "photographs" acquired on trees with natural background (see Fig. 1). The LSIS/DYNI team submitted three ? Granded by COGNILEGO ANR 2010-CORD-013 and PEPS RUPTURE Scale
Swarm Vision 4 http://www.imageclef.org/2012/plant runs, all of them based on local feature extraction and large-scale supervised classi cation. We obtained the best score for the "photographs" category with an automatic method (S = 0:32). A part of Pl@ntLeaves II dataset was provided as training data whereas the remaining part was used later as test data. The training subset was built by including the training AND test subsets of last year's Pl@ntLeaves I dataset, and by randomly selecting 2/3 of the individual plants for each NEW species (several pictures might belong to the same individual plant but cannot be split across training and test data).
{ The training data is comprised of 8422 images (4870 "scans", 1819 "scanlike" photos, 1733 natural photos) with full xml les associated to them (see previous section for few examples). A ground-truth le listing all images of each species was provided complementary. { The test data is comprised of 3150 images (1760 "scans", 907 "scan-like" photos, 483 natural photos) with purged xml les (i.e. without the taxon information that has to be predicted). 2.2
The goal of the task was to retrieve the correct species among the top k species of a ranked list of retrieved species for each test image. Each participant was allowed to submit up to 4 runs built from di erent methods. As many species as possible can be associated to each test image, sorted by decreasing con dence score. However, only the most con dent species were used in the primary evaluation metric described below. Providing an extended ranked list of species was encouraged in order to derive complementary statistics (e.g. recognition rate at other taxonomic levels, suggestion rate on top k species, etc.).
The primary metric used to evaluate the submitted runs was a normalized classi cation rate evaluated on the 1st species returned for each test image. Each test image is attributed with a score of 1 if the 1st returned species is correct and 0 if it is wrong. An average normalized score is then computed on all test images. A simple mean on all test images would indeed introduce some bias with regard to a real world identi cation system. Indeed, we recollect that the Pl@ntLeaves II dataset was built in a collaborative manner; So that few contributors might have provided much more pictures than many other contributors who provided few. Since we want to evaluate the ability of a system to provide correct answers for all users, we would rather measure the mean of the average classi cation rate per author. Furthermore, some authors sometimes provided many pictures of the same individual plant (to enrich training data with less e orts). Since we want to evaluate the ability of a system to provide the correct answer based on a single plant observation, we also decided to average the classi cation rate on each individual plant. Finally, our primary metric was de ned as the following average classi cation score S:
S = 1 XU 1 XPu 1 NXu;p su;p;n; U u=1 Pu p=1 Nu;p n=1 (1) where { U : number of users (who have at least one image in the test data) { P u : number of individual plants observed by the uth user { N u; p : number of pictures taken from the pth plant observed by the u-th user { su;p;n : classi cation score (1 or 0) for the nth picture taken from the pth plant observed by the uthh user Finally, to isolate and evaluate the impact of the image acquisition type ("scans", "scan-like", "photograph"), a normalized classi cation score S was computed for each type separately. Participants were therefore allowed to train distinct classi ers, use di erent training subsets or use distinct methods for each data type.
For all submitted runs, whatever the particular image type, we followed the same pipeline: i) feature extraction coupled with spatial pyramid (SP) for local analysis and a linear large-scale supervised classi cation. For our rst participation, we didn't performe any (supervised) segmentation leading to the extraction of more elaborate and speci c descriptors for leaf classi cation. 3.1
We de ne our SP matrix with L levels such as , [ry; rx; dy; dx; ]. is a matrix of size (L 5). For a level l 2 f0; : : : ; L 1g, the image I, with size (ny nx), is divided into potentially overlapping sub-windows Rl;v of size (hl wl). All these windows are sharing the same associated weight l. In our implementation, hl , bny:ry;lc and wl , bnx:rx;lc where ry;l, rx;l and l are the lth element of vectors ry, rx and respectively. Sub-window shifts in x y axis are de ned by integers y;l , bny:dy;lc and x;l , bnx:dx;lc where dy;l and dx;l are elements of dy and dx respectively. Overlapping can be performed if dy;l ry;l and/or dx;l rx;l. The total number of sub-windows is equal to V =
L 1 L 1 (1 X Vl = Xb l=0 l=0
ry;l) dy;l + 1c:b (1
rx;l) dx;l + 1c: (2) (3)
Let's assume available a training data set fxi; yigi=1, where xi 2 Rd is a
descripN
tor extracted from image Ii and yi 2 f1; : : : ; M g, where M = 126 is the number
of classes and N = 8422 is the number of training samples. As in [
N 1 wT w + C X max 1 2 i=1 yiwT xi; 0 2) ; where the parameter C controls the generalization error and is tuned on a speci c validation set. LIBLINEAR converges to a solution linearly in O(dN ) compared to O(dNs2v).
Moreover, in order to obtain an estimate of p(y = ljx), we performed an SVM regression given the output of the previous classi cation stage for each binary classi er. (4) (5) (6) (7)
Following [
In LPQ, Short Fourier Transforms (SFT) are computed over M M windows centered on z at four frequencies u1 = [a; 0]T , u2 = [0; a]T , u3 = [a; a]T and u4 = [a; a]T with a = M1 such that
F (u; z) = X f (z
y)e j2 uT y ; y2Nz where z 2 R
I. For each pixel, we compute the LPQ code as5 LP Q(z) =
3 X 22i1fRe(F (u;z)) 0g + i=0
3 X 22i+11 i=0 fIm(F (u;z)) 0g; where LP Q(z) 2 f0; : : : ; 255g. Local histograms of LPQ codes are retrieved by counting occurrences of each individual LPQ code j such as: xLP Q(j; R) =
X 1fLP Q(z)=jg; j = 0; : : : ; 255: z2R The local histogram vector is de ned by 5 1fxg = 1 if event x is true, 0 otherwise.
xLP Q(R) , [xLP Q(0; R); : : : ; xLP Q(255; R)] ; where xLP Q(R) is furthermore `2 normalized. The full vector x is obtained by concatenating previous normalized histograms for 4 di erent scales M 2 f3; 5; 7; 9g, = 11 11 11 11 11 (V = 1 + 21) and the 3 (R; G; B) color channels.
2 4 4 8 8 The total dimension of this vector is equal to d = 256:(1 + 21):4:3 = 67584.
Finally, we normalize each element of xi such that xi;l 2 [ 1; 1] ; l = 1; : : : ; d; i = 1; : : : ; N followed by `2 normalization on xi. The a posteriori probabilities associated with the MSCLPQ approach are denoted p1(y = ljx). 3.3
See sec. 3.2
Basically, the operator ILBP encodes the relationship between a central block
of (s s) pixels located in zc = [yx; xc]T with its 8 neighboring blocks [
ILBP (zc; s) = i=7 X 2i1fAi Acg + 281 i=0 7 iP=0 Ai 8Ac ; where 8zc 2 R I, ILBP (zc) 2 N29 .
The di erent areas fAig and Ac in eq.(8) can be computed e ciently using
the image integral technique [
A(y; x; s) = II(y + s; x + s) + II(y; x) (II(y; x + s) + II(y + s; x)): (10) As for MSCLPQ, e cient features are obtained by counting occurrences of the jth visual ILBP at scale s in a ROI R I: xILBP (R; j; s) =
1fILBP (zc;s)=jg; X where j = 0; : : : ; b 1 is the jth bin of the histogram and b = 512. Full histogram of ILBP, denoted zILBP is computed by: xILBP (R; s) , [xILBP (R; 0; s); : : : ; xILBP (R; b Finally, the full vector x is obtained by concatenating previous normalized his1 1 1 1 1 tograms for 4 di erent scales s 2 f1; 2; 3; 4g, = 1 1 1 1 1 (V = 1 + 21) 2 4 4 8 8 and the 3 (R; G; B) color channels. The total dimension of this vector is equal to d = 512:(1 + 21):4:3 = 135168. We also normalize each element of xi such that xi;l 2 [ 1; 1] ; l = 1; : : : ; d; i = 1; : : : ; N followed by `2 normalization on xi. The a posteriori probabilities associated with MSCILBP approach are denoted as p2(y = ljx).
Following the same framework as in [
Speci cally, F ILBP patches zILBP (Ok) of size (m m) centered on ROI's fOkg (possibly overlapping) are extracted (cf. eq. 7) for k = 0; : : : ; F 1 and 8s 2 S (see Fig. 4). For a faster computation for each scale s, the integral image II is rst computed from I.
For a complete dataset containing N images and 8s 2 S, we obtain a collection of P = T S patches Z , fzig, i = 1; : : : ; P , where T = N F . We de ne, the subset of patches zi at scale s by Z(s) Z with T elements. In order to obtain highly discriminative visual features, a common procedure consists of encoding each patch zi 2 Z(s) at scale s through an unsupervised trained dictionary D , [d1; : : : ; dK ] 2 Rb K , where K denotes the number of dictionary elements, and its corresponding weight vector ci 2 RK . In the BoF framework, the vector ci is assumed to have only one non-zero element:
T arg min X
D;C i=1 kzi
Dcik22 s:t: kcik`0 = 1; (12) where C , [c1; : : : ; cK ] and k k`0 de nes the pseudo zero-norm, where here only one element of ci is non-zero. In eq. (12), under these constraints, (D; C) can be optimized jointly by a Kmeans algorithm for example.
In the Sc approach, in order to i) reduce the quantization error and ii) to have
a more accurate representation of the patches, each vector xi is now expressed
as a linear combination of a few vectors of the dictionary D and not only by a
single one. Imposing the exact number of non-zero elements in ci (sparsity level)
involves a non-convex optimization [
T arg min X
D;C i=1 kzi
Dcik22 +
kcik`1
s:t:
kcik`1 = 1;
(13)
where the sparsity in controlled by the parameter . The last equation is not
jointly convex in (D; C) and a common procedure consists of optimizing
alternatively D given C by a block coordinate descent and then C given D by a
LASSO procedure [
For an image I and given a trained dictionary Db (s) for a type of code at
scale s, F sparse vectors fck(s)g are computed by a LASSO algorithm. The
nal e cient descriptor x(s) , x0(s); : : : ; xK 1(s) 2 RK is obtained by the
following max-pooling procedure [
L 1 The spatial pyramid has V = P Vl ROIs fRl;vg with l = 0; : : : ; L 1, l=0 v = 0; : : : ; Vl 1 (see Fig. 5 for an example). The quantity zlj;v(s) for each ROI Rl;v is computed by: xlj;v(s) ,
max kjOk2Rl;v (jcjk(s)j); j = 0; : : : ; K 1: We reinforce our model by an important normalization step that improves con(14) (15) siderably accuracy and consists of the `2 normalization of all vectors fxl;v(s)g, v = 0; : : : ; Vl 1; s 2 S, i.e. belonging to the same pyramidal layer l. This step is also very important and often hidden in the existing literature.
The nal descriptor x( ) will be de ned by the weighted concatenation of all the xl;v(s) vectors, i.e. x( ) , f lxl;v(s)g, l = 0; : : : ; L 1; v = 0; : : : ; Vl 1 and 8s 2 S. The total size of the feature vector x( ) is d = K:V:S, where typically in our simulations, we xed K = 2048, V = 22 and S = 4. A nal `2 clamped normalization step is performed on the full vector x( ). In our experiment, we extracted F = 35 35 patches per scale and per image with m = 26. 2000 patches per class for each scale have been randomly selected to train dictionary ( = 0:2). The a posteriori probabilities associated with MSILBP+ScSPM approach are denoted p3(y = ljx).
To obtain a nal decision, we simple performed an average of all pf (y = ljx) a posteriori probabilities, i.e.
3 p(y = ljx) = 1 X pf (y = ljx): 3
f=1 3.4
The three rst stages are identical as in LSIS DYNI run 2.
As for MSILBP parches, we extracted F = 35 35 SIFT patches (m = 16) per image and for each of the 4 scales ( = f0:5; 0:65; 0:8; 1:0g). 2000 patches per class for each scale have been randomly selected to train dictionary ( = 0:2, K = 2048). The a posteriori probabilities associated with SIFT+ScSPM approach are denoted p4(y = ljx).
We also performed an average of all pf (y = ljx) a posteriori probabilities 4 p(y = ljx) = 1 X pf (y = ljx): 4 f=1 4
Fig. 6 presents the summarized results for the "scans" category. Without any segmentation and/or speci c pre-processing, we obtained a score S = 0:41 with LSIS DYNI run 3, i.e. the 6th best score for all submitted runs (29 in total), relatively close to the top-4 (S = 0:43). Higher scores can probably be obtained with the use of color MBILBP+ScSPM and color SIFT+ScSPM features.
In Fig. 7 we summarize the results for the "scan-like" category. We obtained a score S = 0:42 with LSIS DYNI run 3, i.e. the 7th best score for all submitted runs (29 in total). In this case, with an unsupervised detector to "home-in" leafs more precisely, we could also improve results. Ideally, as for all runs above S = 0:42, a prior segmentation is known to help considerably results for "scans" and "scan-like" categories. Finally, Fig. 8 provides the summarized results for the "photographs" category. We obtained a score S = 0:32 with LSIS DYNI run 3, i.e. the 1th best score for automatic method and for all submitted runs (29 in total). Our 3 runs obtained the best top-3 of all submitted runs. (16) (17) For our rst participation to ImageCLEF plants identi cation 2012 challenge, we demonstrated that for "photographs" category, our framework o ers best performances for automatic method. This category is considered the most challenging due to "real" in-situ conditions and shows that computer vision approaches for image catagorization/ ne-grained visual categorization are well adapted for this challenge. Several improvements can be obtained, for example with some better encoding schemes (Fisher vectors) and/or pooling technics.