This paper describes the participation of Inria within the Pl@ntNet project1 at ImageCLEF2013 plant identi cation task. For the SheetAsBackground category (scans or photographs of leaves with a uniform background), the submitted runs used a multiscale triangle-based approaches, either alone or combined with other shape-based descriptors. For the NaturalBackground category (unconstrained photographs of leaves, owers fruits, stems,...), the four submitted runs used local features extracted using di erent geometric constraints. Three of them were based on large scale matching of individual local feature, while the last one used a sher vector representation. Metadata like the owering date or/and plant identi er were successfully combined to the visual content. Overall the proposed methods performed very well for all categories and sub-categories.
The plant identi cation task of ImageCLEF2013 [
Inria, within the Pl@ntNet project, submitted four runs, in both image categories. But the methods used di erent algorithms: for the SheetAsBackground queries, the submitted runs were based on shape boundary features (see Section 2), while large scale matching approaches or sher vectors with SVM classiers were used for the NaturalBackground category (see Section 3). Results are discussed in Section 4, followed by conclusions and perspectives in Section 5. 1 http://www.plantnet-project.org/
Methods used for the SheetAsBackground category
We used the triangular representations presented in [
Two declinations of multiscale triangle-based descriptors were used:
- TOA: a multiscale triangular shape descriptor, where the triangles are
described by two successive oriented angles [
We experimented also combinations of these representations with two
complementary descriptions:
- DFH - Shapes: a 2D Directional Fragment Histogram (DFH) that computes
directions and relative lengths on a succession of elementary fragments on the
contour, associated with a set of geometric metrics of the shape (Shapes) [
All these descriptors require a preliminary leaf boundary extraction, performed
here by using the Otsu thresholding method [
I1 I2
Ik Same PlantID
RP RP RP {
L1 L2 Lk n o i s u F
Final list of images
Fig. 1. Multiple image queries. I1; :::Ik are leaf images associated to the same P lantID
tag and RP is the retrieval process involving either TOA, TSLA, SC2 or DFH - Shapes.
Multiple image queries. For each run, we used the fact that images in the
test dataset are associated with plant observations in order to perform multiple
image queries for leaf images having the same P lantID value (cf. Figure 1).
More precisely, for each descriptor:
- We rst grouped all the images I1; :::Ik coming from the same plant observation
using the P lantID in metadata.
- Then, we computed the retrieved images similarity ranking lists L1; :::Lk
corresponding to the query images I1; :::Ik.
- Finally, the 100 rst image results were kept for each list and were merged into
a nal list L using a late fusion with the Leave Out algorithm (LO) [
Methods used for the NaturalBackground category For the NaturalBackground category, we explored the following directions: the fact that a plant view organ is generally centred (Section 3.1 and 3.4), the fact that there are sometimes several images from the same plant observation, the fact that species have not the same owering periods (Section 3.2), and nally whether the automatic segmentation improves performances (Section 3.3).
The 4 runs are based on the same local features:
Interest points detection. Harris corners were used at four distinct resolutions
with multiple orientations ([
Local features and weighted descriptor fusion ! RUN1
After a series of tests with the training data, we concluded that not all type
of local features should be used for all views. As might be expected, color is
dominant for the ower but may lead to confusion for leaves, while texture plays
an important role for stem and fruits. The most discriminant features retained
for the methods are ri-LBP, SURF, Fourier, wght-RGB, histo-HSV for Flowers,
Fruit, Stem and Entire, while only ri-LBP, SURF, Fourier, EOH are computed
for Leaves. For each combination of one view and one type of feature, local
(a) Harris points detected with rhomboid-based mask and with grid-based weighting
(b) Points ltered using foreground/background segmentation
features are computed, compressed and indexed using RMMH method [
For a query image Q, according to its view, the basic algorithm applied is:
(i) to retrieve lists of similar images for each type of feature, (ii) transform each
list of images into a probability distribution by species, (iii) merge probabilities
for all types of features.
(i) Image retrieval: A response list R is composed of similar images Ii ranked
by a score Si. The score Si is the number of matches between an image Ii and
the query image Q, more precisely the number of instances of image Ii retrieved
through the nearest neighbors lists of each local feature of the query image Q: the
description of a local feature is compressed with RMMH and its approximate
30-nearest neighbors are searched by probing multiple neighboring buckets in
the consulted hash table (according to the a posteriori multi-probe algorithm
described in [
Instead of using the top-K images for decision (as in the last year's runs
[
Query image from class Cercis siliquastrum:
Pistacia Robinia terebinthus pseudoacacia
1 2 (3 images / obser.) s ()c geam ()d SIi 21 ()e mS 21 f() cS1-im., 1-obser.,
ignore ) ()g k(PC 0 centage of the average size of the training class, while K00 is a percentage of the number of training classes. The response R is scanned from the most to the least similar image, the counter of the number of classes with at least K0 images is incremented accordingly, and when we nd K00 such classes, we stop the scanning of the response. The adaptive criterion is imporant in order to avoid the noise in the nal response: (a) had we searched for a xed (and not dependent on the training data) number of classes with at least K0 plant observations, we would often output classes that are not relevant to ll-in the pre-de ned requirement, or (b) had we output a xed number of most similar classes, we would give more weight to the classes with small number of plant observations and would not reward the fact that some classes are well represented in terms of plant observations. Finally, our K00 per class resulted in the K per image of 85, ranging from 15 to 205, for the most to the least di erent-plant-observation-containing response; organ-wise, the values of K ranged from 66 for Stem to 101 for Fruit.
Moreover, to eliminate the redundant images, we consider only two most similar images per one plant observation: the score per plant observation Sm is a simple average of the image scores Si. The score for a class is a simple sum of the scores of its plant observations Sm: this step actually favorizes the well-represented classes, and penalizes the classes with small number of plant observations. Finally, the classes with only one image comming from one plant observation are removed from the list as outliers.
Figure 3 shows an example of the probability computation from a ranked list of 8 images: the rst returned image has high score Si, however, it is the only image of the only plant observation for it's class and this image is ignored in Organ Wunif SURF Fourier EOH ri-LBP wght-RGB histo-HSV Flower 0.2 0.169 0.171 - 0.209 0.230 0.221 Entire 0.2 0.192 0.175 - 0.196 0.221 0.216 Fruit 0.2 0.188 0.182 - 0.186 0.223 0.221 Leaf 0.25 0.271 0.233 0.237 0.259 - Stem 0.2 0.197 0.216 - 0.210 0.192 0.184 the nal classes list; next three images belong to the same plant observation, so we will keep only the rst two scores Si; nally, the last four images belong to the same class as the query image, however, unlike for the previous group, they are all from di erent plant observations. Thus, we will use all scores in the calculation of the class probability. (iii) Probability fusion: At this step we have several species probability distributions, one for each kind of feature and we use a weighted fusion in order to obtain a nal probability distribution. Let us de ne F as a set of local features and P (Cfk) as probability of class Ck for feature f 2 F . In order to re ect the discriminating power of each local feature, we de ne the nal probability: P (Ck) = P w(f ) P (Cfk), where w(f ) = max P (Cfk)= P max P (Cfk) f2F 8k f2F 8k
Table 1 shows the average weights attributed to each local feature for all test images, displayed per organ. The last two columns show the di erence between the minimal and maximal average weight per feature and the average di erence between the minimal and maximal weight per image. Color-based features have higher weights for Flower, Entire and Fruit, than gray-based ones. For graybased features, ri-LBP contributes more than SURF and Fourier for Flower, while it is Fourier which contributes the most for Stem. For Leaf, SURF and riLBP contributed more than Fourier and EOH. We can note that all the average weights are actually rather close to the Wunif , and that all the chosen local features play an important role in the overall decision. The values avg(W ) Wunif show that on the image level, there was always a local feature that had the weight signi cantly lower than the others, thus the in uence of that, presumably, confusing response was minimized in the overall response. 3.2
Multi-image queries and ltering by owering period ! RUN2 This run is an extension of the RU N 1, with two modi cations making use of metadata: (i) the responses from all images belonging to the same plant observation are merged and (ii) only for the Flower images, owering dates for ltering irrelevant species in responses.
Multi-image queries: As in the training dataset, images from the test dataset are sometimes related to a same plant observation which was explicitly mens e g ) a (a im s ' . v i d n I
Entire 1 image 1st in Stem 2nd in Stem 1st in Fruit 3rd in Stem 4th in Entire 1st in Entire 1st in Leaf tioned in metadata with the tag I ndividualP lantI d. Thus, we have some query images showing the same plant on distinct organs with di erent angles (Figure 4). In such cases, we try to take advantage from the complementarity of the views in order to compute a unique ranking list of species for one plant id. Then, the ranking list was repeated in the run le for each image of the associated plant. For each plant associated to at least two image queries, the fusion is applied in two-stages: 1. single-view level: if several query images (Io) are related to the same organ o, we kept for each class Ck the highest probability: Po(Ck) = max Pi(Ck) 8i2Io 2. multi-organ level: for responses of di erent organs (O), we apply the same the weighted fusion of probabilities used as in Section 3.1 where local features F are replaced with the organs set O. As discussed, this approach underlines the discriminant power of an organ.
Anemone hepatica Aphyllathes monspeliensis Cichorium intybus Scilla bifolia Figure 4 shows an example of the fusion on one set of query images from one plant observation (IndividualP lantID = 4165) from the training dataset in a leave one out procedure during preliminary tests. The correct species is Viburnum tinus and 4 views are represented by 10 images (4(a) shows sample images for each view). 4(b) shows the rst proposed species per image and we can see that Viburnum tinus does not appear in all responses. 4(c) shows the responses combined per view. We can note here that for Entire and Leaf, the proposed species have lower probability with respect to the species proposed for other views, which results to higher weights assigned for Stem and Fruit (line 4(d)). Finally, the ranking list of proposed species 4(e) is lead by the rst species from the, presumably, least confusing view response (Stem) while the correct species is at the 3rd position. Finally, it is not only 2 query images of Fruit, but all the 10 query images which are associated with a relevant species response. Filtering by owering period: Unlike other views, Flower has an important feature: in many species these plant organs are present for just a quite short period of time, and each species has its own owering periods. The metadata contains the date when the photograph was taken, for the training, as well as for the test datasets. A post-processing treatment is applied to the list of species obtained through pure visual search, and only the species for which the date of query image was observed in the training data were retained.
The owering period histogram (Figure 5 (a)) for a species is constructed by week, with 3 additional weeks to account for geographical and year-to-year di erences. Given a training image of class Ck, taken in week w, histogram bins HCk (h); h = w 3; :::; w + 3 are incremented. For a query image Q taken in week wQ, an histogram HQ is constructed in the same manner, and nally a species Ck is retained if 9wjHQ(w) > 0 ^ HCk (w) > 0.
Figure 5 (a) shows the owering periods for four species that have similar color and texture. Cichorium intybus owers appear cleary later than the other species over a year. Thus, any query images in this period will exclude the three other species, even if the visual content are very similar to these species. 3.3
Automatic segmentation ! RUN3
In this run we tested the automatic fore-/back-ground segmentation in an
attempt to reduce the number of the Harris points in the (cluttered) background.
The segmentation algorithm used was Otsu [
Figure 2 shows (a) initial detected points and (b) the ltered set. The distribution in (a) is clearly in a rhomboid shape, while in the case (b) the points are all in the foreground object or on its edges.
Multi-modal image representation through embedded local
features ! RUN4
This run is performed in order to explore combining local features through an
embedding schema for images representation and multi-class learning approach.
We have used sher vectors representation [
Image representation: Fisher vectors representation (FV) extends the BOV
representation where the local patches are described by their deviation from a
generative Gaussian mixture model (GMM). Let X = fxt; xt 2 <D; t = 1:::T g
be the set of T D-dimensional local descriptors of an image I. Assuming that the
generation process of X can be modelled by the GMM model with parameters
, we can characterize an image I by the following gradient vector [
T
X log
t=1
(xt)
where = f!i; i; i; i = 1:::Kg and !i, i and i are respectively the mixture
weight, mean vector and variance matrix (assumed diagonal) of Gaussian i.
We used the gradient vector wrt. the mean only on GMM with K Gaussians as
it was presented in [
Y = fyj ; j = 1:::Dg with D = Pn i=1 di
K
Classi er training: For each sub-category we learned a standard approach
with a linear one-versus-all multi-class SVM classi er. For e ciency, we trained
our classi ers using Stochastic Gradient Descent (SGD) algorithm as in [
SheetAsBackground. The four methods obtained very good results, all of them
being in the top 5 with the run Inria PlantNet Run 2 reaching the second
position. We discuss the fact that the ranking of the 4 runs di ers from what
we observed during preliminary tests with a Leave One Out procedure on the
training dataset (see Table 2). For instance TSLA method, which had the lowest
preliminary results with a 4th position on the training dataset, is nally the 1st
on the testing dataset. As noticed in [
TOA Inria PlantNet Run 1
TSLA Inria PlantNet Run 2 TSLA+DFH-Shapes Inria PlantNet Run 3
TOA+SC2 Inria PlantNet Run 4
Moreover, the combinations TSLA+DFH-Shapes and TOA+SC2 increase the
scores of TSLA or TOA on the training dataset while it is not the case on the test
images. This may be due to some unsuccessful automatic extractions of the leaf
contour on a part of the test dataset as illustrated in Figure 7. The quality of the
automatic segmentation is closely dependent on the noise that may be present in
the image. Multiscale triangular methods are robust to partial occlusion and to
local contour deformations, since the local descriptors associated to the contour
points describe only a portion of the leaf boundary (cf. [
Run score NlabUTokyo Run 3 0.393 Inria PlantNet Run 2 ! 0.385 NlabUTokyo Run 2 0.371 Inria PlantNet Run 1 ! 0.353 NlabUTokyo Run 1 0.341 Inria PlantNet Run 3 ! 0.325 Inria PlantNet Run 4 ! 0.245
Entire Flower Fruit Leaf Stem 0.297 0.472 0.311 0.275 0.253 ! 0.274 0.494 0.260 0.272 0.240
0.273 0.484 0.259 0.273 0.285 ! 0.254 0.437 0.249 0.240 0.211
0.236 0.423 0.209 0.269 0.276 ! 0.216 0.421 0.238 0.195 0.176 ! 0.150 0.327 0.137 0.165 0.171 4th, 5th and 7th. Basically, our scores are closely interlaced with the ones from NlabUTokyo, whereas the runs of all other groups are signi cantly lower.
Inria PlantNet Run 2 had the best scores within our runs as well in details for the 5 sub-categories. We can note also that overall gain for Inria PlantNet Run 2 with respect to Inria PlantNet Run 1 was 0.032, ranging from 0.011 for Fruit to 0.057 for Flower, which shows the bene ts that can be done when using multi-image queries and owering period ltering.
Inria PlantNet Run 3 achieved only 0.028 (8%) smaller score than Inria PlantNet Run 1, while using 30% less interest points, which is an interesting result if we consider memory resources for implementation in a nal system. But we must note that for Flower and Fruit, the losses were much smaller (3-4%) comparing to Entire, Leaf and Stem (15-19%). We suspect that this is due to the segmentation problems with the later three views and that in those cases, alternative segmentation methods could be applied.
Inria PlantNet Run 4 reached a quite good results if we compare with all submitted runs. However, the method experimented obtained lower score than the method used in Inria PlantNet Run 1 which was using the same local features. One explanation can be that the choice of a fusion by concatenation of Fisher Vectors computed in separated features space was not the most relevant fusion scheme because the discriminating powers of each local feature was not visible, or the concatenation led to normalisation problems. Also, the sher embedding itself might degrade performances compared to the independent knn search of each local feature. 5
Inria submitted four runs that used di erent processing methods for SheetAsBackground and NaturalBackgroud categories. The task was more challenging this year, with twice of species and images, and more visual diversity with the 5 di erent plant views. Our methods performed very well and our best runs were 2nd in both categories.
For SheetAsBackground, the results obtained by our four runs (in the top 5
positions) con rmed the e ectiveness of our multiscale triangle based approaches
[
For NaturalBackground, we tested local features matching with several declinations as the use of metadata or automatic segmentation. For the rst three runs, we proposed to use a weighted fusion scheme for combining the responses from multiple indexes build for each combination of type of views and type of local features. It underlines the discriminant power of a non-confusing response. With the use of metadata, such as the plant ids and the date of photograph, we showed that multi-image queries and ltering by owering period can clearly increase the baseline performance. On the other hand, with the use of automatic segmentation, we could reduce the size of the training features, while having a rather small performance decrease. We also note that, as for other participants, the score for Flower category is double the score for other categories and that the best Flower scores are approaching the scores obtained in SheetAsBackground category.
Acknowledgments. Part of this work was funded by the Agropolis foundation through the project Pl@ntNet (http://www.plantnet-project.org/)