This paper describes the participation of Inria within the Pl@ntNet project7 at the LifeCLEF2014 plant identi cation task. The aim of the task was to produce a list of relevant species for each plant observation in a test dataset according to a training dataset. Each plant observation contains several annotated pictures with organ/view tags: Flower, Leaf, Fruit, Stem, Branch, Entire, Scan (exclusively of leaf). Our system treated independently each category of organ/view and then a late hierarchical fusion is used in order to combine the results on visual content analysis from the most local level analysis in pictures to the highest level related to a plant observation. For the photographs of owers, leaves, fruits, stems, branches and entire views of plants, a large scale matching approach of local features extracted using di erent spatial constraints is used. For scans, the method combines the large scale matching approach with shape descriptors and geometric parameters on shape boundary. Then, several fusion methods are experimented through the four submitted runs in order to combine hierarchically the local responses to the nal response at the plant observation level. The four submitted runs obtained good results and got the 4th to the 7th place over 27 submitted runs by 10 participating teams.
The plant identi cation task of LifeCLEF2014 [?] [?] was organized as a plant species retrieval task over 500 species with visual content being the main available information. The aim of the task was to produce a list of relevant species for each observation in a test dataset containing 8163 plant observations based on 13146 images, where each observation can contain several pictures of detailed views on various organs: Flower, Fruit, Leaf, Stem, Entire, Branch and Scan (scans or scan-like pictures of leaf exclusively). The training dataset contains numerous plant observations related to 47815 images tagged with these organ/view information.
We present in this paper the methods used in order to produce the four submitted runs by Inria within the PlantNet project. It is based more or less on the same system used during our previous participations to the ImageCLEF20112013 plant tasks [?] but with some improvements mainly related to new types of combination of image list results provided by the di erent image queries from a same plant observation. 2
Each tested observation can potentially contain several pictures on several organs/views. Our system treated independently each organ/view and then a late hierarchical fusion framework is used in order to combine the visual content analysis from the most local level in pictures to the highest level related to one same plant observation.
{ local level (see subsection ??): we extract various types of local features around interest points on training and test images. Then, considering one single test image with one organ/view tag, and only one type of feature, we search the knn local features in the training dataset with the same organs/views tag, and combine them in order to obtain a ranked training image list. { image level: (late fusion of features, see subsection ??) then we combine the ranked training image list produced for all types of features. { multi-image level: (see subsection ??) the results from multiple test images from a same plant observation within a same organ/view are combined. { multi-image level: (see subsection ??) the results from several organs/views from a same plant observation are combined. { observation level: (see subsection ??) the combined results are transformed in a ranked list of species. 2.1
We explored the fact that in most cases, one organ or one group of organs, is generally centred in the pictures of the dataset. Harris corners were used at four distinct resolutions with multiple orientations ([?]). In addition, as in [?], to minimize the e ect of the cluttered background, a rhomboid-shaped mask was applied to the input image and we used a Gaussian-like distribution in a 7x7 grid in order to take more points at the center of the pictures (the gure ??(a) illustrates the detected points). About 100 points per image were output, and according to the multiple-orientations, between 150 and 200 local features are extracted in patches around these points (see subsection??), depending of the visual content of the picture analysed.
Several local features are extracted around each interest point from an oriented and scaled patch: rotation invariant Local Binary Pattern (ri-LBP) [?]; SURF [?] [?] (sums of 2D Haar wavelet responses); a 20-dim. Fourier histogram [?]; an 8-dim. Edge Orientation Histogram (EOH); a 27-bin weighted RGB histogram (wght-RGB) [?], a 27-bin weighted LUV histogram (wght-LUV) and a 30-bin HSV histogram. After a series of preliminary evaluations tests with the training data, we concluded that not all type of local features should be used for all views (see subsection ??). In the speci c case of Scan we used a concatenation of the SURF, EOH, Fourier and a 16-dim. histogram based on the Hough transform [?] (we will call these concatenated descriptors "SEFH").
For each type of view/organ, and for each type of features, local features are computed, compressed and indexed using RMMH method [?,?] using a 256-bit hash code, which led to a total of 43 unique hash tables (6 organs/views x 7 types of features + 1 separate table for scans), but around half of them are nally considered for computing the submitted runs (see subsection ??).
Considering a test image Iq, its associated organ/view, and considering one visual index of one type of features, the local similarity search produces a response list R composed of similar training images sorted by a score Si related to a matching function on each training image i. Each local query feature q of the image Iq is compressed with RMMH and its approximate k-nearest neighbors are searched by probing multiple neighboring buckets in the consulted hash table (according to the a posteriori multi-probe algorithm described in [?]). Two types of votes and parameters of k were used through in the submitted runs in order to produce scores and sort the training image response list: Vote On Best and Ranking Dominate.
Vote On Best (VOB) (Run 1, 2 & 4 )
The score Si here is directly the number of matches between a training image Ii
and the query image Iq. More precisely it is the number of instances of image Ii
retrieved through the k = 30 nearest neighbors lists of each local feature of the
query image Q. Finally the list is limited to the 300 best training images.
Ranking Dominate(RD) (Run 3 )
This new vote introduces a more advanced image ranking method based on the
number of dominating local features. Inspired by the concept of top-k dominating
queries in the database domain [?], we consider that a given local feature x is
dominating another local feature y if it is closer to a query feature q (according
to the distance used as matching function). Two images I1 and I2 can then be
compared with reference to a query image Iq by counting the number of query
local features qi for which a local feature of I1 is dominating all features of I2
(or conversely). More precisely, images are ranked using a quick sort algorithm
based on the following comparison function:
n
f (I1; I2) = sgn(X sgn(ri1
i=1
ri2))
(
(Run 1, 2, 3 & 4 ) In the case of leaf scans and scan-like pictures, additional boundary descriptors are used since it has been showed to provide a consistent improvement of the global performances in the previous ImageCLEF2011 dataset [?]. Involving a rst step of automatic leaf boundary detection, these descriptors are the Directional Fragment Histogram (DFH) describing the leaf margin, and several geometric parameters embedded in one single vector (Shapes6) of 6 dim. measuring 6 standard geometric parameters on the shape: rectangularity, convexity, solidity, circularity, Sphericity and Ellipse variance. 2.3
At this step, we have now for each image query Iq several image training response lists, one for each type of (local or global) features. The next step is now to combine these responses in order to obtain a single response. Thus, we use a late fusion approach with di erent methods experimented through the four submitted runs.
CIN and WP (Run 1 ) For a query image Iq, according to its organ/view, the basic algorithm applied is: (i) starting from the retrieved lists of similar training images for each type of feature, (ii) transform each list of images into a probability distribution by species with an adaptive rule CIN (see below), (iii) merge probabilities for all types of features with a Weighted Probability approach (WP).
CIN It an adaptive KNN rule established in this particular context of plant observations [?]. For converting an image response list to a species probability distribution, we use this adaptive rule focusing on plant observations (rather than images). Indeed, the more a species will be represented in a response through various plants observed by distinct users at di erent dates and locations, the more the associated images will be informative for predicting a species. In contrast, numerous redundant near duplicate images from the same plant observation will be less informative for predicting a species. Instead of using a basic KNN rule focusing on images, we search for top-K00 classes (species) represented with at least K0 di erent plant observations. The values of K0 and K00 are determined empirically based on the given training database, and are constant for a database: K0 is a percentage of the average size of the training class, while K00 is a percentage of the number of training classes. The response R is scanned from the most to the least similar image, the counter of the number of classes with at least K0 images is incremented accordingly, and when we nd K00 such classes, we stop the scanning of the response. The adaptive criterion is important in order to avoid the noise in the nal response: (a) had we searched for a xed (and not dependent on the training data) number of classes with at least K0 plant observations, we would often output classes that are not relevant to ll-in the pre-de ned requirement, or (b) had we output a xed number of most similar classes, we would give more weight to the classes with small number of plant observations and would not reward the fact that some classes are well represented in terms of plant observations. Finally, our K00 per class resulted in the K per image of 85, ranging from 15 to 205, for the most to the least di erent-plantobservation-containing response; organ-wise, the values of K ranged from 66 for Stem to 101 for Fruit.
Moreover, to eliminate the redundant images, we consider only two most similar images per one plant observation: the score per plant observation Sm is the average of the image scores Si. The score for a class is the sum of the scores of its plant observations Sm: this step actually favours the well-represented classes, and penalizes the classes with small number of plant observations. Finally, the classes with only one image coming from one plant observation are removed from the list as outliers. Weighted Probability combination (WP) At this step we have several species probability distributions, one for each type of features and we use a weighted fusion in order to obtain a nal probability distribution. Let us de ne F as a set of local features and P (Cfk) as probability of class Ck for feature f 2 F . In order to re ect the discriminating power of each local feature, we de ne the nal probability:
P (Ck) = X w(f ) P (Cfk)
f2F w(f ) = max P (Cfk)= X max P (Cfk)
8k f2F 8k where BordaMNZ & IrpMNZ (BordaMNZ: Run 2, 3 & 4, IrpMNZ; Run 4 ) In [?] the authors proposed similarity ranking lists fusion algorithms in order to merge some multi-feature similarity lists into a nal overall similarity ranking list. They showed good performances on their experimental results with basic algorithms working directly with the ranks, i.e. without complicated and empirical transformation of similarity lists into probability distributions. We experimented here with two algorithms in order to merge the image training response lists given by each type of features into one single list: the Borda Count Algorithm taken from social theory in voting, and the Inverse Rank Position Algorithm (Irp). Basically a score is computed for each training image contained in the response lists to combine, and then used for sorting a nal merged list of training images.
Considering an image query Q, a training image Ii, and several image training response lists R, one for each feature f from a set of available features F , the score used in the Borda Count algorithm is the sum of the rank positions of the training image in the lists to merge:
Borda(Q; Ii) =
F X rankposition(Ii; f ) f=1 Irp(Q; Ii) =
1 PF 1
f=1 rankposition(Ii;f) The score used in the Irp algorithm is the inverse of the sum of inverses of the rank positions of the training image in the di erent lists to merge: Like in the previous work mentioned in [?], we used the MNZ extension of these two algorithms (BordaMNZ and IrpMNZ ) in order to heavily weights common training images in the lists to combine. Indeed, the hypothesis is that di erent lists contain similar sets of relevant training images while they contain di erent sets of non-relevant training images. Basically, the weights are based on the number of Non-Zero entries of each training images in the lists to combine, i.e. the number of times a training image appears in the ranked lists.
BordaM N Z(Q; Ii) = Borda(Q; Ii)
(
IrpM N Z(Q; Ii) = Irp(Q; Ii)
(
At this step, we have now a knowledge at the image query level combining the di erent type of features for each image query from a same observation plant query Oq. This knowledge can be a probability distribution if the Weighted Probability approach was used, or a ranked training image list if the BordaMNZ or IrpMNZ methods were used. The image queries from a same plant Oq can be related to the same or distinct organs and views showing di erent angles of the plant and we try here to take advantage from the complementarity of these views in order to compute a nal knowledge at the observation level. Next combinations of knowledge are performed in this order: { image level to organ/view level (if the query plant Oq has several images from one same organ/view Vq), { then, organ/view level to observation level (if the query plant Oq as several organs V ).
Only the Weighted Probability approach (Run 1 ) and the BordaMNZ approach (Run 2, 3 & 4 ) are used in these upper levels of combination following the same formulas in previous subsections ??.
For instance, at organ level with the Weighted Probability approach, given several probability distributions from a set of di erent organs/views V , we apply the same the weighted fusion used as in subsection ?? where local features F are replaced by V . 2.5
(Run 1, 2, 3 & 4 ) At this nal step, we have for one query observation plant Oq, a distribution of probabilities if the Weighted Probability approach was used in the lower levels of fusion (Run 1 ), or a nal list of training image response if the BordaMNZ was used instead.
In the case of the WP approach, the nal list of species is directly given by the probability distribution following a decreasing order of probabilities. In the case of the BordaMNZ approach, we use a last step for converting the image list response into a probability distribution with the CIN rule (see previous section ??). 2.6
Preliminary evaluation and feature selection Considering the available number of visual feature for each images and the distinct methods of combinations, we supposed here that each type of organ/views has a its own relevant subset of visual features exploited no necessarily with the same combination method. We made some intensive preliminary evaluations on the training dataset in order to nd these subsets and methods for each type of organ/views. We used a Leave One (Observation) Out procedure: focusing on one organ/view, for image from the training dataset, we use it as an internal query and excluded from the returned ranked image response list all the images belonging to the same plant observation as the query image, in order to remove a bias related to near duplicate images form a same plant. We concluded that in most of the cases, it was more or less the same subsets of features which was pointed out by the 3 fusion methods BordaMNZ, IrpMNZ and WP. Since the BordaMNZ demonstrated the best performances for mostly every organ/view, we choose to select its subset of visual features. As might be expected, color is dominant for the ower but may lead to confusion for leaves, while texture plays an important role for stems and fruits. The features nally selected are presented in table ??:
SURF Fourier EOH riLbp wRGB wLUV HSV SEFH DFH Shapes6 Branch Entire Flower Fruit Leaf Scan
Stem
di erent methods used at each level of fusion for each submitted run.
PlantNet Run 1 is actually more or less the same method used for the submitted run Inria PlantNet Run 1 during the previous ImageCLEF 2013 Plant Task [?]. We used this con guration as a baseline in order to see the score this year despite 250 more new species, and to see if we can obtain better scores with the 3 new approaches experimented in the 3 other submitted runs.
PlantNet Run2 is the BordaMNZ version of the PlantNet Run 1, where for each step of fusion we changed the Weighted Probability WP combination method by the BordaMNZ method. In this way, only list responses are merged at each step, and nally we use the CIN approach in order to obtain a list of probability of species, and thus, a list of ranked of species according to these probability values.
PlantNet Run 3 is like the PlantNet Run2 except that we changed the local responses combination with the Ranking Dominate vote (see previous subsection ??).
PlantNet Run 4 is supposed to be the best con guration, where for each level of combination we choose the best method according to the preliminary evaluations on the training dataset. The BordaMNZ is most of the time the retained method, except for scans and stem pictures at the feature combination level for which we had noticed better performances with the IprMNZ combination. At the time of writing the paper, we noticed a mistake with the Fruit organ where we selected a Weighted Probability approach and which probably gave bad combinations in upper level since image response lists were expected instead of probability distributions. Submitted "image" runs For each submitted run, it was recommended but not mandatory to submit a second set of run les detailing species prediction at image level like in the previous ImageCLEF 2013 Plant Task [?]. The aim was to see if the participants can take bene t from the combination of multiple images from a same plant. In order to produce these complementary "image" runs, we simplify the hierarchical fusion of treatment, stopping at the feature combination level for one same image query (see Table 3). Unfortunately, we encountered a problem during the generation of the le (Image) PlantNet Run 3 and we were not able to submit it. 3
2. It is disappointing that the new Ranking Vote experimented in PlantNet Run 3 didn't show some improvement and is strangely very close to the second run PlantNet Run 2. Finally, our supposed best run PlantNet Run 4, with the best combination at each level of fusion, didn't show the best performances, but it is still better of our baseline Run PlantNet 1. Table ?? resumes the scores obtained
Run name Score IBM AU Run 4 0,471 IBM AU Run 3 0,459 IBM AU Run 2 0,454 PlantNet Run 2 0,289 PlantNet Run 3 0,289 PlantNet Run 4 0,282
PlantNet Run 1 0,278 for the rst 6 submitted runs out of 14 total. Figure ?? gives an overview of all results obtained by the participants who submitted these optional runs. Our 3 submitted "image" runs by our team are in the top 6. For each method, like for most of other methods used by the other participants, it highlights the fact that we obtained some slightly improvement by combining the pictures and the organs/views from a same plant observation query.
Table ?? resumes the detailed scores by organs/views obtained for the rst 10 submitted runs out of 14 total and Figure ?? gives an overview of all results obtained by the participants who submitted these optional runs. In most cases, our 3 submitted "image" runs by our team are in the top 6, expect for the Entire and Flower where the 4th run of IBM AU obtained best results, and for the Stem where the FINKI team used some better approaches. Inria within the PlantNet project submitted four runs, and 3 complementary "image" runs, that used the same hierarchical fusion framework, and where each run was related to distinct con gurations of methods. All the submitted runs obtained good results and are ranked from the 4th to the 7th place compared to the other submitted run, but clearly behind the best results obtained by the IBM AU team. The rst run was more or less the same approach used in the previous ImageCLEF 2013 plant identi cation task, and was considered as a baseline in our work. We try to improve this approach with simple but e ective fusion approaches of ranked lists of results based on the Borda Count and the Inverse Fig. 3. Overview of the image scores of the LifeCLEF 2014 Plant Task compared with the o cial scores related to observation.
Run Branch Entire Flower Fruit Leaf Leaf Scan Stem IBM AU Run 4 0,292 0,333 0,585 0,339 0,318 0,64 0,269 IBM AU Run 3 0,298 0,34 0,57 0,326 0,304 0,614 0,267 IBM AU Run 2 0,294 0,335 0,555 0,317 0,3 0,612 0,267 PlantNet Run 4 0,112 0,167 0,366 0,197 0,165 0,541 0,152 PlantNet Run 2 0,112 0,181 0,376 0,22 0,164 0,453 0,156 PlantNet Run 1 0,112 0,168 0,366 0,197 0,165 0,449 0,133 IBM AU Run 1 0,103 0,193 0,389 0,161 0,103 0,278 0,138 FINKI Run 1 0,088 0,117 0,255 0,177 0,16 0,4 0,157 FINKI Run 2 0,108 0,099 0,187 0,16 0,14 0,399 0,18
FINKI Run 3 0,088 0,117 0,255 0,177 0,162 0,36 0,159
Rank Position Algorithms. These two approaches performed slightly better, and we obtained the best results for the full BordaMNZ approach.
Acknowledgments. Part of this work was funded by the Agropolis foundation through the project Pl@ntNet (http://www.plantnet-project.org/)