This paper presents the participation of INRIA IMEDIA group and the Pl@ntNet project to ImageCLEF 2011 plant identi cation task. ImageCLEF's plant identi cation task provides a testbed for the system-oriented evaluation of tree species identi cation based on leaf images. The aim is to investigate image retrieval approaches in the context of crowdsourced images of leaves collected in a collaborative manner. IMEDIA submitted two runs to this task and obtained the best evaluation score for two of the three image categories addressed within the benchmark. The paper presents the two approaches employed, and provides an analysis of the obtained evaluation results.
This paper presents the participation of INRIA IMEDIA group and the Pl@ntNet4 project to the plant identi cation task that was organized within ImageCLEF 20115 for the system-oriented evaluation of visual based plant identi cation. This rst year pilot task was more precisely focused on tree species identi cation based on leaf images. The task was organized as a classi cation task over 70 tree species with visual content being the main available information. Three types of image content were considered: leaf scans, leaf photographs with a white uniform background (referred as scan-like pictures) and unconstrained leaf's photographs acquired on trees with natural background. IMEDIA group, in collaboration with the Pl@ntNet project submitted two runs, one based on large-scale local features matching and rigid geometrical models, the other one based on segmentation and shape boundary features. 4 http://www.plantnet-project.org/papyrus.php?langue=en 5 http://www.imageclef.org/2011
The task was evaluated as a supervised classi cation problem with tree species used as class labels. 2.1
Training and Test data A part of Pl@ntLeaves dataset was provided as training data whereas the remaining part was used later as test data. The training subset was built by randomly selecting 2/3 of the individual plants of each species (and not by randomly splitting the images themselves). So that pictures of leaves belonging to the same individual tree cannot be split across training and test data. This prevents identifying the species of a given tree thanks to its own leaves and that makes the task more realistic. In a real world application, it is indeed much unlikely that a user tries to identify a tree that is already present in the training data. Detailed statistics of the composition of the training and test data are provided in Table 1.
Photograph Train
Nb of pictures Nb of individual plants Nb of contributors 2349 151 17 721 55 13 717 51 2 180 13 1 930 72 2 539 33 3 All
Train 3996 269
Test 1440 99 Table 1. Statistics of the composition of the training and test data The goal of the task was to associate the correct tree species to each test image. Each participant was allowed to submit up to 3 runs built from di erent methods. As many species as possible can be associated to each test image, sorted by decreasing con dence score. Only the most con dent species was however used in the primary evaluation metric described below. But providing an extended ranked list of species was encouraged in order to derive complementary statistics (e.g. recognition rate at other taxonomic levels, suggestion rate on top k species, etc.).
The primary metric used to evaluate the submitted runs was a normalized classi cation rate evaluated on the 1st species returned for each test image. Each test image is attributed with a score of 1 if the 1st returned species is correct and 0 if it is wrong. An average normalized score is then computed on all test images. A simple mean on all test images would indeed introduce some bias with regard to a real world identi cation system. Indeed, we remind that the Pl@ntLeaves dataset was built in a collaborative manner. So that few contributors might have provided much more pictures than many other contributors who provided few. Since we want to evaluate the ability of a system to provide correct answers to all users, we rather measure the mean of the average classi cation rate per author. Furthermore, some authors sometimes provided many pictures of the same individual plant (to enrich training data with less e orts). Since we want to evaluate the ability of a system to provide the correct answer based on a single plant observation, we also decided to average the classi cation rate on each individual plant. Finally, our primary metric was de ned as the following average classi cation score S:
S = 1 XU 1 XPu 1 NXu;p su;p;n U u=1 Pu p=1 Nu;p n=1 (1) U : number of users (who have at least one image in the test data) Pu : number of individual plants observed by the u-th user Nu;p : number of pictures taken from the p-th plant observed by the u-th user su;p;n : classi cation score (1 or 0) for the n-th picture taken from the p-th plant observed by the u-th user
It is important to notice that while making the task more realistic, the normalized classi cation score also makes it more di cult. Indeed, it works as if a bias was introduced between the statistics of the training data and the one of the test data. It highlights the fact that bias-robust machine learning and computer vision methods should be preferred to train such real-world collaborative data. Finally, to isolate and evaluate the impact of the image acquisition type (scan, scan-like, photograph), a normalized classi cation score S was computed for each type separately. Participants were therefore allowed to train distinct classi ers, use di erent training subsets or use distinct methods for each data type. 3 3.1
Large-scale local features matching and rigid geometrical
models ! inria imedia plantnet run1
State-of-the-art methods addressing leaf-based identi cation of leaves are mostly
based on leaf segmentation and shape boundary features [
Besides clutter robustness, the method has several advantages: it does not require any complex training phase allowing fast dynamic insertion of new crowdsourced training data, and it is weakly a ected by unbalanced class distribution thanks to the selectivity of the spatial consistency ltering.
Mixed texture & shape local features Rather than using classical SIFT
features computed around DoG points, we employed multi-resolution color
Harris points ([
Local features: hough 4 4, eoh 8, fourier 8 32 are extracted around each Harris
point from an image patch oriented according to the principal orientation and
scaled according to the resolution at which the Harris corner was detected.
hough 4 4 is a 16 dimensional histogram based on ideas inspired from the Hough
transform and is used to represent simple shapes in an image [
Finally, we use as local features the concatenation of these 3 local features,
resulting in a 280-dimensional feature vector extracted around each Harris point.
Local features compression with RMMH Random Maximum Margin
Hashing (RMMH) [
Local features indexing and matching with AMP-RMMH We also used
RMMH for indexing purposes using the multi-probe hashing method described
in [
Classi cation with a top-k decision rule Best species label is nally computed by voting on the top-10 returned training images (ranked by geometrical consistency score).
Training data strategy Since training and test leaf images are categorized in three distinct image types (scans, scan-like photos and unconstrained photos), an important question is which training images types should be used for which test image type. Few leave-one-out experiments performed on the training set itself did show us that using only scans as training images for all test images might be more e ective than other strategies (e.g. using all training images for all test image types or using only the same image type for training and testing). This can be explained by the fact that scan images do not contain any noisy background so that all local features included in the trained index are actually parts of the leave and not distractors as in unconstrained photographs. 3.2
Directional Fragment Histogram and geometric parameters on
shape boundary! inria imedia plantnet run2
The method used in the second run is very distinct from the rst one and is closer
from state-of-the-art methods based on leaf segmentation and shape boundary
features. We use a shape boundary descriptor called Directional Fragment
Histogram introduced in a previous work on botanical images database [
As almost boundary-based shape description methods, the rst step deals
with image segmentation. We use the classical Otsu adaptive thresholding method
[
Boundary description with Directional Fragment Histogram This method
was introduced and applied successfully in a previous work on botanical data in
2006 [
Boundary description with geometric parameters In order to improve
performances in plant identi cation, we chose to combine the DHF descriptor
with 8 morphological features used in plant identi cation literature, like Aspect
Ratio, Rectangularity, Convex Area Ratio, Convex Perimeter Ratio, Sphericity,
Eccentricity and Form Factor. The table 2 gives the 8 geometric parameters
used for the task. Most of these parameters were succesfuly experimented in
[
The two runs submitted by IMEDIA, in spite of theirs theoretical di erences, gave both good results, and obtained the best evaluation scores for two
Dmax Dmin
As Ab As Ah Ps Ph of the three image categories addressed within the benchmark, on scans for the run inria imedia plantnet run1, and on scan-like images for the second run inria imedia plantnet run2.
Considering the rst run, the approach based on large-scale local features matching and rigid geometrical models gives surprisingly better results on scans than state of the arts methods based on shape boundary features.
Considering the image types and the results for all teams, performances are degrading with the complexity of the acquisition image type. Indeed, scans are more easy to identify than scan-like photos and unconstrained photos are much more di cult. This is can be seen in gure 5 where the relative scores of each image type are highlighted by distinct colors. However, if this "rule" is true for the rst run inria imedia plantnet run1 , it is not for the second run inria imedia plantnet run2 (and also for 5 other runs). It is di cult to give a precise reason of these results, but numerous unsuccessful scan tests have a relatively poor quality, coming from a low resolution original scan, noisy with a non uniform and gradually yellow colored background with blurred content. These unsuccessful scans maybe indicate a weakness at the very rst step of automatic segmentation. For IMEDIA these results are very promising considering the complementarity of the two very distinct methods. Surprisingly, the matching approach gives the best evaluation score of the task on scans than state of the arts methods based on shape boundary features. This is an important result that opens further investigations in matching based approaches applied to plant identi cation. Initially aimed at retrieving rigid objects, this original approach for plant leaf identi cation can be certainly improved in order to be more robust to other kind of images as the scan-like pictures and photographs, maybe by considering a part-based model approach. The very good results with the second method based on shape boundary description, let us to plan improvement by combining it with the matching approach. Indeed, by considering all test images, only 22% of the images are successful at the same time for the two methods, which let us to aim a signi cant room for improvement.
This work was funded by the Agropolis fundation through the project Pl@ntNet (http://www.plantnet-project.org/)