ImageCLEF' plant identi cation task provides a testbed for the system-oriented evaluation of tree species identi cation based on leaf images. The aim is to investigate image retrieval approaches in the context of crowdsourced images of leaves collected in a collaborative manner. This paper presents an overview of the resources and assessments of the plant identi cation task at ImageCLEF 2011, summarizes the retrieval approaches employed by the participating groups, and provides an analysis of the main evaluation results.
Convergence of multidisciplinary research is more and more considered as the next big thing to answer profound challenges of humanity related to health, biodiversity or sustainable energy. The integration of life sciences and computer sciences has a major role to play towards managing and analyzing cross-disciplinary scienti c data at a global scale. More speci cally, building accurate knowledge of the identity, geographic distribution and uses of plants is essential if agricultural development is to be successful and biodiversity is to be conserved. Unfortunately, such basic information is often only partially available for professional stakeholders, teachers, scientists and citizens, and often incomplete for ecosystems that possess the highest plant diversity. A noticeable consequence, expressed as the taxonomic gap, is that identifying plant species is usually impossible for the general public, and often a di cult task for professionals, such as farmers or wood exploiters and even for the botanists themselves. The only way to overcome this problem is to speed up the collection and integration of raw observation data, while simultaneously providing to potential users an easy and e cient access to this botanical knowledge. In this context, content-based visual identi cation of plant's images is considered as one of the most promising solution to help bridging the taxonomic gap. Evaluating recent advances of the IR community on this challenging task is therefore an important issue. This paper presents the plant identi cation task that was organized within ImageCLEF 20117 for the system-oriented evaluation of visual based plant identi cation. This rst year pilot task was more precisely focused on tree species identi cation based on leaf images. Leaves are far from being the only discriminant visual key between tree species but they have the advantage to be easily observable and the most studied organ in the computer vision community. The task was organized as a classi cation task over 70 tree species with visual content being the main available information. Additional information only included contextual meta-data (author, date, locality name) and some EXIF data. Three types of image content were considered: leaf scans, leaf photographs with a white uniform background (referred as scan-like pictures) and unconstrained leaf's photographs acquired on trees with natural background. The main originality of this data is that it was speci cally built through a citizen sciences initiative conducted by Telabotanica 8, a French social network of amateur and expert botanists. This makes the task closer to the conditions of a real-world application: (i) leaves of the same species are coming from distinct trees living in distinct areas (ii) pictures and scans are taken by di erent users that might not used the same protocol to collect the leaves and/or acquire the images (iii) pictures and scans are taken at di erent periods in the year. 2 2.1
Building e ective computer vision and machine learning techniques is not the
only side of the taxonomic gap ptoblem. Speeding-up the collection of raw
observation data is clearly another crucial one. The most promising approach in
that way is to build real-world collaborative systems allowing any user to enrich
the global visual botanical knowledge [
Technically, images and associated tags were collected through a crowd-sourcing
web application [
8 http://www.tela-botanica.org/ and between June and September 2010 thanks to the work of active contributors from TelaBotanica social network. The idea of collecting only scans during this rst period was to initialize the training data with limited noisy background and to focus on plant variability rather than mixed plant and view conditions variability. This allowed to collect 2228 scans over 55 species. A public version of the application 9 was then opened in October 2010 and additional data were collected up to March 2011. The new collected images were either scans, or photographs with uniform background (referred as scan-like photos ), or unconstrained photographs with natural background. They involved 15 new species from the previous set of 55 species. The Pl@ntLeaves dataset used within ImageCLEF nally contained 5436 images with 3070 scans, 897 scan-like photos and 1469 photographs. Figure 2 displays samples of these 3 image types for 4 distinct tree species. The full list of species is provided in Figure 1.
{ Date upload date of the image { Type (acquisition type: scan, scan-like or photograph) { Content content type: single leaf, single dead leaf or foliage (several leaves on tree visible in the picture) { Taxon full taxon name (sub-regnum, regnum, class, division, order, family, genus, species) { VernacularNames French or English vernacular names { Author name of the author of the picture { Organization name of the organization of the author { Locality locality name (a district or a country division or a region) { GPSLocality GPS coordinates of the observation These meta-data are stored in independent xml les, one for each image. Figure 3 displays an example image with its associated xml data.
Additional but partial meta-data information can be found in the image's EXIF,
and might include the camera or the scanner model, the image resolution and
dimension, the optical parameters, the white balance, the light measures, etc.
2.3
The main originality of Pl@ntLeaves compared to previous leaf datasets, such
as the Swedish dataset [
The task was evaluated as a supervised classi cation problem with tree species used as class labels. 3.1
A part of Pl@ntLeaves dataset was provided as training data whereas the remaining part was used later as test data. The training subset was built by randomly selecting 2/3 of the individual plants of each species (and not by randomly splitting the images themselves). So that pictures of leaves belonging to the same individual tree cannot be split across training and test data. This prevents identifying the species of a given tree thanks to its own leaves and that makes the task more realistic. In a real world application, it is indeed much unlikely that a user tries to identify a tree that is already present in the training data. Detailed statistics of the composition of the training and test data are provided in Table 1.
The goal of the task was to associate the correct tree species to each test image. Each participant was allowed to submit up to 3 runs built from di erent methods. As many species as possible can be associated to each test image, sorted by decreasing con dence score. Only the most con dent species was however used in the primary evaluation metric described below. But providing an extended ranked list of species was encouraged in order to derive complementary statistics (e.g. recognition rate at other taxonomic levels, suggestion rate on top k species, etc.).
The primary metric used to evaluate the submitted runs was a normalized classi cation rate evaluated on the 1st species returned for each test image. Each test image is attributed with a score of 1 if the 1st returned species is correct and 0 if it is wrong. An average normalized score is then computed on all test images. A simple mean on all test images would indeed introduce some bias with regard to a real world identi cation system. Indeed, we remind that the Pl@ntLeaves dataset was built in a collaborative manner. So that few contributors might have provided much more pictures than many other contributors who provided few. Since we want to evaluate the ability of a system to provide correct answers to all users, we rather measure the mean of the average classi cation rate per author. Furthermore, some authors sometimes provided many pictures of the same individual plant (to enrich training data with less e orts). Since we want to evaluate the ability of a system to provide the correct answer based on a single plant observation, we also decided to average the classi cation rate on each individual plant. Finally, our primary metric was de ned as the following average classi cation score S:
S = 1 XU 1 XPu 1 NXu;p su;p;n
U u=1 Pu p=1 Nu;p n=1 U : number of users (who have at least one image in the test data) Pu : number of individual plants observed by the u-th user Nu;p : number of pictures taken from the p-th plant observed by the u-th user su;p;n : classi cation score (1 or 0) for the n-th picture taken from the p-th plant observed by the u-th user
It is important to notice that while making the task more realistic, the normalized classi cation score also makes it more di cult. Indeed, it works as if a bias was introduced between the statistics of the training data and the one of the test data. It highlights the fact that bias-robust machine learning and computer vision methods should be preferred to train such real-world collaborative data. Finally, to isolate and evaluate the impact of the image acquisition type (scan, scan-like, photogragh), a normalized classi cation score S was computed for each (1) type separately. Participants were therefore allowed to train distinct classi ers, use di erent training subsets or use distinct methods for each data type. 4
A total of 8 groups submitted 20 runs, which is a successful participation rate for a rst year pilot task on a new topic. Participants were mainly academics, specialized in computer vision and multimedia information retrieval, coming from all around the world: Australia (1), Brazil (1), France (2), Romania (1), Spain (1), Turkey (1) and UK (1). We list below the 8 participants and give a brief overview of the techniques they used to run the plant identi cation task. We remind here that ImageCLEF benchmark is a system-oriented evaluation and not a formal evaluation of the underlying methods. Readers interested by the scienti c and technical details of any of these methods should refer to the CLEF2011 working note of each participant.
IFSC (3 runs) [
LIRIS (4 runs) [
UAIC (3 runs) This participant was the only one trying to bene t from metadata associated to the images (location, date, author, etc.). They submitted therefore 3 runs to evaluate the contribution of metadata compared to using visual content only. Their rst run (UAIC2011 Run01) is based on visual content only, the second one (UAIC2011 Run02) uses only metadata based features in the classi cation process, and the third one uses both (UAIC2011 Run03).
SABANCI-OKAN (1 run) [
INRIA (2 runs) [
RMIT (2 runs) [
DAEDALUS (1 run) [
KMIMMIS (4 runs) This participant also used a generalist image retrieval framework based on local features and a nearest-neighbor classi er. They compared di erent con gurations: basic clustered SIFT with 1-NN label transfer (kmimmis run1 and kmimmis run4), simple edge and corner point detection with 1-NN vote label transfer (kmimmis run2), simple edge and corner point detection with 10-NN vote label transfer (kmimmis run3).
Over all runs, the most frequently used class of methods is shape boundary
analysis. 8 runs among 20 are based on some boundary shape features.This is
not surprising since state-of-the-art methods addressing leaf-based identi cation
in the literature are mostly based on leaf segmentation and shape boundary
features [
Figures 12, 13 and 14 present the normalized classi cation scores of the 20 submitted runs for each of the three image types. Alternatively, Figure 15 presents the overall performances averaged over the 3 image types. Table 2 nally presents the same results but with detailed numerical values.
A rst global remark is that, as expected, the performances are degrading with the complexity of the acquisition image type. Scans are more easy to identify than scan-like photos and unconstrained photos are much more di cult. This is can be easily seen in Figure 15 where the relative scores of each image type are highlighted by distinct colors.
A second global remark is that no method provide the best score for all image types. None of the run even belongs to the top-3 runs of all image types (as shown in Table 2). This is somehow disappointing from the genericity point of view but not surprising regarding the nature of the di erent image types. One could expect that scans and scan-like photos lead to similar conclusions but this is actually not the case. The only runs that give quite stable and good performances over the three image types are the two runs of IFSC based on complex network shape boundary analysis method (IFSC USP run1 & IFSC USP run2). This justi es their excellent ranking when averaging the classi cation scores over the three image types. All other methods fail in providing as good results for the unconstrained photographs as shown in Figure 14. This score gap between IFSC run and others has however to be mitigated by a bias introduced by the author normalization of the classi cation score. Indeed, their high score is mostly due to excellent performances on the images of one of the 3 contributors. All these images are very similar and less cluttered than the average of the unconstrained photos (actually they are all close-up of a juda's tree leaf). But still, it seems that this is the only method that have at least perform very well on these images.
A third important remark is that shape boundary analysis methods do not provide the best results on the scan images whereas they are usually considered as being state-of-the-art on such data. They all provide good classi cation scores between 48% and 56% but they are consistently outperformed by two more generic image retrieval approaches (as shown in Figure 12). The best score is achieved by INRIA's run using large-scale matching of local features with rigid geometrical models (inria imedia plantnet run1, 68% classi cation rate). This suggests that modeling leaves as part-based rigid and textured objects might be an interesting alternative to shape boundary approaches that do not characterize well margin details, ribs or limb texture. Second best score on scans is obtained by the run of SABANCI-OKAN (Sabanci-okan-run1) which uses a supervised classi cation approach based on support vector machine (SVM) and a combination of 3 global visual features. This suggests that combining shape boundary features with other color and shape textures is also a promising direction.
Global conclusions on the scan-like photos are quite di erent (Figure 13). The
best score is obtained by INRIA's run (inria imedia plantnet run2) purely based
on a global shape boundary feature (DFH [
In botany, species are described and categorized by morphological features,
frequently on leaves [
To evaluate which leaf organization makes more di cult the species identi cation, we averaged the performances over the runs of all participants for each kind of organization: simple or compound and palmately compound. Figure 18 presents the obtained scores for each kind of image. This graph con rms that species with simple leaves are on average easier. However, compound leaves can be subdivided into sub-categories describing how lea ets are organized: pinnately compound (with lea ets arranged along a the main axis called rachis), or palmately compound (with lea ets attached at one same basal point). Figure 19 presents the obtained scores and illustrates that species with palmately compound leaves can be easier than species with simple leaves, at least for the two tested species "Aesculus hippocastanum" and "Vitex agnus-castus". Another interesting point is the di erence on scores between scan and scan-like for pinnate compound leaves (there were no scan-like images of palmately compound leaves in the test image dataset). One explanation is that scan-like of compound leaves are often in relief involving disturbing shadows and a global shape .
In the case of simple leaves, to evaluate which kind of shape makes more di cult the species identi cation, we averaged the performances over the runs of all participants for each category of shape identi ed in the test image dataset: asymmetrical, elliptic, lanceolate, linear, lobate, obovate and orbicular. Figure 20 presents the obtained scores for each kinds of shape and image. Results show that species with an orbicular shape are easier to identify. This is con rmed in the graph 16 where three of the six orbicular species in the test image dataset ("acer campestre", "cercis siliquastrum", "corylus avellana") give good results (the best one for "corylus avellana").
One last important morphological feature studied by botanists is the margin type. In this case, to evaluate which type of margin makes more di cult the species identi cation, we averaged the performances over the runs of all participants for each category of margin identi ed in the test image dataset: "untoothed", "dentate", "serrate" and "crenate". Figure 21 presents the obtained scores for each kinds of margin and image. Results show that the margin type weakly a ect species identi cation, except maybe for species with crenate margin. To qualitatively assess which kind of images causes good results and which one makes all methods failed, we sorted all test pictures by the number of runs in which they were correctly identi ed. The obtained ranking con rms that scan images are much easier for the identi cation. 99% of 100-top ranked images are actually scans (with 11 to 17 successful runs). Figure 22 displays the 4 best identi ed images (with 17/20 successful runs). They all are very standard leaf images similar to the one found in books illustrations. On the other side, 260 images were not identi ed by any run with a majority of unconstrained photos (63 scans, 27 scan-like photos, 168 unconstrained photographs). The scans and scan-like photos belonging to this category of most di cult images are very interesting. As illustrated in Figure 23, most of them correspond to outlier leaves with defaults or unusual morphology (ill or torn leaves, missing lea ets, etc.). Figure 23 displays 8 of them. 6
This paper presented the overview and the results of ImageCLEF 2011 plant identi cation testbed. A challenging collaborative dataset of tree's leaf images was speci cally built for this evaluation and 8 participants coming from di erent countries submitted a total of 20 runs. A rst conclusion was that identi cation performances are close from mature when using scans or photos with uniform background but that unconstrained photos are still much more challenging. More data and evaluation are clearly required to progress on such data. Another important conclusion was that sate-of-the-art methods based on shape boundary analysis were not the best ones on leaf scans. Better performances were notably obtained with a local features matching technique usually more dedicated to the large-scale retrieval of rigid objects. On the other side, shape boundary analysis methods remain better on scan-like photos due to their better robustness to light re ections and shadow e ects. This suggests that combining shape boundary features with part-based rigid object models might be an interesting direction. Adding texture and color information also showed some improvements. On the contrary, using additional metadata such as geo-tags was not concluding on the evaluated dataset. Probably because the geographic spread of the data was limited.
This work was funded by the Agropolis fundation through the project Pl@ntNet (http://www.plantnet-project.org/) and the EU through the CHORUS+ Coordination action (http://avmediasearch.eu/) Fig. 20. Mean classi cation score per margin type averaged over all participant runs