The LifeCLEF plant identi cation challenge aims at evaluating plant identi cation methods and systems at a very large scale, close to the conditions of a real-world biodiversity monitoring scenario. The 2015 evaluation was actually conducted on a set of more than 100K images illustrating 1000 plant species living in West Europe. The main originality of this dataset is that it was built through a large-scale participatory sensing plateform initiated in 2011 and which now involves tens of thousands of contributors. This overview presents more precisely the resources and assessments of the challenge, summarizes the approaches and systems employed by the participating research groups, and provides an analysis of the main outcomes.
Image-based approaches are nowadays considered to be one of the most
promising solution to help bridging the botanical taxonomic gap, as discussed in [
Noticeable progress in this way was achieved by several projects and apps
like LeafSnap4 [
Contrary to previous evaluations reported in the literature, the key objective of the PlantCLEF challenge has always been to build a realistic task close to real-world conditions (with many di erent contributors, cameras, areas, periods of the year, individual plants, etc.). This was initially achieved through a citizen science initiative that began 5 years ago, in the context of the Pl@ntNet project, in order to boost the production of plant images in close collaboration with the Tela Botanica social network. The evaluation dataset was enriched every year with new contributions and progressively diversi ed with di erent input feeds (annotation and cleaning of older data, contributions made through Pl@ntNet mobile applications). The plant task of LifeCLEF 2015 was directly in the continuity of this e ort. Main novelties compared to the last year were: { the doubling of the number species, i.e. 1000 species instead of 500 { the possibility to use external training data at the condition that the experiment is entirely re-producible 2
More precisely, PlantCLEF 2015 dataset is composed of 113,205 pictures
belonging to 41,794 observations of 1000 species of trees, herbs and ferns living in
Western European regions. This data was collected by 8,960 distinct
contributors. Each picture belongs to one and only one of the 7 types of views reported
in the meta-data (entire plant, fruit, leaf, ower, stem, branch, leaf scan) and
is associated with a single plant observation identi er allowing to link it with
the other pictures of the same individual plant (observed the same day by the
same person). It is noticeable that most image-based identi cation methods and
evaluation data proposed in the past were so far based on leaf images (e.g. in
[
An originality of PlantCLEF dataset is that its "social nature" makes it closer to the conditions of a real-world identi cation scenario: (i) images of the 8 http://www.imageclef.org/ 9 http://www.clef-initiative.eu/ same species are coming from distinct plants living in distinct areas, (ii) pictures are taken by di erent users that might not used the same protocol of image acquisition, (iii) pictures are taken at di erent periods in the year. Each image of the dataset is associated with contextual meta-data (author, date, locality name, plant id) and social data (user ratings on image quality, collaboratively validated taxon name, vernacular name) provided in a structured xml le. The gps geo-localization and device settings are available only for some of the images. Table 1 gives some examples of pictures with decreasing averaged users ratings for the di erent types of views. Note that the users of the specialized social network creating these ratings (Tela Botanica) are explicitly asked to rate the images according to their plant identi cation ability and their accordance to the pre-de ned acquisition protocol for each view type. This is not an aesthetic or general interest judgement as in most social image sharing sites.
To sum up each image is associated with the followings meta-data: { MediaId: id of the image { View Content: Branch or Entire or Flower or Fruit or Leaf or LeafScan or
Stem { ClassId: the class number ID that must be used as ground-truth. It is a numerical taxonomical number used by Tela Botanica { Species the species names (containing 3 parts: the Genus name, the Species name, the author(s) who discovered or revised the name of the species) { Genus: the name of the Genus, one level above the Species in the taxonomical hierarchy used by Tela Botanica { Family: the name of the Family, two levels above the Species in the taxonomical hierarchy used by Tela Botanica { Date: (if available) the date when the plant was observed, { Vote: the (round up) average of the user ratings of image quality { Location: (if available) locality name, most of the time a town { Latitude & Longitude: (if available) the GPS coordinates of the observation in the EXIF metadata, or, if no GPS information were found in the EXIF, the GPS coordinates of the locality where the plant was observed (only for the towns of metropolitan France) { Author: name of the author of the picture, { YearInCLEF: ImageCLEF2011, ImageCLEF2012, ImageCLEF2013, PlantCLEF2014, PlantCLEF2015 specifying when the image was integrated in the challenge { IndividualPlantId2014: the plant observation ID used last year during the LifeCLEF2014 plant task, { ImageID2014: the image id.jpg used in 2014. 3
The challenge was evaluated as a plant species retrieval task based on multiimage plant observation queries. The goal was to retrieve the correct plant species among the top results of a ranked list of species returned by the evaluated system. Contrary to previous plant identi cation benchmarks, queries were not de ned as single images but as plant observations, meaning a set of one to several images depicting the same individual plant, observed by the same person, the same day, with the same device. Each image of a query observation is associated with a single view type (entire plant, branch, leaf, fruit, ower, stem or leaf scan) and with contextual meta-data (data, location, author).
The whole PlantCLEF dataset was split in two parts, one for training (and/or indexing) and one for testing. All observations with pictures used in the previous plant identi cation tasks were directly integrated in the training dataset. Then for the new observations and pictures, in order to guarantee that most of the time each species contained more images in the training dataset than in the test dataset, we used a constrained random rule for putting with priority observations with more distinct organs and views in the training dataset. The test set was built by choosing 1/2 of the observations of each species with this constrained random rule, whereas the remaining observations were kept in the reference training set. Thus, 1/3 of the pictures are in the test dataset (see Table 1 for more detailed stats). The xml les containing the meta-data of the query images were purged so as to erase the taxon names (the ground truth) and the image quality ratings (that would not be available at query stage in a real-world application). Metadata of the observations in the training set are kept unaltered.
Total Branch Entire Flower Fruit Leaf LeafScan Stem Train 91,759 8,130 16,235 28,225 7,720 13,367 5,476 12,605 Test 21,446 2,088 2,983 6,113 8,327 1,423 696 935
All 113,205 10,218 19,218 34,438 16,047 14,790 6,172 13,540
As a novelty this year, participants to the challenge were allowed to use external training data at the condition that (i) the experiment is entirely reproducible, i.e. that the used external resource is clearly referenced and accessible to any other research group in the world, (ii) participants submit at least one run without external training data so that we can study the contribution of such resources, (iii) the additional resource does not contain any of the test observations. It was in particular strictly forbidden to crawl training data from the following domain names: http://ds.plantnet-project.org/ http://www.tela-botanica.org http://identify.plantnet-project.org http://publish.plantnet-project.org/ http://www.gbif.org/
In practice, each candidate system was evaluated through the submission of a run, i.e. a le containing a set of ranked lists of species (each list corresponding to one query observation and being sorted according to the con dence score of the system in the suggested species). Each participating group was allowed to submit up to 4 runs built from di erent methods. The metric used to evaluate the submitted runs is an extension of the mean reciprocal rank [29] classically used in information retrieval. The di erence is that it is based on a two-stage averaging rather than a at averaging such as:
S = 1 XU 1 XPu 1
U u=1 Pu p=1 ru;p (1) where U is the number of users (within the test set), Pu the number of individual plants observed by the u-th user (within the test set), ru;p is the rank of the correct species within the ranked list of species returned by the evaluated system (for the p-th observation of the u-th user). Note that if the correct species does not appear in the returned list, its rank ru;p is considered as in nite. Overall, the proposed metric allows compensating the long-tail distribution e ects occurring in social data. In most social networks, few people actually produce huge quantities of data whereas a vast majority of users (the long tail) produce much less data. If, for instance, only one person did collect an important percentage of the images, the classical mean reciprocal rank over a random set of queries would be strongly in uenced by the images of that user to the detriment of the users who only contributed with few pictures. This is a problem for several reasons: (i) the persons who produce the more data are usually the most expert ones but not the most representative of the potential users of the automatic identi cation tools. (ii) The large number of the images they produce makes the classi cation of their observations easier because they tend to follow the same protocol for all their observations (same device, same position of the plant in the images, etc.) (iii) The images they produce are also usually of better quality so that their classi cation is even easier.
A secondary metric was used to evaluate complementary (but not mandatory) runs providing species prediction at the image level (and not at the observation level). The evaluation metric in that is expressed as:
S = 1 XU 1 XPu 1
Nu;p X
1 U u=1 Pu p=1 Nu;p n=1 ru;p;n (2) where U is the number of users, Pu the number of individual plants observed by the u-th user, Nu;p the number of pictures of the p-th plant observation of the u-th user, ru;p;n is the rank of the correct species within the ranked list of images returned by the evaluated system. 4
123 research groups worldwide registered to LifeCLEF plant challenge 2015 in
order to download the dataset. Among this large raw audience, 7 research groups
succeeded in submitting runs on time and 6 of them submitted a technical report
describing in details their system. Participants were mainly academics,
specialized in computer vision, machine learning and multimedia information retrieval.
We list below the participants and give a brief overview of the techniques used in
their runs. We remind here that LifeCLEF benchmark is a system-oriented
evaluation and not a deep or ne evaluation of the underlying algorithms. Readers
interested by the scienti c and technical details of any of these methods should
refer to the LifeCLEF 2015 working notes of each participant (referenced below):
EcoUAN (1 run) [
INRIA-ZENITH (3 runs) [
MICA (3 runs) [
The feature extraction step in itself is based on kernel descriptors, namely a gradient kernel for the leafscan, fruit, ower, leaf, entire and branch view type, and a LBP kernel for the stem view type. The late fusion of the SVM classi ers of each view type is based on the sum of the inverse rank position in each ranked list of species. The second run, (run 2) di ers from the rst one in that it uses complementary HSV histogram features for the ower and entire view types. The third run (Run 3) di ers from Run 2 in the fact that it uses an alternative fusion strategy based on a weighted probability combination.
QUT RV (3 runs), [
Sabanki-Okan (3 runs) [
SNUMED (4 runs), [
UAIC (1 run), [], Romania. This participant used a content-based image search engine (Lucene Image Retrieval Library []) to retrieve the most similar images of each query image and then apply a two-stage instance-based classi er returning the top-10 most populated species for each image and then the top-10 most populated across all the images of a query observation. 5 5.1
The following graphic 2 and table 2 show the scores obtained on the main task (i.e. at the observation level). It is noticeable that the top-9 runs which perform the best were based on the GoogLeNet [28] convolutional neural network which clearly con rms the supremacy of deep learning approaches over hand-crafted features as well as the bene t of training deeper architecture thanks to the improved utilization of the computing resources inside the network. The score's deviations between these 9 runs are however still interesting (actually 10 points of mAP between the worst and the best one). A rst source of improvement was the fusion strategy allowing to combine the classi cation results at the image
Fig. 2. O cial results of the LifeCLEF 2014 Plant Identi cation Task.
level into classi cation scores at the observation level. In this regard, the best
performing algorithm was a SoftMax function [
The second best approach that did not rely on deep learning (i.e. INRIA ZENITH
run 2) was to use the Fisher Vector model [
The following graphic 3 presents the scores obtained by the additional imagelevel runs provided by the participants. In order to evaluate the bene t of the combination of the test images from the same observation, the graphic compares the pairs of run les on images and on observations produced with the same method.
Basically, for each method, we can observe an improvement by combining the di erent views of the same plant observation. This has to be related to the fact that observing di erent plant organs is the current practice of botanists, who most of the time can't identify a species with only one picture on only one organ. However, we can say that the improvement are not so much high: we guess that there is a room of improvement here, basically with more images and may be with new methods of fusions dealing with this speci c problem of multi-image and multi-organ problem.
The following graphic 4 below show the detailed scores obtained for each type of organs. Remember that we use a speci c metric weighted by authors and plants, and not by sub-categories, explaining why the score on images not detailed is not the mean of the 7 scores of these sub-categories.
Like during the previous Plant Identi cation Task, this detailed analysis shows that LeafScan and the Flower views are far away the most e ective for identifying plant species, followed by the Fruit view, the Leaf view, the Entire view and the Branch view. On the other side, the stem view (or bark view when speaking about trees) is the less informative one particularly when noticing that the number of species represented in that view, and thus the confusion risk, was lower than for the other organs. Interestingly, the hand-crafted visual features of the MICA group perform very well on the Leaf Scan category, with an identi cation score better than most of the runs based on the GoogLeNet CNN. This shows the relevance of the leaf normalization strategy they used as well as the e ectiveness of the gradient kernel for this type of view. For professional use cases in which taking the time to scan the leaf might not be an issue, this method is a serious alternative to the use of the CNN which requires much more resources and training data.
This paper presented the overview and the results of the LifeCLEF 2015 plant identi cation challenge following the four previous ones conducted within CLEF evaluation forum. The main novelty compared to the previous year was the possibility to use external training data in addition to the speci c training set provided within the testbed. The rst objective of this novelty was clearly to encourage the deployment of transfer learning methods, and in particular of deep convolutional neural networks in order to evaluate their ability to identify plant species at a large-scale. In this regard, the results show that such transfer learning approaches clearly outperform previous approaches based on hand-crafted visual features, aggregation models and linear classi ers. The results are as impressive as a 0; 784 identi cation score on the ower category. Now, the second objective of opening the training data was to encourage the integration of new plant data (and not only of the popular generlist dataset ImageNet), particularly for populating the long tail of the less populated species which is an important challenge in terms of biodiversity. Unfortunately, none of the participants addressed this issue. More generally, we believe that collecting and building appropriate training data is becoming one of the most central problem for solving de nitely the taxonomic gap problem. 28. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. arXiv preprint arXiv:1409.4842 (2014) 29. Voorhees, E.M., et al.: The trec-8 question answering track report. In: TREC. vol. 99, pp. 77{82 (1999)