The ImageCLEFs plant identi cation task provides a testbed for a system-oriented evaluation of plant identi cation about 250 species trees and herbaceous plants based on detailed views of leaves, owers, fruits, stems and bark or some entire views of the plants. Two types of image content are considered: SheetAsBackgroud which contains only leaves in a front of a generally white uniform background, and NaturalBackground which contains the 5 kinds of detailed views with unconstrained conditions, directly photographed on the plant. The main originality of this data is that it was speci cally built through a citizen sciences initiative conducted by Tela Botanica, a French social network of amateur and expert botanists. This makes the task closer to the conditions of a realworld application. This overview presents more precisely the resources and assessments of task, summarizes the retrieval approaches employed by the participating groups, and provides an analysis of the main evaluation results. With a total of twelve groups from nine countries and with a total of thirty three runs submitted, involving distinct and original methods, this third year task con rms Image Retrieval community interest for biodiversity and botany, and highlights further challenging studies in plant identi cation.
Convergence of multidisciplinary research is a key to answer profound challenges of humanity related to health, biodiversity or sustainable energy. The integration of life sciences and computer sciences has a major role to play towards managing and analyzing cross-disciplinary scienti c data at a global scale. More speci cally, building accurate knowledge of the identity, geographic distribution and uses of plants is essential if agricultural development is to be successful and biodiversity is to be conserved. Unfortunately, such basic information is often only partially available for professional stakeholders, scientists and citizens, and often incomplete for ecosystems that possess the highest plant diversity. A noticeable consequence, expressed as the taxonomic gap, is that identifying plant species is usually impossible for the general public, and often a di cult task for professionals, such as farmers or wood exploiters and even for the botanists themselves. The only way to overcome this problem is to speed up the collection and integration of raw observation data, while simultaneously providing to potential users an easy and e cient access to this botanical knowledge. In this context, content-based visual identi cation of plant's images is considered as one of the most promising solution to help bridging the taxonomic gap. Evaluating recent advances of the Image Retrieval community on this challenging task is therefore an important issue.
This paper presents the plant identi cation task that was organized for the third year running within ImageCLEF6 [10] dedicated to the system-oriented evaluation of visual based plant identi cation. Like previous year, the task is more related to a retrieval task instead of a pure classi cation task in order to consider a ranked list of retrieved species rather than a single brute determination. Visual content was being the main available information but with additional information including contextual meta-data (author, date, locality name and geotag, names at di erent taxonomic ranks) and some EXIF data. Each year try to take to the next level the challenge to a more realistic scenario by covering progressively one entire ora at the scale of one wide region like France. After two years focused exclusively on leaves mainly from Mediterranean tree species, the task focused this year on 250 species of herbs and trees species living in France with di erent views or organs of plants: photographs of owers, fruits, barks, leaves and the entire view of the plants. Finally, it was two types of content which were considered: a SheetAsBackground category containing scans and scan-like photographs of leaves in a front of a generally white uniform white background, and a NaturalBackground with most of the time a cluttered natural background of the 5 types of organs. The main originality of this data is that it was speci cally built through a citizen sciences initiative conducted by Tela Botanica7, a French social network of amateur and expert botanists. This makes the task closer to the conditions of a real-world application: (i) organs of the same species are coming from distinct plants living in distinct areas and with at distinct growing stages, (ii) pictures and scans are taken by di erent users that might not used the same protocol to collect the leaves and/or acquire the images, (iii) pictures and scans are taken at di erent periods in the year.
Building e ective computer vision and machine learning techniques is not the only side of the taxonomic gap problem. Speeding-up the collection of raw observation data is clearly another crucial one. The most promising approach in that way is to build real-world collaborative systems allowing any user to enrich the global visual botanical knowledge [15]. To build the evaluation data of ImageCLEF plant identi cation task, we therefore set up a citizen science project around the identi cation of common woody species covering the Metropolitan French territory. This was done in collaboration with Tela Botanica social network and with researchers specialized in computational botany.
Technically, images and associated tags were collected through a crowd-sourcing web applications [15], [13] and were all validated by expert botanists. Several cycles of such collaborative data collection and taxonomical validation occurred. Scans of leaves were the rst type of pictures collected thanks to the work of active contributors from Tela Botanica since the summer 2009. The idea of collecting only scans of leaves rst was to initialize training data with limited noisy background and to focus on plant variability rather than mixed plant and view conditions variability. This allowed to collect a rst dataset of 2228 scans over 55 species. A rst public crowd-sourcing web application8 was then opened in October 2010 and additional data were collected up to March 2011. The new collected images were either scans, or photographs with uniform background (referred as scan-like photos), or unconstrained photographs with natural background. It involved besides 15 new species from the previous set of 55 species. In April 2011 a new version of the web application has opened 9 and the acquisition protocol was extended to 4 more types of views with a natural background mentioned below, and focusing to the same limited set of species. During the last two years, members from Tela Botanica contribute regularly every month on more and more species, the nal ambition being to cover the entire vascular French ora (around 6000 species) with numerous pictures of di erent plant organs, with numerous plant observations spread all over France at di erent growing stages photographed by a crowd of photographers, introducing slowly over the months great visual and morphological variabilities. However, for each year task, we decided to limit the number of species in the task by keeping only the most populated ones in terms of images and plant observations. This is why like the rst year we decided to focus again only on leaves during the ImageCLEF 2012 Plant Identi cation task with a number of 125 species, because at the time of the task we didn't collected su ciently pictures of complementary organs. This year, we decided to propose to add these complementary views while we added to the previous dataset Pl@ntLeaves 125 new more species more focusing on herbaceous plants than threes in order to cover more diversity of the French
project.org/en/base/plantscan
ora. Complementary views concerning owers, fruits, stems and entire views
associated to previous species and previous plant observations yet contained in
the 2012 dataset were also added. Finally, the Pl@ntView dataset used within
ImageCLEF2013 plant task contained 26077 images collected by 327 distinct
contributors: 11031 for the SheetAsBackground category and 15046 for the
NaturalBackground (in more details 16% of leaves, 18% of owers, 8% of fruits, 8%
of stems and 8% of entire plant). The gure 2.1 gives some examples illustrating
the type of views, but illustrating also the fact that a species does not contain
systematically at least one image for each organ.
{ IndividualPlantID : plant observation identi er
{ Date: date and time of plant observation
{ Type: SheetAsBackground or NaturalBackground
{ Content : Flower, Fruit ), Leaf, Stem or Entire
{ Taxon: full taxon name according the botanical database[
Subclass, Superorder, Order, Family, Genus, Species) { ClassId : species identi er { VernacularNames : English common name { Author name of the author of the picture { Organization name of the organization of the author { Locality locality name (a district or a country division or a region) { GPSLocality GPS coordinates of the locality.
Concerning the locality information, note that sometimes the GPS can be very imprecise when the locality was not mentioned: in this case we used the GPS coordinates of the district or the country division or a region, according to the level of information available. Metadata is stored in independent xml les, one for each image. Additional but partial meta-data information can be found in the image's EXIF, and might include the camera or the scanner model, the image resolution and dimension, the optical parameters, the white balance, the light measures, etc. 2.3
A crucial added-value of this collection over older ones used in the literature
(such as Swedish [22], ICL [
Flower There is a great diversity within owers and it is an intensive subject of studies by botanists since the ower is often the key for identify a species. Flowers of the dataset can be categorized according to the color (see gure 2), the symmetry (see gure 3), the number of petals (see gure 4) and the size (see gure 5). Most of the time one species is associated to one category, but there are exceptions like in gure 6 where for one same species the owers can have di erent colors. Besides this rst categorisation, botanists studied the Brown
White
Green
Rose
Blue
Yellow in orescence, i.e. the internal structure of the ower and the organisation of the owers on a plant. Species from a same taxonomical group generally share a same organization, and thus a same visual appearance. Figure 7 gives all the type of in orescence contained in the dataset. Some type of in orescence can be very noticeable and very typical of a group of species. However, at the opposite some very distinct groups of species in the taxonomical hierarchy can have a very distinct visual appearance, but sharing a same type of in orescence. Fruit The fruit is the transformation of the ower and it can be also categorized into distinct types. The gure 8 shows the great diversity of type of fruits that small
middle are represented through the 250 species of the Pl@ntView dataset. The di erent modes of dissemination gives a second complementary and interesting way to show the visual diversity of the fruits in the dataset. Indeed, a same mode of dissemination of (even very) distinct species involves generally some same morphological features. For instance, for the endozoochory dissemination (seed dispersal by animals) the fruits are generally colored for attracting birds for instance.
Stem The stem is generally a di cult plant sub-part for identifying a species, maybe because the visual information is mainly expressed with the texture, less Achene
Berry
Capsule
Cone
Drupe Folicle
Legume
Samara
Silique by the color and or the shape. Age of the plant is second di culty for analysing the stem, more precisely for the trees and theirs barks. Through the collaborative process, the dataset contains for numerous species, di erent plants at di erent ages and ll partially the wide diversity of the barks. The gure 10 shows a representative example for the species Robinia pseudo-acacia with young and old trees: more the tree is young more it has some thorns as a strategy of defence. very young young adult old Entire The entire view is maybe the most di cult view for identifying with precision a species, because this kind of view generally does not contain su ciently information, and because a same species can have a very di erent general appearance depending to the geographical and climatic conditions. However, more the plant is small (young or intrinsically small), more the "useful" organs for identi cation are visible. The gure 11 shows one big tree of Magnolia grandiora L. where it is very di cult or even impossible for identifying the pant if we look the entire view. The second plant is a small herbaceous species of Gentiana pneumonanthe L. where we can see that the ower is very visible on the entire view for identi cation.
Species
Flower
Magnolia grandi ora L.
Gentiana pneumonanthe L.
The precise goal of the task was to retrieve the correct species among the top k species of a ranked list of returned species, one list for each image of a test dataset. Participants received a rst training set of annotated images in order to explore di erent techniques and train their system. Six weeks later participants received the test set containing images without species labels, but with the view type, organ type, author, organization and plant identi er tags. Then, 2 months later, participants were allowed to submit up to 4 run les, most of the time related to variations of the same method. A particular attention was paid when splitting the data into training and test subsets to avoid any bias. Several pictures in the dataset might actually depict the same individual plant (or neighboring plants) observed in the same conditions (same person, day, device, lightening conditions, etc.). Randomly splitting images in a nave way would therefore favor having such near-duplicate images in both the training and the test subsets, making the recognition much more easy. To avoid this bias, we therefore performed our random split at the observation level rather than at the image level thanks to associated metadata (observation id when available, author, date, etc.). Numerous images of the di erent views were automatically integrated in the training dataset since the associated plant observations were yet integrated last year task with the leaves. The training data nally resulted in 20985 images while the test data resulted in 5092 images. Detailed statistics of the composition of the training and test data are provided in Table 1.
SheetAsBackground TTreastin NaturalBackground TTreastin According to similar concerns, the primary metric used to evaluate the submitted runs uses a two-stage average of raw image scores, one at the observation level (i.e. we compute the average score of all images belonging to the same observed plant), and one at the user level (i.e. we average the scores of the observations of a given user). A at mean would actually have introduce some new bias with regard to a real world identi cation system. Indeed, as the dataset was built in a collaborative manner, it appears that few contributors often provide much more pictures than many other contributors who provided few (long tail distribution). Since we want to evaluate the ability of a system to provide correct answers to any user, we rather measure the mean of the average classi cation score per author. Furthermore, some authors sometimes provided many pictures of the same individual plant (to enrich training data with less e orts). Since we want to evaluate the ability of a system to provide the correct answer based on a single plant observation, we also decided to average the classi cation rate on each individual plant. The raw image score itself is computed for each test image as the inverse of the rank of the correct species in the list of retrieved species. More formally, our primary metric was de ned as the following average score S: S = 1 XU 1 XPu 1 NXu;p su;p;n U u=1 Pu p=1 Nu;p n=1 (1) U : number of users (who have at least one image in the test data) Pu : number of individual plants observed by the u-th user Nu;p : number of pictures taken from the p-th plant observed by the u-th user su;p;n : score between 1 and 0 equals to the inverse of the rank of the correct species for the n-th picture taken from the p-th plant observed by the u-th user It is important to notice that while making the task more realistic, the normalized classi cation score also makes it more di cult. Indeed, it works as if a bias was introduced between the statistics of the training data and the one of the test data. It highlights the fact that bias-robust machine learning and computer vision methods should be preferred to train such real-world collaborative data. Finally, to isolate and evaluate the impact of the image acquisition type (SheetAsBackground, NaturalBackground, a normalized classi cation score S was computed for each type separately. Participants were therefore allowed to train distinct classi ers, use di erent training subsets or use distinct methods for each data type. 4
With 12 nalist groups coming from all around the world over 9 countries and 33 submitted runs, the 2013 edition of the task con rmed its increasing attractiveness (respectively 10 and 11 groups crossed the nish line in 2011 and 2012) although its complexity was higher (with heterogeneous view types). Participants were mainly academics, specialized in computer vision and multimedia information retrieval. We list below the participants and give a brief overview of the techniques they used in their runs. We remind here that ImageCLEF benchmark is a system-oriented evaluation and not a deep or ne evaluation of the underlying algorithms. Readers interested by the scienti c and technical details of any of these methods should refer to the ImageCLEF 2013 working notes of each participant (referenced below): AGSPPR (3 runs) [25], China. AGSPPR team focused their work on the SheetAsBackground category and submitted 3 runs using distinct visual features and approaches: a global shape feature (i.e. the leafs area length of major axis and length of minor axis), SIFT features (run 2), and an extension of the CENTRIST approach (CENsus Transform hISTogram []) called SPACT designed for reducing the number of descriptors with a PCA algorithm (run 3). For run 1 and 3, they used a multiclass Support Vector Machine (SVM) classi er with a radial basis kernel function, while they used a pure matching approach for the 2nd run. DBIS (4 runs) [20], Germany. DBIS team runs are based on global visual features and a multiclass SVM classi er. These participants experimented numerous (early) combinations of about thirty global features, in order to select the best combination for each type of view. The selected features are predominantly based on color (Auto Color Correlogram, Border Interior Color, Color Histogram, Color Layout, Color Structure, EdgeHistogram, tamura, CEDD, FCTH). They also experimented several SVM parameters in order to boost theirs results. I3S (2 runs) [16], France. I3S team used a popular approach in the eld of image classi cation: they extracted SIFT features in order to produce Bag of visual Words (BoW), one BoW vector by picture, from a 1000 visual words dictionary (built with kMeans clustering algorithm). BoW's are then exploited to train species model with SVM classi ers, one for each species and type of view. Species prediction of test images are produced with a one-against-all procedure. INRIA PLANTNET (4 runs) [8], France. For the SheetAsBackground category, after a basic Otsu segmentation, INRIA team used multiscale triangle representations, alone and combined with other shape-based descriptors (Directional Fragment Histogram and shape parameters). In addition, multi-image queries were considered, by using images belonging to the same plant observation in order to boost the results. For the NaturalBackground category, all the 4 submitted runs are based on local features (SURF, Fourier, rotation invariant Local Binary Patterns, Edge Orientation Histogram, weighted RGB and HSV histograms). The last one uses a multi-cue Fisher Vector embedding [] with a one-against-all multiclass SVM classi er. The three rst runs use Hamming embedding and hash-based approximate knn matching: all local features are hashed, indexed and searched in separate indices (one for each each type of view and type of feature) and retrieved images are scored by the number of matches. A two-stage late fusion scheme is then apply to combine the image response lists of the di erent modalities and of the di erent types of view. Metadata was also successfully used (in run 2), in particular the date for the ower category and the plant observation identi ers (to share the query images of the same plant). LAPI (1 run) [9], Romania. LAPI team proposed to exploit a complex approach for image description based on contour extraction, curve partitioning and abstraction. They suggest that their approach is a "structural alternative" to the prevailing gradient-based features (e.g. SIFT). Contrary to other teams, they considered a more di cult task by automatically recognizing the view type before recognizing the plant species. They used a classical Linear Discriminant Analysis (LDA) as classi er for both the view type recognition and the species prediction.
LIRIS REVES (2 runs) [11], France. ReVes team used the same supervised model-based segmentation strategy than the one they used during the 2012 leaforiented campaign and tried to extend it to the other types of view (although it was more di cult to build a priori shape models of that organs). They used a late fusion approach to combine the decisions of the classi ers of each modality as well as to combine the multiple images of a given individual plant when this occurred in the query set. They nally attempted to use the geo-tags available in the metadata by interpolating them thanks to external climatic data. MICA (3 runs) [17], Vietnam. MICA team experimented 3 distinct approaches. Run 1 used a GIST descriptor with a k-nearest neighbors rule on all types of view. Run2 was based on the same approach but with additional color and texture features for the Flower and Entire types of view. Run3 used a Bag of visual Words (BoW) approach based on SURF local features and an "un-sharp masking" pre-processing step to lter some background information. Classi cation was achieved through a multi-class SVM.
NLAB UTOKYO (3 runs) [18], Japan. NLAB participant focused his work on visual features learning for building accurate image descriptions. A set of local features, mostly SIFT variations and a Self Similarity descriptor, were densely extracted in each picture according to a regular grid and then "augmented" with a supervised polynomial embedding technique taking into account neighboring local features. Further, these locally embedded and augmented features were encoded into a global Fisher Vector representation which allows an accurate classi cation with any linear classi er. In this work, linear logistic regression models were used. An independent classi er was trained for each raw descriptor and a late-fusion based an average log-likelihood of posterior probabilities was used to merge independent classi er results.
SABANCI-OKAN (1 run) [24], Turkey. This team submitted only one run using distinct features for the two categories. For the SheetAsBackground category, an automatic segmentation was performed using edge preserving morphological simpli cation by means of area attribute lters, followed by an adaptive threshold. Then, a variety of shape and texture features were extracted (the same than the ones used during the 2012 campaign). For the NaturalBackground category, a set of global features was extracted: HSV color auto-correlograms, weigthed-saturation hue histogram and other texture descriptors, depending on the considered organ. For the Flower, Fruit and Entire view types only color features were used, while for the Stem view type, texture features were used after a segmentation preprocessing step. The dates provided in the metadata were also exploited for the three rst view types (that are likely to be more time dependent). Classi cation was performed through independent SVM classi ers, one for each view type.
SCG USP (4 runs), Brazil. This team submitted one run with a fully automatic approach (run 1) and three other ones runs involving human assistance for a background/foreground segmentation. More precisely, training pictures were segmented with the semi-supervised Grabcut algorithm, while test images were manually segmented. Then, numerous features were extracted: Gabor, LBP, fractal, geometrical features. The nal classi cation step was performed with a LDA classi er, except for the 3rd run where a SVM classi er was used. Only the 4th run tried to train independent classi ers (i.e. one for each view type). UIAC (3 runs) [21], Romania. Unlike the other groups, UAIC explored the strategy of integrating additional external training data to boost their performances. They actually crawled 507 additional pictures from Wikimedia Commons with relevant annotations. And this con rms the di culty of collecting dense and accurate data speci c to a given ora. From the technological point of view, they used the LIRe (Lucene Image Retrieval) engine and, after preliminary tests, they selected the Joint Composite Descriptor (JCD). The LIRe engines gives for each test image a list of training images where a candidate species potentially appears several times. Thus, they used 3 distinct approaches of combination in order to obtain a single score for each species: a max operator, a normalized sum, and a naive Bayes classi er. The results were further re ned and ranked based on GPS metadata, author names and organization tags, assuming that certain authors and organizations would have a greater interest in certain plant species.
Table 2 attempts to summarize the methods used at di erent stages (feature, classi cation, subset selection,...) in order to highlight the main choices of participants. This table should be used in next section on result analysis, in order to see if there are some common techniques which tend to lead to good performances. 5
We present here an overview of the o cial results of the task and discuss the main ndings. SheetAsBackground: Table 3 and gure 12 present the identi cation scores of the 33 submitted runs for the SheetAsBackground category. As expected, results on scans and scan-like images of leaves are generally higher than the photographs of the NaturalBackground category. The Sabanci Okan teams reached the highest scores of 0.607 with an approach mainly centered on leaf shape boundary features. Using contour-based approaches is con rmed to be an e ective strategy by the good performances of the Inria PlantNet group and the Liris team. Interestingly, one team which used a more generic approach in computer vision (the NLabUTokyo team working with Fisher Vector representations), also obtained very good identi cation scores whereas they used exactly the same technique for the NaturalBackground category. Other teams who attempted to use non-contour based approaches obtained signi cantly lower scores. Compared to the raw identi cation scores obtained during the 2012 campaign, we only noticed a slight increase (0.607 vs 0.58 for scans and 0.55 for scan-like). But it is important to remark that the task itself was more complex in several aspects: (i) scans and scan-like pictures have been merged in a single category (ii) the number of species was increased from 115 (scans) or 83 (scan-like) to 126 this year (iii) test images themselves were more complex (weaker lighting conditions, more shadows, more old dried leaves and less uniform background. NaturalBackground: Table 4 and gure 13 present the identi cation scores of the 33 submitted runs for the NaturalBackground category. As expected, results are signi cantly lower than the SheetAsBackground category due to the noisy backgrounds and clutter e ects. The highest scores, obtained by the NLabUTokyo team, reached equivalent values than the 2012 task, but without any human assistance in the work ow, contrary to last year best runs that involved semi-automatic segmentation mechanisms. This is even more remarkable . . t t a a c c b b
i-d tan g In lp n i r l e , a w e u o t - - , r , - - - - - - , rrtseeeapn itehdFHD rtse tsnSFRU+ i,rreuEHOrecyvnu+ PCA+ ,lrsooabG iirtteccecyn ittrseexuh ,)eSFRU+ ,teoppnnO iItrySSMirseechFV fseeaabd lirrsecoodp irseogundh lrrrseoogam,rtagom lirtca lrau edw eamiop ,oFVanC iton co ,tenm radn i,rten -ISFT liisamignd+ rtcooun ,i)rrrce fr/odun -tcoau isehuh eeogm ragmrtsgo ,reE .TH h s g n r y S th rca ab n i a e Hi ia b p k , w ts L o ts r me e B t o p ik n ab :s c a l- Go e
Rit d l ,l,IogaSbFPCTTA lltrrreooooouCC il,lirrroooooCCH ,ltrtrtcoouSuuC ,,raauFECCDDmIFBT+OW :liltteeehupMilltseeoavynnu itrrsscoapndh l:itrrsraaauH ,iteeghdPBwtrrtecooaxnu iiitrtoagannn ifrtegggaanu l,tseeavSFRU :treeceeaphndmireee/okunnZmm i,,lt(r)s1oo2nuC r(rreagoo2unwm(r)o3nuwi,IrseegndSFTC ,,lf--IIseSSFFTHV lileeooaypnbdmmro:iftrteeeaovyh ,rttr(seeeoxuuuF ,rrseecogogakdb .ragoml:ltrrcooaaSuHV ii-tttrseegaoahdnu ,,l,frtrcoaaabLPB S A r o T S S t s N L c p B w S H G t Bd S + t S t t t N w G r a M e V
V n o i s u f e t a l , n o i s s e r g e R c i t s i g o L n i
O
s , e M is ru lt ) s ( u id a ilraadb rcoapph fronMl,isgom lssaSVM rcaoapph iltsscaSV trsecoV lissayn a a ru ia ilc g lu re A ith ign y om tu in msh t
b n n wM tcah lse loy lla aMehmtsa to la lu ion rnu itcao ssSV len .)3M rekn ,pFB itsn lea scn igan lied iirscm i lca re & tc (R ga ceS ifsou ean app D s i k 1 n s t i
r a le ts M e n rg e O re ea n i i V n re
n M tch a ll iF a n i aL lta ro is n o i t a c i s s a l c d e s a b e c n a t s i d e v i a P I a t a d a t e M s t I P P o t
h a G G t z b n n ll ll u o u S ( r A A
y m g u e s h t ro cao t a r x r s a p s
p s ma a l c ith ian i
t w s l
e u n y
m io a s b ll fu e -a
v s t i
v is a l n 1 ltu ro M se ,r V r o S
t r e a a g r e a e n imop iL p i h s n o i r t o la t e p r i r e c s p e a
h D s e , t i m s r o o p f
s m n o a
r C t t e n c i a o r J t n o i t a t n e m g e s r o l o c
M C O H given that their approach was purely based on the visual content contrary to the second best run of the task (by Inria Plantnet team) which did make use of the date and the plant identi er tags. The contribution of using the metadata can be observed by comparing this run with the second best one of that team (Inria Plantnet run 1) that was purely based on visual data. Overall, the runs of these two teams represent the head of the pack with six (or even seven) runs clearly outperforming the other runs.
Detailed results: Figures 17, 14, 18,19,16, and gure 15 display the identi cation scores for each view type separately (still for the NaturalBackground category). It shows that the average identi cation scores are signi cantly boosted by the good performances obtained on the ower images. Most techniques used
Run name run lename Entire Flower Fruit Leaf Stem Nat. NlabUTokyo Run 3 run3 0,297 0,472 0,311 0,275 0,253 0,393 Inria PlantNet Run 2 plantnet inria run2 0,274 0,494 0,26 0,272 0,24 0,385 NlabUTokyo Run 2 run2 0,273 0,484 0,259 0,273 0,285 0,371 Inria PlantNet Run 1 plantnet inria run1 0,254 0,437 0,249 0,24 0,211 0,353 NlabUTokyo Run 1 all siftcopphsv cca 0,236 0,423 0,209 0,269 0,276 0,341 Inria PlantNet Run 3 plantnet inria run3 0,216 0,421 0,238 0,195 0,176 0,325 Inria PlantNet Run 4 plantnet inria run4 0,15 0,327 0,137 0,165 0,171 0,245 Sabanci Okan Run 1 Sabanci-Okan-Run1 0,174 0,223 0,194 0,049 0,106 0,181 DBIS Run 2 DBISForMaT run2 0,102 0,264 0,082 0,034 0,095 0,159 train2012 svm Scan12
Photo4 - 1 4 DBIS Run 3 DBISForMaT run3 cross- 0,109 0,256 0,079 0,035 0,095 0,158 val2013 svm feature4 con
g60 1 2 3 DBIS Run 4 DBISForMaT run4 cross- 0,152 0,206 0,104 0,027 0,042 0,141 val2013 svm feature5 con
g80 Photo14 1 3 3 UAIC Run 4 run wiki max 1 0,09 0,136 0,12 0,08 0,128 0,127 DBIS Run 1 DBISForMaT run1 0,067 0,168 0,1 0,052 0,103 0,12 train2012 svm Scan4
Photo2 1 2 3 UAIC Run 1 run wiki sum 3 0,089 0,109 0,132 0,093 0,104 0,119 UAIC Run 2 run author10 GSP10 lire80 0,092 0,105 0,127 0,096 0,11 0,117 Liris ReVeS Run 2 LirisReVeS run2 0,026 0,102 0,082 0,161 0,166 0,092 Liris ReVeS Run 1 LirisReVeS run1 0,021 0,098 0,081 0,151 0,153 0,089
UAIC Run 3 run lire naivebayes 0,068 0,055 0,111 0,049 0,102 0,081 Vicomtech Run 1 outputCLEFTestMean 0,095 0,117 0 0 0,1 0,081 Vicomtech Run 2 outputCLEFTestMax 0,091 0,116 0 0 0,094 0,08 LAPI Run 1 LAPI run1 0,026 0,073 0,025 0,084 0,043 0,058 Mica Run 2 MICA-run2 0,016 0,086 0,048 0,014 0,014 0,053 Mica Run 3 Run3 0,016 0,013 0,048 0,11 0,014 0,042 SCG USP Run 3 SCG USP run3 0,017 0,025 0,042 0,047 0,054 0,03 I3S Run 1 new 100 0,017 0,023 0,041 0,038 0,025 0,026 I3S Run 2 new2 100 0,017 0,023 0,041 0,038 0,025 0,026 SCG USP Run 1 SCG USP run1 0,02 0,026 0,027 0,02 0,037 0,025 SCG USP Run 2 SCG USP run2 0,027 0,029 0,02 0,018 0,019 0,025
Mica Run 1 MICA-run1 0,016 0,013 0,048 0,014 0,014 0,023 SCG USP Run 4 SCG USP run4 0,019 0,014 0,022 0,031 0,021 0,017 AgSPPR Run 1 AgSPPR run1 0 0 0 0 0 0 AgSPPR Run 2 AgSPPR run2 0 0 0 0 0 0 AgSPPR Run 3 AgSPPR run3 0 0 0 0 0 0 Table 4. Normalized and detailed cores for each run for the NaturalBackground. HA=humanly assisted, Auto=full automatic. by most participants were signi cantly more accurate on that image type. This con rms the botanical expertise on the important role of owers in the identication mechanisms as this organ was historically used as the primary one to distinguish species between each others (for owering plants of course). This is good news that computer vision methods go in the same direction.
Besides the Flower category, there was no clear second best organ or view type. Stem images provided surprisingly good results relatively to the botanist knowhow. Bark morphology is actually not considered as a the most accessible identi cation criterion for non-specialists. The texture itself is for instance highly correlated with the age of the plant. Identi cation results on the Entire plant views are also rather surprising regarding their higher complexity and variability. Overall, an important remark is that the ranking of the runs did not change much from an organ to another one, fostering the idea that generic methods might solve heterogeneous ne-grained classi cation problems.
Metadata: Regarding the use of metadata, two runs (Sabanci Okan run 1 and Inria Plantnet run 2) exploited successfully the date for improving the results. Using the observation date complementary to the visual content was a simple and e cient way to obtain a gain of up to 4 points on the Flower view type (thanks to the relatively short ourishing period of many species). Inria Plantnet run number 2 exploited also the observation identi er tag in order combine the results of the query images coming from the same plant. But since the whole NaturalBackground test dataset did contain only a few plant observations with multiple images, the impact of using this tag is much lower than the impact of using the date eld. On the other side, this multiple-image strategy was much more bene cial for the SheetAsBackground category as a signi cant number of plants were represented by several images (leaves used for scans are actually more likely to be collected in mass from the same plant). The runs of Inria
Plantnet and Liris ReVes teams exploited successfully this information for the SheetAsBackground category.
As the previous years, several teams, like Liris Reves or UIAC, attempted to exploit the geo-localization information in order to re ne candidate species list.
In particular, Liris teams proposed to use the raw GPS data of the training set complementary to external environmental data in order to interpolate them and build coarse species distribution maps. These maps where used afterwards to prune the species returned by the visual search and keep only the most probable ones. Unfortunately, the results do not show a great improvement over the purely visual runs of these teams. This can be explained by the fact that the database doesn't yet contain enough numerous and dense observations to build an accurate geographic repartition of the species. Also, the geo-localization data is partially noisy due to heterogeneous precisions in the localization (points, cities, departments).
Finally, the UAIC team tried to explore author and organization tags assuming that an authors or a group of author from a same organizations have more interest on speci c groups of species. However the results did not show clearly some gain by using these user informations. None of teams neither explored the hierarchical taxonomy structure, nor the common names, which could be source of improvements.
External data: UAIC explored the strategy of integrating additional external training data to boost their performances. They focused their search on Wikimedia Commons which contain more and more reliable contents related to species of life in general. They managed to crawl 507 additional pictures which is ne but clearly not su cient to make a strong di erence compared to tens of thousands of images in the training set. This con rms the di culty of collecting dense and accurate data, speci c to a given ora, and with relevant annotations (like organ and view type).
Impact of the global training strategy: Whereas some of the teams used a classical leave-one-image-out strategy cross-validate their training, some other ones used a more sophisticated leave-one-plant-out strategy that is closer to the real-world problem evaluated by the task. This second option seems to have take bene ts to the teams using it, namely Sabanci Okan, NLabUTokyo, Liris ReVes and Inria Plantnet. Indeed, they all mentioned that they did not split images from the same individual plant in the training set, in order to avoid over tting problems (images of the same plant can actually be very similar). Back to purely visual approaches: I3S and MICA teams experimented, at least through one run, a standard approach in image categorization with SIFT or SURF features, visual bag of words (BoW) representations and SVM multiclass. MICA team obtained intermediate scores contrary to I3S team. Explanations for this di erence, can be that MICA use of preprocessing step for unsharp mask of Leaf images from SheetAsBackground and NaturalBackground (see gure12 MICA run 3 and I3S runs where scores are very di erent on Leaf ). The more recent approach in image categorization based on Fisher Vector (FV) representations, which can be see as an extension of BoW, showed a clear gain regarding to the BoW runs as we can see with the 3 NLabUTokyo runs and the Plantnet Inria run 4 on the NaturalBackground category. NLabUTokyo obtained the best scores, maybe because they capture local spatial information by enriching dense local descriptors with polynomials, contrary to the Inria Plantnet run where patch are extracted around Harris corners and descriptors are directly embedded in Fisher Vector representations. Moreover NLabUTokyo used also a late fusion where classi ers are trained independently for each descriptor, while Inria Plantnet run 4 used an intermediate fusion by concatenating FV representations from the di erent type of descriptors. Besides, late fusion is also are shared approach for the best runs of Inria Plantnet team.
The fact that NLabUTokyo runs obtained almost the best results for all subcategories, con rms the idea that FV representation is a successful generic approach in spite of di erent type of visual contents. It is important to notice that the run 2 obtained close scores to the best one (run 3) without considering subcategory tags, which show that views tags are may be not essential for succeeding the task. This is an important conclusion since image tagging is an heavy process with users. However, this generic approach is not the most e cient on SheetAsBackground compared to contour based approaches dedicated to leaf shape analysis. This may show that generic approaches like the one used by the NLabUTokyo team is dependent to the background, and that through a dense grid patch extraction, their system learn a contextual information o the background. In particular this can be observed with the Fruit subcategory where NLabUTokyo run 3 outperforms other methods: fruits are generally small elements in the pictures di cult to capture, and also these organs appear often after the leafage, thus we can suppose these cluttered backgrounds have a non negligible contribution in the species contribution.
Like in the previous working note with the leaf [14], we try here to present some complementary results by analysing some morphological features, more precisely on the sexual organs which are the ower and the fruit. Beyond the methods used, we try to analyse which feature, which kind of ower or fruit is intrinsically more di cult than the others. The gure 20 shows detailed results by category of color. The graph to the left shows the proportion of image test by color used for computing and displaying the graph to the right. Results in this second graph are sorted in a decreasing order of mean performance over all the submitted run (except the AgSPPR's runs which not really participate to the NaturalBackground ). One can notice that the two most represented colors, the yellow and the white (more than 50% of the database) are not the ones which enables the best results, maybe rigthly because there is more species and thus more confusions and ambiguity. Green owers, which is not so rare, seem to be the most di cult color maybe because it expresses no color in a sense and thus it is a di cult information to capture, notably if the owers hidden with a background of leafage or grass. Similarly, the brown owers may also be very confused with barks for trees where owers appear before leaves, which can explain the performances on this color. The gure 21 attempts to give a Fig. 20. Detailed results by ower color. The graph to the left represent the proportions of the tested images used for computing the detailed results in the graph to the right. This second graph gives the minimum, the median, the mean and the maximum scores over all the submitted runs for each ower color. complementary perspective of results about ower according to the in orescence structure as mentioned in section 2.4. First of all we can see that the in orescence categories are strongly unbalanced in terms of image number. Thus the best scores obtained by Cyathium, Panicle, Umbel and to a lesser extent Umbel, Captitulum and Solitary are no very representative for making some relevant conclusions on these types of in orescences. Concerning the most representative ones the Cyme seems to be the type where the runs performed the best on average. The gure 22 gives the results according to the fruit type. As for the in orescence, unfortunately, some types of fruits are not well represented in the dataset like Silique, Cone, Folicle and to a lesser extent Legume. Even if this last type of fruit is not so well represented, it is interesting to note that all teams seem to have the best scores on Legume because the associated species are from a very large family of plants called Fabaceae which is spread all other the world. Then, it is di cult to highlight one type of fruit over the others, because there is always one best method at the same score around 0.4. We have just to note that the Samara (like "helicopters" from maple for instance) seem to be clearly the most di cult type of fruit, even when we look at the best run (not over 0.2).
This paper presented the overview and the results of ImageCLEF 2013 plant
identi cation testbed following the two previous one in 2011 and 2012. The
number of participants increased from 8 to 12 groups showing an increasing
interest in applying multimedia search technologies to environmental challenges.
This year the challenge climb one step by considering multiple type of view and
organs of plants while the number of species increased from 125 to 250 species
and plant observations densely covered the French territory. Results are
encouraging by scaling state-of-the-art plant recognition technologies to a real-world
application with thousands of species might still be a di cult task. Despite
increasing di culties on SheetAsBackground images and the number of species,
scores are high and show that leaf analyses is still the best way for identifying a
plant, even if collecting new scans is more di cult than shooting photographs.
Performances obtained on NaturalBackground category of unconstrained
pictures of plant organs are very encouraging especially for the Flower when we
look detailed results and where best methods can compete with scores
SheetAsBackground. It corroborates a well-know usage of botanists for identifying plants
and this is good news in a sense that computer vision methods go in the same
direction. An interesting conclusion is that these good results on
NaturalBackground images are obtained with generic visual classi cation technique without
any speci city related to plants. With the emergence of more and more plant
identi cation apps [
This work was funded by the Agropolis fundation through the project Pl@ntNet (http://www.plantnet-project.org/) and the EU through the CHORUS+ Coordination action (http://avmediasearch.eu/). Thanks to all participants. Thanks to Jennifer Carre, Violette Roche and all contributors from Tela Botanica. Thanks to Souheil Selmi from Inria and Julien Barbe from Amap for their help. 7. Agarwal, G., Belhumeur, P., Feiner, S., Jacobs, D., Kress, J.W., R. Ramamoorthi, N.B., Dixit, N., Ling, H., Mahajan, D., Russell, R., Shirdhonkar, S., Sunkavalli, K., White, S.: First steps toward an electronic eld guide for plants. Taxon 55, 597{610 (2006) 8. Bakic, V., Mouine, S., Ouertani-Litayem, S., Verroust-Blondet, A., Yahiaoui, I., Goeau, H., Joly, A.: Inria's participation at imageclef 2013 plant identi cation task. In: Working notes of CLEF 2013 conference (2013) 9. C., R., L., F., C., V.: Has an image classi cation approach any chance at all (in plant classi cation)?... In: Working notes of CLEF 2013 conference (2013) 10. Caputo, B., Muller, H., Thomee, B., Villegas, M., Paredes, R., Zellhofer, D., Goeau, H., Joly, A., Bonnet, P., Gomez, J.M., Varea, I.G., Cazorla, M.: ImageCLEF 2013: the vision, the data and the open challenges. In: Proc CLEF 2013. LNCS (2013) 11. Cerutti, G., Tougne, L., Sacca, C., Joliveau, T., Mazagol, P.O., Coquin, D., Vacavant, A.: Late information fusion for multi-modality plant species identi cation.
In: Working notes of CLEF 2013 conference (2013) 12. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large
Scale Hierarchical Image Database. In: CVPR09 13. Goeau, H., Bonnet, P., Barbe, J., Bakic, V., Joly, A., Molino, J.F., Barthelemy, D., Boujemaa, N.: Multi-organ plant identi cation. In: Proceedings of the 1st ACM international workshop on Multimedia analysis for ecological data. MAED '12 (2012) 14. Goeau, H., Bonnet, P., Joly, A., Yahiaoui, Itheri Boujemaa, N., Barthelemy, D., Molino, J.F.: The ImageCLEF 2012 plant identi cation task. In: ImageCLEF (2012) 15. Goeau, H., Joly, A., Selmi, S., Bonnet, P., Mouysset, E., Joyeux, L.: Visual-based plant species identi cation from crowdsourced data. In: Proceedings of ACM Multimedia 2011 (2011) 16. Issolah, M., Lingrand, D., Precioso, F.: Sift, bow architecture and one-against-all support vector machines. In: Working notes of CLEF 2013 conference (2013) 17. Le, T.L., Pham, N.H.: Imageclef2013 plant identi cation mica. In: Working notes of CLEF 2013 conference (2013) 18. Nakayama, H.: Nlab-utokyo at imageclef 2013 plant identi cation task. In: Working notes of CLEF 2013 conference (2013) 19. Nilsback, M.E., Zisserman, A.: A visual vocabulary for ower classi cation. In:
CVPR06 20. Saretz, S., Bottcher, T.: Btu dbis' at imageclef2013 plant identi cation task. In:
Working notes of CLEF 2013 conference (2013) 21. Serba, C., Siriteanu, A., Gheorghiu, C., Iftene, A., Alboaie, L., Breaban, M.: Combining image retrieval, metadata processing and naive bayes classi cation at plant identi cation 2013. In: Working notes of CLEF 2013 conference (2013) 22. Soderkvist, O.J.O.: Computer Vision Classi cation of Leaves from Swedish Trees.
Master's thesis, Linkoping University, SE-581 83 Linkoping, Sweden (September 2001), liTH-ISY-EX-3132 23. Wu, S.G., Bao, F.S., Xu, E.Y., xuan Wang, Y., fan Chang, Y., liang Xiang, Q.: A leaf recognition algorithm for plant classi cation using probabilistic neural network (2007) 24. Yanikoglu, B., Aptoula, E., Yildiran, S.T.: Sabanci-okan system at imageclef 2013 plant identi cation competition. In: Working notes of CLEF 2013 conference (2013) 25. Zhang, L., Cai, C.: Agsppr at imageclef 2013 plant identi cation task. In: Working notes of CLEF 2013 conference (2013)