Plant identi cation in an open-world (LifeCLEF 2016)

Abstract

The LifeCLEF plant identi cation challenge aims at evaluating plant identi cation methods and systems at a very large scale, close to the conditions of a real-world biodiversity monitoring scenario. The 2016-th edition was actually conducted on a set of more than 110K images illustrating 1000 plant species living in West Europe, built through a large-scale participatory sensing platform initiated in 2011 and which now involves tens of thousands of contributors. The main novelty over the previous years is that the identi cation task was evaluated as an open-set recognition problem, i.e. a problem in which the recognition system has to be robust to unknown and never seen categories. Beyond the brute-force classi cation across the known classes of the training set, the big challenge was thus to automatically reject the false positive classi cation hits that are caused by the unknown classes. This overview presents more precisely the resources and assessments of the challenge, summarizes the approaches and systems employed by the participating research groups, and provides an analysis of the main outcomes.

Keywords

• LifeCLEF
• plant
• leaves
• leaf
• ower
• fruit
• bark
• stem
• branch
• species
• retrieval
• images
• collection
• species identi cation
• citizen-science
• ne-grained classi cation
• evaluation
• benchmark

-

Image-based plant identi cation is the most promising solution towards bridging the botanical taxonomic gap, as illustrated by the proliferation of research work on the topic [ 7 ], [ 4 ], [ 13 ], [ 10 ], [ 1 ] as well as the emergence of dedicated mobile applications such as LeafSnap [ 14 ] or Pl@ntNet [ 12 ]. As promising as these applications are, their performance is still far from the requirements of a fully automated ecological surveillance scenario. Allowing the mass of citizens to produce accurate plant observations requires to equip them with much more effective identi cation tools. As an illustration, in 2015, 2,328,502 millions queries have been submitted by the users of the Pl@ntNet mobile apps but only less than 3% of them were nally shared and collaboratively validated. Allowing the exploitation of the unvalidated observations could scale up the world-wide collection of plant records by several orders of magnitude. Measuring and boosting the performance of automated identi cation tools is therefore crucial. As a rst step towards evaluating the feasibility of such an automated biodiversity monitoring paradigm, we created and shared a new testbed entirely composed of image search logs of the Pl@ntNet mobile application (contrary to the previous editions of the PlantCLEF benchmark that were based on explicitly shared and validated plant observations).

As a concrete scenario, we focused on the monitoring of invasive exotic plant species. These species represent today a major economic cost to our society (estimated at nearly 12 billion euros a year in Europe) and one of the main threats to biodiversity conservation [22]. This cost can even be more important at the country level, such as in China where it is evaluated to be about 15 billion US dollars annually [23], and more than 34 billion US dollars in the US [ 17 ]. The early detection of the appearance of these species, as well as the monitoring of changes in their distribution and phenology, are key elements to manage them, and reduce the cost of their management. The analysis of Pl@ntNet search logs can provide a highly valuable response to this problem because the presence of these species is highly correlated with that of humans (and thus to the density of data occurrences produced through the mobile application). 2 2.1

Dataset

Training dataset

For the training set, we provided the PlantCLEF 2015 dataset enriched with the ground truth annotations of the test images (that were kept secret during the 2015 campaign). More precisely, PlantCLEF 2015 dataset is composed of 113,205 pictures belonging to 41,794 observations of 1000 species of trees, herbs and ferns living in Western European regions. This data was collected by 8,960 distinct contributors. Each picture belongs to one and only one of the 7 types of views reported in the meta-data (entire plant, fruit, leaf, ower, stem, branch, leaf scan) and is associated with a single plant observation identi er allowing to link it with the other pictures of the same individual plant (observed the same day by the same person). An originality of the PlantCLEF dataset is that its social nature makes it close to the conditions of a real-world identi cation scenario: (i) images of the same species are coming from distinct plants living in distinct areas, (ii) pictures are taken by di erent users that might not used the same protocol of image acquisition, (iii) pictures are taken at di erent periods in the year. Each image of the dataset is associated with contextual meta-data (author, date, locality name, plant id) and social data (user ratings on image quality, collaboratively validated taxon name, vernacular name) provided in a structured xml le. The gps geo-localization and device settings are available only for some of the images. More precisely, each image is associated with the followings meta-data: { ObservationId: the plant observation ID from which several pictures can be associated { FileName { MediaId: id of the image { View Content: Entire or Branch or Flower or Fruit or Leaf or LeafScan or

Stem { ClassId: the class number ID that must be used as ground-truth. It is a numerical taxonomical number used by Tela Botanica { Species the species names (containing 3 parts: the Genus name, the speci c epithe, the author(s) who discovered or revised the name of the species) { Genus: the name of the Genus, one level above the Species in the taxonomical hierarchy used by Tela Botanica { Family: the name of the Family, two levels above the Species in the taxonomical hierarchy used by Tela Botanica { Date: (if available) the date when the plant was observed { Vote: the (round up) average of the user ratings of image quality { Location: (if available) locality name, a town most of the time { Latitude & Longitude: (if available) the GPS coordinates of the observation in the EXIF metadata, or if no GPS information were found in the EXIF, the GPS coordinates of the locality where the plant was observed (only for the towns of metropolitan France) { Author: name of the author of the picture { YearInCLEF: ImageCLEF2011, ImageCLEF2012, ImageCLEF2013, PlantCLEF2014, PlantCLEF2015 specifying when the image was integrated in the challenge { IndividualPlantId2014: the plant observation ID used last year during the LifeCLEF2014 plant task { ImageID2014: the image id.jpg used in 2014. 2.2

Test dataset

For the test set, we created a new annotated dataset based on the image queries that were submitted by authenticated users of the Pl@ntNet mobile application in 2015 (unauthenticated queries had to be removed for copyright issues). A fraction of that queries were already associated to a valid species name because they were explicitly shared by their authors and collaboratively revised. We included in the test set the 4633 ones that were associated to a species belonging to the 1000 species of the training set (populating the known classes). Remaining pictures were distributed to a pool of botanists in charge of manually annotating them either with a valid species name or with newly created tags of their choice (and shared between them). In the period of time devoted to this process, they were able to manually annotate 1821 pictures that were included in the test set. Therefore, 144 new tags were created to qualify the unknown classes such as for instance non-plant objects, legs or hands, UVO (Unidenti ed Vegetal Object), arti cial plants, cactaceae, mushrooms, animals, food, vegetables or more precise names of horticultural plants such as roses, geraniums, cus, etc. For privacy reasons, we had to remove all images tagged as people (about 1:1% of the tagged queries). Finally, to complete the number of test images belonging to unknown classes, we randomly selected a set of 1546 image queries that were associated to a valid species name that do not belong to the Western European ora (and thus, that do not belong to the 1000 species of the training set or to potentially highly similar species). In the end, the test set was composed of 8,000 pictures, 4633 labeled with one of the 1000 known classes of the training set, and 3367 labeled as new unknown classes. Among the 4633 images of known species, 366 were tagged as invasive according to a selected list of 26 potentially invasive species. This list was de ned by aggregating several sources (such as the National Botanical conservatory, and the Global Invasive Species Programme) and by computing the intersection with the 1000 species of the training set. 3

Task Description

Based on the previously described testbed, we conducted a system-oriented evaluation involving di erent research groups who downloaded the data and ran their system. To avoid participants tuning their algorithms on the invasive species scenario and keep our evaluation generalizable to other ones, we did not provide the list of species to be detected. Participants only knew that the targeted species were included in a larger set of 1000 species for which we provided the training set. Participants were also aware that (i) most of the test data does not belong to the targeted list of species (ii) a large fraction of them does not belong to the training set of the 1000 species, and (iii) a fraction of them might not even be plants. In essence, the task to be addressed is related to what is sometimes called open-set or open-world recognition problems [ 3,18 ], i.e. problems in which the recognition system has to be robust to unknown and never seen categories. Beyond the brute-force classi cation across the known classes of the training set, a big challenge is thus to automatically reject the false positive classi cation hits that are caused by the unknown classes (i.e. by the distractors). To measure this ability of the evaluated systems, each prediction had to be associated with a con dence score in p 2 [0; 1] quantifying the probability that this prediction is true (independently from the other predictions).

Each participating group was allowed to submit up to 4 runs built from different methods. Semi-supervised, interactive or crowdsourced approaches were allowed but compared independently from fully automatic methods. Any human assistance in the processing of the test queries had therefore to be signaled in the submitted runs.

Participants to the challenge were allowed to use external training data at the condition that the experiment is entirely re-producible, i.e. that the used external resource is clearly referenced and accessible to any other research group in the world, and, the additional resource does not contain any of the test observations. It was in particular strictly forbidden to crawl training data from the following domain names: http://ds.plantnet-project.org/ http://www.tela-botanica.org http://identify.plantnet-project.org http://publish.plantnet-project.org/ http://www.gbif.org/ 4

Metric

The metric used to evaluate the performance of the systems is the classi cation mean Average Precision, called hereinafter "mAP-open", considering each class ci of the training set as a query. More concretely, for each class ci, we extract from the run le all predictions with P redictedClassId = ci, rank them by decreasing probability p 2 [0; 1] and compute the Average Precision for that class. The mean is then computed across all classes. Distractors associated to high probability values (i.e. false alarms) are likely to highly degrade the mAP, it is thus crucial to try rejecting them. To evaluate more speci cally the targeted usage scenario (i.e. invasive species), a secondary mAP ("mAP-open-invasive") was computed by considering as queries only a subset of the species that belong to a black list of invasive species. 5

Participants and methods

94 research groups registered to LifeCLEF plant challenge 2016 and downloaded the dataset. Among this large raw audience, 8 research groups succeeded in submitting runs, i.e. les containing the predictions of the system(s) they ran. Details of the methods and systems used in the runs are further developed in the individual working notes of the participants (Blue eld [ 9 ], Sabanci [ 5 ], CMP [ 20 ], LIIR, Floristic [ 6 ], UM [ 15 ], QUT [ 16 ], BME [ 2 ]). Table 1 provides the results achieved by each run as well as a brief synthesis of the methods used in each of them. Complementary, the following paragraphs give a few more details about the methods and the overall strategy employed by each participant. Blue eld system, Japan, 4 runs, [ 9 ]: A VGGNet [ 19 ] based system with the addition of Spatial Pyramid Pooling, Parametric ReLU and unknown class rejection based on the minimal prediction score of training data (Run 1). Run 2 is the same as run 1 but with a slightly di erent rejection making use of a validation set. Run 3 and 4 are respectively the same as Run 1 and 2 but the scores of the images belonging to the same observation were summed and normalised. BME TMIT system, Hungary, 4 runs, [ 2 ]: This team attempted to combine three classi cation methods: (i) one based on dense SIFT features, sher vectors and SVM (Run 2), (ii) the second one based on AlexNet CNN (Run 1) and (iii), the last one based on a SVM trained on the meta-data. Run 3 corresponds to the combination of three classi ers (using a weighted average) and Run 4 added two rejection mechanisms to Run3 (a distance-based rejection for mopAePn- imnovpAaesPni-v-e cmloAsPedthe sher vectors and the minimal prediction score of training data for the CNN). CMP system, Czech Republic, 3 runs: This team built their system with the very deep residual CNN approach ResNet with 152 layers [ 11 ] which achieved the best results in both ILSVRC 2015 and COCO 2015 (Common Objects in Context) challenges last year. They added a fully-connected layer with 512 neurons on top of the network, right before softmax classi er with an maxout activation function [ 8 ]. They obtained thus a rst run (run 2) by using all the 2016 training dataset while they used only the 2015 training dataset in run 3. Run 1 achieved the best performances by using a bagging approach of 3 ResNet-152: the training dataset was divided into three folds, and each CNN was using a di erent fold for validation and the remaining two folds for ne tuning.

Floristic system, France, 3 runs, [ 6 ]: This participant used a modi ed GoogleNet architecture by adding batch normalisation and ReLU activation function instead of the PReLU ones (run 1). In Run 2, adaptive thresholds (one for each class) based on the prediction of the training images in the ne tuned CNN were estimated for removing too low prediction on test images. Run 3 used a visual similarity search for scaling down the initial CNN prediction when a test image gives inhomogeneous knns according to the metadata (organ tags, GPS, genus and family levels).

LIIR KUL system, Belgium, 3 runs: This team used a ensemble classi er of 5 ne-tuned models: one Ca eNet, one VGGNet16 and 3 GoogLeNet. They added 12k external training data from Oxford owers set, LeafSnap and trunk12 and attempted to exploit information in the metadata, mostly range maps from GPS coordinates comparing predictions with content tags. As a rejection criteria, they used a threshold on con dence of best prediction, one di erent threshold for each run (run 1: 0.25, run 2: 0.2, run 3: 0.15).

QUT system, Australia, 4 runs, [ 16 ]: This participant compared a standard CNN ne tuned approach based on GoogleNet (run 1) with a bagging approach "mixDCNN" (run 2) built on the top of 6 ne tuned GoogleNet on the 6 training subsets corresponding to the 6 distinct organs ("leaf" and "leafscan" training images are actually merged into one subset). Outputs are weighted by "occupation probabilities" which give for each CNN a con dence about their prediction. Run 3 merged the two approaches, run 4 too but with a threshold attempting to remove false positives.

Sabanci system, Turkey, 4 runs, [ 5 ]: This team used a CNN-based system with 2 main con gurations. Run 1: an ensemble of GoogleLeNet [ 21 ] and VGGNet [ 19 ] ne-tuned on LifeCLEF 2015 data (for recognizing the targeted species), as well as a second GoogleNet ne-tuned on the binary rejection problem (using 70k images of PlantCLEF2016 training set for the known class label and 70K external images from the ILSCVR dataset for the unknown class label). Run 2 is the same than Run 1 but without rejection. Run 3 is the same than Run 1 but with manual rejection of 90 obviously non plant images. UM system, Malaysia & UK, 4 runs: This team used a CNN system based on a VGGNet 16 layers. They modi ed the higher convolutional level in order to learn at the same time combinations of species and organs. VGGNet16 with dedicated and combined organ & species layers with seven organ labels: branch, entire, ower, fruit, leaf, leafscan and stem. 6

O

cial Results We report in Figure 1 the scores achieved by the 29 collected runs for the two o cial evaluation metrics (mAP-open and mAP-open-invasive). To better assess the impact of the distractors (i.e. the images in the test set belonging to unknown classes), we also report the mAP obtained when removing them (and denoted as mAP-closed). As a rst noticeable remark, the top-26 runs which performed the best were based on Convolutional Neural Networks (CNN). This de nitely con rms the supremacy of deep learning approaches over previous methods, in particular the one bases on hand-crafted features (such as BME TMIT Run 2). The di erent CNN-based systems mainly di ered in (i) the architecture of the used CNN, (ii) the way in which the rejection of the unknown classes was managed and (iii), various system design improvements such as classi er ensembles, bagging or observation-level pooling. An impressive mAP of 0:718 (for the targeted invasive species monitoring scenario) was achieved by the best system con guration of Blue eld (run 3). The gain achieved by this run is however more related to the use of the observation-level pooling (looking at Blue eld run 1 for comparison) than to a good rejection of the distractors. Comparing the metric mAP-open with mAP-closed, the gure actually shows that the presence of the unknown classes degrades the performance of all systems in a roughly similar way. This di culty of rejecting the unknown classes is con rmed by the very low di erence between the runs of the participants who experimented their system with or without rejection (e.g. Sabanci Run 1 vs. Run 2 or FlorisTic Run 1 vs. Run 2). On the other side, one can remark that all systems are quite robust to the presence of unknown classes since the drop in performance is not too high. Actually, as all the used CNNs were pre-trained on a large generalist data set beforehand (ImageNet), it is likely that they have learned a diverse enough set of visual patterns to avoid under ting. 7

Complementary Analysis: Impact of the degree of novelty

Within the conducted evaluation, the proportion of unknown classes in the test set was still reasonable (actually only 42%) because of the procedure used to create it. In a real mobile search data stream, the proportion of images belonging to unknown classes could actually be much higher. To simulate such a higher degree of novelty, we progressively down sampled the test images belonging to known classes and recomputed the mAP-open evaluation metric. Results of this experiment are provided in Figure. For clarity, we only reported the curves of the best systems (for various degrees of novelty). As a rst conclusion, the chart clearly shows that the degree of novelty in the test set has a strong in uence on the performance of all systems. Even when 25% of the queries still belong to a known class, none of the evaluated systems reach a mean average precision greater than 0:45 (to be compared to 0:83 in a closed world). Some systems do however better resist to the novelty than others. The performance of the best run of Blue eld on the o cial test set does for instance quickly degrade with higher novelty rates (despite the use of a rejection strategy). Looking at the best run of Sabanci, one can see that the use of a supervised rejection class is the most bene cial strategy for moderate novelty rates but then the performance also degrades for high rates. Interestingly, the comparison of LIIR KUL Run1 and LIIR KUL Run3 show that simply using a higher rejection threshold applied to the CNN probabilities is more bene cial in the context of high unknown class rates. Thus, we believe there is still rooms of improvements in the design of adaptive rejection methods that would allow to automatically adapt the strength of the rejection to the degree of novelty. This paper presented the overview and the results of the LifeCLEF 2016 plant identi cation challenge following the ve previous ones conducted within CLEF evaluation forum. The main novelty compared to the previous year was that the identi cation task was evaluated as an open-set recognition problem, i.e. a problem in which the recognition system has to be robust to unknown and never seen categories. The main conclusion was that CNNs appeared to be naturally quite robust to the presence of unknown classes in the test set but that none of the rejection methods additionally employed by the participants improved that robustness. Also, the proportion of novelty in the test was still moderate. We therefore conducted additional experiments showing that the preformance of CNNs is strongly a ected by higher rates of images belonging to unknown classes and that the problem is clearly still open. In the end, our study shows that there is still some room of improvement before being able to share automatically identi ed plant observations within international biodiversity platforms. The proportion of false positives would actually be too high for being acceptable for biologists. 22. Weber, E., Gut, D.: Assessing the risk of potentially invasive plant species in central europe. Journal for Nature Conservation 12(3), 171{179 (2004) 23. Weber, E., Sun, S.G., Li, B.: Invasive alien plants in china: diversity and ecological insights. Biological Invasions 10(8), 1411{1429 (2008)

References

1. Aptoula , E. , Yanikoglu , B. : Morphological features for leaf based plant recognition . In: Proc. IEEE Int. Conf. Image Process ., Melbourne , Australia. p. 7 ( 2013 )

2. Balint Pal Toth , Marton Toth , D.P. , Szucs , G.: Deep learning and svm classi cation for plant recognition in content-based large scale image retrieval . In: Working notes of CLEF 2016 conference ( 2016 )

3. Bendale , A. , Boult , T.E.: Towards open world recognition . CoRR ( 2014 ), http: //arxiv.org/abs/1412.5687

4. Cerutti , G. , Tougne , L. , Vacavant , A. , Coquin , D.: A parametric active polygon for leaf segmentation and shape estimation . In: International Symposium on Visual Computing . pp. 202 { 213 ( 2011 )

5. Ghazi , M.M. , Yanikoglu , B. , Aptoula , E.: Open-set plant identi cation using an ensemble of deep convolutional neural networks . In: Working notes of CLEF 2016 conference ( 2016 )

6. Goeau, H., Champ , J. , Joly , A. : Floristic participation at lifeclef 2016 plant identi cation task . In: Working notes of CLEF 2016 conference ( 2016 )

7. Goeau, H., Joly , A. , Selmi , S. , Bonnet , P. , Mouysset , E. , Joyeux , L. , Molino , J.F. , Birnbaum , P. , Bathelemy , D. , Boujemaa , N.: Visual-based plant species identi - cation from crowdsourced data . In: ACM conference on Multimedia . pp. 813 { 814 ( 2011 )

8. Goodfellow , I.J. , Warde-Farley , D. , Mirza , M. , Courville , A. , Bengio , Y.: Maxout Networks. ArXiv e-prints ( Feb 2013 )

9. Hang , S.T. , Tatsuma , A. , Aono , M. : Blue eld (kde tut) at lifeclef 2016 plant identi cation task . In: Working notes of CLEF 2016 conference ( 2016 )

10. Hazra , A. , Deb , K. , Kundu , S. , Hazra , P. , et al.: Shape oriented feature selection for tomato plant identi cation . International Journal of Computer Applications Technology and Research 2 ( 4 ), 449{meta ( 2013 )

11. He , K. , Zhang , X. , Ren , S. , Sun , J.: Deep residual learning for image recognition . arXiv preprint arXiv:1512.03385 ( 2015 )

12. Joly , A. , Goeau, H., Bonnet , P. , Bakic , V. , Barbe , J. , Selmi , S. , Yahiaoui , I. , Carre , J. , Mouysset , E. , Molino , J.F. , et al.: Interactive plant identi cation based on social image data . Ecological Informatics 23 , 22 { 34 ( 2014 )

13. Kebapci , H. , Yanikoglu , B. , Unal , G. : Plant image retrieval using color, shape and texture features . The Computer Journal 54 ( 9 ), 1475 { 1490 ( 2011 )

14. Kumar , N. , Belhumeur , P.N. , Biswas , A. , Jacobs , D.W. , Kress , W.J. , Lopez , I.C. , Soares , J.V.B. : Leafsnap: A computer vision system for automatic plant species identi cation . In: European Conference on Computer Vision . pp. 502 { 516 ( 2012 )

15. Lee , S.H. , Chang , Y.L. , Chan , C.S. , Remagnino , P. : Plant identi cation system based on a convolutional neural network for the lifeclef 2016 plant classi cation task . In: Working notes of CLEF 2016 conference ( 2016 )

16. McCool , C. , Ge , Z. , Corke , P. : Feature learning via mixtures of dcnns for negrained plant classi cation . In: Working notes of CLEF 2016 conference ( 2016 )

17. Pimentel , D. , Zuniga , R. , Morrison , D.: Update on the environmental and economic costs associated with alien-invasive species in the united states . Ecological economics 52(3) , 273 { 288 ( 2005 )

18. Scheirer , W.J. , Jain , L.P. , Boult , T.E.: Probability models for open set recognition . IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 36 ( 2014 )

19. Simonyan , K. , Zisserman , A. : Very deep convolutional networks for large-scale image recognition . CoRR abs/1409 .1556 ( 2014 )

20. Sulc , M. , Mishkin , D. , Matas , J.: Very deep residual networks with maxout for plant identi cation in the wild . In: Working notes of CLEF 2016 conference ( 2016 )

21. Szegedy , C. , Liu , W. , Jia , Y. , Sermanet , P. , Reed , S. , Anguelov , D. , Erhan , D. , Vanhoucke , V. , Rabinovich , A. : Going deeper with convolutions . In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . pp. 1 { 9 ( 2015 )