Overview of LifeCLEF Plant Identi cation task 2020

Abstract

Automated identi cation of plants has improved considerably thanks to the recent progress in deep learning and the availability of training data with more and more photos in the eld. However, this profusion of data only concerns a few tens of thousands of species, mostly located in North America and Western Europe, much less in the richest regions in terms of biodiversity such as tropical countries. On the other hand, for several centuries, botanists have collected, catalogued and systematically stored plant specimens in herbaria, particularly in tropical regions, and the recent e orts by the biodiversity informatics community made it possible to put millions of digitized sheets online. The LifeCLEF 2020 Plant Identi cation challenge (or "PlantCLEF 2020") was designed to evaluate to what extent automated identi cation on the ora of data de cient regions can be improved by the use of herbarium collections. It is based on a dataset of about 1,000 species mainly focused on the South America's Guiana Shield, an area known to have one of the greatest diversity of plants in the world. The challenge was evaluated as a cross-domain classi cation task where the training set consist of several hundred thousand herbarium sheets and few thousand of photos to enable learning a mapping between the two domains. The test set was exclusively composed of photos in the eld. This paper presents the resources and assessments of the conducted evaluation, summarizes the approaches and systems employed by the participating research groups, and provides an analysis of the main outcomes.

Keywords

• LifeCLEF
• PlantCLEF
• plant
• domain adaptation
• cross-domain classi cation
• tropical ora
• Amazon rainforest
• Guiana Shield
• species identication
• ne-grained classi cation
• evaluation
• benchmark

-

Automated identi cation of plants and animals has improved considerably in the last few years. In the scope of the LifeCLEF 2017 Plant Identi cation challenge [ 14 ] in particular, impressive identi cation performance were measured thanks to recent deep learning models (e.g. up to 90 % classi cation accuracy over 10K species), and it was shown in [ 13 ] that automated systems are nowadays not so far from the human expertise. However, these conclusions are valid for species that are mostly living in Europe and North America. Therefore, the LifeCLEF 2019 Plant identi cation challenge was focused on tropical countries, where there are typically much less of collected observations and images and where the ora is much more di cult to identify for human experts [ 7 ]. In the meantime, biodiversity informatics initiatives such as iDigBio5 or eReColNat6 made available online millions of digitized herbarium sheets collected over several centuries and conserved in many natural history museums over the world. During centuries, botanists have collected, catalogued and systematically stored plant specimens in herbaria. These physical specimens are used to study the variability of species, their phylogenetic relationship, their evolution, or phenological trends. One of the key step in the work ow of botanists and taxonomists is to nd the herbarium sheets that correspond to a new specimen observed in the eld. This task requires a high level of expertise and can be very tedious. Developing automated tools to facilitate this work is thus of crucial importance. In the continuity of the PlantCLEF challenges organized in previous years [4{12], the LifeCLEF 2020 Plant identi cation challenge presented in this paper was designed to evaluate to what extend automated identi cation on the ora of data de cient regions can be improved by the use of natural history collections of herbarium sheets. Many species in tropical countries are not easily accessible, resulting in a very limited number of photos collected in the eld, while several hundred or even several thousand of herbarium sheets have been collected over the centuries. Herbaria collections represent potentially a large pool of data to train species prediction models, but they also induces a much more di cult problem usually referred as a cross domain classi cation task. Indeed, a plant photographed in the eld may have a very di erent visual appearance than its dried version placed on a herbarium sheet (as illustrated in Figure 1). 2 2.1

Datasets and task description

Training set

The conducted study was based on a newly created dataset of 997 species mainly focused on the Guiana shield and the Northern Amazon rainforest (see Figure 2), an area known to have one of the greatest diversity of plants and animals in the world. The dataset contains 321,270 herbarium sheets (see Table1 for detailed information). About 12% were collected in French Guyana and hosted in the "Herbier IRD de Guyane" (IRD Herbarium of French Guyana). These herbarium sheets were digitized in the context of the e-ReColNat6 project. The 5 http://portal.idigbio.org/portal/search 6 https://explore.recolnat.org/search/botanique/type=index remaining herbarium sheets come from the iDigBio5 portal (the US National Resource for Advancing Digitization of Biodiversity Collections).

In order to enable learning a mapping between the two domains (i.e. between the "source" domain of herbarium sheets and the "target" domain of eld photos), a relatively smaller set of 6,316 photos in the eld was provided additionally to the large herbarium sheets dataset. About 62 % of them also come from he iDigBio portal and were acquired by various photographers related to numerous institutes and national museums that share their data in iDigBio. Besides, two highly trusted experts of the French Guyana ora, Marie-Francoise Prevost "Fanchon" [ 3 ] and Jean-Francois Molino7 provided the remaining eld photos that were divided between the training set and the test set.

A valuable asset of the training set is that a set of 354 plant observations are provided with both herbarium sheets and eld photos for the same individual plant. This potentially allows a more precise mapping between the two domains (see previous Figure 1 as an example).

It should also be noted that about half of the species in the training set (495 to be precise) is only represented by herbarium sheets and therefore it is not possible to learn to recognize them directly from eld photos. 2.2

Test set

The test set was composed of 3,186 photos in the eld related to 638 plant observations (about 5 pictures per plants on average). To avoid bias related to similar pictures coming from neighboring plants in the same observation site, we ensured that all observations of a given species by a given collector were either in the training set or in the test set but never spread over the two sets. For instance, for the observations of J.F. Molino, the 166 species in the test set are di erent from the 125 species in the training set.

Most importantly, plant species in the test set were selected according to the number of eld photos illustrating them in the training set. As it can be 7 https://scholar.google.fr/citations?user=xZXYc4kAAAAJ&hl=fr observed in Figure 3 (a), the priority was given to species with few or no eld pictures at all. Such a choice may seem drastic, making the task extremely di cult, but the underlying idea was to encourage and promote methods that are as generic as possible, capable of transferring knowledge between the two domains, even without any examples in the target domain for some classes. The second motivation of this choice, was to impose a mapping between herbarium and eld photos and avoid that classical methods based on CNNs perform well because of an abundance of eld photos in the training set rather than the use of herbarium sheets above all. 2.3

External training sets

Participants to the evaluation were allowed to use complementary training data (e.g. for pre-training purposes) but on the condition that (i) the experiment is entirely reproducible, i.e. that the used external resource is clearly referenced and accessible to any other research group, (ii) the use of external training data or not is clearly mentioned for each evaluated method, and (iii) the additional resource does not contain any of the test observations. External training data was thus allowed but participants had to provide at least one submission that used only the training data provided this year. The goal of the task was to identify the correct species of the 638 plant of the test set. For every plant, the evaluated systems had to return a list of species, ranked without ex-aequo. Each participating group was allowed to submit up to 10 run les built from di erent methods or systems (a run le is a formatted text le containing the species predictions for all test items).

The main evaluation measure for the challenge was the Mean Reciprocal Rank (MRR), which is de ned as the mean of the multiplicative inverse of the rank of the correct answer: where Q is the number of plant observations and rankq is the predicted rank of the true label for the qth plant observation.

A second evaluation measure was again the MRR but computed on a subset of observations of di cult species that are rarely photographed in the eld. The species were chosen based on the most comprehensive estimates possible from di erent data sources (iDigBio, GBIF, Encyclopedia of Life, Bing and Google Image search engines, that were actually provided by the organizers or some participants of previous editions of PlantCLEF and ExpertCLEF challenges). It is therefore a more challenging metric because it focuses on the species which impose a mapping between herbarium and eld photos. Figure 3 (b) revises the previous Figure 3 (a) according to the considered external data sources and shows that many plant observations in the di cult test subset are related to species estimated to have less than 10 eld photos.

Course of the challenge: The training data was publicly shared in early February 2020 through the AICrowd platform8. Any research team wishing to participate in the evaluation could register on the platform and download the data. The test data was shared in mid-April but without the species labels, which were kept secret. Each team could then submit up to 10 submissions corresponding to di erent methods or di erent settings of the same method. A submission (also called a run) takes the form of a csv le containing the predictions of the method being evaluated for all observations in the test set. For each submission, the calculation of the evaluation metrics is then done automatically and visible to the participant. Once, the submission phase was closed (mid-June), the participants could also see the evaluation metric values of the other participants. As 8 https://www.aicrowd.com/challenges/lifeclef-2020-plant a last important step, each participant was asked to provide a working note, i.e. a detailed technical report containing all technical information required to reproduce the results of all submissions. All LifeCLEF working notes are reviewed by at least two members of LifeCLEF organizing committee to ensure a su cient level of quality and reproducibility. 3

Participants and methods 68 participants registered for the PlantCLEF challenge 2020 and downloaded the data set, and 7 research groups succeeded in submitting a total of 49 runs, i.e.

les containing the predictions of the system(s) they ran. Details of the methods are developed in the individual working notes of most of the participants (LU [ 21 ], ITCR PlantNet [ 20 ], Neuon AI [ 2 ] and SSN [ 16 ]). The other teams did not provide a detailed description of their systems, but some informal descriptions were sometimes provided in the metadata associated with the submissions and partially contributed to the comments below.

LU, Lehigh University, USA, 10 runs [21]: this team used a Partial Domain

Adaptation (PDA) approach corresponding to the scenario where target categories are only a subset of source categories to promote positive transfer. They rst extracted deep features from a pre-trained NASNetLarge model [ 22 ] and nd the shared categories between the two domains. They then develop an novel Adversarial Consistent Learning (ACL) approach through an uni ed deep architecture which combine a source domain classi cation loss, an adversarial loss and a feature consistent loss. The adversarial loss helps to learn domain-invariant features while the feature consistent loss aims to preserve the ne-grained feature transition between the two domains.

Neuon AI, Malaysia, 7 runs [ 2 ]: this team developed a triplet loss network [ 1, 18 ] between herbarium sheets and eld images, trained to maximize the embedding distances of di erent species while minimizing the embedding distances of same species. First, two InceptionV4 CNNs [ 19 ] were ne-tuned, one exclusively with the herbarium sheets related to the 997 classes, and an other one with more than 1 million pictures related to 10k classes from the PlantCLEF 2017 dataset [ 5 ]. The networks are then fused in a nal embedding layer trained to optimize the distances between two embeddings based on triplet loss, but only on the subset of 435 classes that contain both herbarium sheets and eld photos in the PlantCLEF 2020 training set. Then, each class is associated to a single embedding computed as the average of embeddings from random herbarium sheets of the class. For inference, a plant observation is then associated to a single embedding computed as the average of the embeddings from all eld photos of the observation. The Cosine similarity is used as a distance metric between the embeddings of all the herbarium classes and the embedding of the tested eld observation. It is then transformed with Inverse Distance Weighting into probabilities for ranking the classes. The best results was reached with an ensemble of 3 triplet loss networks, without any frozen layers and many data augmentations techniques on both sides (training and test pictures).

ITCR - PlantNet, Costa Rica & France, 10 runs, [20]: this team based

most of its runs on a Few Shot Adversarial Domain Adaptation approach [ 15 ] (FSADA) where the purpose is to learn a domain agnostic feature space while preserving the discriminative ability of the features for performing the species classi cation task. First, a ResNet50 is netuned in the herbarium sheets only. Then, the encoder part of the ResNet50, without the classi er last layers, is frozen and used to extract features on herbarium sheets or eld photos. Then, given random pairs of extracted features, a discriminator is trained to distinguish 4 categories: (1) di erent domains and di erent classes, (2) di erent domains and same class, (3) same domain and di erent classes, (4) same domain and same classes. Finally, during a last stage, the encoder, the discriminator and the classier are trained together. Domain adaptation is achieved once the discriminator is not able to distinguish samples from categories (1) and (2) and categories (3) and (4), when the discriminator is not able to tell which was the original domain. This participant attempted several improvements based on a jigsaw self-supervision technique or/and the use of the taxonomic information provided in the metadata (with multi-task classi ers and multiple discriminators). The best result was obtained with an ensemble of several variations of FSADA models, while the best single model was using 3 taxonomic levels (species, genus, family) and external datasets (PlantCLEF 2019 [ 7 ] and GBIF [ 17 ]).

SSN College of Engineering, India, 2 runs, [16]: this team used a classical

CNN approach based on a ResNet50 which didn't give very good results given the limited number of eld photos in the training set. It seems they didn't use any external data.

The 3 other teams did not provide an extended description of their system. According to the description provided in the submission system, the UWB team (3 runs) used a classical CNN approach based on ResNet18 with various combinations of external data [ 17 ]. Unfortunately, at the time of writing, there is not enough information about the submissions from the "To Be" (10 runs) and the "Domain" teams (7 runs). 4

Results

We report in Figure 4 and Table 4 the performances achieved by the 49 evaluated runs. Figure 5 reorganizes the results according to the second MRR metric focusing on the most di cult species.

The main outcomes we can derive from that results are the following ones:

A very di cult task even with the most advanced deep learning tech

niques. The best MRR value obtained across all evaluated methods was equal to 0.18. This is quite low compared to the MRR value measured within more classical plant identi cation benchmarks, e.g. MRR=0.92 in LifeCLEF 2017 Plant Identi cation challenge [ 14 ] and MRR=0.36 LifeCLEF 2019 Plant Identi cation challenge focused on tropical ora [ 7 ]. As already noticed, tropical ora is inherently more di cult than generalist ora (even for experts), which partially explains the low performance achieved. The asymmetry between training data based on herbarium sheets and test data based on eld photos adds a considerable di culty. It is important to note that the low scores achieved do not mean that the use of herbarium sheets does not improve the identi cation. The species in the test set were actually selected as the ones having very few eld pictures in the training set. The performance that would have been obtained on that species without using any herbarium sheet would have been considerably weaker. Traditional CNNs performed poorly. Figure 4 shows a great disparity between the performances obtained by the di erent evaluated methods. To explain that we have rst to distinguish between approaches based on classical CNNs alone (typically pre-trained on ImageNet and ne-tuned with the provided training data) and approaches that additionally incorporate an explicit and formal domain adaptation technique between the herbarium and eld domains. As expected regarding the low number of eld photos in the training set for numerous species, directly ne-tuned CNNs with the PlantCLEF 2020 training set obtained the lowest scores (ITCR PlantNet Run 1, SSN Run 1&2, UWB Run 1).

Moreover, the use of external training data with classical CNNs did not

greatly improve performances. It provides some improvements on the main evaluation metric as demonstrated with the UWB runs 2 & 3 and ITCR PlantNet Run 2. All these runs extended the training data with the PlantCLEF 2019 training data [ 7 ] and the GBIF training data provided by [ 17 ]). ITCR PlantNet Run 2 made a greater improvement on the main MRR metric probably because they used a two stage training strategy: they rst ne-tuned a pre-trained ResNet50 with all the herbarium sheets from PlantCLEF 2020, and then netuned it again with all the eld photos PlantCLEF 2020 and the external training data (PlantCLEF 2019 and GBIF). This two stages strategy can be seen eventually as a rst naive domain adaptation technique because the second stage shifts the learned features in an initial herbarium feature space to a eld feature space. However, regarding the second MRR metric focusing on the most di cult species with few eld photos in the training set, performances for all the aforementioned runs is still quite low. This means that the performances of a classical CNN approach, without a formal domain adaptation technique, is too dependent from the number of eld photos available in the training data, and is not able to e ciently transfer visual knowledge from the herbarium domain to the eld photos domain. An adversarial domain adaptation technique performed the best. Among other submissions, two participants stood out from the crowd with two di erent domain adaptation techniques. ITCR PlantNet team based all its remaining runs on a Few Shot Adversarial Domain Adaptation approach [ 15 ] (FSADA), directly applied in the ITCR PlantNet Run 3. FSADA approach uses a discriminator that helps the initial encoder trained on herbarium sheets to shift the learned feature representations to a domain agnostic feature space where the discriminator is no longer able to distinguish if a picture comes from the herbarium or the eld domain, while maintaining the discriminative power regarding the nal species classi cation task. The basic FSADA approach (ITCR PlantNet Run 3) clearly outperformed the traditional CNN approach (ITCR PlantNet Run 1), while both approaches are based on the same initial netuned ResNet50 model on the herbarium training data. It should be noted that the LU team also used an adversarial approach but with less success.

A mapping domain adaptation techniques reached an impressive genericity on di cult species. While the adversarial domain adaptation technique used by the ITCR PlantNet team obtained the best results on the overall MRR metric, the Neuon AI team obtained the best results on the second MRR metric focusing on the most di cult species in the test set. Contrary to the approach used by ITCR which tries to learn a common and agnostic feature space to both domains, the Neuon team tries on its side to netune two networks dedicated each to one of the domain and to optimize a distance that maps the two domains for the purpose of classifying species. The Neuon AI Run 5, which is an ensemble of 3 instances of their approach, gave particularly impressive results with fairly high and, more importantly, equivalent values for both MRR measures. It means that Neuon AI's approach is very robust to the lack of training eld photos and able to generalize on rare di cult species in the test set. In other words, their approach is able to transfer knowledge to rare species which was the underlying objective of the challenge.

External data improved domain adaptation approaches. ITCR Plant

Net Run 4 shows a signi cant impact on the main MRR metric from using external training data compared to the same adversarial domain adaptation approach (ITCR PlantNet Run 3), while maintaining the same level of genericity on rare species with similar MRRs value on the second metric. Unfortunately it is not possible to measure this impact on the Neuon AI method because they did not provide a run using only this year's training data.

Multi-task approaches have a positive impact on performance. Some teams implemented multi-task approaches, i.e. they added additional tasks than the main species identi cation task in the global optimization problem. Such approaches are known to potentially improve the performance of the main task by providing additional knowledge to the model and help the extraction of potentially useful common features. The use of upper taxon level information, in particular, was successful in ITCR PlantNet Run 6 (using a multi-classi cation task integrated to the FSADA approach) compared to ITCR PlantNet Run 4 using only the species classi cation task. Noticeably, yhis is the rst time over all LifeCLEF plant identi cation challenges that we clearly observe an important gain of the use of genus and family information to improve the species identi cation. Many species with few training data have apparently been able to bene t indirectly from a "sibling" species with many data related to a same genus or family. The impact is probably enhanced this year because of the lack of visual data on many species. To a lesser extent, self supervision auxiliary task such as jigsaw solving prediction task (ITCR PlantNet Run 5) improved a little the baseline of this team (ITCR PlantNet Run 4), and the best submission over all this year challenge is an ensemble of all FSADA approaches, combining self supervision or not, upper taxons or not. 5

Conclusion

This paper presented the overview and the results of the LifeCLEF 2020 plant identi cation challenge following the 9 previous editions conducted within CLEF evaluation forum. This year's task was particularly challenging, focusing on species rarely photographed in the eld in the northern tropical Amazon. The results revealed that the last advances in domain adaptation enable the use of herbarium data to facilitate the identi cation of rare tropical species for which no or very few other training photos are available. A mapping domain adaptation technique based on a two-streamed Herbarium-Field triplet loss network reached an impressive genericity by obtaining quite high similar results regardless of whether the species have many or very few eld photos in the training set. On the other hand, a Few Shot Adversarial Domain Adaptation technique outperformed all the other approaches according to the main metric but not with the same genericity according to the second metric, even if the use of taxonomic information can improve the genericity. The results are thus contrasted and allow us to hope for improvements in the near future on both aspects: raw performances and genericity. We believe that the proposed task may be in the future a new baseline dataset in the eld of domain adaptation, and motivate new contributions through a realistic and crucial usage for the plant biology research community.

Acknowledgements This project has received funding from the French National Research Agency under the Investments for the Future Program, referred as ANR-16-CONV-0004 and from the European Union's Horizon 2020 research and innovation program under grant agreement No 863463 (Cos4Cloud project). This work was supported in part by the Microsoft AI for Earth program.

References

1. Argueso, D. , Picon , A. , Irusta , U. , Medela , A. , San-Emeterio, M.G. , Bereciartua , A. , Alvarez-Gila , A. : Few-shot learning approach for plant disease classi cation using images taken in the eld . Computers and Electronics in Agriculture 175 , 105542 ( 2020 ). https://doi.org/https://doi.org/10.1016/j.compag. 2020 . 105542 , http:// www.sciencedirect.com/science/article/pii/S0168169920302544

2. Chulif , S. , Chang , Y.L. : Herbarium- eld triplets network for cross-domain plant identi cation - neuon submission to lifeclef 2020 plant . In: CLEF working notes 2020 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2020 , Thessaloniki, Greece. ( 2020 )

3. Delprete , P.G. , Feuillet , C. : Marie-francoise prevost \fanchon"( 1941 { 2013 ). Taxon 62 ( 2 ), 419 { 419 ( 2013 )

4. Goeau, H., Bonnet , P. , Joly , A. : Plant identi cation in an open-world (lifeclef 2016) . In: CLEF task overview 2016 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2016 , Evora, Portugal. ( 2016 )

5. Goeau, H., Bonnet , P. , Joly , A. : Plant identi cation based on noisy web data: the amazing performance of deep learning (lifeclef 2017) . In: CLEF task overview 2017 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2017 , Dublin, Ireland. ( 2017 )

6. Goeau, H., Bonnet , P. , Joly , A. : Overview of expertlifeclef 2018: how far automated identi cation systems are from the best experts ? In: CLEF task overview 2018, CLEF: Conference and Labs of the Evaluation Forum , Sep. 2018 , Avignon, France. ( 2018 )

7. Goeau, H., Bonnet , P. , Joly , A. : Overview of lifeclef plant identi cation task 2019: diving into data de cient tropical countries . In: CLEF task overview 2019 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2019 , Lugano, Switzerland. ( 2019 )

8. Goeau, H., Bonnet , P. , Joly , A. , Bakic , V. , Barthelemy , D. , Boujemaa , N. , Molino , J.F. : The imageclef 2013 plant identi cation task . In: CLEF task overview 2013 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2013 , Valencia, Spain. Valencia ( 2013 )

9. Goeau, H., Bonnet , P. , Joly , A. , Boujemaa , N. , Barthelemy , D. , Molino , J.F. , Birnbaum , P. , Mouysset , E. , Picard , M.: The imageclef 2011 plant images classi cation task . In: CLEF task overview 2011 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2011 , Amsterdam, Netherlands. ( 2011 )

10. Goeau, H., Bonnet , P. , Joly , A. , Yahiaoui , I. , Barthelemy , D. , Boujemaa , N. , Molino , J.F. : Imageclef2012 plant images identi cation task . In: CLEF task overview 2012 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2012 , Rome, Italy. Rome ( 2012 )

11. Goeau, H., Joly , A. , Bonnet , P. : Lifeclef plant identi cation task 2015 . In: CLEF task overview 2015, CLEF: Conference and Labs of the Evaluation Forum , Sep. 2015 , Toulouse, France. ( 2015 )

12. Goeau, H., Joly , A. , Bonnet , P. , Selmi , S. , Molino , J.F. , Barthelemy , D. , Boujemaa , N. : The lifeclef 2014 plant images identi cation task . In: CLEF task overview 2014 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2014 , She eld , United Kingdom. She eld , UK ( 2014 )

13. Joly , A. , Goeau, H., Botella , C. , Glotin , H. , Bonnet , P. , Vellinga , W.P. , Muller, H.: Overview of lifeclef 2018: a large-scale evaluation of species identi cation and recommendation algorithms in the era of ai . In: Jones, G.J. , Lawless , S. , Gonzalo , J. , Kelly , L. , Goeuriot , L. , Mandl , T. , Cappellato , L. , Ferro , N. (eds.) CLEF: CrossLanguage Evaluation Forum for European Languages . Experimental IR Meets Multilinguality, Multimodality, and Interaction , vol. LNCS . Springer, Avigon, France ( Sep 2018 )

14. Joly , A. , Goeau, H., Glotin , H. , Spampinato , C. , Bonnet , P. , Vellinga , W.P. , Lombardo , J.C. , Planque , R. , Palazzo , S. , Muller, H.: Lifeclef 2017 lab overview: multimedia species identi cation challenges . In: International Conference of the CrossLanguage Evaluation Forum for European Languages . pp. 255 { 274 . Springer ( 2017 )

15. Motiian , S. , Jones , Q. , Iranmanesh , S. , Doretto , G.: Few-shot adversarial domain adaptation . In: Advances in Neural Information Processing Systems . pp. 6670 { 6680 ( 2017 )

16. Nanda H Krishna , R am Kaushik R , R.M.: Plant species identi cation using transfer learning - plantclef 2020 . In: CLEF working notes 2020 , CLEF: CLEF: Conference and Labs of the Evaluation Forum , Sep. 2020 , Thessaloniki, Greece. ( 2020 )

17. Picek , L. , Sulc , M. , Matas , J.: Recognition of the amazonian ora by inception networks with test-time class prior estimation . In: CLEF working notes 2019 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2019 , Lugano, Switzerland. ( 2019 )

18. Schro , F. , Kalenichenko , D. , Philbin , J.: Facenet: A uni ed embedding for face recognition and clustering . 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2015 ). https://doi.org/10.1109/cvpr. 2015 . 7298682 , http://dx.doi.org/10.1109/CVPR. 2015 .7298682

19. Szegedy , C. , Io e, S., Vanhoucke , V. , Alemi , A. : Inception-v4, inception-resnet and the impact of residual connections on learning . arXiv preprint arXiv:1602.07261 ( 2016 )

20. Villacis , J. , Goeau, H., Bonnet , P. , Mata-Montero , E. , Joly , A. : Domain adaptation in the context of herbarium collections: a submission to plantclef 2020 . In: CLEF working notes 2020 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2020 , Thessaloniki, Greece. ( 2020 )

21. Zhang , Y. , Davison , B.D. : Adversarial consistent learning on partial domain adaptation of plantclef 2020 challenge . In: CLEF working notes 2020 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2020 , Thessaloniki, Greece. ( 2020 )

22. Zoph , B. , Vasudevan , V. , Shlens , J. , Le , Q.V. : Learning transferable architectures for scalable image recognition ( 2017 )