This paper presents the implementation and performance of a Herbarium-Field triplet loss network to evaluate the herbarium- eld similarity of plants which corresponds to the cross-domain plant identi cation challenge in PlantCLEF 2020. A two-streamed triplet loss network is trained to maximize the embedding distance of di erent plant species and at the same time minimize the embedding distance of the same plant species given herbarium- eld pairs. The team submitted seven runs which achieved a Mean Reciprocal Rank score of 0.121 and 0.111 for the whole test set and the sub-set of the test set respectively.
Plant specimens in herbaria have been used by novices and experts alike to
study and con rm plant species as well as many other useful applications as
described in [
The objective in PlantCLEF 2020 [
We adopt triplet loss function to optimize the plant embeddings which regulates the measure of plant similarity. The implemented network is trained to maximize the embeddings of di erent herbarium- eld species pairs and minimize
Fig. 1. The triplet loss concept mainly revolves around minimizing the distances
between same class and maximizing the distances between di erent classes. (a) shows
two classes with its herbarium counterpart, the image embedding is compared with its
own herbarium and the herbarium from another class (as indicated by the arrows). (b)
The distances between herbarium- eld pairs of the same species has to be less than
the herbarium-pairs of di erent species (red and blue box denotes the class label).
the embeddings of same species pairs. It learns the similarity between herbarium
sheets and eld images instead of directly classifying plant species as
conventional convolutional neural networks (CNN) [
The authors in [
Plant Disease Recognition with Siamese Network The authors in [
Non-Frozen Layers This method on the other hand trains all layers in the network. It allows the network to relearn and recompute the embeddings of herbarium and eld images with respective to their optimized embedding space from the triplet loss. The new layers are set to have a higher learning rate than the migrated layers. 3.2
Herbarium Network As mentioned in 3.1, a Herbarium network based on
the Inception-v4 model [
As mentioned in the task description, only a subset of species for eld images were provided to allow learning a mapping between the herbarium and eld domain. We separated the species which possess both herbarium and eld images to be used for mapping. Out of 997 classes, 435 classes were identi ed having both herbarium and eld images. These classes were then used for training. Although the total number of classes was reduced from 997 to 435 species, the network was still trained to map the embedding space of 997 classes.
During the training of the Herbarium-Field triplet loss network, the images used for each batch were picked to be balanced for each class. For instance, in a batch of size 16, each class may not comprise more than 4 images, meanwhile the minimum number of images in each class is 2. This allows a balanced selection of anchors for the triplet loss. 4.2
In order to increase the network generalization and increase training sample size,
data augmentation was applied on the training images. Random cropping,
horizontal ipping and colour distortion (brightness, saturation, hue, and contrast)
of images were performed on the training dataset. As a result, features and
various transforms that are invariant to their original locations can be learned by
the network, consequently reducing the chance of over tting [
The training dataset distributions and network setup parameters are summarized in Table 1 and Table 2 respectively. 5
The experiments were conducted using Tensor ow 1.13 [
Batch Size Input Image Size
Optimizer Initial Learning Rate
Weight Decay
Loss Function 5.1
Due to the limited eld training samples, prior to training, a sample of images from each of the \herbarium photo associations" and \photo" folders were randomly segregated for validation purposes. 1,219 eld images were separated from the test set leaving 5,038 eld images for training instead of 6,257 as stated in Table 1. The number of images and classes present in the experimented training and testing dataset are summarized in Table 3. Nevertheless, the class number for the Herbarium-Field triplet loss network remains 997 and 10,000 in the Herbarium and Field network stream respectively.
For inference, the embeddings from 997 herbarium classes were rst extracted using the trained Herbarium-Field triplet loss network to form the reference embeddings served as a herbarium dictionary. Random samples from each class were picked and fed into the network to obtain the embeddings. The extracted embeddings were then averaged to get a single embedding representation for each class. The embedding for each class was subsequently saved as a dictionary.
Note that the extraction was done with two di erent types of image cropping, namely, Center Crop and Center and Corner Crop. The Center Crop approach crops the centre region of the herbarium sample. Meanwhile, the Corner Crop approach on the other hand crops the top left, top right, bottom left, and bottom right region of the herbarium sample. Each region was cropped and resized then passed into the network for the extraction of herbarium embeddings.
After obtaining the single embedding representation of each class, the saved dictionary is then used to compare the embedding distance between the 997 herbarium representation and the test image. During validation, Center and Corner Crop were also applied together with horizontal ip in obtaining the test images' embeddings. This resulted in 10 di erent variations for each image which was then averaged to obtain their similarity probability. Cosine similarity was used as the distance metric in measuring the embedding similarity. Then, the cosine distance was obtained by subtracting the cosine similarity from 1. Finally, inverse distance weighting was performed on the cosine distance to obtain the probabilities of each class. 5.3
The experimented results are tabulated in Table 4 and Table 5 for Center Crop and Center Crop and Corner Crop herbarium extraction methods respectively. The networks were tested on the same validation set of 1,219 images in which the Top 1 and Top 5 predictions were evaluated. Center Crop and Corner Crop were also applied on the eld test set before validation. 5 di erent Herbarium
layers.
layers, or to put simply, trained with all layers.
ble of 3 di erent models trained on all layers.
A network trained with all layers whereby the training images were pre-processed with more transformations and augmentation.
semble (NFL-AUG-ENS) An ensemble of Network 3 and Network 4. 5.4
From the experiments, it can be seen that the NFL ensemble models performed the best among the networks. The ensemble of these networks increased the robustness of the system and returned better predictions. On the other hand, the FL network performed the worst among the networks. It can be suggested that the training of all layers does help the prediction model instead of freezing the front layers or extractor layers of the network. In can be seen that the ensemble models with increased augmentation performed equally as to the ensemble model without increased augmentation. It can be suggested that the increased augmentation may have not produced enough new signi cant information for the network to learn. Since a portion of eld images were separated from the training set to serve as test set, some of the classes may miss some eld information. In addition, the trained model does not represent the entire classes as some classes miss eld images. Consequently, the networks did not performed as well as it was not fed with su cient images to represent the eld domain. An approach to increasing the prediction accuracy would be increasing the training samples of the eld images that are not present in the training set. 6
The procedure adopted to produce the submitted results are as follow: (i) Construct herbarium dictionary by extracting samples of herbarium embeddings for all 997 plant species using the trained Herbarium-Field triplet loss network. (a) Apply Center and Corner Crops on the images before extraction. (b) Average the cropped herbarium embeddings for each species and save them. (ii) Group the test images belonging to the same observation ID. (iii) For each image under the same observation ID, apply Center and Corner
Crops which result in 5 images each. (iv) Subsequently ip the images horizontally resulting in 10 images each. (v) Average the 10 images and pass them to the Herbarium-Field triplet loss network. (vi) Obtain the image embeddings. (vii) Compute cosine similarity between each of the extracted embeddings with the saved 997 herbarium embeddings. (viii) Obtain cosine distance by subtracting the cosine similarity from the value of 1. (ix) Apply inverse distance weighting on the cosine distance. (x) Obtain the probabilities of the embedding distance. (xi) Average the probabilities over the total number of images for each observation ID. (xii) Repeat steps (iii) to (xii) for the remaining observation IDs. (xiii) Collect the predictions, probabilities and ranks for each observation ID. 6.2
The team submitted a total of seven runs based o the networks mentioned in Section 5.3.
Run 1 This model was based o (FL). Unlike the rest of the runs, this network was trained with frozen front layers and does not apply image ipping during validation. Moreover, the embedding distances were normalized, inversed then applied with softmax to obtain the probabilities. In addition, the probabilities were based o the averaged embedding instead of all embeddings for each observation ID.
Run 2 This model was based o (NFL). Similar to Run 1 however it was trained with all layers of the network, the embeddings of each observation IDs were averaged and then applied with Cosine Similarity and Inverse Distance Weighting to obtain the probabilities.
Run 3 This model was based o (NFL). Similar to Run 2 however by using Cosine Similariy and Inverse Weighting, the probabilities of each embeddings were rst computed then averaged for each observation IDs .
Run 4 This model was based o (NFL). Similar to Run 3 however the probabilities take into account the total embeddings of each observation IDs multiplied by their croppings which consist of 10 variations. Run 5 This model was based o (NFL-ENS). Unlike Run 1 to 4, the network was trained together with the full dataset as stated in Table 1. It is also an ensemble of the predictions from 3 models of the same network. Run 6 This model was based o (NFL-AUG). Similar to Run 5 which was trained with the full dataset however it is not an ensemble of models and trained with increased image processing transformations and augmentations. Run 7 This model was based o (NFL-AUG-ENS). This run is the ensemble of the predictions from Run 5 and Run 6. 6.3
Our best submitted runs scored a Mean Reciprocal Rank (MRR) of 0.121 and 0.108 for the rst and second metric respectively. Our results are tabulated in Table 6. The results by all the participating teams are summarised in Fig. 3 and Fig. 4. 6.4
Similar to the experiment results, the ensemble models performed the best among the networks. The ensemble model with increased augmentation on the other hand performed best in the whole test set. In addition, the MRR score of the networks for the rst and second metric are relatively close despite the few training photos in the sub-set species. It can be suggested that the number of training samples for each class does not directly in uence the performance of the model. Other than lling the missing training samples of the eld classes, the methods in obtaining the herbarium embedding representation can also be looked into to increase prediction accuracy. Such methods involve nding the best herbarium dictionary representation. Various image processing methods like ipping can be performed before extracting the herbarium embeddings. Meanwhile, nding the best model of the Herbarium-Field Triplet Loss Network and using it for the extraction of the herbarium embeddings would be signi cant as well.
Fig. 3. O cial Results of PlantCLEF 2020. In addition to the submitted results, the team trained another 3 runs which was based o the continuation of Run 6. However, the results did not performed better than the submitted runs. Since the runs were trained with the whole dataset, we believe the drop in performance is due to over tting as there was no baseline to determine when to stop training the model. The MRR score of the runs are tabulated in Table 7.
Run 8 This model was based o (NFL-AUG). This run was a continuation of the training from Run 6 which was trained with increased iterations. Run 9 This model was based of (NFL-AUG-ENS). This run was an ensemble of Run 8 and Run 5 predictions.
Run 10 This model was based o (NFL-ENS). This run was an ensemble of 3 di erent models from Run 8.
We tested the post-challenge runs on our segregated test set as well and the results are tabulated in Table 8 and Table 9 for Center Crop and Center and Corner Crop herbarium dictionary construction methods respectively. In contrast with its MRR score, Run 8 shows the best performance in the experimental validation setup when in fact it performed the worst among the post-challenge runs. This is likely due to over tting as mentioned. 8
In this paper we have presented our approach in PlantCLEF 2020 which focused on the cross-domain plant identi cation between herbarium sheets and in- eld photos. We adopted a two-streamed Herbarium-Field triplet loss network which performed relatively equal regardless if few eld training images were given. Based on the similar score between MRR metric 1 and 2, it is proven that the proposed network feature is not directly a ected by the plant class but it learns to perceive the similarity between a given eld image with herbarium images. It is shown that even with a minimal amount of eld images for each species, crossdomain plant identi cation can be performed. The identi cation of real-world plants based on herbarium sheets alone is indeed a challenging task. Although our machines did not performed as well with missing eld classes which is the case in real-world, it shows that with su cient data, it o ers a step in alleviating the tedious task of herbarium- eld classi cation which requires high level expertise. For future work, the eld images that are not present among the training dataset can be added to improve the predictions. This would allow the model to learn the whole representation of plant species with respect to their herbarium and eld domain. Furthermore, the extraction of herbarium embeddings to form a more powerful dictionary can be investigated to nd the best representation of herbarium embeddings for the herbarium- eld similarity comparison.
The resources of this project is supported by NEUON AI SDN. BHD., Malaysia.