The LifeCLEF Plant Identification Challenge (PlantCLEF) [1] is part of the Conference and Labs of the Evaluation Forum (CLEF), which tackles various multilingual and multimodal information access evaluations [2]. This year, the focus of PlantCLEF 2022 was to classify 80,000 plant species. Compared to its past editions, PlantCLEF 2022 offered the largest ever number of classes for training, making it resource-intensive, which is often the case in real-world applications. In the context of global plant identification, the aggregation of various data from different sources poses many challenges. It is a real problem to deal with big data consisting of many classes, unbalanced classes, inaccuracies, duplications, and a diversity of visual contents and quality. The total training datasets provided in this challenge consisted of 4 million images and were grouped as "trusted" or "web". In addition, the metadata included more levels of plant taxonomy, i.e., class, order, family, genus, and species. This paper presents our approach, the network architectures used, the training setup implemented, and the results obtained from our submissions to PlantCLEF 2022.
Two training datasets were downloaded: "trusted" and "web". After removing some duplicates with the same name, the trusted dataset totalled 2,885,052 images. It is derived from academic sources and collaborative platforms, signifying a higher certainty of quality. Meanwhile, the web dataset totalled 1,071,627 images. It is based on search engine queries and suffered from notable errors, which were then semi-automatically cleaned by the organisers. In addition, from the trusted dataset, we segregated 63,119 images to serve as our validation dataset. This validation dataset consists of unique species belonging to one plant observation id from our trusted train dataset. Lastly, the test set consisted of 55,306 images from 26,868 plant observations. The details of the datasets used are represented in Table 1.
The networks implemented in our approach were based on the Inception-v4 and Inception-ResNet-v2 architectures [3]. Two types of networks were constructed: a single convolutional neural network (CNN) and a triplet network. They were either initialised on weights pre-trained from the ImageNet dataset [4] or the weights pre-trained from PlantCLEF 2022 dataset.
Motivation for Triplet Network Convolutional Neural Networks (CNNs) have effectively solved classification tasks in various domains and even perform comparably equal or better than humans. However, a pre-defined number of classes is required before training. If the classification task is assigned with new labels, the network has to be retrained. In addition, CNNs work best when there is sufficient training data.
In a global-scale plant identification task, it is impractical to retrain the network every time a new species is discovered. Furthermore, many plant species, especially those in remote areas, are rarely photographed. More often than not, the available data for training follows a long-tail Considering these concerns, we opted to implement the metric-learning-based triplet network. Its goal is to learn the similarity and dissimilarities between classes instead of directly classifying them. The network accomplishes this by minimising the embedding distance of the same species while maximising the embedding distance of different species. A small embedding distance indicates the same species. Meanwhile, a large embedding distance indicates different species. Besides, it does not require a pre-defined number of classes before the training. Moreover, as shown in the previous PlantCLEF editions [5,6], our triplet networks [7,8] can generalise plant species equally well with or without less training data than a conventional CNN.
Single CNN This network resembles a conventional Inception-v4 and Inception-ResNet-v2 neural network. Similarly, it consists of convolutional layers, pooling layers, dropout layers and fully-connected layers, which return the softmax probabilities of its prediction. The multi-task classification is adopted in this network by utilising the five taxonomy labels: Class, Order, Family, Genus, and Species. Table 2 shows the multi-task classification labels and their number of classes. The network architecture is visualised in Figure 1 (A).
Triplet Network This network resembles the single CNN mentioned above. However, it consists of two streams and instead of using its fully-connected layer for its predictions, it is used to compute the plants' image embedding representation. In addition, a batch normalisation layer is added, followed by L2-normalisation, and finally, a triplet loss layer1 to train the optimum embedding representation of the plants. Furthermore, instead of its original 1536 features in the fully-connected layer, we reduced its final feature vector to 500. Due to resource limitation, we did not adopt the multi-classification approach in this training. Only the Species taxonomy label is utilised. The network architecture is visualised in Figure 2 (A).
The networks were set up using Tensorflow 1.12 [9] and TF-Slim [10] library with the hyperparameters described in Table 3. Random cropping, colour distortion, and horizontal flipping were also applied to the images during training of the networks. The scripts and lists used are available at https://github.com/NeuonAI/plantclef2022_challenge.
The two main evaluation methods of the models used the Argmax function and embedding dictionary similarity comparison. Argmax is used in the single CNN, while the embedding dictionary similarity comparison is used in the triplet network. The inference procedure for the single CNN and triplet network are visualised in Figures 1 (B) and 2 (B), respectively. The following steps describe the overall inference procedure.
1. Group the images based on the same observation id (if the test set is used). image, there will be 10 predictions for each image for each label (Class, Order, Family, Genus, Species). 5. Average the 10 prediction probabilities of each image classification. 6. Obtain the Top-1 and Top-5 accuracy (if the validation set is used). 7. Obtain the Top-30 accuracy (if the test set is used).
Triplet Network Before inference, a sample from the training dataset (trusted) is randomly chosen to create a dictionary list. The dictionary list totalled 592,258 images which consist of 80,000 species. A maximum of ten images represent each species in the dictionary. 1. Group the dictionary images based on the same species. 2. Augment the dictionary images to 10 variations through cropping and flipping. Note that the 10 different variations include the cropping of the top-right, top-left, bottom-right, bottom-left, and centre of the image. Then, these 5 images are horizontally-flipped to obtain a total of 10 image variations. 3. Feed the augmented dictionary images into the network. 4. Obtain the image embeddings of each image. Note that since there are 10 image variations, there will be 10 embeddings for each image. 5. Average the 10 embeddings and save them as a dictionary reference. 6. Repeat steps 2 to 5 until all the 80,000 species embeddings are collected. 7. Group the test image(s) based on the same observation id (if the test set is used). 8. Augment the validation / test image(s) to 10 variations through cropping and flipping as previously. 9. Feed the augmented validation / test images into the network. 10. Obtain the image embeddings of each image. Note that since there are 10 image variations, there will be 10 embeddings for each image. 11. Average the 10 embeddings to obtain the single embedding of each validation / test image. 12. Compute the cosine similarity between the single image embedding and the saved dictionary. 13. Obtain the cosine distance by subtracting the computed cosine similarity from the value of 1. 14. Employ inverse distance weighting on the cosine distance. 15. Acquire the probabilities of the single image embedding mapped to the dictionary. 16. The species mapped to the highest probability denotes the class of the species. 17. Obtain the Top-1 and Top-5 accuracy (if the validation set is used). 18. Obtain the Top-30 accuracy (if the test set is used). 19. Repeat steps 8 to 18 until all the images are evaluated.
The networks we experimented with were variations of single CNN and triplet networks. They differ in their network architecture, training data, and weights initialised. Table 4 shows the details of our experimented networks. As seen in Table 4, models 1 to 5 are single CNNs, models 6 to 8 are triplet networks, and models 9 to 12 are single CNN whose weights were initialised from the triplet network.
From the validation dataset, we computed the Top-1 and Top-5 accuracy of the models and tabulated them in Tables 5 and 6, respectively. Due to the large training dataset and time constraints, most of the models trained were not saturated before we evaluated them. In addition, not all the models experimented with were used in the submissions.
Based on our experiments, most of the models trained with a higher number of iterations performed better in the validation dataset. Nevertheless, the higher number of iterations does not necessarily depict higher performance. Comparing Model 2 (421,517 steps) and Model 4 (522,583 steps), Model 2 performed better with a Top-1 accuracy of 0.462 compared to Model 4 of 0.4545. Since the validation set was built from the trusted dataset, which was what Model 2 was trained on, it may have resulted in this bias. Furthermore, we find that the single CNN initialised from the triplet network (Model 12b) performed slightly better than the single CNN initialised from ImageNet (Model 5b). Since our models did not saturate, we cannot give a definite answer
We trained several Inception-v4 and Inception-ResNet-v2 single and triplet deep learning models for the plant identification task in PlantCLEF 2022. Due to its large number of species and training data, it was indeed resource-intensive to experiment. Due to resource and time constraints, our models were not rightfully saturated, and we did not experiment as intended. Therefore, we would like to look into the performance between our single CNN and our triplet models for future work. It is worth looking into their performance when they are both saturated and compared with the same evaluation methods and on different validation sets focusing on unbalanced classes. Nevertheless, we submitted nine runs and our best submission achieved a Macro Averaged Mean Reciprocal Rank score of 0.6078, placing 4th among the 45 submitted runs. In addition, we have shown that web or noisy data does improve generalisation in the identification. Furthermore, the ensemble of models from different network architectures, i.e., Inception-v4 and Inception-ResNet-v2, give higher accuracy than a single model.
The resources of this project is supported by NEUON AI SDN. BHD., Malaysia.
if it improves the model. However, if we were to compare both models in the same training iteration, the model initialised from the triplet network weights (Model 12b) did perform better than the model initialised on ImageNet weights (Model 5b). Among the different networks trained, the triplet network performed the worst, but this is because their evaluation methods differ from one another (Argmax vs dictionary similarity comparison). Moreover, they were trained significantly less than their single CNN counterpart. In addition, the triplet network relies on the image dictionary for inference. The classes with fewer image samples, especially those with a single sample, may not have provided enough features in the dictionary.
We submitted nine runs to PlantCLEF 2022, and its details are tabulated in Table 7. Apart from Run 1, our submissions were constructed from the ensemble of various models. Run 7, which constitutes three single CNNs based on the Inception-v4 and Incpetion-ResNet-v2 architectures, and trained on the trusted and web data, performed the best among our submissions.
The test set was evaluated based on the average Mean Reciprocal Rank (MRR) per species, also known as the Macro-Averaged Mean Reciprocal Rank (MA-MRR) score. Our best Run (7) achieved an MA-MRR score of 0.6078 and was the fourth-highest among the 45 submissions. On the other hand, the top submission scored an MA-MRR of 0.6269. The results of the overall submissions are summarised in Figure 3.
Comparing our Run 2 (trusted only) and Run 3 (trusted and web), the models trained on both trusted and web datasets performed better. Once again, showing that web or noisy data does improve the generalisation of deep learning as in PlantCLEF 2017 [12]. Furthermore, combining models of different architectures, i.e., Inception-v4 and Inception-ResNet-v2 did slightly boost the performance from 0.5461 (Run 1: Inception-v4 only) to 0.5536 (Run2: Inception-v4 and Inception-ResNet-v2). Since our triplet models and single CNN initialised on triplet weights did not saturate, it did not help the ensemble models. Consequently, Run 3 and 7 dropped in performance when added with the triplet or single CNN triplet initialised models, as observed in Run 5 and 8.