This paper presents the methods and submissions made by our team to PlantCLEF 2023, an imagebased plant identification task involving 80,000 plant species. Like PlantCLEF 2022, the task aimed to identify species based on 26,868 multi-image plant observations from a test set of 55,306 images. Given a training dataset of 4 million photos, we trained several Inception-v4 and Inception-ResNet-v2 models and submitted ten runs. Compared to the previous year, we experimented with more data augmentations, diferent batching methods and trained separate organ models, specifically for the labels: bark, flower, fruit, habit, and leaf. Our highest-performing run, comprising several ensembled models, achieved a macro-averaged mean reciprocal rank of 0.61813, increasing from 0.6078 in last year's performance (PlantCLEF 2022) through increased data augmentations.
Accurate identification of plants on a global scale helps us understand and conserve Earth’s
biodiversity. It allows naturalists to study species diversity [
This paper describes our training strategies, submissions, and results obtained in PlantCLEF
2023. In short, our training strategies include increasing data augmentation, applying
balanced batching, and employing multi-organ and single-organ training schemes. Firstly, data
augmentation is a common technique to increase the training dataset by performing various
transformations on the images to tackle the class-imbalance problem and to avoid overfitting
[
From the above approaches, we find that the data augmentation method enhances our model the best and has helped us achieve a macro-averaged mean reciprocal rank (MA-MRR) of 0.61813, increasing from 0.6078 in last year’s performance (PlantCLEF 2022), producing the second best results in the PlantCLEF 2023 challenge.
Two datasets were provided in PlantCLEF 2023, the training dataset and the test dataset. The training dataset was grouped into two: Trusted and Web. The Trusted dataset comprised 2,886,761 higher-quality images from academic sources and collaborative platforms. On the other hand, the Web dataset included 1,071,627 images from search engine queries sufering significant inaccuracies. The Trusted and Web datasets contain metadata such as Class, Order, Family, Genus, Species, manual tag, predicted tag, and predicted tag probability. Nevertheless, not all images were annotated with the manual and predicted tags. Meanwhile, the test dataset consists of 55,306 images categorised into 26,868 plant observations used for the challenge’s observationlevel plant classification. Like the Trusted and Web datasets, the test dataset contained the manual tag metadata. In addition, we partitioned separate datasets from the training dataset provided to cater to our multi-organ and single-organ training schemes. Since not all images were annotated with tags, some images were not included in the single-organ training. We used the images annotated with the tags of “bark”, “flower”, “fruit”, “habit”, and “leaf”. Furthermore, we excluded the tags with less than 0.7 in their predicted probability score. Table 1 shows the training datasets used for our multi-organ and single-organ training approaches. Note that we pre-processed the training datasets and removed all additional duplicate images sharing the same filename, resulting in a reduced training dataset size compared to the original training dataset.
Our models were set up using Tensorflow 1.12 and TF-Slim library with the hyperparameters
described in Table 2. Like our submissions to PlantCLEF 2022 [
Our models were trained with random cropping, horizontal flipping, and colour distortion applied to our images. In contrast to our models from last year, we increased the data augmentations in our training by performing more bi-cubic resizing, random hue and random contrast to our images.
In addition, we adopted a balanced batching method in training our models. We limit the selection of training images for a species in an epoch to a maximum of 16 samples. We chose 16 as it would be suitable since the lowest sample number in the training dataset is 1, while the highest sample number in the dataset is over 100. It aims to avoid bias towards any particular species and prevents poor performance on underrepresented species.
Furthermore, we implemented two training schemes: multi-organ and single-organ. In multiorgan training, we trained our model with the entire dataset (like our submissions last year). Using all the available organ parts, the classification is trained on the species. On the other hand, in single-organ, we trained our model based on the individual organ tags provided as mentioned in Section 2, which includes Bark, Flower, Fruit, Habit, and Leaf. The classification of a single-organ model is based on the specific organ and their respective species.
The model predictions are obtained by the Softmax function as well as the feature embedding comparison method.
The softmax function is commonly used as the activation function in the output layer of a neural network for multi-class classification. It allows the network to produce class probabilities over multiple classes. It accepts a vector of input data and assigns a probability value to each class. The probability generated is a vector of numbers between 0 and 1 and these values would sum up to 1. The class with the highest probability indicates the most confident predicted class. The Softmax function is represented in Equation 1 where x represents the input vector, and denotes the exponential function raised to the power of x. The denominator, ∑︀ , calculates the sum of the exponentiated values over all elements of the input vector.
softmax() = ∑︀ (1)
This method refers to comparing the test image’s features to the train images’ features. It involves transforming the train and test images into lower-dimensional feature vectors (embeddings) using the learned representation obtained from the fully-connected layers of our model, then comparing them to measure their similarity. Essentially, this method include two processes: the feature dictionary generation and feature similarity comparison.
1. Feature dictionary generation
First, we generated a list of train images (feature dictionary) to compare with the test images. Since the training data consists of over 3 million images and it is computationally intensive to compare all the photos, we only selected a maximum of ten images for each of the 80,000 plant species from the Trusted training dataset. Consequently, our feature dictionary is made from a list of 592,258 training images. These images are then transformed into feature embeddings and averaged according to their respective classes. Therefore, the feature dictionary is composed of 80,000 feature embeddings. Note that to obtain the feature embedding of each training image, we applied ten crops to the image and then averaged them to get a single feature embedding. The ten crops involve cropping the top-left, top-right, bottom-left, bottom-right, and centre, all of which are also horizontally flipped to get ten variations. The total feature embeddings of each species are then averaged over the total number of images it has in the list of the dictionary, making a feature dictionary of 80,000 feature embeddings. 2. Feature similarity comparison
After acquiring the train feature dictionary of 80,000 feature embeddings, we obtain the test image embeddings by performing the same ten crops technique to the test images. Since the test images’ classes are unknown, the embeddings are averaged according to the test observation id instead of the species class id as in the train feature dictionary generation. Therefore, since there are 26,868 observation ids, the total test embeddings generated result in 26,868 test feature embeddings.
Next, we used cosine similarity to measure the similarity between the train feature dictionary and the test feature embeddings. The calculated cosine similarity score is then transformed with Inverse Distance Weighting into probabilities for ranking the classes. The weights for the transformed embedding vector were calculated as in Equation 2 where is the weight assigned to a specific test feature embedding, is the distance between the test feature embedding and the dictionary feature embedding, is the distance between the test feature embedding and the dictionary feature embedding , raised to the power of , while ∑︀ is the sum over all dictionary feature embeddings. Similar to the Softmax function method, the class with the highest probability indicates the most confident predicted class.
( 1 ) = ∑︀( 1 ) (2)
We submitted ten runs to PlantCLEF 2023, consisting of several models described in Table 3.
Essentially, they difer in their network architecture, data augmentation, batching method, and
training data. The evaluation metric implemented in this challenge is the Macro Average (by
species) Mean Reciprocal Rank (MA-MRR). The MA-MRR is a statistic measure for evaluating
any process that generates a list of possible responses to an index of queries ordered according
to the likelihood of correctness as defined in [
Our runs are tabulated in Table 4. Run 1, 2, 3, 5, 6, and 8 were based on the predictions from a single model, while Run 4, 7, 9, and 10 were based on several models ensembled. As expected, the ensembled models are more accurate than a single model prediction. Our highest-performing run (Run 9), an ensembled model, obtained an MA-MRR score of 0.61813. From the new strategies experimented with, we found that increasing data augmentation helped improve the models’ accuracy. It is confirmed by comparing our run from PlantCLEF 2022 (Run 7 MA-MRR: 0.6078) and our current run (Run 9 MA-MRR: 0.61813), which difer by only adding the new model (Multi-IR-AUG), which is trained with increased augmentations.
On the other hand, the balanced batching method did not help in improving the model’s performance. Comparing Run 1 and 2, we see that the model without the balanced batching strategy (Run 1 MA-MRR: 0.54242) achieved a higher MA-MRR score compared to the model with the balanced batching method (Run 2 MA-MRR: 0.46606). Although it was not what we expected, it could be due to the reduction of training samples which may be inherently more important than others. Furthermore, the single-organ training scheme did not help in the predictions. Run 4, which comprised the single-organ models (bark, flower, fruit, habit, and leaf), performed the worst among all other runs based on the multi-organ training scheme. It suggests that training with multi-organ data is more efective than single-organ data as a combination of organs can provide more information to the model. Finally, we show that the predictions obtained by the feature embedding matching method (Run 3, 6, and 8 MA-MRR: 0.45242, 0.46476, 0.4591) are lower compared to the predictions obtained by the Softmax function method (Run 1 and 5 MA-MRR: 0.54242, 0.5504). It is likely due to the clearer decision boundary obtained when using the Softmax function, as this approach relies on the confidence of the network’s prediction. On the other hand, feature embedding comparison might involve more nuanced comparisons between feature vectors, which can lead to a loss of discriminative information. This loss could result in reduced accuracy and lower performance.
We trained several Inception-v4 and Inception-ResNet-v2 models and submitted ten runs to
PlantCLEF 2023. To improve our performance in this year’s challenge, we tried several new
strategies that difered from our previous submissions in PlantCLEF 2022. These strategies
included three key approaches: increased data augmentation, a balanced batching method,
and a multi-organ and single-organ training scheme. Among these approaches, increased data
augmentation improved our model predictions by increasing our MA-MRR score from 0.6078 to
0.61813. The balanced batching method did not work out as intended but is possibly the result
of the degradation of the head classes with many training samples at the expense of improving
the tail classes with fewer training samples from under-sampling [
The resources of this project is supported by NEUON AI SDN. BHD., Malaysia.