We present a transfer learning approach using a self-supervised Vision Transformer (DINOv2) for the PlantCLEF 2024 competition, focusing on the multi-label plant species classification. Our method leverages both base and ifne-tuned DINOv2 models to extract generalized feature embeddings. We train classifiers to predict multiple plant species within a single image using these rich embeddings. To address the computational challenges of the large-scale dataset, we employ Spark for distributed data processing, ensuring eficient memory management and processing across a cluster of workers. Our data processing pipeline transforms images into grids of tiles, classifying each tile, and aggregating these predictions into a consolidated set of probabilities. Our results demonstrate the eficacy of combining transfer learning with advanced data processing techniques for multi-label image classification tasks. Our code is available at github.com/dsgt-kaggle-clef/plantclef-2024.
Our approach leverages the embedding space learned by DINOv2 as a generalized feature representation
of images, which are used to train models with higher bias (i.e., linear classifiers), as illustrated in Figure
1. DINOv2 learns robust feature representations by processing images as sequences of fixed-size patch
tokens with an additional [CLS] token for classification tasks [
DINOv2 is a state-of-the-art vision transformer encoder model, similar to BERT [
The DINOv2 architecture comes in diferent sizes, each with its respective embedding dimensions:
small (S) with 382, base (B) with 768, large (L) with 1024, and giant (g) with 1536. We chose the ViT-B/14
(distilled) base model as it provides a balance of computational eficiency, performance, and feature
representation suited for our use case, ensuring the model is powerful enough to extract meaningful
features while remaining computationally feasible. The base model produces embeddings with a fixed
size of R257× 768, regardless of the input image dimensions. This fixed size is due to the base model’s
architecture: each image is divided into 256 fixed-size patches, and each patch is embedded into a
768-dimensional vector. Additionally, a [CLS] token, also with a 768-dimensional embedding, is added
to the sequence, resulting in a total of 257 vector embeddings per image [
The organizers provided two DINOv2 models based on the ViT-B/14 (distilled) architecture, each
using a self-supervised learning method [
We conducted experiments on the embedding datasets to maximize our performance on the public leaderboard. Initially, we establish a baseline with minimal modifications, performing multi-class classification on the multi-label test dataset. Subsequently, we introduced more complex inference approaches, such as grid-based image prediction for multi-label classification.
Our two main approaches were as follows: (1) Extract embeddings using both base and finetuned DINOv2 models from the cropped and resized single-label image dataset and the multi-label test dataset, train classifiers on the training embeddings, and perform classification on the test embeddings. (2) Perform inference using the fine-tuned ViT model with both full-image and grid-based image prediction approaches. We measure model performance using the metrics referenced in Section 3.3. The results of our derived datasets and models are summarized in Table 3 and 4, respectively.
To address the computing and memory constraints of the large-scale training data, our solution leverages several technologies. We use Google Cloud Platform (GCP) for computing and storage, Apache Spark for distributed data processing, Petastorm for distributed data loading, PyTorch Lightning for model training, and Weights and Biases for experiment tracking. Our primary compute resources include the n2-standard-4 VM instance (4vCPU, 2 core, 16GiB memory) and the g2-standard-8 GPU instance (8vCPU, 4 core, 32GiB memory), scaling up as needed for the dataset’s magnitude.
Apache Spark was crucial to our entire pipeline, especially for preprocessing and modeling tasks. PyTorch Lightning provides a high-level interface for our deep learning workflows, enabling eficient model training and hyperparameter tuning. Our end-to-end pipeline comprises four main components: downloading, preprocessing, modeling, and inference, illustrated in Figure 2.
We utilize aria2, a lightweight multi-protocol and multi-source command-line download utility that facilitates fast and reliable downloading of large datasets. The images and metadata files were downloaded and stored in Google Cloud Storage for subsequent preprocessing.
The preprocessing phase involves two main steps: converting images to Apache Parquet format and performing cropping and resizing operations. We concatenate image data in batches to optimize cloud computation, as reading millions of images incurs significant network overhead. Columnar data formats like Parquet eficiently represent data for batch processing, handling both binary and metadata.
Many images in the dataset are rectangular and inconsistent in resolution. To improve processing eficiency, we crop and resize all images to ensure the subject remains in focus, as shown in Figure 3. Each image is cropped to a square centered at the midpoint and resized to ℛ128× 128 pixels. This step reduced the dataset size from 281GiB to approximately 15GiB, achieving more than an order of magnitude reduction. This reduction facilitated faster embedding extraction using DINOv2 and decreased computational load during training and inference. We choose a relatively small, square dimension to allow for constructing a multi-class dataset from collages of smaller images.
We preprocess each image into a grid of tiles and extract DINOv2 features for each tile (Figure 1). Using the base DINOv2 model, we extract full-size embeddings ℛ257× 768 from each image, including DCT coeficients of the tile tokens and the [CLS] token. With the fine-tuned DINOv2 model, we focus on extracting the [CLS] token embeddings. We create the following embedding datasets in Table 1.
Base DINO DCT. The base DINOv2 model extracts embeddings from the cropped and resized
singlelabel image dataset. The embeddings have dimensions of ℛ257× 768. We apply the DCT algorithm for
dimensionality reduction with an 8x8 filter size, resulting in a ℛ1× 64 tensor. The DCT captures
multidimensional low-rank structures in frequency and is known for its compressive properties on datasets,
such as JPEG and MP3 for images and audio respectively [
Base DINO [CLS] token. The base DINOv2 model extracts the [CLS] token embeddings from the cropped and resized dataset. The [CLS] token is a special token added to the input sequence of the Vision Transformer, aggregating information from all tiles and providing a generalized feature representation of the entire image for classification tasks. The resulting [CLS] token embedding is a tensor of ℛ1× 768 dimensions.
Fine-tuned DINO. We use the fine-tuned model dinov2-onlyclassifier-then-all discussed in Section 2.1 to extract [CLS] token embeddings from the cropped and resized single-label image dataset, resulting in a tensor of shape ℛ1× 768.
We train linear classifiers on both DCT-reduced embeddings and [CLS] token embeddings using PyTorch Lightning and the negative log-likelihood loss. For inference, we performed multi-class classification on full-size images to predict a single plant species per image.
To address multi-label classification, each test image is divided into a grid of tiles, with embeddings extracted for each tile using the fine-tuned DINOv2 model. We then perform inference on these tile embeddings using argmax and top probabilities for prediction aggregation.
Our inference workflow employs a Luigi [
We employ two distinct approaches to image prediction on the test dataset, as shown in Figure 4: full-image and grid-based image prediction. The test dataset is not cropped and resized to preserve the high quality of multi-label images.
Full-Image Prediction. In this approach, the entire test image is processed in its original dimension. The fine-tuned ViT model evaluates the image and outputs probabilities for each of the 7806 plant species classes. We then extract the top 20 probabilities, representing the most likely species present, and map these probabilities to their corresponding species IDs.
Grid-based Image Prediction. We segment the image into an × grid of tiles, resulting in tiles. Each tile is independently processed using the fine-tuned ViT model, which outputs probabilities for each species class. We then select the top probabilities (default = 10) for each tile and map them to their respective species classes. This produces a nested array of prediction lists per tile, which we flatten into a single array, further limiting to the top probabilities (default = 5) from each tile. For example, a 3 × 3 grid yields nine tiles, each with ten top probabilities, selecting the top five logits in each tile, totaling 45 species IDs and probability mappings.
Let , be the probability of species in tile . For the argmax approach, we select: For the top probability approach, we aggregate the top logits per tile, and select the top species across all tiles, where = 5, and is the total number of tiles: ^ = argmax,
⎛ ⎞ ^ = Top- ⎝ ⋃︁ Top-(:, )⎠
=1
We generate a consolidated array of predictions whether processing the entire image or employing the grid-based approach. This array, a collection of species ID and probability mappings, is sorted in descending order based on the probability scores. This prioritization helps accurately identify the most likely species in each image tile.
The training dataset exhibits significant class imbalance, with a highly right-skewed distribution of species. Of the approximately 7,800 species, nearly half have fewer than 100 images. To mitigate this imbalance, we select a subset of plant species with at least 100 images for training While this approach may overlook rare species, we hypothesize that focusing on more represented species will increase the confidence in species identification.
By selecting species with a minimum of 100 images, we aim to improve the balance between precision and recall for the included species, directly impacting the Macro F1 Per Species and Macro F1 Per Plot metrics. While this may lead to a lower recall for rare species in the test set, it enhances the precision for the more common species, resulting in higher Macro F1 scores for those species. For the Micro F1 score, which is more sensitive to the performance of common species due to the overall count of true positives, false positives, and false negatives, our approach is likely to result in higher scores as well.
The metrics used to evaluate our model are Macro F1 Averaged Per Plot, Macro F1 Averaged Per Species, and Micro F1 scores.
The F1 score is the harmonic mean of precision and recall, defined as:
· 1 = 2 · + where and denote precision and recall, respectively.
The Macro F1 Averaged Per Plot and Macro F1 Averaged Per Species are calculated as follows: Macro F1 Per Plot = 1 ∑︁ 1(, ^) =1
Macro F1 Per Species = 1 ∑︁ 1(, ^) =1 (1) (2) (3) (4) (5) where is the number of plots, is the number of species, is the true label for plot , and ^ is the predicted label for plot .
The Micro F1 score aggregates the contributions of all classes to compute the average F1 score: Micro F1 =
2 ∑︀=1 2 ∑︀=1 + ∑︀=1 + ∑︀=1 (6)
We present our best results on the public and private leaderboards. Our best model utilizes a linear classifier trained on the fine-tuned DINO embeddings (Table 1) and performs inference using a gridbased approach with argmax logit per tile. We achieved public scores of 20.77 for Macro F1 Averaged Per Plot, 47.42 for Macro F1 Averaged Per Species, and 19.67 for Micro F1, as shown in Tables 2 and 3.
The linear classifier trained on the fine-tuned DINOv2 embeddings consistently outperforms base DINOv2 models across all metrics, demonstrating enhanced feature representation from additional training (Table 3). The base DINOv2 models show relatively low performance, with the model trained on the [CLS] token embeddings performing slightly better than the DCT embeddings. The lower performance of the DCT embeddings is due to their reduced dimensionality ℛ1× 64 compared to the [CLS] token embeddings ℛ1× 768, leading to information loss. While base [CLS] token embeddings are more efective than DCT embeddings, both are less efective than fine-tuned model embeddings.
The multi-class, full-image inference method with the linear classifier trained on fine-tuned DINOv2 embeddings shows significant improvement over the base models. For a more sophisticated approach, we utilize a grid-based inference method to improve multi-label classification capability, employing two strategies: argmax logit per tile and top 5 species per tile. The grid-based approach with argmax logit per tile achieves the highest scores across all metrics. The grid-based approach with top 5 species per tile also performs well with slightly lower performance due to less confident predictions introducing noise and ambiguity, and the complexity of aggregating multiple logits, resulting in suboptimal predictions.
For inference using the fine-tuned model, we found that a 3 × 3 grid size struck a balance between computational eficiency and species distribution, having experimented with 2 × 2 and 5 × 5 grid sizes without observing significant improvements, as shown in Table 4. We chose 5 species per tile to maximize the Macro F1 Score Per Species, focusing on the most represented species in the dataset. However, this approach leaves room for improvement, particularly for rare species that are underrepresented.
By leveraging transfer learning, we use embeddings from both base and fine-tuned DINOv2 models to train linear classifiers, addressing the challenge of single-label training data versus multi-label test data. Our preprocessing pipeline manages the large-scale dataset by converting images to Parquet format, reducing dimensionality through cropping and resizing, and decreasing computational load for extracting embeddings and making inferences.
We demonstrate the efectiveness of using DCT for dimensionality reduction and the [CLS] token for generalized feature representation. The fine-tuned DINOv2 model provides enhanced feature representations crucial for multi-label classification. Our grid-based inference approach with argmax and top probability aggregation enables accurate prediction of multiple species within each image.
We did not test the base DINOv2 in a grid-based manner or the fine-tuned DINOv2 with full-image inference due to computational constraints and our hypothesis that fine-tuning would yield better results when exploiting local features through tiling. While capturing the top 5 logits per tile provides broader species identification, it also introduces less confident predictions, complicating final aggregation. Conversely, the argmax method directly selects the highest probability, leading to better predictions.
In Figure 5, we visualize the spatial separation between species using the embeddings extracted from
the base and fine-tuned ViTs. The UMAP [
The image token DCT coeficients exhibit the poorest species separation among the three approaches. Although efective for dimensionality reduction and capturing low-frequency information, DCT does not inherently learn discriminative features for classification tasks. In contrast, the base DINOv2 [CLS] token embeddings show better species separation, as the [CLS] token aggregates information from all image patches, providing a more comprehensive representation of the entire image.
The fine-tuned DINOv2 model’s [CLS] embeddings demonstrate the clearest species separation. The additional training on the specific task data refines the model’s feature representations, making them more relevant and discriminative for classification. Fine-tuning efectively bridges the gap between general-purpose feature extraction and task-specific discriminative power, resulting in more accurate and reliable species classification, as shown in Table 3.
One direction for future work is to generate a collage dataset by tiling individual species that are
likely to co-occur. We propose a collaborative filtering approach to address the multi-label problem,
leveraging Locality Sensitive Hashing (LSH) [
We present a robust multi-label plant species classification approach using self-supervised Vision
Transformer (DINOv2) models. Our study highlights the potential of combining self-supervised learning
and transfer learning with data processing techniques to tackle large-scale biodiversity challenges. We
ofer a scalable solution for multi-label image classification using only single-label training data. Future
work could enhance model training and inference by integrating additional data augmentation
techniques, experimenting with various grid sizes, exploring other dimensionality reduction methods, and
utilizing alternative loss functions such as binary cross-entropy and asymmetric loss [