FungiCLEF 2024 addresses the fine-grained visual categorization (FGVC) of fungi species, with a focus on identifying poisonous species. This task is challenging due to the size and class imbalance of the dataset, subtle inter-class variations, and significant intra-class variability amongst samples. In this paper, we document our approach in tackling this challenge through the use of ensemble classifier heads on pre-computed image embeddings. Our team (DS@GT) demonstrate that state-of-the-art self-supervised vision models can be utilized as robust feature extractors for downstream application of computer vision tasks without the need for taskspecific fine-tuning on the vision backbone. Our approach achieved the best Track 3 score (0.345), accuracy (78.4%) and macro-F1 (0.577) on the private test set in post competition evaluation. Our code is available at https://github.com/dsgt-kaggle-clef/fungiclef-2024.
The featured dataset for the FungiCLEF competition [
These two datasets do not have the same distribution of classes (Figure 2). Moreover, there was significant class imbalance in both datasets, with the most common class having 1,913 images, and the least common class only ~30 images in DF20, and down to only one observation for some species in DF21. There were also significant variations in terms of lighting, background, and clarity, due to the real-world conditions under which fungi were photographed (Figure 1). This adds another level of complexity on this task - Fungi classes are not only hard to distinguish due to inter-class variations or intra-class variance, but also due to varying image quality and image features. This highlights the need for a robust model in efectively performing fine-grained classification on this rich and varied dataset.
State-of-the-art work on this dataset primarily utilizes models such as Swin Transformer [
Beyond FungiCLEF, Wei et al. [
Our overall approach to this challenge of fine-grained classifying of fungi species was to: 1. Incorporate metadata as additional input / prediction targets for the model. 2. Learn the concept of unknown classes by incorporating the validation dataset into training. 3. Experiment with objective functions to induce model capability in fine-grained classification task. 4. Train only on metadata and image embeddings for rapid prototyping and model optimization.
Cloud computing resources were supplied by Data Science @ Georgia Tech. Data was hosted on Google Cloud Storage, and models developed on virtual instances with NVIDIA L4. For more memory intensive experiments, GPUs used in model development include NVIDIA RTX 4090 and a distributed cluster with 2x NVIDIA V100.
Libraries used include pandas [
To improve the eficiency of experiments, we built a data preprocessing pipeline with PySpark (Figure 3). We appended the 300 pixel and full versions of image data with associated metadata, and stored them as parquet files for faster I/O. Embeddings were also precomputed and stored as parquet files separately. A custom PyTorch dataset object was created to serve the image and embedding data alongside metadata.
The metadata columns were grouped based on their potential use as either model inputs or prediction
targets. For the validation set / public test set, only substrate, metasubstrate, habitat, date, and location
were provided [
Given that unknown classes were only present in the validation dataset, we divided DF21 into three equal sections of 20,000 cases, stratified by species. One of the sections was designated as the held out test set, with the remaining two sections utilized as validation set / addition to the training set in a two-fold cross validation training. By percentage, this gives us a ratio of 90.4%, 4.8%, 4.8% for training, validation, and testing over the entire dataset. Significantly, the test and validation sets in each validation fold have the same class distribution.
Embeddings are the learned intermediate representation of deep learning models that capture structure
about the input domain. We experimented with two models as the vision model backbone to generate
embeddings - DINOv2 [
For training, the image embeddings were precomputed. For the testing set and for our competition submission, the vision backbone model was frozen, and embeddings were generated during inference and fed into our trained classifier heads.
We explored two separate approaches in model development (1) Training a computer vision model from
end-to-end, and (2) Training a classifier head only on precomputed embeddings. While approach (1) is
the more traditional method for computer vision tasks, it is much more compute intensive due to the
number of parameters to be trained [
For transfer learning with the embedding model, we use a traditional MLP classifier head with a hidden
dimension of 4096, with metadata directly concatenated to the embedding. Inspired by Diao et al. [
All models were first trained on a smaller, exploratory development set, before undergoing training runs on the full dataset. For experiments that appeared promising in its initial outcomes, training parameters were further tuned using Optuna to generate a full model for benchmark. Training performance were logged on Weights & Biases, with the top 2 performing models saved as checkpoints. A two-fold cross-validation was used, where each fold had 1/3rd of DF21 dataset as the validation set, and another 1/3rd incorporated into the training data. Our experiment logs can be viewed at https://wandb.ai/chiu/FungiClef.
All experiments were trained 20 to 50 epochs each, with batch sizes of 64 to 512. Initial learning rates
ranged from 1 · 10− 5 to 1 · 10− 3, with AdamW [
Metrics recorded during training include training / validation loss, top-1, top-3 accuracy, macro F1
score, and accuracy for correct identification of poisonous species. Calculation for specific track scores
were adapted from the FungiCLEF competition [
The baseline loss function for model development was unweighted, multi-class cross entropy loss. We
also explored incorporating class weights in cross-entropy loss, and other loss functions such as focal
loss [
For our benchmark model, the model was trained with a custom loss function:
composite = seesaw + · poison
Where seesaw is the seesaw loss of the class prediction, poison is the binary cross entropy loss of the model’s prediction in whether the fungi is poisonous, and an adjustable weighting factor for the composite loss function.
While weighted sampler is usually utilized to overcome class imbalance [
Our best performing model was an ensemble model on DINOv2 embeddings consisting of two classifier
heads (180MB each) from the two-folds of cross-validation training. The model was trained on image
embeddings precomputed from DINOv2-large. The weighting for poison loss was 0.1. The initial
learning rate was 1 · 10− 4, with AdamW [
The results of our team’s experiments are outlined in Table 4. For our first submission during competition,
we used a pre-trained MetaFormer model from the previous year’s competition as a baseline. In
postcompetition evaluation, our best model achieved an accuracy of 78.4% and a macro F1 score of 0.577 in
the private test set. Our model’s performance was comparable to previous years’ winners [
We intially experimented with vision models including EficientNet [ 38], VisionTransformer [39],
and MetaFormer [
Overall, while DINOv2 embeddings proved to be good input for image classification, our embedding
model using ResNet embeddings did not perform well, with best validation accuracy at 25%. This was
likely due to DINOv2 being a class agnostic, self-supervised model, whereas ResNet was trained on
ImageNet with specific classification targets. As such, the features extracted from ResNet would be
more tailored to the dataset, whereas DINOv2 features were more representative of the underlying
image [
We experimented with using metadata as additional prediction targets as seen in our ablation in Table 3. However, this did not yield additional performance, but the additional overhead required to tune the weighting of various targets was not worth the complexity. As such, we did not utilize metadata in our final model. The inclusion of metadata appeared to provide some marginal benefits in validation accuracy and F1 score. This echoes the finding from previous research on the dataset, where the incorporation of metadata as input had a positive contribution to model performance. 1Due to numerous issues with the HuggingFace platform, our best results were not recorded in the oficial competition. Our post competition evaluation was performed under the same constraints as the oficial competition. Post-competition results were provided and verified by the organiser of FungiCLEF. 2Oficial competition results are from test submissions with an under-tuned vision model. These results are included for completeness.
Whilst using embeddings allowed for much faster model development time, there is still an additional gap in the performance of the embedding classifier compared to traditional image-based models. It is likely that the information loss in the transformation of image to embeddings was too significant for the simple classifier architecture to overcome. It would be interesting to further fine-tune DINOv2 on the DanishFungi dataset, and repeat our experiments. Moreover, a more rigorous incorporation of metadata into our models could provide a more holistic understanding of the data, leading to more accurate and reliable classification systems.
In summary, we addressed the complex task of fine-grained visual categorization (FGVC) for identifying poisonous fungi using transfer learning and advanced deep learning methodologies. The Danish Fungi 2020 dataset presented significant challenges such as class imbalance, subtle inter-class variations, and high intra-class variability, necessitating a comprehensive data preprocessing and augmentation pipeline.
Our experiments with various deep learning models, including vision transformers, convolutional neural networks, and linear classifiers with embeddings, highlighted the potential of DINOv2 embeddings combined with a multi-layer perceptron. Integrating multimodal metadata further enhanced classification performance, emphasizing the value of auxiliary information. Despite promising results, embedding-based classifiers faced limitations due to potential information loss, suggesting the need for ifne-tuning self-supervised models on domain-specific datasets and improved metadata incorporation. Overall, our research advances FGVC technical capabilities, providing valuable methodologies for mycological safety and educational applications, and contributes to the broader field of fine-grained classification tasks.
We thank the DS@GT CLEF team for providing the development and research environment for our machine learning experiments as well as valuable comments and suggestions. problem, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 875–884. [38] M. Tan, Q. V. Le, Eficientnetv2: Smaller models and faster training, CoRR abs/2104.00298 (2021).
URL: https://arxiv.org/abs/2104.00298. arXiv:2104.00298. [39] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, CoRR abs/2010.11929 (2020). URL: https://arxiv.org/ abs/2010.11929. arXiv:2010.11929. [40] L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction (2018). arXiv:1802.03426.