Introduction

Correctly identifying mushroom species and distinguishing between poisonous and edible varieties are critical for public health. In 2023 alone China had 1,303 reported cases of mushroom poisonings traced to 97 species of mushroom, of which 12 were newly discovered species [1]. FungiCLEF 2024 [2] is a competition held as part of the LifeCLEF 2024 [3] lab 1 . FungiCLEF is a long-tailed fine-grained open-set classification task with an additional asymmetrically weighted edible-poisonous confusion component. In this work, I propose a novel solution for fine-grained classification of fungi species that simultaneously minimizes confusion between edible and poisonous species as well as detecting species of fungi unknown to the training dataset. The primary contributions of this work are 1) the use of an open-set recognition classifier trained using the embeddings from the closed-set classifier applied to fine-grained open-set recognition as well as 2) leveraging an ensemble of computationally lean models with carefully selected test-time augmentations. Extensive experimentation was used to improve the training methodology of this discriminator. I refer to this optimized open-set classifier training paradigm as OpenWGAN-GP.

CLEF 2024: Conference and Labs of the Evaluation Forum, September 09-12, 2024, Grenoble, France * Corresponding author. Envelope jack.etheredge@gmail.com (J. N. Etheredge) GLOBE https://github.com/Jack-Etheredge (J. N. Etheredge) Orcid 0000-0001-5467-3866 (J. N. Etheredge) is a modification to standard cross-entropy loss. Given predicted logits 𝑧 and predicted probabilities 𝜎 from the classifier, and 𝑦 𝑖 is the one-hot encoded ground truth label with 1 <= 𝑖 <= 𝐶, seesaw loss is defined as

𝐿 seesaw (𝑧) = − 𝐶 ∑ 𝑖=1 𝑦 𝑖 log(𝜎 𝑖 ),(1)

over classes 𝐶 where 𝜎 𝑖 is defined as

𝜎 𝑖 = 𝑒 𝑧 𝑖 ∑ 𝐶 𝑗≠𝑖 𝑆 𝑖𝑗 𝑒 𝑧 𝑗 + 𝑒 𝑧 𝑖 . (2)

𝑆 𝑖𝑗 is a balancing coefficient between different classes. 𝑆 𝑖𝑗 is determined by a combination of a mitigation factor 𝑀 𝑖𝑗 and a compensation factor 𝐶 𝑖𝑗 :

𝑆 𝑖𝑗 = 𝑀 𝑖𝑗 ⋅ 𝐶 𝑖𝑗(3)

𝑀 𝑖𝑗 mitigates the penalty on tail classes based on their instance ratio compared to head classes by decreasing the penalty on class 𝑗 relative to the ratio of instance counts between the less abundant tail class 𝑗 and the more abundant head class 𝑖. Conversely, 𝐶 𝑖𝑗 increases the penalty on class 𝑗 whenever a misclassification occurs from class 𝑖 to class 𝑗. This dual-factor approach in 𝑆 𝑖𝑗 allows Seesaw loss to dynamically adjust penalties based on both instance distribution and misclassification behavior, optimizing the learning process in long-tailed multi-class classification tasks. The loss function is explained in greater detail in the original paper [20].

The closed-set image classification models used to classify the images belong to a family of hybrid convolutional and self-attention transformer models known as Metaformers [21]. These models are explained in detail in Section 3.5.

The other challenge is the recognition of the open-set "unknown" class. OpenWGAN-GP was used to classify images as belonging to the closed-set or open-set datasets. The architecture of the open-set discriminator and the OpenWGAN-GP training methodology are described in greater detail in Section 3.6.

Evaluation metrics

The public leaderboard for the competition reported multiple metrics for each submission. Track 1 was a classification loss that included unknowns. Track 2 was an edible-poisonous confusion loss with a ×100 weight for poisonous → edible misclassification. Track 3 was the sum of the Track 1 and Track 2 losses. Additionally, the macro-averaged F1 score and accuracy were reported. The accuracy has been ignored for all experimental results reported here. Apart from the macro-averaged F1 score, none of the metrics correct for class imbalance. Thus the impact of classification performance on each class impacts final performance for Tracks 1-3 proportional to the number of observations for that class.

Track 1 loss is a standard classification error with an additional "unknown" class:

𝐿 1 = ∑ 𝑖 𝑊 1 (𝑦 𝑖 , 𝑞(𝑥 𝑖 )),(4)

for class predictions 𝑞(𝑥) for observations 𝑥 from a classifier 𝑞 and true labels 𝑦. The cost function 𝑊 1 is defined as

𝑊 1 (𝑦, 𝑞(𝑥)) = { 0 if 𝑞(𝑥) = 𝑦 1 otherwise . (5)

Track 2 loss penalizes the confusion of edible and poisonous species. Consider a function 𝑝 that indicates poisonous species as 𝑝(𝑦) = 1 if species 𝑦 is poisonous, and 𝑝(𝑦) = 0 otherwise. Let 𝑐 𝑃𝑆𝐶 denote the cost for poisonous → edible misclassification (a poisonous observation was predicted as edible) and 𝑐 𝐸𝑆𝐶 the cost for edible → poisonous misclassfication. 𝑐 𝐸𝑆𝐶 = 1 and 𝑐 𝑃𝑆𝐶 = 100. Track 2 loss is defined as:

𝐿 2 = ∑ 𝑖 𝑊 2 (𝑦 𝑖 , 𝑞(𝑥 𝑖 )),(6)

for class predictions 𝑞(𝑥) for observations 𝑥 from a classifier 𝑞 and true labels 𝑦 as in 𝐿 1 . The cost function 𝑊 2 is defined as

𝑊 2 (𝑦, 𝑞(𝑥)) = ⎧ ⎨ ⎩ 0 if 𝑝(𝑦) = 𝑝(𝑞(𝑥)) 𝑐 𝑃𝑆𝐶 if 𝑝(𝑦) = 1 and 𝑝(𝑞(𝑥)) = 0 𝑐 𝐸𝑆𝐶 otherwise (7)

Track 3 (the "user-focused loss") is simply the sum of Track 1 and Track 2 losses:

𝐿 3 = 𝐿 1 + 𝐿 2 .(8)

Custom poison loss

A custom poison loss was used for all of the final models. The poison loss was formulated as a class weighted binary cross entropy loss.

Given the set of poisonous classes 𝑃 and the set of edible classes 𝜀, sum the probabilities of each class independently where 𝑃 𝑖 is the softmax probability output for class 𝑖.

𝑃 𝑝𝑜𝑖𝑠𝑜𝑛𝑜𝑢𝑠 = ∑ 𝑖∈𝑃 𝑃 𝑖 .(9)𝑃 𝑒𝑑𝑖𝑏𝑙𝑒 = ∑ 𝑖∈𝜀 𝑃 𝑖 .(10)

Let 𝑦 ∈ {0, 1} be the binary ground truth label for an image, where 1 indicates a poisonous class and 0 indicates an edible class. 𝛼 = 100 is the weight assigned to the edible class to penalize edible → poisonous misclassifications. Thus, the weighted cross-entropy loss is as follows:

𝐿 𝑝𝑜𝑖𝑠𝑜𝑛,𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑 = −[𝑦 log(𝑃 𝑝𝑜𝑖𝑠𝑜𝑛𝑜𝑢𝑠 ) + 𝛼(1 − 𝑦) log(𝑃 𝑒𝑑𝑖𝑏𝑙𝑒 )](11)

Since the softmax probabilities output by the model sum to 1, the probabilities for all the poisonous classes and all the edible classes were summed independently and used as the prediction for the binary cross entropy criterion. A weight of 100 was assigned to the edible class, since edible → poisonous misclassifications (true label is edible, predicted label is poisonous) was penalized ×100 in the Track 2 loss. The total training loss was the sum of the seesaw loss and the custom poison loss. This approach ensures that the training process emphasizes correctly classifying edible species as edible, thus reducing the risk of mistakenly classifying edible species as poisonous, which is heavily penalized in the evaluation metrics.

Model architectures

All experiments were performed on a machine with a single NVIDIA RTX 3090 graphics card and all models were trained using PyTorch [22]. Given that I was working on a single computer for this competition, efficient use of the limited compute available was of critical importance. Ensembling and test-time augmentations were used to increase performance while keeping training efficiency in check. An ensemble of computationally lean models have been shown to outperform a single larger model with respect to both training and inference cost [23]. Model architectures were chosen based on their performance on ImageNet and/or iNaturalist relative to the computational complexity of the models in TFLOPs, aiming for a final ensemble of at least two models. Test time augmentations allow for much greater performance without requiring any additional training of the models, making them a particularly attractive target for optimization when compute and time are both limited.

Metaformers [21] are a family of models that combine different tokenizers with a transformer backbone. A collection of Metaformer models (Metaformer-0, Metaformer-1, and Metaformer-2) were created in [4] that combine metadata with the images to improve the classification performance of the models on multiple fine-grained image datasets. CAFormer [24] models are very similar in architecture to the Metaformer variants proposed in [4], but do not make use of metadata. In the final ensemble, Metaformer-0, Metaformer-2, and CAFormer-S18 were used. Hereafter Metaformer refers to the Metaformer models from [4] which incorporate metadata information.

I fine-tuned CAFormer-S18 with weights pretrained on ImageNet-21K [25] while Metaformer-0 and Metaformer-2 models were pretrained on iNaturalist2021 [5]. CAFormer-S18 was fine-tuned on a different train-validation split than the two Metaformer models. Metaformer-0 and Metaformer-2 differ only in the number of channels in the convolutional and transformer blocks, with Metaformer-2 having more channels in every block. The S18 variant of CAFormer refers to a specific combination of convolutional and self-attention token mixers. CAFormer-S18 utilizes a total of 18 blocks: 3 convolution blocks with 64 channels, 3 convolution blocks with 128 channels, 9 attention blocks with 320 channels, and 3 attention blocks with 512 channels.

OpenGAN and OpenWGAN-GP

To my knowledge, this is the first time that OpenGAN has been utilized for open set recognition of fine-grained images beyond digit recognition. In order to improve training stability, I incorporated the Wasserstein GAN loss and gradient penalty (WGAN-GP) [26] into the training of OpenGAN [13] to create OpenWGAN-GP. In addition to incorporating WGAN-GP, batch normalization layers were replaced with layer normalization layers for the discriminator as suggested in [26].

OpenGAN proposes selection of the discriminator against a validation set of open-and closed-set examples, selecting the discriminator with the best validation ROC-AUC. However, since the ROC-AUC is calculated using a range of classification thresholds, the best classification threshold would also need to be determined for each OpenGAN discriminator. As such, the macro-averaged F1 was used as the selection metric instead of ROC-AUC for OpenWGAN-GP. Additionally, if the OpenWGAN-GP discriminator is selected based on macro-F1, an ensemble can be averaged without the need to calibrate the classification threshold.

Another difference from the original OpenGAN paper is that models were selected based on their macro-F1 performance rather than the proposed ROC-AUC. Since models would be used in an ensemble and the OpenWGAN-GP probabilities would be averaged, it was important to assume that the classification threshold for all of the individual OpenWGAN-GP models would be the same. An alternative strategy would have been voting, which would have allowed for different classification thresholds per model, but this was not explored in this study.

OpenGAN is a methodology for training a lightweight discriminator that utilizes the intermediate representation of an image to generate a binary classification for open-set recognition. Several related methods were proposed, but the one that performed best in their experiments and the one that I focus on in this work is OpenGAN fea with the inclusion of open-set training data, which I will simply refer to as OpenGAN. This paradigm allows for training a classification without initial consideration of the open set data. The discriminator is a multilayer perceptron consisting of fully connected layers with sizes 𝐷 → 𝐻 × 8 → 𝐻 × 4 → 𝐻 × 2 → 𝐻 → 1, where 𝐷 represents the dimension of the intermediate representation from the closed-set classifier and 𝐻 is a hidden dimension multiplier. 𝐻 = 64 unless otherwise specified. The output layer uses a sigmoid activation function. Batch normalization [27] and LeakyRELU [28] are used between each dense layer. During training, the generator generates a feature vector of length 𝐷 from a 100-dimensional input vector with each value sampled independently from a standard normal distribution (mean 0, variance 1). The generator is also a multilayer perceptron with batch normalization and LeakyRELU. It has a similar architecture, but there are some critical differences. The output dimension of the generator must match the input dimension 𝐷 [29]. As such, the discriminator is updated twice per update of the generator since the discriminator is trained adversarially against the generator (real vs fake) as well as supervised (openvs closed-set). This training paradigm is illustrated in Figure 1. The Adam optimizer [30] was used with a learning rate of 1e-4 for the discriminator and 2e-4 for the generator. The higher learning rate is used for the generator to account for the 2:1 updates of the discriminator vs the generator.

Metadata

Metaformer-0 and Metaformer-2 allow the fusion of metadata information with the vision information. Metadata was utilized to provide the model with information concerning location, local growth conditions, and temporal information by including the country code, substrate, and habitat, and observation date (month and day). Example substrates include "fruits", "wood", "cones", "soil", and "peat mosses", while example habitats include "bog", "dune", "meadow", and "roof". There are 34, 32, and 31 categories for country code, substrate, and habitat, respectively. Metadata was preprocessed for Metaformer-0 and Metaformer-2 according to [17]. The month and day were transformed by periodic encoding into [sin ( 2𝜋 month

12

) , cos ( 2𝜋 month

12

)] and [sin (

2𝜋 day 31 ) , cos ( 2𝜋day

31 )] respectively to preserve temporal relationships. Geographical information in the form of country codes, habitat, and substrate were all one-hot encoded. Metaformer-0 and Metaformer-2 utilize trainable embeddings to project this encoded metadata to the same dimensionality as the image features in order to fuse them with the latent representation of the images.

Training settings

I used the AdamW optimizer [31] with an initial learning rate of 1e-3 on only the classification output dense layer with the pretrained model frozen for the first 5 epochs, then reduced to 5e-5 for subsequent epochs. CAFormer-S18 models were trained with a batch size of 40, while Metaformer-0 models were trained with a batch size of 32, and Metaformer-2 models were trained with a batch size of 12. A weight decay of 0.05 was used for all of these models. The learning rate was reduced by a factor of 0.1 if the model did not improve the validation loss for 5 consecutive epochs. Early stopping was also employed when training all models to prevent wasting compute time on models which were no longer improving in their generalization to the validation set as measured by the validation loss.

LogitNorm [32] describes a technique that applies an L2 norm to the logits during training (the norm is not applied during inference). It was included with the hope that it should improve the separation of the classes in the embedding space relative to standard seesaw loss, which might enable OpenWGAN-GP to leverage the embedding space for more accurate open-set recognition. Additionally, LogitNorm was shown to act similarly to temperature scaling [33] to create models that generate less overconfident predictions. This would be important for maximum softmax probability or entropy thresholding, which were explored as alternatives to OpenWGAN-GP.

Training data augmentation

Training was performed with a square random crop, TrivialAugment [34], horizontal flip with 50% probability, and GridMask [35] with a probability of 20%, applied in that order.

Test-time augmentations

At test-time, all images were resized with bicubic interpolation along the shortest dimension to 384 or 576 depending on the model followed by a square center crop of the same size. Test-time augmentations and ensembling were instrumental techniques for the final inference performance. Using a larger image size for inference than training was shown to improve accuracy for multiple datasets in FixRes [36].

The strategies employed were averaging horizontal flips, multi-instance averaging, ensemble averaging, and inference at a higher resolution (576x576) for CAFormer-S18 relative to training (384x384). The overall inference pipeline is summarized in Figure 2. Fivecrop was also investigated, but could not be incorporated in the allowed compute budget. Since it did not yield as significant an improvement relative to a larger ensemble with horizontal flipping according to local evaluation and public leaderboard scores for Track 3, the chosen configuration was preferred. It could be useful in the future to experiment with ensembling techniques that are more sophisticated than simple averaging, but none were attempted in this study.

Due to the open set "unknown" class being implicitly edible, the penalty for misclassifying poisonous mushrooms as unknown was greater than the decrease in misclassification loss. To mitigate this in the proposed solution, if the top prediction of the classification network was a poisonous mushroom, the prediction from the OpenGAN open set classifier was ignored for that observation.

Experimental results

My best-performing ensemble and OpenWGAN-GP combinations achieved 1st place on Track 1, F1, and Accuracy on the private leaderboard. All results reported are for the public leaderboard test set unless explicitly stated otherwise.

Open-set recognition

Different open set detection methods were evaluated against the public leaderboard and these results can be seen in Table 1. Experiments with a local validation set showed that temperature scaling improved the performance of maximum softmax probability (MSP) thresholding as well as softmax entropy thresholding (experiments not shown). OpenWGAN-GP consistently performed better on the leaderboard than softmax thresholding or entropy thresholding, even after temperature scaling [33] the probabilities. As can be seen in Table 1, the performance of either of these methods depends on optimizing a threshold. The optimal entropy threshold for local validation was 6, which did not appear to be optimal for the public leaderboard. This suggests that this method may not generalize well between test sets. OpenWGAN-GP is a binary classifier that is selected using the macro-F1 score with a classification threshold of 0.5, which means that no additional thresholding should be needed to generalize between test sets. Table 1 shows that despite not tuning the classification threshold, OpenWGAN-GP shows the best Track 1 and Track 3 performance while maintaining a similar Track 2 performance to MSP and entropy thresholding after avoiding poisonous → unknown misclassification as explained below. It can be seen from the results in Table 1 that ignoring OpenWGAN-GP predictions for open-set recognition in the cases when the highest predicted probability belongs to a poisonous species ("ignore poison pred") is critical to preventing the open-set recognition from degrading performance on Track 3. OpenWGAN-GP without ignore poison pred achieves a better Track 1 score than OpenWGAN-GP with ignore poison pred, but a much higher Track 2 score. This suggests that in many cases OpenWGAN-GP is correctly identifying unknowns that the classification network is predicting to be poisonous, but that the poisonous → edible cost for the poisonous closed → open misclassifications overwhelms the improvement in classification loss. This reinforces how challenging it is to simultaneously optimize classification performance, identification of unknown species, and avoidance of misclassifying poisonous species as edible.

Following [17], I explored fine-tuning the models through outlier exposure after first training the models without the inclusion of unknowns, but validation loss failed to improve after the first epoch upon inclusion of unknowns (results not shown). It appears that unknowns were included for the entire duration of training in their work. Unfortunately, I was unable to complete this experiment before the competition concluded.

Model architectures and ensemble selection

Multiple architectures were evaluated for this study. Efficientnet-B0 [37] and Efficientnetv2-S [38] were both experimented with but their results were not as promising as Metaformer and CAFormer against a local validation set in early experiments. Results are not shown for these experiments since they are not directly comparable to the experiments reported. CAFormer-S18 has better performance than Metaformer-0 when the image resolution is increased at inference time despite belonging to the same family of models as Metaformer-0 and Metaformer-2, which leverage metadata information. CAFormers performed almost as well as Metaformer despite not utilizing information from the metadata. Future work could evaluate merging the two architectures into a CAFormer with a head for the metadata information. An ensemble of three CAFormer-S18 models that vary only by their training and validation data split performs nearly as well as ensembles of Metaformer-0, Metaformer-2, and CAFormer-S18. The best performing ensemble was Metaformer-0, Metaformer-2, CAFormer-S18 split C. Split C performs better than the other two CAFormer-S18 data splits, which provides a likely explanation as to why this ensemble outperformed Metaformer-0, Metaformer-2, and CAFormer-S18 split A.

Optimization of OpenWGAN-GP training

OpenWGAN-GP training was optimized with respect to the hidden dimension size, ratio of closed and open set samples, and whether training augmentations were applied. Table 3 shows that the best Track 3 performance is achieved when using training augmentations and oversampling the open-set data to roughly the same number of samples as the closed-set data. This case is represented as weighted undersampling (w.u.) for closed-set sampling and "3x all" sampling for open-set sampling, which represents the case of oversampling the open-set dataset completely 3 times with training augmentations to increase the diversity of representations of the limited open-set data. These settings are used for all results shown in Table 2 and the OpenWGAN-GP results in Table 1. For sampling the closed-set data, weighted undersampling outperforms random undersampling and balanced undersampling. In cases where there is a >5% disparity between the number of samples in the open and closed sets, training is performed with balanced sampling between the open and closed sets. This pertains to all combinations except the 3x oversampling of the open set and weighted undersampling of the closed set. Local experimentation suggested that using the entire closed set dataset could yield a slight increase in Track 3 performance, but this dramatically increases the training time for the OpenWGAN-GP classifier (experiments not shown). While the Track 3 performance does not appear to be particularly sensitive to the hidden dimension size, the trend suggests that a smaller hidden dimension may have slightly improved performance, as shown in Table 4. More exhaustive combinations could not be performed due to competition submission limitations.

Test-time augmentations

Several test-time augmentations were evaluated. The performance of each of these augmentations is shown in Table 5. Multi-instance averaging, averaging of horizontal flips of the same image, averaging of multiple crops of the same image, and averaging multiple image sizes are explored with Metaformer-0 trained on split D. Since Metaformer-0 does not support inference at a different resolution than the training resolution (in this case 384x384), the image size in Table 5 refers to the image resolution of the shorter dimension before a square crop of 384. For example, if the image size is 441, then the image is resized to 441 along the shorter dimension (assuming it is a rectangular image) and then a square center crop of 384 is taken. As such, image size must be at least 384 for Metaformer-0. Each augmentation improves performance individually and in combination. Of the test-time augmentations that were experimented with, multi-instance averaging has the greatest impact of any individual transformation.

Leaderboard performance

Public leaderboard performance is shown in Table 10 for teams that have selected models for private leaderboard evaluation. Private leaderboard performance for the selected models for each team are shown in Table 11. The best performance for each metric is independently reported for each team, which means that results for each team may represent distinct solutions for each metric. My models achieved the best performance for Track 1 and accuracy in both the public and private leaderboards and the best F1 for the private leaderboard. My models placed 3rd on the private leaderboard for Track 2 and 2nd for Track 3, indicating that the poisonous → edible misclassification could be improved for the methods presented here.

Discussion

The proposed methodology for open-set recognition of fungi species addresses the critical challenge of distinguishing between edible and poisonous mushrooms while effectively identifying unknown species. This study demonstrates the potential of combining Metaformer and CAFormer models to achieve robust classification performance. The integration of metadata in Metaformer models significantly enhances the model's ability to leverage additional contextual information, thereby improving classification accuracy. However, one notable challenge is the current evaluation metrics, which assume unknown species are edible. This assumption may not be ideal if the models are intended to be used for foraging contexts, where new poisonous species of mushrooms are continually discovered. Re-evaluating these metrics to consider unknown species as potentially poisonous could mitigate the tension between open-set classification and the misclassification of poisonous species, thereby enhancing the practical applicability of the models in real-world scenarios. The current structure of the metrics which treats unknown species as edible puts open-set recognition and poisonous species identification in direct opposition with each other since misclassifying a closed-set poisonous species as unknown is heavily penalized, making their joint optimization challenging. If the intention is to build a system which displays both high detection rates for unknown species and high recall for poisonous mushrooms, the high penalty for poisonous → edible misclassification would work in favor of rather than against identification of unknown species if unknowns were assumed poisonous instead of edible. This may ultimately improve the model's performance in both aspects. Intuitively, the open-set should be more diverse and thus sharing the label with the inherently less realistic fake generated data seems the more logical choice in most scenarios when greater diversity is expected to be observed in the open-set data.

Future work

Future research should explore the redefinition of evaluation metrics to account for the possibility of unknown species being poisonous. This adjustment could reduce the conflict between optimizing for open-set classification and minimizing poisonous species misclassification. Additionally, investigating more sophisticated ensembling techniques and incorporating advanced data augmentation strategies could further improve model performance. Exploring the use of few-shot learning techniques might address the challenge posed by classes with very few observations. Finally, expanding the application of the proposed OpenWGAN-GP framework to other domains with similar classification challenges could validate its versatility and robustness.

Conclusions

This paper presents a novel method for open-set recognition of fungi species. The integration of WGAN-GP training optimizations into OpenGAN, resulting in OpenWGAN-GP, enhances training stability and enables lightweight discriminators to effectively identify unknown fungi species. An ensemble of Metaformer and CAFormer models is leveraged to classify fungi accurately while avoiding the misclassification of poisonous mushrooms as edible. The application of carefully chosen testtime augmentations, such as image resolution adjustments, horizontal flipping, and multi-instance averaging, dramatically improves classification performance. These techniques collectively contributed to achieving 1st place in the FungiCLEF 2024 competition for Track 1, F1, and Accuracy and 2nd place for the final ranking metric Track 3, which combines edible → poisonous confusion loss Track 2 with standardard misclassification loss including the unknown class Track 1.