The FungiCLEF 2024 challenge aims to foster research in the field of application-oriented fine-grained open-set classification. Particularly, it sets the challenge to optimize fungi species classification while recognizing unknown species with the evaluation of multiple metrics targeting the problems of actual use-cases, e.g., the risk of a highly detrimental confusion of a poisonous species for an edible species. To develop a well-performing approach, we focus on reducing this particular risk by introducing multiple improvements. The major improvements are a poisonous reranking which prevents predicting an edible species while a significant chance of the sample being poisonous exists and a genus loss which provides additional training information improving the regularization of the feature space. The advancements provide a large improvement in terms of poisonous confusion but also in terms of overall classification accuracy. With this approach, we achieved the 1 st place in the challenge's main metric. Code is available at https://huggingface.co/stefanwolf/fungi2024.
Fine-grained open-set classification is an important topic in the biology context in order to find samples of rare species and to provide inexperienced citizen scientist with a support to identify species of plants and animals. Particularly, fungi species classification has an additional use case: identifying poisonous species to reduce the risk of accidental eating poisonous fungi. Thus, the FungiCLEF 2024 challenge [1], part of the LifeCLEF 2024 lab [2], sets up the task of open-set fungi species classification with an emphasis on correctly identifying fungi species while preventing confusing poisonous species for edible species.
While the task of open-set fungi classification has been intensively investigated in the recent iterations of the FungiCLEF challenge [3,4], optimizing the poisonous confusion error has only been lightly approached rendering the room for improvements quite large. Thus, we focus on the poisonous confusion error and can achieve significant gains with multiple advancements. These advancements are • a poisonous reranking which predicts the highest ranking poisonous species if its confidence is only lower by a certain factor than the overall highest ranking species. • a genus loss that regularizes the feature space by incorporating the genus label in training.
• a second open-set threshold to reduce the risk of misclassifying a poisonous sample as an unknown species. • a two-stage metadata integration that enhances the overall classification accuracy.
CLEF 2024: Conference and Labs of the Evaluation Forum, September 09-12, 2024, Grenoble, France stefan.wolf@kit.edu (S. Wolf); philipp.thelen@iosb.fraunhofer.de (P. Thelen); juergen.beyerer@iosb.fraunhofer.de (J. Beyerer)
A wide range of approaches have been proposed targeting fine-grained fungi classification in wildlife images. Sulc et. [5] employ an ensemble of CNNs to classify images of fungi. Picek et al. [6] propose a simple but effective probabilistic strategy to exploit metadata in order to improve the accuracy of fine-grained fungi classification. Kiss and Czùni [7] provide a study about a broad-range of design choices optimizing mushroom type classification accuracy. The 2022 [3] and the 2023 [4] iterations of the FungiCLEF challenge summarize a variety of approaches with the 2022 iteration being focused on improving open-set fungi classification and the 2023 iteration emphasizing the importance of choosing metrics based on use cases to focus research on relevant aspects, e.g., focusing on reducing the confusion of poisonous species with edible species.
The 2024 iteration of the FungiCLEF challenge [1] aims to stimulate research about efficient open-set fungi classification. The target is to distinguish 1,604 fungi species by using an observation consisting of one or multiple images with different perspectives and additional metadata information such as habitat, substrate, time and location. Apart from distinguishing the known species, the submitted approach needs to be able to solve an open-set scenario, i.e., it needs to recognize whether a sample is of a species not part of the training samples. The provided training data consists of the Danish Fungi 2020 dataset [6], the validation data consists of the test set of FungiCLEF's 2022 iteration [3] and the test data consists of new data for the 2024 iteration. The evaluation is done based on three metrics:
• Track 1: Classification error -standard classification with "unknown" category. • Track 2: Poisonous confusion error -cost for confusing edible species for poisonous and vice versa (with 100× weight for confusing edible species for poisonous). • Track 3: User-focused error -user-focused loss composed of both the classification error and the poisonous/edible confusion.
Our challenge submission is based on the approach by Wolf and Beyerer [8]. To simplify the training and due to insignificant impact, we refrain from applying the resampling-based class balancing. We extend the approach by applying several improvements as described in this section, i.e., a two-stage integration of metadata, a genus loss, a poisonous reranking and a two-threshold open-set recognition strategy. The overall architecture of our approach is shown in Figure 1.
We employ a Swin Transformer V2 Base [9] as an image feature extractor backbone. Additionally, we use the metadata information provided by the Danish Fungi dataset [6] to improve the classification accuracy. The metadata is encoded similarly to the approach by Ren et al. [10] We encode the month 𝑚 and day 𝑑 of each observation as a vector
) )︀ ⊺ . The geographical locations country code, substrate and habitat are encoded as one-hot vectors. All metadata vectors are concatenated and fed into two fully connected layers with an output size of 64 and each being followed by a ReLU activation and a layer norm [11]. The resulting metadata feature vector and the image feature vector are concatenated and fed into a final linear classification layer followed by a softmax activation. We apply an auxiliary second classification head for predicting the genus of a sample during training which is fed the image features.
For training the model, we apply two losses: a classification loss on the species level as commonly used and an auxiliary classification loss on the genus level. Both are label-smooth losses [12] with a smoothing value of 0.9. To prevent a degradation of the image features when training with metadata, we use a two-stage training with the first stage only training the image classification stream of the network and the second stage training the complete network including the metadata feature extractor with a newly initialized species classifier.
During inference, we extract the feature vectors of all images of an observation and concatenate the observation-wise mean of the image features with the metadata features before feeding the result into the species classification head. Based on the resulting softmaxed confidence scores, we apply our poisonous reranking which reranks the poisonous species with the highest confidence to the top of the species ranking if its confidence is higher than the actual top-1 species' confidence divided by a poisonous reranking factor 𝛼. After the poisonous reranking, we apply the entropy-based open-set thresholding based on the approach by Ren et al. [10] If the entropy of the output confidences is above a certain threshold 𝜏 , we predict the observation to be out-of-distribution. We extend this approach by employing two thresholds 𝜏 𝑒 and 𝜏 𝑝 , which are applied if the predicted species is edible or poisonous, respectively. The threshold 𝜏 𝑝 is selected higher than the threshold 𝜏 𝑒 to reduce the risk of misclassifying a posionous species as an out-of-distribution species. Both, the poisonous reranking and the second threshold, are improvements targeting the challenge's Track 2 metric which is measuring the poisonous confusion with a significantly higher weight for mispredicting poisonous species than edible species.
We use the official FungiCLEF 2024 [1] datasets. The Danish Fungi 2020 [6] dataset is used for training. All metrics reported in this study are based on the official validation set which is the test set of FungiCLEF's 2022 iteration [3]. The test set for the official ranking is a set of images which has not been disclosed publicly prior to the end of the challenge. Only the results of the public part of the test set was publicly visible with the results on the private part only being disclosed after the challenge deadline.
We use the MMPreTrain [13] classification framework based on PyTorch [14] for the training and the inference of the models. All models are pre-trained on the ImageNet-21k dataset [15] and trained for 24 epochs with an AdamW optimizer [16], a base learning rate of 6.25 • 10 −5 , a learning rate warm-up for 2100 iterations and a cosine learning rate decay. We train with a total batch size of 128. The metadata training is performed for two epochs in a second stage with a frozen image encoder.
Our image pre-processing pipeline for training includes a random crop of an image area between 8% and 100%, a resize to 384×384 pixels, a random horizontal image flip, RandAugment [17] and random erasing [18]. We use 8 Nvidia A100 GPUs for training. The pre-processing pipeline for the inference includes an image scaling with 438 pixels output size on the shorter edge and a center crop of size 384×384 pixels.
We compare a training with and without our genus loss in Table 1. All metrics are improved by the application of the genus loss. Particularly, the Track 2 error focused on identifying poisonous samples as such improves significantly with a drop of 0.22 to 0.18. Nonetheless, also the classification-focused F1 score and Track 1 error show an improvement. The strong increase in terms of identifying poisonous species is likely due to most genus containing only edible species. Thus, considering the genus level in training the feature space results in a denser feature representation of these poisonous-wise uniform genus. Therefore, the chance of misclassification of a species of a uniformly edible genus with a nonedible species is heavily reduced. The improvements in terms of classification accuracy are probably induced by species with a low number of samples. The risk of misclassifying them with species from other genus due to a lack of variance in the data is reduced when also training the feature space on the genus level.
We apply the poisonous reranking additional to the genus loss and evaluate it with different factors. It reranks poisonous species to the top-1 if their confidence is higher than the actual top-1's confidence divided by an 𝛼 > 1 in order to prevent costly confusions of poisonous fungi with edible fungi. The results are shown in Table 2. As expected, it drastically reduces the Track 2 error from 0.181 to 0.076 due to a lower number of samples misclassified as an edible species. While the Track 1 error is slightly increased from 0.41 to 0.416 due to some so far correctly classified samples from edible species now being misclassified as a poisonous species, the increase is small compared to the drop in terms of Track 2 error resulting in a significant drop in the overall Track 3 error from 0.591 to 0.492. Out of the evaluated values of 𝛼, the Track 2 and 3 errors are reduced until an 𝛼 of 10 while a value of 20 is leading to an increase in all error metrics. Particularly, also the Track 2 error increases showing that even the comparatively lowly weighted case of mispredicting an edible species for a poisonous species now playing a significant role.
We employ the entropy-based open-set recognition by Ren et al. [10] and extend it by a second threshold for poisonous species. If the predicted species is poisonous, a higher entropy is needed to classify the sample as out-of-distribution since out-of-distribution samples are considered edible by the Track 2 metric and thus, misclassifying a poisonous sample as out-of-distribution increases the Track 2 error heavily. We compare it to applying no open-set recognition and applying a simple softmax-based thresholding. The results including genus loss and poisonous reranking as baseline are shown in Table 3 and indicate an improvement in all metrics for both open-set recognition methods. The entropy-based approach with two thresholds provide an additional improvement over the softmax-based thresholding.
We integrate metadata information in the inference process by feeding the encoded metadata through two fully connected layers and concatenating the resulting vector to the feature vector of the image encoder before the final linear classification layer. The impact of this metadata exploitation strategy is shown in Table 4 including all previously mentioned improvements. The results show a significant improvement across all evaluated metrics.
The final best-performing model includes all proposed improvements with the following adjustments:
1. a poisonous reranking factor 𝛼 of 13, the overall best performing value on the public test set.
2. an open-set entropy thresholds of 𝜏 𝑒 = 2.5 for edible species and 𝜏 𝑝 = 7 for poisonous species.
3. including the validation set in training with unknown samples being assigned a vector with each element having the same value as target label similar to the approach by Ren et al. [10] and the true genus label for the genus loss.
The final private test set results of the top-5 challenge participants are shown in Table 5. We ranked first with the lowest error in the main metric Track 3 due to a high emphasis on optimizing Track 2 while not lacking too far behind in terms of Track 1. Particularly, we achieve the first place with an efficient solution that consists of only a single model. While the runner-up team achieved a better Track 1 error, the Track 2 error is almost twice as high. In contrast, the third-placed team achieved an even better Track 2 metric than our approach. However, this achievement comes at a large Track 1 error outweighing the advantage.
In this study, we described our top-ranking approach for the FungiCLEF 2024 challenge. With a high emphasis on reducing the risk of confusing poisonous species for edible species, we propose several advancements which improve the respective error drastically while also improving the overall classification accuracy. Particularly, we introduced a poisonous reranking, a genus loss, two-threshold open-set recognition and an efficient two-stage metdata exploitation strategy.
This work was supported by the Helmholtz Association's Initiative and Networking Fund on the HAICORE@FZJ partition.