FungiCLEF 2022 [4] is a competition held jointly by CLEF 2022 conference [5,6] and FGVC9 workshop at CVPR 2022 conference. The competition release the train data based on Danish Fungi 2020 [7], which aims at fine-grained fungi recognition. It includes both image and meta-information such as habitat, substrate, time, longitude, latitude etc, and contains 295,938 samples belonging to 1,604 species. The competition also release test data which contains 59,420 observations with 118,676 images and 3,134 species, it includes meta-information as train data but miss some attributes such as longitude and latitude. After data analyze, we found the category distribution of the train dataset is long-tailed, as a result, in this work we tackle the competition as a fine-grained, long-tailed, open-set classification task.
Different from common classification tasks that try to distinguish objects with large inter-class variations, Fine-Grained Visual Classification (FGVC) aims at capturing the subtle difference within similar categories, such as differentiating bird species, car types, etc. It is acknowledged that FGVC is a challenging task due to small inter-class variations and large intra-class variations.
Numerous methods for FGVC are mainly focused on modeling discriminative regions, such as part-based model [8,9,10] and attention-based model [11,12]. Recently, inspired by the fact that human experts use meta-information to distinguish visually similar species, there are many works [13,14,1,15] utilize additional information to enhance fine-grained classification performance. Among them, Metaformer [1] is the state-of-the-art work proposed recently, it is a hybrid framework that convolution and transformer are both used. In this work, we taken Metaformer as a strong baseline and improve it progressively.
Recently, transformers have leading the research in the filed of computer vision, starting from Vision Transformer [16], there are many various transformer backbones achieve SoTA performance in a wild range of vision tasks, such as Swin Transformer [17], CSwin Transformer [18], etc. On the other side, ConvNext [2] is a pure convolution backbone, it applies modern training techniques, macro and micro design of the network architecture, achieves comparable results with transformers. In this work, in order to obtain models with distinct difference and enhance the performance of model ensemble, we taken ConvNext as another baseline backbone.
In real world scenarios, the distribution of the categories is often long-tailed. It is well known that major class will dominate the training process and suppress the performance of tail class. Many works designed loss function to deal with the problem of long-tail classification, such as Adaptive Class Suppression Loss [19], Equalization loss [20], Seesaw Loss [3], etc. In our method, we utilize Seesaw Loss to dynamically balance the training process between head classes and tail classes.
For practical application, it usually faces with open-set recognition challenge, the classifier should not only recognize the classes which have been seen during training, but also notice that if a instance comes from unknown classes. Motivated by [21], in this work, we trained the model on known classes to obtain a good close-set classifier, and determine whether a instance belonging to open-set based on the maximum value of it's logit score vector. Furthermore, to avoid tail categories being misclassified as open-set categories, we intuitively design a post process to alleviate the confusion.
Our main contributions in FungiCLEF 2022 competition can be summarized as follows:
β’ We take Metaformer and ConvNext as our strong baseline, then we apply hyper-parameter tuning and some modern training techniques to improve it's performance on fungi dataset.
Motivated by [21], we divide the fine-grained, open-set recognition problem into two parts. Firstly we are attended to lift up the close-set recognition performance, including network architecture,
As shown in Figure 1, we taken MetaFormer [1] and ConvNext [2] as our initial baseline.
MetaFormer is a hybrid framework that combines convolution and vision transformer, it also proposes a simple and effective solution for adding meta-information using the transformer layer.
In our approach, we directly use MetaFormer and modify the input of meta-information. We perform the mapping
), πππ ( 2πππππ‘β
), π ππ( 2ππππ¦ 31 ), πππ ( 2ππππ¦ 31 )] to encode temporal information. We use one-hot encoding to encode category meta-information such as countryCode, Substrate and Habitat. To enhance the model diversity for later model ensemble, we use ConvNext as another network architecture. We apply hyper-parameter tuning to improve their performance, and we will illustrate the ablation studies in Sec 3.2 to show the progressive process.
It is known that instances from head categories dominate the training process, the biased learning lead to misclassification for tail categories. In this work, we borrow the idea from Seesaw Loss [3] to alleviate this problem. During training process, Seesaw Loss dynamically balances positive and negative gradients for each category with a dynamic factor, it reformulate the Cross Entropy loss as
where π¦ is the category label, usually represented by one-hot, π§ is the outputs of model, ΜοΈ π is the probability calculated by πππ π‘πππ₯(π§) with a dynamic factor π. For more detail, please refer to Seesaw Loss [3].
The post process is intuitively designed based on several observations, and applied on final ensemble results. In order to be as clear as possible the process, we write the post process in python-style pseudocode, refer to the Algorithm 1. We detailed it in the following.
Threshold for selecting open-set samples. It is acknowledged that the predict confidence score of open-set samples are relatively low. This phenomenon can be used as the criterion for their recognition. Specifically, we draw the logit (the direct output of the model) frequency distribution of both validation set and test set, shown in Figure 2. As the open set samples are only contained in the test set, we can compare the low confidence areas of the two distributions to approximately get the logit threshold for open set samples. For example, we can set threshold to 5 as an approximate for the Alleviate the influence of microscopy images. In the test set, we find that one test sample may contains several images as shown in Figure 3. For such case, we average the model outputs of them to get the confidence and use argmax to get predicted category. During the above process, we also find that there are images showing huge visual discrepancy with the majority. Specifically, we find that some test samples contain microscopy images such as sample_c shown in Figure 3, these microscopy images tend to produce low confidence due to little training data, it will influence the naive average strategy. To cope with this problem, we delicately design the post process. As shown in Algorithm 1, from line 29 to line 35, if the maximum logit of averaged outputs is lower than a certain threshold, we will look into the maximum logit of all images from a test sample, if it is greater than a certain threshold, we will get the corresponding category as the prediction. In this case, we think that the test samples with low averaged outputs may be caused by containing too many microscopy images, the high confidence prediction of one image from test sample is sufficient to infer the category, and the test sample should not be considered as open-set categories. We found many test samples contain several images, we predict the category of the test sample by averaging the model outputs of them. We also notice that some test samples such as sample_c contains one image pictured from natural environment, and the other images come from microscope view, it will disturb the average results.
Distinguish tail categories and open-set categories. We put the tail categories that are never been predicted by the model into hard tail categories, we argue that there are many hard tail categories misclassified as open-set categories. To deal with the problem above, we design the post process, refer to the Algorithm 1. From line 18 to line 27, to avoid the misclassification, we mining hard tail categories from top-3 predictions with low threshold filtering.
Algorithm 1 Pseudocode of Post Process in a python-like style.
In this section, we first elaborate on the implementation and training details. Then we introduce ablation studies on loss functions and bag of training settings. Then we list some other attempts and it's results. Finally we study on different test time augmentations, and show the effectiveness of post process for tail categories recognition and open-set categories recognition.
We trained the model on Danish Fungi 2020 dataset [7] which contains 295,938 training images belonging to 1,604 species observed mostly in Denmark, the dataset has been divided into train and validation set. We use both train and validation for training in most settings. We report the results on test set which contains 59,420 observations with 118,676 images and 3,134 species. The test set is divided into 2 parts, the public set contains 20% of the data, the private set contains 80% of the data. As the performance of open-set recognition affects the mean π 1 score, to make relative fair comparison in ablation studies, based on the observation illustrated in Sec 2.3, we utilize threshold to select βΌ1000 samples which have low confidence score as open-set samples for most experiments. We conduct all the experiments with Tesla V100 (32G). We use AdamW optimizer with cosine learning scheduler, initialize the learning rate to 5π β5 and scale it by batch size, we follow most of the augmentation and regularization strategies of [17] in training.
As shown in Table 1, we train MetaFormer-0 for 100 epochs, with Soft Target Cross Entropy loss and mixup [22] augmentation to build our baseline. For ablation studies, it should be noted that except the parameter to be compared, there are little other not consistent parameters, such as the accumulate steps in last row in Table 5, we argue that it will not affects the conclusion largely.
Losses. As shown in Table 2, we compare different losses with several common augmentation techniques. Specifically, we compare cross entropy loss with either mixup or label smoothing [23] and Seesaw loss. They are all devoted to alleviate the long-tail problem in training. It is found that label smooth converges faster than mixup in our experiments. The best performance is achieved when Seesaw loss is adopted. Batch size. Table 3 illustrates that larger batch size improves the performance. in detail, by increasing batch size from 32 to 64, we improved π 1 score from 76.90% to 77.49% on public set, consistently improve π 1 score from 72.48% to 74.27% on private set. Similar techniques is to increase the accumulate steps. As shown in Table 4, enlarging accumulate steps improves mean π 1 score in private test set consistently with MetaFormer-0 and MetaFormer-1.
Training epochs. We found the longer training epochs will not definitely improve the performance. As shown in Table 5, for MetaFormer-0 and MetaFormer-2, it is consistent that proper epochs is essential for better result. Following this line, we did not train models with dozens of epochs such as 100 epochs and above. Image size. Usually, training with larger image size improves the overall performance, especially for fine-grained tasks. We use the 384 as the baseline image size and try several other larger settings. As shown in Table 6, the larger image size 448 does not consistently bring improvements on public test set. We blame it to the coupled training schedules with the image size, which we do not investigate into it. Finally, we adopt the image size 384 in all settings. Nevertheless, the performance with this baseline is satisfactory enough. Pretrain dataset. We transfer MetaFormer-2 pretrained on different dataset such as herbarium, imagenet22k and inaturalist21. The results are shown in Table 7. Experimentally, we do not directly choose the best-performed pre-training model. Instead, we use ensemble techniques to combine them. We find that ensemble will produce a consistent improvement compared with single model. Even combining the best-performed single model with other slightly poorerperformed models will not affect the conclusion.
ConvNext. In addition to MetaFormer, We also train ConvNext. The results are listed in Table 8.
The experiments in ConvNext is not fully explored compared to MetaFormer. Although the results of ConvNext are inferior to MetaFormer, we still add it to the model ensemble process and the performance is also improved. Pseudo label. After training models with various settings, we use model ensemble to get the best model currently, and take the model predictions on test samples as their label. We select top βΌ 50% test samples by their confidence score. We trained MetaFormer-2 and ConvNext-large with train+val+pseudo, the results are listed in Table 9. For open-set recognition, we intuitively select samples with lower confidence score as open-set samples. As shown in Table 12, by increasing open-set sample from βΌ 1000 to βΌ 1500, we improve mean π 1 score from 83.26% to 83.50% on public test set, from 79.38% to 79.60% on private test set. It demonstrates the effectiveness to select a proper open-set threshold. Compare the average ensemble (v3) with average ensemble (v2), v3 improves the ensemble performance, the only difference between them is that v3 contains the models trained with pseudo label. It is acknowledged that the tail categories tend to have lower confidence score compared to head categories, so the tail categories are easier to be misclassified as head categories or wrongly identified as open-set categories. As shown in Table 12, with our post process for tail categories applied on average ensemble (v3), which termed as average ensemble (v4), the mean π 1 score improves a lot on both public test set and private test set.
In this paper, we introduce our solution for FungiCLEF 2022 competition. To solve this challenging fine-grained, open-set problem, we try a bunch of techniques, such as different network baseline, hyper-parameters tuning, modern training techniques, loss for long tail recognition and specially designed post process. With these endeavours we achieved 1st place among the participators. The experimental results show the progressive process for single model, and the effectiveness of test time augmentation and post process for tail categories. For future work, it is valuable to study the method that fuse meta-information and visual information for Fine-Grained Visual Classification, and the problem of distinguish between tail categories and open-set categories is also worth exploring.