SnakeCLEF 2022 is a fine-grained image classification benchmark for snake identification. Recently, the masked autoencoder (MAE) has shown superior performance on fine-grained image classification tasks. As a result, we use the MAE pretrained ViT models and refine them on the SnakeCLEF 2022. Overall, the learning process contains two difficulties: 1) dealing with fine-grained species that are visually similar and 2) a long-tailed distribution. To address these issues, we propose using statistic-aware post-processing to process the metadata and refine image predictions. Next, we improve an effective logit adjustment loss (ELAL) to alleviate the classification bias toward the head class. Notably, we achieve 2nd place on the SnakeCLEF 2022 benchmark with a 0.84565 top F1 score. Codes and models are available at https://github.com/ylingfeng/snakeclef2022_fgvc9.
Fine-grained visual categorization [1,2,3,4,5,6,7] is a popular task to identify fine categories out of coarse divisions. Recently, there is an increasing necessity to develop a fine-grained visual categorization algorithm for various species of snakes for biodiversity, conservation, and global health. The SnakeCLEF 2022 benchmark 1 [8] aims to tackle this requirement, which is held by LifeCLEF [9,10] jointly with FGVC9 2 of the CVPR 2022.
The difficulty in fine-grained snake identification lies in the high intra-class and low interclass differences in appearance, and many species are visually similar to others. Moreover, the species distribution in terms of geographical location is irregular, and some countries (e.g., US) contain hundreds of species while some (e.g., Vatican) have only a few types. In addition, the dataset suffers from a severe long-tailed problem in which two-thirds of categories contain less than 100 instances.
In terms of the above problems, we propose to solve them individually. First, as for the visually similar samples which are confusing for image-only predictions, we utilize the metadata [11,12,13,14,15,16] provided in the dataset to form a prior distribution of whole species. Different from previous multi-modal methods which embed the metadata to the feature space, we design a parameter-free post-processing structure to refine the predictions. To be specific, we record the number of occurrences of metadata corresponding to each species as the priors. More details can be found in Sec. 4.1. Secondly, in Sec. 4.2 we propose the effective logit adjustment loss (ELAL) to alleviate the prediction bias along with training the long-tailed samples by increasing the optimization weight of the tailed classes while reducing the head.
Our contributions can be summarized as:
β’ We improve a new way to process the metadata by recording statistics referring to each category and a post-processing algorithm is designed to refine the image predictions. β’ We propose the effective logit adjustment loss (ELAL) to alleviate the prediction bias resulting from the long-tailed dataset. β’ Based on our algorithm, we achieve 2nd place on the SnakeCLEF 2022 benchmark with a 0.84565 top F1 score.
Fine-grained image classification: To deal with the fine-grained property which is hard to recognize merely through the visual clues, there are three workable solutions: 1) to detect the discriminative regions of an image and pass all parts through the networks for joint classification [1,3,4,6,7]. 2) Design a robust feature extraction architecture to capture the subtle representations from an image [17,5,2,18,19]. 3) Utilize the metadata (e.g., shooting date, latitude, longitude, country, and a brief description of the image) [11,12,13,14,15,16]. However, the region detector and feature extractor are heavily designed and thus not suitable for our task. Meanwhile, the existing metadata fusion methods all deal with the multimodal feature by embedding them to higher semantic representations before interaction. Specifically, in SnakeCLEF 2022, the types of metadata are discrete (e.g., country, endemic, and code), which is different from the continuous latitude, longitude, or date hypothesized in the previous works.
To make use of this metadata, we calculate the existence label within a certain country for all country values in the metadata and form the prior matrix regarding all species.
Long-tailed distribution: In terms of the long-tailed classification, the data re-sampling [20,21] seeks to change class sampling probability based on the number of samples to get a classbalanced dataset, which includes over-sampling and under-sampling. [22] develop a twostage paradigm to re-balanced the classifier in the second stage with a frozen backbone. Reweighting [23,24,25,26] aims to assign the loss weight class-wise to reduce the optimization bias between head-tail classes. The logit adjustment loss [23] encourages a large relative margin between logits of rare versus dominant labels. Based on this work, we modify the margin coefficient and propose an effective logit adjustment loss (ELAL) to solve the long-tailed problem efficiently.
The SnakeCLEF 2022 dataset [8,9,10] is based on observations of 187,129 snakes, containing 318,532 photographs, belonging to 1,572 snake species, observed in 208 countries. The data comes from the online biodiversity platform, iNaturalist. The provided dataset has a heavy long-tailed class distribution (see Fig. 1), where the most frequent species (Natrix natrix) is represented by 6,472 images and the least frequent species by just 5 samples.
The evaluation metric for this competition is Mean (Macro) F1-Score. The F1 score, commonly used in information retrieval, measures accuracy using the statistics precision (P) and recall (R).
The macro F1 score is not biased by class frequencies and is more suitable for the long-tailed class distributions observed in nature. This metric raises a higher requirement for classification accuracy on tailed categories.
Given the metadata-label mapping, we count the instance number for all categories if it is attached to a certain metadata value. Then, we obtain the statistic in form of a metadata-wise category matrix P β R πΓπ , where π is the value number within one type of metadata, and π is the number of classes. Next, we transform P to a one-hot form P π , known as the prior statistic, which represents whether a specific category would appear in a certain place. Finally, P π is utilized to refine the prediction from the visual networks via the Hadamard product. The whole structure is illustrated in Fig. 2.
In this section, we introduce our new effective logit adjustment loss (ELAL) function which addresses the performance drop resulting from the prediction bias brought by the long-tailed distribution. First, we give a brief review of the existing loss functions, and then we show how ELAL is developed based on them. The vanilla softmax cross-entropy can be derived by:
where π¦ denotes the ground-truth label. The logit adjustment loss [23] adds a label-dependent offset to each of the logits, and modifies Eq. 1 with the shifted coefficient π :
where π = ππ¦β² ππ¦ , π π¦ = ππ¦ βοΈ π¦ β² π π¦ β² β (0, 1), and π π¦ is the total number of instances in each class. Class-balanced Loss [24] proposes the concept of an effective number to replace the direct label-wise instance number to represent the volume of samples. The definition of the effective number is shown as:
Inspired by the conception of effective number, which is an improved representation of the vanilla number, we modify the logit adjustment loss by changing the shifted coefficient π to π = ππ¦β² ππ¦ , π π¦ = πΈπ¦ βοΈ π¦ β² πΈ π¦ β² β (0, 1) and propose ELAL. Notably, we set π½ = 1π β 6 by default in our experiments.
In this section, we first elaborate on our experimental settings, then ablation studies are conducted to demonstrate the performance of each component. Finally, we list the top results of our methods and give a considerable analysis.
In this paper, we use the Masked autoencoder (MAE) [27] pretrained ViT [28] models conducted on ImageNet-1K [29] training set for 800 epochs. The fine-tuning codes and checkpoints refer to the MAE repository 3 . The ImageNet-1K dataset has 1.3M images with 1K categories for training and 50K images for validation. Notably, we do not use the larger ImageNet-22K (IN22K) dataset, which contains 14.2M images and 22K classes. Based on the MAE pretrained models, we finetune 50 epochs on the SnakeCLEF 2022 dataset, and the default setting is depicted in Table 1. We randomly select 1/10 of the training dataset to form the validation set to update our algorithm, and a full set is used to train models which present the final submissions. To be specific, we set batch size per GPU to 2 to avoid exceeding the GPU memory. The effective learning rate is obtained following MAE: lr= base_lrΓglobalbatchsize / 256. We apply random resizing/cropping, random horizontal flipping [30], label-smoothing regularization [31], Mixup [32], CutMix [33], RandomErasing [34], and RandAug [35] as the standard data augmentations. Notably, all ablation studies are conducted under ViT-L for fair comparisons. The ViT-large and ViT-huge models are trained on eight NVIDIA TITAN Xp GPUs (12G) and eight GeForce RTX 3090 GPUs (24G), respectively.
First, we compare the performance with different sets of metadata for post-processing. Table 2 shows that refining predictions with "endemic" and "code" metadata perform the best. Next, we conduct the ablation on two losses. Table 3 shows our ELAL achieves a higher F1 score under two sets of input resolution. To demonstrate the effectiveness of ELAL on tail class and the potential side effect on head class, we calculate the validation accuracy on the top 10/50/100/500 class from the head and tail classes, respectively (Table 4).
Based on the strong ViT-L and ViT-H [28], we conduct experiments with an input resolution of 384/392/448 learned on a full training set. We adopt multi-crop [36] as post-processing strategies, which would crop the given image into four corners and the central crop plus the flipped version and average the predictions of whole crops. The model ensemble is an averaging operation over each prediction score after the softmax of the selected models. Our final submissions come from the ensemble of models w/o and w/ multi-crop, which receives 0.84409 and 0.84565 F1 scores on the private benchmark.
We attempt to run a ViT-H with a 448 resolution, which is capable of reaching a higher accuracy theoretically, however, due to the resource limitation we only present the result of a 392 resolution. Also, we notice that the effect of post-processing on the private benchmark is not as significant as on the public benchmark. We suspect that there is distribution gap between the train and test set of metadata while the public benchmark is less affected.
In this paper, we give our solution to the Snake Recognition Competition (SnakeCLEF 2022) in FGVC9, which is challenging due to the fine-grained categorization and long-tailed classes. To deal with the difficulties, we utilize statistic-aware metadata to refile image predictions through post-processing and propose the effective logit adjustment loss (ELAL) to handle the long-tailed problem, respectively. Our team achieves the 2nd place on the private benchmark with a 0.84565 top F1 score.