Introduction

A robust image-based identification system for snake species is an important goal for biodiversity, conservation, and global health. With over half a million victims of death and disability from venomous snakebite annually, such a system could significantly improve eco-epidemiological data and treatment outcomes (e.g. selection of specific antivenoms) [1,2]. Importantly, most herpetological expertise and most snake images are concentrated in developed countries where snake diversity is relatively low and snakebite is not a major public health concern. In contrast, remote parts of developing countries tend to lack expertise and images, even in areas where snake diversity is high and snakebites are common [3,4]. Thus, snake species identification assistance has a bigger potential to save lives in areas with the least information.

A primary difficulty of snake species identification lies in the high intra-class and low interclass variance in appearance, which may depend on geographic location, color morph, sex, or age. At the same time, many species are visually similar to other species, i.e., mimicry (Figure 1). Furthermore, our knowledge of which snake species occur in which countries is incomplete, and it is common that most or all images of a given snake species might originate from a small handful of countries or even a single country. Furthermore, many snake species resemble species found on other continents, with which they are entirely allopatric. Incorporating metadata on the geographic origin of an unidentified snake almost always narrows down the possible correct identifications considerably because only about 125 of the approximately 3,900 snake species co-occur in any given location [5]. It is known that more widespread species with more images are over-predicted relative to rare species with few images [6], and this can be a particularly vexing problem when trying to predict the identity of species that are widespread across areas of the world with few images.

The main goal of the SnakeCLEF 2022 competition was to provide a reliable evaluation ground for automatic snake species recognition. Like other LifeCLEF competitions, the SnakeCLEF 2022 competition was hosted on Kaggle1 primarily to attract machine learning experts to participate and present their ideas.

Dataset

For this year, the dataset used in previous editions [7,8] has been extended with new and rare species. The number of species was doubled and the number of images from remote geographic areas with no or just a few samples was increased considerably, i.e., the uneven species distributions across all the countries was straightened. The SnakeCLEF 2022 dataset is based on 187,129 snake observations -including some instances of multiple images of the same individual (refer to Figure 2) -with 318,532 photographs belonging to 1,572 snake species and observed in 208 countries. The dataset has a heavy long-tailed class distribution, where the most frequent species (Natrix natrix) is represented by 6,472 images and the least frequent species just by 5 samples. The difference in the number of images between the species with the most and fewest was reduced by an order of magnitude relative to SnakeCLEF2021. All the data were gathered from the online biodiversity platform iNaturalist2 . Additional dataset parameters and comparison with previous editions are listed in Table 1. Large countries in the tropics (Brazil, Mexico, Colombia, India, and Indonesia) have more than 300 species. Right: Percentage of snake species per country included in the SnakeCLEF 2022 dataset. The countries with adequate species coverage are those from Europe, Oceania, and North America, i.e., the countries with the smallest diversity.

For testing, two sets were created: (i) the full test set for a machine evaluation, with 48,280 images from 28,431 observations, and (ii) the subset from the full test set with 150 observations, tailored for the human performance evaluation. Unlike in other LifeCLEF competitions, where the final testing set remained undisclosed, we provided the test data without labels to the participants. To prevent over fitting to the leaderboard, the evaluation method was composed of two stages; the first being the public leaderboard where the user scores were calculated on an unknown 20% of the test set, and the second a private leaderboard where participants were scored on the remaining part of the test set. In addition to image data, we provide:

• human verified species labels that allow up-scaling to higher taxonomic ranks, • the country-species mapping file describing country-species presence to allow better regularization towards all geographical locations, based on The Reptile Database [9], and • information about endemic species -species that occur only in one geographical region, e.g., Australia or Madagascar.

Geographical information, i.e., state/province and country labels, was included for approximately 95% of the training and test images. Additionally, we provide a mapping matrix (MM 𝑐𝑠 ) describing country-species presence to allow better worldwide regularization.

MM 𝑐𝑠 = {︃ 1 if species S ∈ country C, 0 otherwise.(1)

Unlike last year's dataset, where the vast majority (77%) of all images came from the United States and Canada, the SnakeCLEF 2022 dataset includes just a fraction of the data (28.3%) from the United States and Canada. The rest of the data is distributed across remaining regions, e.g., Europe, Asia, Africa, Australia and Oceania. This was achieved by limiting the number of observations per species in a given country to 400. The estimated worldwide snake distribution and their coverage is visualized in Figure 3.

Timeline

The SnakeCLEF 2022 competition was announced together with the data in late February 2022 through the LifeCLEF, Kaggle, and FGVC challenge pages. Anyone with research ambitions was allowed to register and participate in the competition. The test data were provided jointly with the training data allowing continuous evaluation. Each team could submit up to 2 submissions a day. The competition deadline was May 16, setting the competition for roughly three months. Participants had to submit CSV files containing the Top1 prediction for each snake observation. Once the submission phase was closed (mid-May), the participants were allowed to submit post-competition submissions to evaluate their ablation studies.

Evaluation Protocol

The main goal of this challenge was to build a system that is capable of recognizing 1,572 snake species based on the given snake observation -unseen set of images -and relevant geographical location. As a main metric, we use the macro F1 score (F 𝑚 1 ). The F 𝑚 1 is defined as the mean of class-wise F1 scores:

F 𝑚 1 = 1 𝑁 𝑁 ∑︁ 𝑠=0 𝐹 1𝑠 , 𝐹 1𝑠 = 2 × 𝑃 𝑠 × 𝑅 𝑠 𝑃 𝑠 + 𝑅 𝑠 , (2)

where 𝑠 is species index, 𝑁 equals to the number of classes in a training set. The F1 score for each class represents harmonic mean of the class precision 𝑃 𝑠 and recall 𝑅 𝑠 . This type of evaluation is suitable for data with long-tail distribution, since the quantity of samples from individual classes does not effect the outcome. For the additional evaluation of the performance of the teams we also compute the micro classification accuracy, which is a ratio between the number of correctly classified samples and all samples.

Working Notes

All participants were asked to provide code and a Working Note paper -a technical report with information needed to reproduce the results of all submissions. All submitted Working Notes were reviewed by 2-3 reviewers with a decent publication history in the field of Computer Vision and Machine Learning, ensuring a sufficient reproducibility and quality. The review process was single-blind and offered up to two rebuttals. The acceptance rate was 66.66%.

Results

The best performing team achieved F 𝑚 1 of 86.47% on the private part of the test set and 94.01% classification accuracy on the full test set. Both scores are the best ones in the given category. We observe a steady decrease in the F 𝑚 1 score for the first three teams achieving 84.87% and 82.65% F 𝑚 1 score. Then there are several significant drops in performance. Similar behavior is present in the classification accuracy with the second team achieving 93.29% and the third one achieving 93.53%. It can be seen that the accuracy is not correlated with the F 𝑚 1 score which suggests that some teams did not cope well with the long tail distribution. Further performance evaluation for Top-25 teams is provided in Figure 4.

On the expert set, the best performing team -4 th in terms of F 𝑚 1 score -achieved impressive accuracy of 94.67%. Eight teams achieved satisfactory accuracy on the expert set, around 90%. Further performance evaluation for the Top-25 teams is provided in Figure 5.

GG

Participants and Methods

A total of 31 teams participated in the SnakeCLEF 2022 challenge and submitted 676 submissions. Everyone who submitted a solution better than baseline submission, i.e., random predictions, was considered a participant. The number of participants quadrupled since last year, primarily because Kaggle was used as an evaluation platform. More details can be found in the individual working notes of participants [10,11,12,13,14,15] who passed the review process, ensuring a sufficient level of reproducibility and quality. The main outcomes we can derive from the achieved results are as follows:

GG [10]: The winner of the challenge. They have introduced a novel architecture CoLKA-Net based on VAN [16] (Visual Attention Network) and CoAtNet [17]. It is a combination of large kernel attention and vision transformer. In an ablation study, the model outperforms other tested models -ViT [18], Swin [19], VOLO [20], and ConvNeXt [21] by around 2 points of F 𝑚 1 score (78.00% to 80.10%). In addition, the team used techniques such as Label Aware Smoothing [22], Pseudo labelling for tail classes, FixRes mitigation [23], and augmentations. When TrivialAugment [24] was deployed during the middle stage of experimentation the team observed a rise in F 𝑚 1 of around 0.5%. Progressively, Random Erasing [25], CutMix [26] and Mixup [27] were added which helped with regularization. The final submission score was achieved by an ensemble of six models 2× ConvNeXt, VOLO, CoLKANet, Swin, and ViT. The novel CoLKANet is an interesting contribution with potential outside of the scope of this competition. [11]: The runner-up focused solely on ViT models, ensembling three different variants; two Large models trained on different resolutions (384; 432) and one Huge model (392). The observed trend from the ablation study is that larger models with higher input resolution perform better. The ViT models were pretrained as Masked autoencoders [28] on ImageNet-1K [29]. The team designed a new Effective Logit Adjustment Loss combining Logit adjustment loss [30] and Class-balanced Loss [31]. They observe a better overall performance when compared to Cross Entropy and further analyze that the improvement comes from the tail classes. The metadata was integrated by applying an estimated species a priori distribution in individual regions to the ViT estimates. Interestingly, when all metadata (code, endemic, country) were added the F 𝑚 1 score on validation data rose but dropped rapidly on the test data. The best combination on the test data was when the country code was omitted. This hints at a distribution gap between the training and test metadata. SAI [12]: The team used MetaFormer [32]. The metadata passes through an Embedding, individual tags (code, endemic, country) are concatenated and the final MLP is used to produce the Meta token. The Metaformer seems suitable for integrating categorical tokens. The team shows in an ablation study a significant raise in F 𝑚 1 score from 66.64% to 74.18%. The Logit adjustment loss function is used to optimize the model. It works better than Seesaw [33] loss but is outperformed by a post-hoc logit adjustment. Filtering the predictions based on possible locations of species raises the F 𝑚 1 score from 64.32% to 69.09%. SimCLR [34] with InfoLoss [35] is used to train the model. The team shows significant improvement when compared to only supervised models pre-trained on different datasets (eg. from ImageNet-1k pre-train F 𝑚 1 of 63.76% to 68.83% when SimCLR is used). In accordance with the findings of previous teams, a bigger resolution of input images leads to better scores. The final score was achieved by the multi-scale, multi-crop, and multi-resolution model ensemble. [13]: The team combines models of Swin Transformer, EfficientNet, and BEiT [36]. They experiment with several loss functions (CE Loss, Seesaw Loss, Focal Loss [37]) and show that Focal Loss performs the best, but only marginally (less than 1% F 𝑚 1 score improvement when compared to CE). From the individual models, the EfficientNet-L2 is the best one. Given the highest number of parameters (480M), it is expected. From this point of view, it is interesting that Swin with 197M parameters achieves the worst score (F 𝑚 1 of 61.70%) and is outperformed by EfficientNet-B7 (F 𝑚 1 of 64.99%) with only 66M parameters. The authors experimented with a fine-grained classification technique -PIM [38] -to find informative and discriminative features. They show the improvement on the Swin model (F 𝑚 1 from 62.38% to 63.80%) but then do not use it because of the relatively big computational burden when training the model. Finally, the models are ensembled by averaging their softmax outputs. The authors observe a steady increase in F 𝑚 1 score when adding the models to the ensemble one by one. But an unexpected drop is observed when the last model -Swin -is added. The drop is present only in the test data and not the validation data. This opens the question of how, when, and what models to add to the ensemble. [14]: The team adopted the two-stage principle of the best solution from Snake-CLEF2021 [39]. In the first stage, the snake is detected and in the second stage, the detected region is classified by a CNN. However, this year it was not the best choice, substantially lacking in performance when compared to the winning team (70.7% F 𝑚 1 versus 85.4% F 𝑚 1 ). Even so, the detection helps and the YOLOv5 [40] detection network improved the private F 𝑚 1 by 13% on average. The team experimented with two CNN models -EfficientNet and ConvNeXt. They provide an in-depth study of the behavior of the models when combined with different techniques of optimizing, adding metadata, types of augmentations, and so on. The best competition score was achieved by ensembling seven models -EfficientNet-B4 with and without object detection, EfficientNet-v2-m with no object detection, and EfficientNet-v2-m with and without object detection. However this score of 70.79% F 𝑚 1 is only a bit ahead of a single EfficientNet-v2-m model with score 70.23% F 𝑚 1 . This begs the question of how important it is to use ensemble in a real-life application when the improvement is limited. Lastly, the team experimented with different types of metadata representation. In all cases, the metadata were used as a priori distributions of snake species in different locations that were multiplied with the resulting softmax. The worst option was to use the estimated a priori distribution, followed by the binarized version (with a threshold of 0) and finally the post-competition best result (73.90% F 𝑚 1 ) when the binarized distribution was multiplied by the a priori country distribution of the training dataset.

Secret;Weapon

USTC-IAT-United

FHDO-BCSG

anonymous_rice [15]: The team experimented with several CNN models -ResNet, ResNext, and EfficientNet to produce deep features of the images and concatenate them with the categorical representation of metadata. In the final solution, ResNet-101 and EfficientNet-B0 were used. The concatenated features are inputted into the XGBoost Ensemble Classifier [41] to produce the final classification. The nice property of the XGBoost algorithm is that the relative importance of the ensembled features is computed. Unfortunately, the team achieved low score of 3.6% F 𝑚 1 but after the competition when the backbones were trained further, the score raised significantly to 51.39% F 𝑚 1 . This shows some potential of the XGBoost algorithm that may be interesting to study in the future.

Conclusions and Perspectives

This paper presents an overview and results evaluation of the third edition of the SnakeCLEF challenge organized in conjunction with the Conference and Labs of the Evaluation Forum (CLEF3 ) and LifeCLEF4 research platform [42], and FGVC9 Workshop5 -The Ninth Workshop on Fine-Grained Visual Categorization organized within the CVPR conference. The main outcomes we can derive from the this year's evaluation are as follows.

Transformer-based architectures outperformed CNNs. This year various deep neural network architectures -Convolutional Neural Networks and Transformers -were evaluated; ConvNext [21], EfficientNet [43], Vision Transformer [18], Swin Transformer [19], and MetaFormer [32]. Unlike last year, where the CNN architectures overwhelmed the performance, Vision Transformer architectures were a vital asset for most methods submitted this year. The second best method with F 𝑚 1 score of 84.56% was based on an ensemble of exclusively ViT models and performed slightly worse (-0.9%) than the best performing system that used a combination of Transformer and CNN models. An ensemble of MetaFormer models achieved the third-best score of 82.65%. It seems that Transformers and CNNs benefit from each other in an ensemble, whereas a standalone Transformer ensemble performs better than a pure CNN ensemble which achieved an F 𝑚 1 score of "only" 70.80%

Loss Function matters. Several loss functions were evaluated: Label Aware Smoothing [22], (modified) Categorical Cross-Entropy, Seesaw [33], and Focal Loss [37]. Overall, any Loss function if used is better than standard CrossEntropy. The wining team used Label Aware Smoothing. The runner-up used an Effective Logit Adjustment Loss and showed an improvement of around 2% of F 𝑚 1 score when compared to Cross Entropy, reducing the error rate by 15%. The the third team used Logit adjustment to outperform the Seesaw loss from an F 𝑚 1 score of 76.49% to 78.57%. The team USTC-IAT-United compared CE Loss, Seesaw Loss, and Focal Loss with EfficientNet model. Focal Loss performed the best, but only with marginal improvement over CE Loss and Seesaw Loss.

Self-supervision has potential. Adding unlabeled data to the train set is a welcome option when not many observations of a species are available. The third team used the SimCLR [34] method with InfoNCE [35] loss function to increase the F 𝑚 1 score from 63.76% to 68.83% when compared to an ImageNet-1k pretrained models. Overall, performance on tail classes was higher this year.

Geographical metadata improves classification performance. Most teams report accuracy improvement when adding the metadata into the learning process. The second team achieved an improvement of 10.89% in terms of the F 𝑚 1 score using a simple location filtering approach. The third team described an absolute improvement of 7.54% when adding the metadata into the MetaFormer. Using the a priori country distribution from the training data outperformed other approaches tried by the fifth team, but in real life this will frustrate users in countries that lack many snake images to use as training data. Representation of many locations in both training and testing data remain important and challenging.

Ensemble helps, but at what cost?

Most teams used ensembling to increase the accuracy of classification. The standard approach was to compute an average of the individual models' decisions. Some teams used a late fusion of deep features by concatenation as an ensemble technique. Even though the improvement in accuracy is observable (around 1 percentage point of F 𝑚 1 across the board), it would be interesting to measure the added computational complexity vs the added accuracy. In the case of snakebite, the system's inference time plays a crucial role.