The paper presents a method for image-based snake species identification. The proposed method is based on deep residual neural networks - ResNeSt, ResNeXt and ResNet - fine-tuned from ImageNet pre-trained checkpoints. We achieve performance improvements by: discarding predictions of species that do not occur in the country of the query; combining predictions from an ensemble of classifiers; and applying mixed precision training, which allows training neural networks with larger batch size. We experimented with loss functions inspired by the considered metrics: soft F1 loss and weighted cross entropy loss. However, the standard cross entropy loss achieved superior results both in accuracy and in F1 measures. The proposed method scored third in the SnakeCLEF 2021 challenge, achieving 91.6% classification accuracy, Country F1 Score of 0.860, and F1 Score of 0.830.
The paper describes a method for automatic image-based snake species identification submitted
by the CMP team to the SnakeCLEF 2021 challenge [
Taxonomic knowledge about snakes is crucial in diagnosis and medical response to snakebites.
Accurate identification of the snake species is important for the appropriate treatment of
snakebite victims since specific antivenoms are efective against specific venomous snakes.
Moreover, antivenoms should not be used to treat bites from non-venomous snakes because of
side efects such as allergic reactions [
This paper is structured as follows: Section 2 describes related work focusing on snake species identification. Section 3 introduces the input data and evaluation methodology of the SnakeCLEF 2021 challenge. Section 4 describes the adopted architecture of deep neural network and the optimization procedure. Section 5 covers all experiments, ranging from preliminary experiments to the final challenge submissions. Finally, the results are summarized in Section 6.
Before the existence of large-scale image datasets for snake species classification, Abeysinghe
et al. [
In 2020, the first year of the SnakeCLEF challenge [
The best scoring team in SnakeCLEF 2020, gokuleloop [
The second team in SnakeCLEF 2020, FHDO_BCSG [
The training dataset provided by SnakeCLEF 2021 covers 772 snake species and contains annotated images from three diferent sources: iNaturalist, HerpMapper and Flickr. Examples of images are in Figure 1. The majority of images are from iNaturalist and HerpMapper, with 277,025 and 58,351 images, respectively. Their labels are confirmed by human annotators. The Flickr dataset is the smallest, with 50,630 web-scraped images that contain noisy data.
In total, 386,006 images with annotations were provided. Training with external data was not allowed.
The challenge organizers suggested a subset of 70,208 images, referenced as a mini-subset in the rest of this paper, made of samples from INaturalist and HerpMapper. The experiments described in Section 5 are based on the said subset.
In addition to the images, the dataset contains metadata with information about the country where the image was taken. In total, the training dataset includes images from 188 countries. The dataset is fine-grained with a long tail class distribution. More than 22,000 images represent the most frequent species, while the least frequent species have only 10 images. The least represented species are often found in regions such as Middle and South America, South Africa and Australia. Table 1 shows the distribution of images in geographical regions. For some images, information about the geographical location is missing.
Furthermore, the challenge organizers provided 28,418 images without annotations. Top one species predictions for the test images were sent to the organizers to participate in the challenge.
During the data exploration phase, we discovered that training and validation datasets contain noisy data from Flickr. The noisy data are non-relevant images with various animal species or objects. We estimate1 that the percentage of non-relevant images is 10.6 ± 0.1, with 95% confidence interval. We decided to remove all Flickr images and proceeded with verified images from iNaturalist and HerpMapper.
1We used the Student’s t-distribution with = 20 samples and − 1 degrees of freedom where each sample denotes the percentage of non-relevant images in a set of randomly selected 100 images.
The challenge organizers suggested a data split with 90% training and 10% validation samples. However, after removing Flickr images, it turned out that some species were not represented in the proposed validation set. Table 2 displays the number of snake classes represented, i.e. classes with at least one image, in the dataset sources. iNaturalist and HerpMapper combined have 768 classes which are all represented across the training set but only 733 classes in the validation set. We thus created a new dataset split where all classes are represented in both training and validation splits if more than one image of the species is available. If not, the image is placed in the training set.
Technically, the last 10% of images, ordered by metadata ID, for every species and country combination were selected as the validation data. One validation image was selected for the cases that had fewer than 10 images. We assume the ID ordering is random w.r.t. image content and properties. Mini-subset (introduced in Section 3.1) iNaturalist + HerpMapper (new split)
The challenge used two metrics for the final evaluation. The primary metric is the macro averaged F1 Score across countries ("Country F1 Score"), shown in equation 4. The secondary metric is the macro averaged F1 Score ("F1 Score"), shown in equation 2.
The F1 Score for each species = 1, 2, ..., is computed as a harmonic mean of precision and recall :
The macro averaged F1 Score is the average of the 1 scores of all species: where is a × matrix with elements = {︃1, country is a habitat of species 0, otherwise Similarly, macro averaged Country F1 Score is obtained by averaging CF1 over all countries: macro(1) = 1 ∑︁ 1.
Country F1 Score 1 for each country = 1, 2, ..., is the macro averaged F1 Score computed only for species living in country : The macro averaged Country F1 Score thus increases the importance of species that appear in more countries.
The proposed method is based on the state-of-the-art Convolutional Neural Networks (CNNs) for image classification, described in Subsection 4.1. The following subsections describe the optimization procedure, loss functions, the post-processing of the predictions, applying mixed precision training and implementation details.
All experiments are based on deep residual neural networks, namely the original ResNet [
All networks in our experiments were fine-tuned from ImageNet-1K [
We use two optimization algorithms for training CNN models: stochastic gradient descent with
momentum (SGD) and Adam [
The training hyper-parameters, such as learning rate, momentum and weight decay, are listed in Table 3 and were set the same as in the network pre-training. Batch sizes were adjusted to fit the network on the graphics processing unit (GPU). The input image size stays the same as in the pre-trained networks.
During the training, we select the best checkpoint based on the highest validation Country F1 Score.
For each image, the dataset metadata include the country where the image was taken. Additionally, the dataset comes with a list of countries and snake species that live there. We utilize this information to adjust the model predictions to the country of the query as follows: The classifier predictions are set to 0 for all species that do not live in the country of the query. This adjustment is applied only at test time.
When training large CNN architectures, fitting the model into limited GPU memory is a
bottleneck. We considered the following workarounds: selecting a smaller batch size or applying
mixed precision training [
Mixed precision training is a technique that combines single-precision (32-bit floats, "FP32") and half-precision (16-bit floats, "FP16") float numbers. In order to lower the memory requirements, the forward and backward pass with the large batch size only use a half-precision version of the model. Then, the gradient descent is applied to the single-precision version of the model. In every training step following procedure is applied: 1. Apply the forward pass, compute the loss and apply backward pass on a model in FP16. 2. Convert the gradients from FP16 to FP32. 3. Apply the update on the primary model in FP32.
4. Create a copy of the primary model in FP16.
The baseline loss function for training the classifiers is the standard cross entropy loss: ℓce = −
∑︁ log , , =1 (5) where is the ground truth target and y are the classifier predictions for the -th example, and , is the prediction for the ground truth class of the -th example.
The following subsections describe the loss functions proposed to use the challenge metrics, described in Section 3.3, as a loss measure.
The F1 Score from Equation 2 is not diferentiable and thus cannot be utilized as a loss function for back-propagation. We use an approximation of the F1 Score, referenced as soft F1 loss in the rest of this paper, which uses soft assignments that make the function diferentiable: • the true positives for species are estimated using the softmax predictions y and one-hot encoded target vector t as follows: T̂︁P = ∑︀ yt
=1 • the false positives for species are estimated using the softmax predictions y and one-hot encoded target vector t as follows: F̂P︁ = ∑︀ y(1 − t)
=1 • the false negatives for species are estimated using the softmax predictions y and one-hot encoded target vector t as follows: F̂N︁ = ∑︀ (1 − y)t =1 Notice, that T̂︁P, F̂P︁, and F̂N︁ are now real valued. Soft F1 Score for species , ̂1︁, is obtained by computing the harmonic mean of precision ̂︀ and recall : ̂︀ ̂︀ = T̂︁PT̂+︁PF̂P︁ , ̂︀ = T̂︁PT̂+︁PF̂N︁ , ̂1︁ = ̂︀2̂+︀̂︀̂︀ .
The macro averaged soft F1 Score is obtained by averaging ̂1︁ over all species: macro(̂︁1) = 1 ∑︁ ̂1︁. (9)
The final loss function is ℓ̂︀1 = 1 − macro(̂︁1), so that it ranges from 0 (perfect) to 1 (worst).
Because the macro averaged Country F1 Score from Equation 4 increases the importance of species appearing in more countries, we propose a weighted variant of the cross entropy loss with species weights based on the number of countries in which it appears: ℓwce = − ∑︁ log , , (8)
=1
The Maximum Likelihood Estimation (MLE) of would simply count the relative frequencies in the provided species-country incidence list. In order to avoid zero weights, we add Laplace smoothing: =
+ 1 ∑︀ ( + 1) =1 .
The proposed method was developed using the PyTorch [
2https://github.com/chamidullinr/snake-species-identification
As observed in the previous section, ResNeSt-101 with a higher input size achieves the highest scores of the experimented residual networks. Since its deeper version, ResNeSt-200, does not fit into our GPU memory with larger batch sizes, we experiment with the mixed precision training from Section 4.4.
Table 5 compares the training time and accuracy of ResNeSt-101 and ResNeSt-200 when training with and without the mixed precision technique. Note that in our computational environment, mixed precision runs slower than single precision. The prediction scores after 10 epochs show that mixed precision has little impact on prediction accuracy in setups with the same architecture and batch size. Increasing the batch size from 32 to 64 has a much larger impact on the accuracy. Thus the network trained on a larger batch size with mixed precision achieves better scores than the single-precision network with a smaller batch size.
The loss functions introduced in Section 4.5, namely the soft F1 loss and the weighted cross entropy loss, resulted in inferior classification scores compared to cross entropy loss, see Table 6. We, therefore, fine-tune the CNN classifiers with cross entropy loss, and then choose the best training checkpoint based on the highest validation Country F1 Score.
One possible explanation for the failure of the soft F1 loss is that the batch size of 64 is significantly smaller than the total number of classes, 772. This leads to the classes not being represented in every mini-batch, making the approximation of the F1 loss inaccurate. Figure 2 illustrates the inaccurate approximation of the F1 loss on an example, where the loss values are mostly 0s or 1s.
We measure the prediction scores of ResNeSt-200 with and without the removal of species predictions based on the country incidence information. Table 7 compares the prediction scores on our validation set. The improvement is 0.150 in F1 Score and 0.193 in Country F1 Score.
We submitted the following five runs to the SnakeCLEF 2021 challenge: CMP_S1: ResNeSt-200 fine-tuned for 20 epochs on the full dataset with SGD. CMP_S2: ResNeSt-200 from CMP_S1 fine-tuned for additional 10 epochs on the full dataset with SGD.
CMP_S3: ResNet-101 fine-tuned for 25 epochs on the full dataset with Adam and one cycle. CMP_S4: ResNeXt-101 fine-tuned for 30 epochs on the mini-subset from Section 3.1 with Adam and one cycle.
CMP_S5: An ensemble of all four previous runs, combining the top one predictions by majority voting strategy. In case of ties, predictions of CMP_S1 are preferred.
Table 8 shows the final challenge scores on the test set. While diferent in accuracy, the CNN architectures ResNeSt-200, ResNeXt-101 and ResNet-101 achieve similar results in the primary challenge metric, the Country F1 Score. The highest scores are achieved by the ensemble.
We recognize a shortcoming of the ensemble submission (CMP_S5), which inclines towards the ResNeSt-200 submissions related to each other (CMP_S2 is fine-tuned from CMP_S1). The remaining networks cannot outvote an agreement of CMP_S1 and CMP_S2.
The paper presents a deep learning method for image-based snake species identification, a finegrained classification problem with a long tail class distribution. The method is based on deep residual neural networks – ResNeSt, ResNeXt and ResNet – fine-tuned from ImageNet pre-trained checkpoints. We achieve performance improvements by: discarding predictions of species that do not occur in the country of the query; combining predictions from an ensemble of classifiers; and applying mixed precision training, which allows training neural networks with larger batch size.
The experimented soft F1 loss and weighted cross entropy loss produced inferior results compared to the standard cross entropy minimization. Thus, the competition submissions are ifne-tuned with the standard cross entropy loss.
The proposed method scored third in the SnakeCLEF 2021 challenge, achieving 91.6% classification accuracy, Country F1 Score of 0.860, and F1 Score of 0.830.
This research was supported by the OP VVV funded project CZ.02.1.01/0.0/0.0/16_019/0000765. LP was supported by the UWB grant, project No. SGS-2019-027. [23] J. Howard, S. Gugger, Fastai: A Layered API for Deep Learning, Information 11 (2020) 108.
URL: http://dx.doi.org/10.3390/info11020108. doi:10.3390/info11020108. [24] R. Wightman, PyTorch Image Models, https://github.com/rwightman/ pytorch-image-models, 2019. doi:10.5281/zenodo.4414861, visited on 2021-0628.