Introduction

Bird species diversity and its change in time serve as good indicators of ecosystem state. Traditional methods of monitoring require the presence of a qualified observer who can manually identify the bird. It's quite hard and expensive to conduct such surveys regularly, especially in the case of large areas. Many birds are small and hard to notice, but they have a loud voice. So, it seems promising to use small and cheap audio recording devices with omnidirectional microphones instead of human observers and to process the recordings using modern methods based on deep learning.

Creating and training such algorithms is a difficult task, however. The first problem is the diversity of bird species and their recording conditions. There are many birds that can imitate other birds or even repeat a sound they once heard and liked. Many animals and insects sound like birds. The second problem is the difference between the available training data and the real recordings to be processed. Usually, the training data is a set of short bird call recordings made by different people at different locations. They try to make the recordings clean and loud, without noise or interference. So, good equipment is used, including directional microphones, and bad recordings are dropped. The third problem is that only weak labels are given that identify the bird's existence in each recording but not the exact call position.

BirdCLEF 2024 is a competition that is supposed to address the mentioned problems [1,2]. It is a part of the LifeCLEF 2024 conference [3]. The task is to identify bird calls in a set of recordings made in the Western Ghats, India. The training dataset consists of recordings from the xeno-canto project [4]. Each of them has primary and secondary labels. The primary label corresponds to the main bird that can be heard, and the secondary labels are used to mark additional birds that can accompany the main bird. 182 bird species, whose existence needs to be predicted, were selected by the organizers. The predictions must be made for each of the 5-second-long intervals of about 1100 recordings that form the hidden dataset. An additional constraint is that the task must be completed in 2 hours using CPU only. The macro-averaged AUC ROC that discards classes with no true labels was used as a metric in the competition. To prevent overfitting, the full hidden dataset is split into the public and private parts

Table 1

The CV accuracy score calculated using xeno-canto data for the models with different parameters: the backbone, the embedding dimension 𝐷, the number 𝐻 of attention heads, and the number 𝐵 of multi-head attention blocks. (approximately 35% and 65% of the data, respectively). The corresponding public and private scores are calculated independently, and only the public score is known at the competition time. This article presents the methods that can be used to overcome the aforementioned difficulties and improve the results of bird call recognition.

Methods

Model architecture

The used model is based on the model proposed in [5]. Its scheme is presented in Fig 1 . After the signal is loaded and normalized, the spectrogram transform is performed. Then, it is fed to a backbone CNN, followed by 𝐵 multi-head attention blocks. Finally, a log-sum-exp pooling layer is used to extract the label. The multi-head attention mechanism is described in the paper [6], and it is implemented in PyTorch by the torch.nn.MultiheadAttention class. Its main parameters are the number 𝐻 of parallel attention heads in each of the blocks and the embedding dimension 𝐷.

To find the best combination of the model parameters, a number of tests were conducted. Instead of using the macro-averaged AUC ROC score, which became very close to one after a few epochs, the accuracy score was used in a 5-fold cross-validation (CV) scheme. In the tests conducted, the backbone as well as the values 𝐵, 𝐻, and 𝐷 were varied. The results are presented in Table 1.

The simple resnet18 [7] backbone was used to check different parameters. Using it, the best results were achieved with 𝐷 = 256, 𝐻 = 64 and 𝐵 = 2. This slightly differs from the parameters presented in [5], where they were set as 𝐷 = 768, 𝐻 = 8 and 𝐵 = 2. The results significantly improve with heavier backbones, among which seresnext26t_32x4d [8] seems the best.

Dealing with different lengths of the recordings

An important problem with the recordings is that they have different lengths. The shortest of them lasts only 0.5 seconds, while the longest is more than 1.5 hours. Different lengths don't allow using these recordings in batches while training the model. This causes the training to be slow.

There are several possible ways to overcome this difficulty. Let 𝑡 0 be the fixed final length of each recording processed. If the initial recording is shorter, then it is simply padded with zeros. If it is longer, the "First" approach is to use only the first 𝑡 0 -long interval of the recording. The "First and last" approach is to use the first 𝑡 0 /2 and last 𝑡 0 /2 intervals stacked together. This seems reasonable because, as a rule, the recordings were processed by their authors before being uploaded to the website. So, one can suspect that irrelevant sounds were cut off from the beginning and the end of each recording. These two approaches are quite popular among the competition solutions. However, a lot of information is lost. Instead, the third approach is proposed, which is called the "Sum" approach and illustrated in Fig 2 . It consists of the following steps.

1. Each recording, having length 𝑡, is padded with 𝑡 z = 𝑡 − 𝑡 0 𝑛 int zeros, where 𝑛 int = ⌈𝑡/𝑡 0 ⌉, i.e., the resulting recording contains a whole number of intervals with the length of 𝑡 0 . 2. As an augmentation, a random circular shift is performed. 3. The recording is split into 𝑛 int intervals, and they are summed together. The length of the result is equal to 𝑡 0 .

There is no information loss in the third approach. The overlapping of bird calls that may occur doesn't seem to be a problem since it corresponds to a situation when many birds vocalize at the same time.

The same model (seresnext26t_32x4d; 𝐷 = 768, 𝐻 = 8, 𝐵 = 2) was trained using all the described approaches with different 𝑡 0 values. Every training was repeated three times with different random seeds, and the "best" of them with the highest score on the public dataset was selected. The resulting public and private scores are presented in Table 2.

The results produced with different approaches are close to each other. However, the scores of the "First" approach are slightly better with low 𝑡 0 values. The growth of 𝑡 0 doesn't improve the scores of the "First" and "First and last" approaches, but the scores of the "Sum" approach increase, and it becomes preferable with large 𝑡 0 . At the same time, increasing 𝑡 0 makes the model training longer.

Train data selection

The train dataset suggested at BirdCLEF 2024 competition contains approximately half of all the recordings from xeno-canto project [4], with primary labels corresponding to 182 target birds. So, the obvious step is to download the absent data and create an additional dataset. Merged together, these two datasets contain about 40 000 Using the whole dataset, however, doesn't improve the model score. It seems strange because the higher diversity of data usually causes an increase in the generalization ability of the model. So, one may expect that the additional data corrupts the dataset somehow, making it not correspond to the hidden dataset.

To find out the reason for the described behavior, the AUC ROC probing technique is proposed. This technique is based on splitting the target data into several parts and masking the model predictions so that only one part of the data is scored each time. In the BirdCLEF competitions, the bird species can be grouped together. For example, let 𝑁 = 182 be the total number of bird species. One can select a group of 0 < 𝑛 < 𝑁 bird species. During the submission, the predictions corresponding to the rest 𝑁 − 𝑛 species are set to zero. The constant prediction produces an AUC ROC score of 0.5. So, the resulting AUC ROC score 𝑆 is equal to 𝑆 = (𝑆 𝑛 𝑛 + 0.5(𝑁 − 𝑛))/𝑁 , and the AUC ROC score 𝑆 𝑛 of the selected group is equal to 𝑆 𝑛 = (𝑆𝑁 − 0.5(𝑁 − 𝑛))/𝑛. Using this simple formula, it is possible to estimate the model performance for different groups of species. If these scores differ significantly and the size of each group is large enough, one may assume that the feature used for group selection is important.

One of the possible features that can affect the model's performance is the number 𝑁 rec of available recordings of each bird species, i.e., the frequency of its occurrence in the dataset. Following the proposed AUR ROC probing technique, the species were split into five groups (Table 3). It can be assumed that the model will work well for those birds for which the training set contains many records and poorly in the other case. However, the situation is different. Group 5 with at least 200 recordings of each bird species, has almost as low scores as Group 1 with a maximum of 20 recordings.

One can conclude that, for some reason, the model has significant difficulties when dealing with common birds. One of the reasons for this is that common birds may be present in many recordings in the background while they are not marked, even with secondary labels. Another possible reason is the geographical distribution of the places at which the recordings were made. Indeed, many common birds were recorded in Europe, America, or Africa, far away from the region of interest. Birds can have local dialects. Also, a bird, which is common in Europe, can be rare in India.

The geographic information can be easily taken into account since GPS coordinates are provided. The locations of places at which five bird species were recorded are presented in Fig 3 . Here, "zitcis1", "commoo3" and "barswa" are the primary labels of common birds, with the number of recordings equal to 500 in the competition dataset. The fourth bird, "revbul" is medium-rare and has 101 recordings, while the fifth, "maltro1", is a rare bird with 17 recordings. However, it is noticeable that almost all recordings of common birds were made outside India, while "revbul" and "maltro1" are endemic birds. As a result, the number of recordings of all common and rare species that are made in the Indian region is quite low and significantly less than that of medium-rare birds. This observation explains the differences in model scores across different groups of species.

To handle this observation, the algorithm for train data selection is proposed, which consists of the following steps.

1. Prepare the whole dataset with all the recordings from the xeno-canto project [4] that contain the target bird species calls. 2. For each recording, calculate its distance 𝐿 WG to the Western Ghats region. This can be done, for example, by placing a large number of points in the Western Ghats region and calculating the minimal distance between these points and the point where the recording was made. 3. Specify the maximum distance 𝐿 max and drop all the recordings with larger distances.

Calculate the number of recordings ̃︀

𝑁 rec for each bird species in the resulting dataset. 5. Calculate the distance weight 𝑤 𝐿 for each of the recordings. This weight is a decreasing function of 𝐿 WG . For example, 𝑤 𝐿 = 1 + cos(𝜋𝐿 WG /𝐿 max ) may be used. 6. Calculate the class imbalance weight for each of the recordings. This weight is a decreasing function of ̃︀ 𝑁 rec . For example, 𝑤 imb = 1/ ̃︀ 𝑁 rec may be used. 7. The weight of each of the recordings in the final dataset is the product of distant and class imbalance weights:

𝑤 = 𝑤 𝐿 • 𝑤 imb .

The value of 𝐿 max is important. In the current competition, setting 𝐿 max = 4000 km was a good choice. From a geographical point of view, it allows to discard European, American, and most African data while covering Southern Asia. Using the lower 𝐿 max decreases the diversity of species, and with its higher value, the training set includes irrelevant data. As a result, the public score of the model worsens in both cases.

Additional noise sources

As it was mentioned earlier, there is a huge domain shift between the competition data used for training and for model scoring. The training dataset contains the recordings from the xeno-canto project [4]. As a rule, these are recordings of high quality. However, the model is supposed to work well with the data recorded with an omnidirectional microphone in a noisy environment. The unlabeled soundscapes dataset is provided with recordings similar to those used for model scoring.

Listening to the unlabeled recordings, it is possible to make a list of present noise sources. They include car traffic and horns, aircraft noises, sirens, human voices, music, frogs, and cicadas, as well as broadband noises of rain, wind, or even uncertain nature. The example spectrogram of such an unlabeled sounscape is presented in Fig 4 . One has to introduce all these kinds of interferences to the model to increase its generalization ability and reduce the domain shift. At the same time, the addition of an extra sound to the recording must not contain bird calls that can disorient the model.

There are several datasets that can help introduce these noises; for example, the Vehicle Type Sound Dataset [9], the Noise Audio Data Dataset with short sounds of different natures [10], the Rain Forest Dataset [11] with recordings of several frog species, and the Hindi Speech Classification Dataset with recordings of short phrases [12]. As an addition, manual selection of recordings not containing bird calls can be used [13]. Although the use of a dataset with the regional dialects spoken in the Western Ghats alongside Hindi may seem more appropriate, it's quite hard to find a sufficient number of such recordings distributed freely. At the same time, the influence of including these dialects on model performance seems minor.

The broadband noises are quite hard to add. On the one hand, many existing recordings of rain and wind noises contain bird calls, which have to be manually filtered. On the other hand, these noises are nonstationary, so they can't be precisely modeled with any kind of simple stationary noise. A similar situation takes place with the sounds produced by cicadas.

To deal with this situation, it is proposed to use unlabeled soundscapes. These recordings, however, contain bird calls that must be excluded. The idea of doing it is based on the fact that background noise as well as cicadas form patterns on the spectrogram that slowly change over time. The patterns of bird calls and other noises are irregular. The first step of the algorithm is taking the 1D Fourier transform of the spectrogram along the time axis. On the second step, only the components with the largest absolute values remain, while the others are put to zero. In the third step, the inverse 1D Fourier transform is performed, and the result is multiplied by random noise. The described filtering procedure significantly reduces the amount of information a spectrogram contains, and its "thin structure" disappears, including bird calls. The example is presented in Fig 5.

Standard augmentations and post-processing

The "standard" augmentations can be used in the BirdCLEF competition. They are performed on spectrograms and include XY masking, random grid shuffle, and recording mixing. XY masking selects a few rectangular areas in the spectrogram and sets the data inside it to a constant. Random grid shuffle splits the spectrogram into a grid and shuffles all its cells. This transform can be used only along the time axis, and the size of each cell must be greater than the length of a potential bird call, say, 5 seconds. Recording mixing is the technique of adding two or more recordings together before passing them to a model. In this case, the resulting recording contains all the birds from the initial recordings. Its weight is also the sum of the initial weights. These augmentations make the training dataset more diverse.

The model predictions may be post-processed using sliding window averaging. This approach assumes that if there is a bird call in a certain time interval, the probability of the same bird call in the neighboring intervals is also high. So, the final prediction for the current interval is a sum of predictions for the current, previous, and next intervals with the weights of 1 − 2𝛼, 𝛼, and 𝛼 respectfully. The coefficient 𝛼 is an averaging parameter, which is often set to 0.25 in BirdCLEF competitions. The results of using different 𝛼 are presented in Table 4. It can be seen, that the value of 𝛼 = 0.25 is indeed optimal, however, the public score is maximized by 𝛼 = 0.3.

Inference time optimization

The inference time on the CPU is one of the crucial factors in the competition. However, it was noticed that the models became extremely slow after training. For example, the used model (seresnext26t_32x4d; 𝐷 = 768, 𝐻 = 8, 𝐵 = 2) processed the 240-seconds-long recording in 60-70 seconds after training, while the untrained model did the same job in 3.5 seconds. The training procedure doesn't change the model architecture and only adjusts its weights. After some research, the problem was localized. In the model computational graph, some of the paths are unnecessary, and the corresponding weights must be set to zero during training. However, using L2 regularization makes these weights very low but not exactly zero. As a result, not only do these paths consume computational resources, but the computations with such low values are extremely slow. To prevent this behavior, one has to retrain the model with an additional L1 regularization term, which causes the small weights to be exactly zero. A simple solution for an already-trained model is weight rounding. To do it, one has to convert the model precision from float32 to float16 and then back to float32. Weight rounding can be performed with one line of code in PyTorch: model.half().float().