This research paper presents deep learning techniques for bird recognition to classify 397 species in the BirdCLEF 2021 challenge. The proposed method was inspired by the DCASE2019 audio tagging challenge, which classifies and recognizes diferent sound events. Data augmentations methods like noise augmentation, spectrogram augmentation techniques are used to avoid overfitting and hence generalize the model. The final solution is based on an ensemble of diferent backbone models and splitting the dataset based on geographic locations provided in the test set. Furthermore, framewise post-processing predictions are used to identify the bird events. The best results were obtained from 12 model ensembles with a public and private score of 0.6487 and 0.6034, respectively.
The paper is organized as follows, section 2 describes the recognition of the birds including data preparation. Section 3 describes feature extraction, data-augmentation, neural network architecture and training steps. Section 4 presents the evaluation results followed by the conclusion.
All the recordings are first converted from ogg to wav format with a sampling rate of 32 kHz. The soundscape recordings are prepared for validation by cutting them into 5 s chunks according to the annotations. The background noises are separated from the soundscape recordings based on parts without bird activity using the provided metadata. Later, these background noises are used for data augmentation.
Some parts of validation soundscape recordings are merged with Xeno-canto training set [
To create more diverse models, 6 diferent sub-datasets are formed targeting diferent locations, which are later ensembeled.
As presented in Table 1, sub-datasets are divided based on locations, e.g. Dataset-1 and 2 are prepared based on the locations of test set. With the given latitude and longitude from test data, a 200 and 400 km radius is marked and most likely occurring species within the radius are taken into account.
Dataset-3 consist of species that mainly occur in the given test recording locations. Dataset-4 belongs to species that mainly present in the recording locations of the United States. Dataset-5 belongs to species that mainly present in the recording locations of Costa Rica. Dataset 6 belongs to species that mainly present in the recording locations of Colombia.
The recordings are sampled at 32 kHz sample rate and trimmed to 30 s long chunks because if we use a shorter window size, it may not include any sound events or include some sound events which may be a noisy event or a background species as shown in Figure 1, for this reason longer chunks are preferred. To make the model learn correctly, we need to make each label correspond to call-events of each species. First, we compute a Short Time Fourier Transform (STFT) with a Hann window of 1024 samples and hop size of 384 samples and mel bins of 64, retain only the magnitude and then followed by applying a log-mel filter banks from 150 Hz to 15 kHz.
Diferent data augmentation techniques are performed to increase the model performance and
improve its generalization to real time data. The following data augmentation methods are
applied to raw audio recordings:
• 30 s chunk at random position for training
• Gaussian Noise
• Gaussian Signal to Noise Ratio (SNR)
• Adding primary background noise
• Adding secondary background noise
• Mixup augmentation
• Spectrogram augmentation
Primary background noise: The train recordings and test recordings contain domain shifts.
To make train recordings more robust, diferent noises are incorporated as background noise.
Besides the noise extracted from soundscape recordings, recordings without bird activity from
BAD (Bird Audio Detection) are used [
Secondary background noise: Mixing various (bursts of overlapping) short audios in the train recording with random pauses between. Noises like wind, car sounds, insects, rain and thunder are used.
Mixup: Audio chunks from random files are mixed together, and their corresponding labels are
added as shown in Figure 2. The mixup augmentations are constructed using the formulae [
Gaussian SNR: The Gaussian noise applied to the samples with random signal to Noise Ratio (SNR).
Spectrogram augmentation Time stretching and pitch shifting are the augmentations tried
on spectrograms. Time stretching is the process of changing the speed/duration of sound
without afecting the pitch of sound. It takes the wave samples and a factor by which the input is
stretched by a factor of 0.4 which has a small diference with the original sample. Pitch shifting
is the process of changing the pitch of the sound without afecting the speed. It takes wave
samples, sample rate and number of steps(-4 to +4) through which the pitch must be shifted.
These methods are performed using LIBROSA library [
In recent years Convolutional Neural Networks (CNNs) have been successfully used for audio
recognition and detection. The architecture design for the bird recognition task was inspired
from DCASE2019 PANNs (Large-Scale Pretrained Audio Neural Networks for Audio Pattern
Recognition) [
From the previous BirdCLEF challenges, deeper CNN networks performed well when compared
with wider or shallow CNNs. Hence the backbone network for this challenge used are ResNets
[
ResNet: Deeper CNNs perform well on audio recognition tasks. The challenge in very deep CNNs is that the gradients do not propagate properly. To solve this issue, ResNets introduced shortcut connections between convolutional layers.
DenseNets: DenseNets were designed to improve the information flow between layers, a diferent connectivity pattern was introduced with direct connections from any layer to all subsequent layers. The change of feature maps is facilitated by down-sampling the architecture by dividing the network into multiple densely connections, making the network deeper.
In this task, after log-mel feature extraction, the inputs are passed to ResNets/DenseNets by removing the last fully connected layers and extract only features. Then, a modified 1D attention based fully connected layer is attached to ResNet. The output of this network is a dictionary which contains clipwise and framewise outputs. Table 3 illustrates the modified networks used in this research.
Our CNNs used a model pretrained on ImageNet [
The networks are trained for 75 epochs without mixup augmentation and 150 epochs with
mixup augmentation. The loss function used here is BCE-focal-2way loss (binary cross entropy)
and sed- scaled-pos-neg-focalloss(FL) [
SED-Scaled-Pos-Neg-Focal loss: It focuses on primary labels and secondary labels loss.
= (, ℎ) − − = (1 − ()) · (1 − ) · − − = () · (1 − ) · = (4) + (5) = ( > 0.0, − , − ) − = ·
Oneslike are tensors filled with the scalar value ‘1‘and zerolike are tensors filled with the scalar value ‘0‘.
The grouped output losses are Focalloss-scaled, bceloss, Focalloss.
The optimizer used here is AdamW optimizer with weight decay 0.1. The learning rate scheduler is a combination of merging Cosine Annealing Scheduler with warmup (cycle-size is epochlength number of epochs) + LinearCyclicalScheduler (cycle-size is epoch-length 2). The initial learning rate is 0.001. Background species metadata are not taken into account. (3) (4) (5) (6) (7) (8)
This section illustrates the combination of models, ensemble techniques and evaluation score on the test set. Table 4, Table 5, and Table 6 illustrate the diferent strategies used in the model and their respective results based on public and private leadership board. The ensemble method used here is voting.
Ensemble RUN 1: Models M1, M2, M9, M10, M14, and M15 are used. This model contains 397 classes and used diferent background noises. The clipwise threshold, framewise threshold and number of votes are discussed in the table 7. Ensemble RUN 2: Models M3, M4, M5, M11, M12, M13, M16, M17, and M18 are used. This model contains 391, 345 and 273 classes, and used diferent background noises. The clipwise threshold, framewise threshold and number of votes are discussed in the table 8. The 9 model ensemble is a combination of diferent classes which are split based on location and achieved the top score of 0.6741 in our public score with less False Positives and 0.6024 in the private score. Ensemble RUN 3: Models M7, M8, and M9 are used. This model contains 162, 187 and 263 classes and used diferent background noises. The clipwise threshold, framewise threshold and number of votes are discussed in the table 9. This ensemble method comprises of 3 diferent locations. The 3 model ensemble based on location split has a score of 0.6799 on public score with less FP compared to 0.5951 on private score.
Ensemble RUN 4: Models M2, M3, M4, M6, M7, M8, M10, M11, M12, M15, M16, and M17 are used. This RUN takes best performing models which used diferent background noises. The clipwise threshold, framewise threshold and number of votes are discussed in Table 10. This ensemble method comprises of 12 diferent models with diferent backbone models and diferent classes split based on location yields the best private score of 0.6034.
The current approach attained an F1 score of 0.6034 in the private leadership board. Recognizing all bird species is still challenging because of domain shift in train (clean audio) and test (noisy audio) data. The train dataset consists of weakly labeled (clipwise labeling) and there were many background species present. A multi-label annotation of train files could have significantly improved the models in bird recognition.
There are several techniques to improve this bird recognition task, methods like vision transforms and removal of no bird activity from the train dataset. A promising approach would be the feature extraction by merging two diferent features in combination with polyphonic event detection. Better inference techniques could focus more on locations, e.g. using the ebird API and thresholds for each species separately, to achieve better recognition of bird events.