There are about 10,000 different bird species in this world, and they all play an important role in the natural world. They serve as good indicators of declining habitat quality and pollution. It is often easier to hear birds than it is to see them. BirdCLEF 2021 [1] -Birdcall Identification is a Kaggle competition organized by The Cornell Lab of Ornithology in collaboration with LifeCLEF 2021 [1] whose challenge is to identify which birds are calling in long recordings, given training data generated in meaningfully different contexts. This paper is structured in a way that it first gives details of the competition and the given data so that there is a clear understanding of the challenges posed by the train and test data. Also, we will provide a detailed solution to the approaches we have used for this challenge including data preparation, augmentations, model building, training procedure, and post-processing techniques.
This section gives a brief overview of the data provided in the competition. Training on the data posed a lot of challenges since the train and test data were of different types.
The training data is mainly comprised of two types of audio recordings:
Train short audio: The bulk of the training data consists of short recordings of individual bird calls generously uploaded by users of xeno-canto.org. These files have been downsampled to 32 kHz where applicable to match the test set audio and converted to the ogg format. Information of 397 unique species has been given. Along with audio files, metadata is also provided which consists of primary label, secondary labels, type, latitude, longitude, scientific name, common name, author, date, filename, license, rating, time, and URL.
Train Soundscapes: There is a distinct shift in acoustic domains between the training and test set. So, some examples of soundscape recordings from the test set have been provided for training and validation purposes. These 20 recordings represent 2 of the 4 test recording locations and are of length 10 minutes each. In the metadata, information has been given as to which birds are present in each of the 5 second timestamps in the training soundscapes. Nocall label has been assigned if no bird is present.
All labels for train short audio had to be considered as weak labels since we did know which species is audible in the recording, but we did not know the exact timestamps of the vocalizations. Training with weakly labeled data was one of the core challenges of this competition. Secondary label lists the number of audible background species as annotated by the author. These lists might be incomplete and not very reliable. Also, the training data had a long-tail distribution making the dataset highly imbalanced as the head classes contained some species having train sequences more than 500 whereas some in the tail region had around mere 10 -20 sequences.
It has approximately 80 audio recordings similar to train soundscapes. They are of 10 minutes each. We need to identify the birds present in each of the 5 second timestamps throughout the audio. These recordings are from 4 locations .
We used 4 different approaches to train our model
In this approach, we trained the model on Mel-Spectrograms. We trained 2 types of modelsmodel A and model B. Model A was trained to predict whether a bird is present or not in an audio clip i.e. it was a binary classification model. To train the model, we used an external dataset Freefield1010 [2] along with competition data. Model B was trained to classify the birds' species. Official competition data [3] was used for this model and we tried not to input any case of nocall making use of the weak labels generated by Model A when run on the competition dataset 1.
Transfer learning from State of The Art Image-net Models to Sound Classification.
For Model type A we took 3 pretrained models i.e. Efficient B0 [4], Resnet50 [5] and Densenet [6]. We noticed that SpecAugment [7] was not giving good results, but SpecChannelShuffle increased the model performance by 0.07. We got the highest score of 0.91 F1 score by Efficientnet-b0 and by blending three models we got 0.93 F1 Score on Freefield1010 [2] Dataset to classify if there is any bird present or not.
For Model type B we experimented with many pretrained models including Efficientnet B0, B1, B2, B3, B4, Resnet 50, Nfnet [8] and Resnet WSL [9] . We mention the result of this in result section 1. Here SpecAgument worked very well.
We executed data augmentation during the training stage.
β’ SpecAugment: SpecAugment is a popular augmentation technique applied on spectrogram.
The spectrogram is transformed by warping it in the time direction, masking blocks of consecutive frequency channels, and masking blocks of utterances in time. We noticed that SpecAugment increased model performance without requiring any further model or training parameter tweaks.
-TimeMasking: In time masking, t consecutive time steps [π‘ 0 , π‘ 0 + t) are masked where t is chosen from a uniform distribution from 0 to the time mask parameter T, and π‘ 0 is chosen from [0, π β t) where π is the time steps.
-FrequencyMasking: In frequency masking, frequency channels [π 0 , π 0 + f) are masked where f is chosen from a uniform distribution from 0 to the frequency mask parameter F, and π 0 is chosen from (0, πβ f) where π is the number of frequency channels.
β’ SpecChannelShuffle: Shuffle the channels of a multichannel spectrogram (channels last). This can help combat positional bias. β’ MixUp [10]: We did mixup according to primary labels that is we combined the melspectrograms according to a parameter alpha which had been taken from beta distribution and also took weighted average of the target label according to the same alpha. Mixup helps in reducing memorization of corrupt labels and acts as a good regularizer during training.
Here Image represents the raw input image array and target represents the label (one-hot encodings) of the corresponding image.
The training procedure used for both the models is as follows:
Model A: The model was fed with both Freefield1010 as well as Competition data and the above augmentations were applied on them. Smaller models were trained for 15 epochs while larger models were trained for 8 epochs. We used linear learning rate for the first few epochs to provide warmup and after reaching its peak i.e. 0.002, it was linearly reduced. Adam [11] optimizer was giving the best result for this model.
Model B: The model was fed with competition data only and augmentations similar to that of Model A were applied. Smaller models were trained for 40 epochs while larger models were trained for 25 epochs. A similar strategy was used for learning rate scheduler as that of Model A. The optimizer used was Adam. While training we froze all the layers but the last few for the initial few epochs to help the model converge faster. Then all the layers were unfrozen and trained for the remaining epochs.
In this approach, we trained the model on raw audio sample in Waveform domain. Here also we train 2 types of models-model A and model B 1. Model A was trained to predict whether a bird is present or not in an audio clip i.e. it was a binary classification model. Model B was trained to classify the birds' species.
We resampled the raw wave to 16000 Hz sampling rate. Then we let πππ₯ π be the max length of the audio. If the length was less than πππ₯ π , we padded it with 0 at one end whereas if the length was greater than πππ₯ π , we cut the audio from both the side.
This Model is highly motivated by ReSE-2-Multi [12]. With this frame-level raw waveform input, the bottom layer filters should learn all conceivable phase variations of (pseudo-)periodic waveforms that are likely to be present in audio signals. This has hampered the usage of raw waveforms as input over spectrogram-based representations in which the phase fluctuation within a frame is taken into account (i.e. time shift of periodic waveforms) is removed by taking merely the magnitude. So we added an Attention layer between two FC (Fully Connected Layer) 2. It's a simple Convolutional Long short-term memory Deep Neural Network (CLDNN) Model [13], with residual Connections which will impact high level features of Audio data.
β’ AddImpulseResponse: Convolve the audio with a random impulse response.
β’ TimeMask: Make a randomly chosen part of the audio silent.
β’ AddGaussianSNR: Add gaussian noise to the samples with random Signal to Noise Ratio (SNR) [14] β’ AddGaussianNoise: Add gaussian noise to the samples β’ We add pink noise at variable volumes, as well as random soundscape β’ We also used a Butterworth filter with stochastic cutoffs (randomly lowpass, highpass, bandpass, bandstop).
Model A used both Freefield1010 data as well as competition data for training whereas Model B used for competition data. the augmentations already stated above were applied to these raw audio samples.The rest of the training procedure is very similar to that of Wave-gram model.
After training the whole dataset in spectrograms domain and waveform domain, we check our hypothesis of combining the result from both domains so that both the models have other model's domain knowledge. For training, we froze all layers but the last 5 layers for Wave-gram Training Model π π , and in case of Wave-Form Domain Training Model π π , we froze all the layers except the last 3 layers. We calculated the loss using the below method which would back-propagate through both the models.
π ππ = π π (ππππ(π π ))
We got a boost by 0.05 in Cross Validation Score with this technique.
In test set and train soundscapes, the audio file contains different types of birds. We thought of separating them and then training the classification models. We decided to introduce the music source separation concept in the multi-class classification task and experimented on it. The model is highly motivated by Demucs [15]. We provide the code in our GitHub repository ??.
We discovered that an audio sample in Train Soundscapes data typically contained a maximum of 5 birds. So we took a hyper-parameter πππ π π to mix πππ π π short audios of birds. We did another experiment of mixing the nocall data from Freefield1010 and considered nocall as another bird that needs to be separated. We did the same steps for data preparation as in the Wave Form Domain dataset. We took different πππ π π of Short Audio of Data and mixed it according to βοΈ πππ π π π=1 π΄ π For second stage training of this model, we prepared the train soundscapes data by dividing it into chunks of data of length πππ₯ π and trained with the pseudo labels predicted by the first-stage model.
What is the difference between Demucs and ATDemucs? In Demucs there is downsample block and then a BiLSTM [16] layer and then upsample block. In ATDemucs Figure 3 there is attention in the LSTM layer and upsample block. In our method, we did cross attention between downsample output and upsample output. Downsample Block: The downsample block is made up of a convolution with kernel size K=8, stride S=4, πΆ πβ1 input channels, πΆ π output channels, and ReLU activation, followed by a 1x1 convolution with GLU [17] activation. We doubled the number of channels in the 1x1 convolution since the GLU outputs C/2 channels with C channels as input. Horizontal Trans Block: We replace the Bi-LSTM Layer with Self Attention [18] Layer consisting of 8 heads and Dropout 0.2 and hidden size πΆ πΏ . This block outputs 2πΆ πΏ channels per time position. We use a 1x1 convolution with ReLU activation to take that number down to πΆ πΏ . Upsample Block: The Upsample Block is nearly symmetrical to the Downsample Block. It is made up of a convolution with kernel size 3 and stride 1, as well as input/output channels πΆ π and a ReLU [19] activation. By eliminating simple concatenation like Demucs, we introduce a cross attention layer in which we take a query from the downsample block and a key and value from the upsample block. In addition, return the number of channels πΆ π by doing a 1x1 convolution using GLU activation. Finally, we employ a transposed convolution with K = 8 kernel size and S = 4 stride, πΆ πβ1 outputs, and ReLU activation. Instead of using an activation function, we output 4πΆ 0 channels for the final layer.
β’ Shift: Randomly shift audio in time by up to 'shift' samples.
β’ FlipChannels: Flip left-right channels.
β’ FlipSign: Random sign flip.
β’ Remix: Within a batch, shuffle the sources. Each batch is divided into groups of size group size, and shuffling is done separately inside each group.
We trained this model in two stages.
First Stage: First we trained the model on mixing short audio. In this, we have training data input as a combination of 5 different birds' short audios. We train our model to differentiate between these different recordings and separated them. We trained the model for 150 epochs with a learning rate 0.003. We used cosine annealing as the LR Scheduler which starts with a large learning rate which is relatively rapidly decreased to a minimum value before being increased rapidly again. AdamW [20] optimizer gave good results than others.
Second Stage: In the train soundscapes, we were given primary labels for the audio recordings at each timestamp of 5 seconds. So, after the first stage we took inference of our model on the train soundscapes and did pseudo labeling so as to finetune the model. We trained the model for 5 epochs with a low learning rate taking AdamW as optimizer. During training, we froze some of the initial layers.
Once our model has been trained to separate the different bird sounds from the main audio recording, we run a classification model on the separated audios so as to classify which bird species it is. For this, we used Resnet50 model with pre-trained weights. We trained the model for approx 20 epochs with Adam optimizer. We got a Cross Validation Score of 0.62.
We used two Post Processing Technique Scaling Method: We noticed different models have different best thresholds. So we decided to take them into some scale then add the logits. Let π ππ π β be the minimum threshold of all the models to be ensembled. Then we scaled the logits such a way that all the models' predictions below their respective best thresholds are converted into range 0 to π ππ π β , whereas all the models' predictions above their best threshold are converted int the range π ππ π β to 1. Then we average all the logits thus obtained after scaling and predict all the birds which have more probability than π ππ π β .
Voting Ensemble: Let π ππ πΆ is the minimum count of bird should present in all models N. We predict all those birds which have βοΈ π π=1 π ππππ π > π ππ πΆ .
We submit three type of inference models:
β’ Spectrograms Model + Waveform Model: We ensemble all the models by above scaling method, Which gave us Cross Validation Score of 0.732 and LeaderBoard Score of 0.6179. β’ Multi-Domain Meta Trained Model: We optimize the best threshold for the CV and get Cross Validation Score of 0.705 and LeaderBoard Score of 0.6167 by 0.15 threshold. β’ ATDemucs: We get the Cross Validation Score score of 0.623 and LeaderBoard Score of 0.59. There are many whereabouts to increase the model accuracy.
Table 1 Shows Cross Validation Score of Spectrograms Based Model (Type Model B). After Scaling All the models, we ensembled with a threshold of 0.20 and we got 0.716 accuracy. Direct Validation Score of 0.732 and LeaderBoard Score of 0.6179. We are figuring out the method a good method to ensemble other than averaging of all 3 method along with ATDemucs. We will update all our key findings in the Source code 1 .
We compose several approaches, specifically a spectrogram architecture, a raw-waveform architecture, and multi-domain meta training. In the spectrogram model as well as the raw waveform model, we used two downstream modules: one for predicting whether a bird is present or not and the other for multi-label classification of the birds. We then combined both these approaches using a loss method that back-propagates through both the models. Also, we experimented with the Demucs model and extended the model architecture by adding an attention layer in upsampling block. Ensembling methods including voting and scaling methods helped achieve better results than any individual model. The spectrogram model along with scaling and downstream modules gave us the best result on the Private Leaderboard which helped us reach 67th position in the competition.