This paper presents deep learning techniques for audio-based bird identification at very large scale. Deep Convolutional Neural Networks (DCNNs) are fine-tuned to classify 1500 species. Various data augmentation techniques are applied to prevent overfitting and to further improve model accuracy and generalization. The proposed approach is evaluated in the BirdCLEF 2018 campaign and provides the best system in all subtasks. It surpasses previous state-of-the-art by 15.8 % identifying foreground species and 20.2 % considering also background species achieving a mean reciprocal rank (MRR) of 82.7 % and 74.0 % on the official BirdCLEF Subtask1 test set.
A system for audio-based bird identification has proven to be particularly useful for
biodiversity monitoring and education. It can help professionals working with
bioacoustic sounds by processing large audio collections in a fast and automated way. Or
help amateurs to identify birds in self-made recordings. The LifeCLEF bird
identification task challenges participants to identify different bird species in a large collection
of audio recordings provided by Xeno-Canto [
Distinguishing between so many different birds is a challenging fine-grained
classification problem. Until 2016, the best system used template matching of small sound
elements combined with a large number of acoustic features and a random forest
ensemble learning method for classification [
Audio files provided in the training and test sets are not only of different duration and
quality, they also have different formats regarding sample rate, bit depth and number
of channels. In a first step, two data sets are formed with homogeneous file properties.
For the first set all files are resampled to 44.1 kHz followed by normalization to a
maximum signal amplitude of -3 dB. For a second set all files are first high pass
filtered at a frequency of 2 kHz (Q = 0.707), resampled to 22050 Hz, mixed to mono
and finally also normalized to -3 dB. The training set is augmented by additionally
extracting 381 audio files using the time coded annotation of species in the metadata
of the newly provided soundscape validation set. All files belonging to the training set
are further processed to create additional data sets with different content described
more detailed below:
BirdsOnly
NoiseOnly
AtmosOnly
LowQuality
Via segmentation, the audio content of each training file is separated in signal and
noise parts. Segmentation is done in frequency domain applying image processing
methods like median clipping [
Many recordings from the community-based online platform Xeno-Canto are created by amateurs with non-professional equipment. To simulate low quality or highly compressed recordings a fourth data set (LowQuality) has been created by encoding all training files to MP3 with a low target bit rate (lame parameter: -V7) and decoding them back to WAV format afterwards.
For model evaluation the training data is split into training and validation sets. This is done by creating 8 stratified folds. For model training only three of them are actually used with individual validation sets of ca. 10 to 20 percent of the training set. To get a more realistic estimation of model performance validation sets are created with respect to bird individuals. An IndividualID was assigned to each audio file in order to achieve all files belonging to the same bird are either part of the training or the validation set. Files got the same unique IndividualID if they belong to the same class and were recorded by the same author at the same day.
For audio-based bird identification DCNNs are trained via PyTorch [
chunks are extracted successively from each file and channel (if not mono) with an
overlap of 10 % for validation files during training and 80 % for files in the test set.
Predictions are averaged for each file (Subtask1: monophone recordings) and time
interval (Subtask2: soundscape recordings) by taking the mean over all chunks and
channels. To emphasize chunks with higher prediction values, suggesting a more
confident identification, all predictions are squared before taking the mean. A similar
technique called mean exponential pooling is employed in the baseline system
provided by [
If species composition is known in advance, e.g. based on knowledge about recording habitat or time, removing predictions of unlikely species can further improve identification performance. To exploit this fact, predictions of species, not part of data sets in previous years, are removed for corresponding files of the validation and test sets. Other metadata besides year of recording or data set membership is not utilized and none of the provided metadata is used for model training.
Various data augmentation techniques are employed to prevent overfitting and
improve model accuracy. The following augmentation methods are applied in time
domain regarding audio chunks:
use files from different data sets (Original, BirdsOnly, LowQuality)
extract chunks from random position in file (wrap around if end of file is reached
and continue reading from beginning)
apply jitter to duration (ca. ± 0.5 s)
add 2 audio chunks from random files of the NoiseOnly set
add 2 audio chunks from random files of the AtmosOnly set
add 2 audio chunks from random files belonging to the same bird species
apply random factor to signal amplitude of all chunks before summation
apply random cyclic shift
skip random number of samples (time interval dropout)
reconstruct audio signal by summing up individual sound elements
Some of the above mentioned techniques were used before in previous BirdCLEF
tasks for example adding background noise or sounds from other files belonging to
the same bird class with random intensity [
Reconstruction via sound elements with dropout. With a chance of ca. 30 % the audio chunk is (re)constructed using the information extracted in the segmentation preprocessing step. Single sound elements are cut from the audio file, multiplied by a short fade in and out envelope, band pass filtered and summed up to recreate the original audio signal. With this procedure it is possible to vary position and length of each sound event individually by adding a small offset to its starting time or by applying a small amount of time stretching to it. Furthermore sound element dropout can be implemented by choosing some elements to be skipped and not added to the audio signal sum.
All above described operations are performed in time domain. The final audio chunk
is than transformed to frequency domain by applying a short-time Fourier transform.
Different FFT parameters are used depending on the sample rate and whether
frequencies are chosen to be linear or mel scaled. Normalization and logarithm is applied
yielding a dynamic range of approximately 100 dB. Low and high frequencies are
removed to get a spectrogram representing a frequency range of about 150 to 10500
Hz. Furthermore the spectrogram is resized to fit the input dimension of the DCNN
used for training (e.g. 299x299 pixels for InceptionV3 [
Piecewise time stretch. Besides manipulating the duration or speed of an entire audio chunk, time stretching is also applied piecewise with a 50 % chance to change speed at multiple times within a chunk. This is accomplished by dividing the spectrogram image in several vertical pieces, each having a width randomly chosen between 10 and 100 pixel. Then, pieces are resized individually by a factor randomly chosen between 0.9 and 1.1 along the horizontal (time) axis.
Piecewise frequency stretch. The same procedure is applied in an analogous way to the frequency axis with a 40 % chance and a stretch factor between 0.95 and 1.15. Random choice of interpolation filter. For resizing, the high-quality Lanczos filter of the Python Imaging Library is used by default. However in 15 % of the cases a different resampling filter is chosen from the library with equal chance: Nearest, Box, Bilinear, Hamming and Bicubic.
As a last augmentation step color jitter is applied to the final spectrogram image. This
is a very common method during training for image classification (e.g. ImageNet).
Brightness, contrast and saturation of the spectrogram image are slightly varied
(factor 0.3) as well as hue (factor 0.01). Color jitter was also used by [
Table 1 demonstrates the effect of augmentation on model performance. All models in the table are trained using the same parameter settings (if not mentioned otherwise) with a learning rate of 0.01 until performance on the validation set stopped improving (ca. 200 epochs). The first two models in the table show a MRR gain of almost 10 % for using no augmentation at all compared to applying all above described techniques. For the other models augmentation methods are separately turned off to show their individual influence. The greatest impact on identification performance is gained by adding background noise from the NoiseOnly and AtmosOnly sets. The least influence is achieved by randomly choosing chunks from the LowQuality set leading even to a better MRR score if this method is turned off. Pred. squared & 80 % overlap of chunks Pred. squared & 80 % overlap & unlikely species excluded
For the first run different models were used for Subtask1 and Subtask2 whereas for runs 2, 3 & 4 the same models or ensemble of models were used for both subtasks. The main properties of models for the first 3 runs are listed in table 3. Results on the official BirdCLEF test sets are summarized in table 4. Run 1 A single model was trained for each subtask (M1 for Subtask1, M2 for Subtask2) by fine-tuning pre-trained InceptionV3 networks with mel spectrograms of 5 seconds audio chunks as inputs. Models M1 and M2 mainly differ in the pre-processing of audio files and choice of FFT parameters (see table 3). For Subtask1 pre-filtered, mono audio files were used with a sample rate of 22050 Hz. For Subtask2 audio files were not filtered or mixed down to mono.
Run 2 For the second run an ensemble of the two models from the first run was used for both subtasks by averaging their predictions. Furthermore predictions of new species were removed for recordings belonging to older BirdCLEF data sets.
Run 3 For this run a third model (M3) was added to the ensemble of models from the previous runs. M3 used 6 seconds audio chunks, a larger number of mel bands and slightly different FFT parameters (see table 3). It was trained on a different training/validation split than M1 and M2. Furthermore predictions of the two best snapshots per model (regarding performance on the validation set) were chosen for averaging instead of just one, resulting in 6 predictions per species and file for Subtask1 (or file and time interval for Subtask2). For this and the next run, unlikely species were also removed for older recordings.
Run 4
Again all models and snapshots from the previous runs were reused plus two
additional snapshots per model. Furthermore snapshots of 4 models created in an earlier
phase were added to the ensemble. Those early models were trained using
spectrogram images with linear instead of mel frequency scaling and did not have the later
developed piecewise time and frequency stretch augmentation. One of the models was
trained by fine-tuning a SENet154 network [
Fig. 1. Official scores of the BirdCLEF 2018 “Subtask1: monophone recordings” bird identification task. The above described methods and submitted runs belong to MfN.
In the following paragraph some insights are summarized about what worked and what did not work so well during the experiments and processing of the task. Although low frequencies are discarded, pre-process the audio files by applying a soft high pass filter led to slightly better identification results. Also mel spectrograms performed better compared to spectrograms with linear frequency scaling. The best network for the task seemed to be the InceptionV3 architecture. Other networks were also tested for a few training epochs (ResNet152, DualPathNet92, InceptionV4, DensNet, InceptionResNetV2, Xception, NasNet). But even the more recent or larger architectures that are superior in other image classification tasks could not meet the performance of the InceptionV3 network with attention branch. Of course that doesn’t mean that other networks are not potentially able to give better results with the proper tuning.
For model training extracting audio chunks at random position on the fly worked better than using pre-sliced, overlapping chunks. Squaring predictions of chunks before taking the mean was beneficial when summarizing the results per file or time interval.
It proved to be a good decision to create training and validation splits with respect to bird individuals recorded over several files. For Subtask1 the MRR scores were in most cases even better on the test set than on the validation set.
If species composition is known in advance based on knowledge about habitat or recording season and time, removing unlikely species leads to less confusion and significantly helps classification. It would be interesting to see to what extent identification performance can be even further improved if metadata was additionally used and incorporated into the training process.
Augmentation prevents overfitting and strongly improves accuracy of models or model ensembles. A few techniques were adapted from previous work and could be successfully complemented with some new approaches. In an extra experiment it was tried to show their individual influence by turning them off one by one. Another maybe even better approach would have been to start from the model without augmentation and turning them on individually. For example the impact of duration jitter cannot be really verified if piecewise time stretching is also applied.
Some augmentation methods did not work as well as expected like e.g. deliberately degrading the audio quality. Nevertheless it still might be a beneficial training strategy if the model is deployed for bird recognition on mobile or embedded devices where recordings need to be highly compressed because of energy and storage constrains.
Acknowledgments. I would like to thank Hervé Goëau, Alexis Joly, Hervé Glotin
and Willem-Pier Vellinga for organizing this task. I especially want to thank Elias
Sprengel for sharing his knowledge and the fruitful cooperation during the bird
detection challenge [