This paper describes a convolutional neural network based deep learning approach for bird song classification that was used in an audio record-based bird identification challenge, called BirdCLEF 2016. The training and test set contained about 24k and 8.5k recordings, belonging to 999 bird species. The recorded waveforms were very diverse in terms of length and content. We converted the waveforms into frequency domain and splitted into equal segments. The segments were fed into a convolutional neural network for feature learning, which was followed by fully connected layers for classification. In the official scores our solution reached a MAP score of over 40% for main species, and MAP score of over 33% for main species mixed with background species.
Identification and classification of bird species can greatly help to explore biodiversity
and to monitor unique patterns in different soundscapes [
The LifeCLEF challenge allows manually aided solutions (like crowdsourcing), however we have chosen state-of-the-art deep learning techniques to address the problem. Our solution uses two dimensional convolutional neural networks, that is trained with preprocessed bird songs transformed to the frequency domain.
The outline of this paper is as follows. Section 2 briefly overviews the application of convolutional neural networks in speech recognition and sound classification, furthermore investigates some solutions for the previous BirdCLEF challenges. Section 3 describes the data preparation method we applied. Section 4 introduces the applied deep learning technique and neural network architectures for bird song classification. Section 5 presents our results and Section 6 draws conclusions. 2
Besides image classification one of the main propelling force of deep learning is speech
recognition. In speech recognition different deep learning techniques, like deep belief
networks, deep neural networks and convolutional networks, are proven to surpass the
accuracy of ‘traditional’ Gaussian Mixture Models [
The task of bird song classification with neural networks has been investigated even
back in 1997 [
There have been a number of competitive approaches in the BirdCLEF challenges
of previous years, however deep learning was not applied in the BirdCLEF competition
before. The winning solution of 2014 used a robust feature extraction (including
MFCC, fundamental frequency, zero crossing rate, energy features, etc. - altogether
6669 features per recordings), feature selection (reducing the number of features from
6669 to 1277) and template matching. The last year’s challenge was won by the same
competitor. His work described in [
As a result, the vertical dimension (frequency) of the spectogram was 200, and the horizontal dimension (time) depended on the length of the recording. In the time domain (horizontal axis) we split the spectograms into 30 sample long columns (that corresponds ~0.5 seconds) and in the frequency domain (vertical axis) we split the spectrograms into 10 sample high rows. As a result, every spectogram was split into 30✕10 sized cells. We used these cells to remove the irrelevant parts (that is likely not to contain any bird call) of the spectrogram based on the mean and the variance. We calculated the mean and variance of every 10 sample high row (that corresponds a frequency range). If a cell's mean is less than 1.5 times the addition of mean plus variance of the actual row, than we dropped the cell. In case of Run 1, 3 and 4 we also removed those parts of the filtered spectrogram where 95% of the column vectors were zeros (see Figure 2). This step was skipped at our second submission (referred to as ‘BME TMIT Run 2’ in the official results; see Table 1). After these preprocessing steps we split the remaining parts of the spectrogram to five seconds long pieces. Thus the dimensions of the resulting arrays were 200✕310 (310 samples corresponds to five seconds). We used this as the input of the convolutional neural network.
For classifying the bird songs we used convolutional neural networks. The resulting
200✕310 arrays of the spectograms after data preparation were fed into the
convolutional neural network and was treated like grayscale images. We used two different
CNN architectures: the first one was inspired by the winner architecture of 2012
ImageNet competition [
In the first type of neural network we modified the shape of the input and the
convolutional layers of AlexNet. We also added batch normalization layers before the
maxpooling layers. Experiments show that with batch normalization significantly better
accuracy can be achieved on MNIST and ImageNet datasets with faster convergence [
The second type of neural network used a simpler architecture, it consisted four
convolutional layers and the fully connected layers had less neurons. We used ReLUs as
activation functions [
To train the model we used RMSProp adaptive algorithm as optimizer [
Batch norm.
Max-pooling 3x3, s:2x2
Batch norm.
Max-pooling 3x3, s:2x2
Batch norm.
Max-pooling 3x3, s:2x2 ... 256@16x25 Batch norm.
Max-pooling 2x2 1 3 x1 16x16 s:8x8 1@200x310
31 256@12x19 1 3 x1 Fig. 4. CNN-Bird-1 (above) and CNN-Bird-2 (below) convolutional neural networks for bird species identification based on the spectogram of bird song recordings. (A@BxC refers to A number of planes with size BxC. The DxD refers to the kernel size.) Due to the fact that we split each audio file to smaller pieces (that were fed to the CNNs) if a recording was longer than five seconds we had to combine the multiple predictions of the neural network. In case of ‘BME TMIT Run 1, 2, 3’ we simply calculated the mean of the classification results. In case of ‘BME TMIT Run 4’ we used a custom calculation method for submitting the classification results: if the recording was split into more parts than we calculated the variance of the CNN’s outputs of each predicted class throughout the 5 seconds long split parts. Next the six predictions with the highest variance were selected. The predicted bird species came from the mean of these predictions. 5
The hardware we used for training were a NVidia GTX 970 (4 GB) and a NVidia
Titan X (12 GB) GPU card hosted in two i7 servers with 32 GB RAM. Ubuntu 14.04
with Cuda 7.5 and cuDNN 4.0 was used as general software architecture. For data
preparation, training and evaluating deep neural networks the Keras [
We investigated the accuracy of the model on a separated test set. The least average precisions (AP) were achieved by Ochre-rumped Antbird (AP=0.00067), Santa Marta Antpitta (AP=0.00136) and Rufous-breasted Leaftosser (AP=0.0015) bird calls. Yellow-eared Parrot (AP=0.692), Lesser Woodcreeper (AP=0.796) and Spillmann's Tapaculo (AP=0.899) species scored the best in the test. Furthermore, a lot of bird calls were misclassified to Orange-billed Nightingale-Thrush (AP=0.229). Analyzing the waveforms and the spectrograms of these species we couldn’t find any particular feature. Hence we suppose the significant difference in AP and the misclassification are generally caused by some shortcomings of the proposed CNN architectures.
The MAP (Mean Average Precision) values of our submission in the official results are shown in Table 3. The first MAP value corresponds to the recordings in which there was a dominant singing bird in the foreground with some other ones in the background. The second MAP is for recordings with only one singing bird. And the third MAP value is for the soundscape audio, that was not targeting specific species and these recordings might have contained an arbitrary number of singing birds. The results show that the smaller convolutional neural network (CNN-Bird-2; Run 3 and 4), which was faster to train performed similarly as the bigger CNN. However, the gain in AUROC on the validation database is not reflected in the official results (MAP values) in case of Run 3 and 4. Moreover the difference in the combination methods of Run 3 and 4 could be measured on the validation set, but in the official results Run 4 didn’t outperform our other approaches. According to the official results we resulted the 4th place out of 6. It should be noted that we joined the competition only on April and we had no previous experience with bird call recognition.
MAP MAP MAP (with background species) (only main species) (‘soundscape’ recordings) 0.323 0.407 0.054 0.338 0.426 0.053 0.337 0.426 0.059 0.335 0.424 0.053
In this paper a deep learning based approach was presented for large-scale bird species identification based on their songs. In the data preparation process the spectogram for every recording was calculated and the irrelevant parts were removed. The resulting spectogram was sliced into five seconds long segments, these segments were used as input of the CNN. Two different types of CNNs were used that achieved about the same accuracy, while one of them had much less parameters. At the final step the predictions of the slices were combined. The results show that the deep learning based approach is well suitable for the task, however fine-tuning is necessary to reach better accuracy, like separating time and frequency in the CNN feature learning part and applying recurrent architectures, e.g. Long Short-Term Memory (LSTM).
Bálint Pál Tóth gratefully acknowledges the support of NVIDIA Corporation with the donation of an NVidia Titan X GPU used for his research.