We present an adaptation of the deep convolutional network Inception-v4 tailored to solving bioacoustic classi cation problems. Bird sound classi cation was treated as if it were an image classi cation problem by a transfer learning of Inception. Inception, the state-of-the-art in image classi cation, was used together with an attention algorithm, to (multiscale) time-frequency representations or images of bird sounds. This has resulted in an e cient pipeline, that we call Soundception. Soundception scored highest on all tasks in the BirdClef2017 challenge. It reached 0.714 Mean Average Precision in the task that asked for classication of 1500 bird species. To our knowledge Soundception is currently the most e ective model for biodiversity monitoring of complex soundscapes.
The main objective of our approach is to create an easy-to-use pipeline of an
acoustic model from an image model. We want to stay in the same framework of
the state-of-the art deep learning [7] Inception-v4, and to transfer it to
soundscape classi cation. Inception-v4 has been pre-trained on imagenet, then we
adapt its inputs and learn a bird acoustic activity detector thanks to the
classi cation outputs joint to an attention mechanism. Then we process a transfer
learning from pre-trained weights on imagenet to bird classi cation. The paper
describes our methodology to e ciently build this model, that we call
'Soundception', in few weeks a reduced GPU resources. We show that Soundception
gives to the best scores of the BirdClef 2017 challenge [
The data representation is a crucial step in any learning process. In our approach,
the representation must be scalable and based on image processing including
some acoustic speci cities, because we take advantage of Google Inception that
is fed by a RGB image. Therefore, we generate three log-spectrograms by fast
Fourier transform at three scales : window size of wi2f0;1;2g = 2(2 i) 128 (i.e.
128, 512, 2048). This fast computation approximates a compressed multi-scale
representation of voices and chirps of birds. We think it improves usual spectral
representations that deal with either temporal or frequency resolution (see usual
representation in [
During the training stage, we run data augmentation. Therefore, we use standard transformations in computer vision. More precisely we run the Inception preprocessing on the spectrograms Si as random hue, contrast, brightness, and saturation, plus random crop in time and frequency, as follows : 1. Random choice of an image I in the dataset. 2. Random crop a subimage Ic from I: { let hIc = initial height of Ic = 299, { let dIc = initial duration of Ic = 15sec: 299 10, { set random temporal dilatation factor of Ic uniformly sampled in [0:95; 1:05], { set random top of Ic uniformly sampled in [0:96; 1] hIc, { set random bottom of Ic uniformly sampled in [0; 0:01] hIc, { crop and resize Ic from I with above parameters and random time o set. 3. Vision preprocessing of Ic by hue, contrast, brightness, saturation variations. 4. Add random noise or process local brightness to Ic.
Inception-v4 is the state-of-the-art in computer vision [
We set dIc = 15 seconds, min d = 60 sec., scale = 1:5 sec. for 299 pixels in respect to baseline Inception inputs of 299 299 pixels, and the available VRAM per GPU (12 Go). We do not use speci c bird detection in order to avoid handcrafted detection which could weaken the complete pipeline. The detection is processed by an attention mechanism as presented in the next section. It runs on a large time window (dIc = 15sec:) to increase the probability of bird activity in the image. The main process results into one time frequency RGB image (as Fig. 1) per audio le, thus 36492 in BirdClef 2017. 3.2
Attention mechanisms are gaining popularity. The goal is to focus attention
somewhere or on something. We can learn detection from classi cation by adding
a soft attention mechanism [
Each attention mechanism is an element wise product of the detector feature map with the feature maps of the previous Inception layers. These mechanisms learn how to pass the information and thus play the role of bird activity detectors.
The rst attention mechanism is the temporal attention de ned by a sigmoid activation because the bird temporal activities are expected to be binomial in time.
The second attention mechanism is de ned by the softmax of the outputs1, yielding to smooth neuron time-frequency activity distribution.
Next, based on the sigmoid or softmax of the outputs, we compute the element wise product between the feature maps of the signal and the feature maps of the detector.
In Fig. 4 we show how Soundception focuses in time on bird activities. In Fig.
5 we show that in time-frequency Soundception indeed focuses on the loudest
formant/frequency of the bird call.
The training of the model took several days depends of hyper-parameters, using
randomly split training (90%) and (10%) validation sets as in [
There are di erent options to evaluate the prediction on the development set. We could consider the audio le transform into images I as previously described with arbitrary temporal size. Here, we optimize the model according to the average score of the predictions from the subimages Ic of the main images I. 4
We report Fig. 6 the o cial scores on the four tasks of each of the challenger. Our model Soundception wins this challenge in the four tasks. The DYNI UTLN RUN1 depicted in this paper is the best model in three of the tasks, with 0.714 Mean Average Precision (MAP) on the 1500 species 'traditional records' task, 0.616 MAP with 'background species' task, and 0.288 MAP on the 'Soundscapes with time-codes' task. It is third in the 'Soundscape without time-codes' task, for which our other run (DYNI UTLN Run2) which explored di erent parameters is rst.
These results are good despite the fact that we had not time to completely train Soundception on di erent topologies and to develop associated preprocessings in the four weeks of the challenge. 5
In this paper we show how we transferred Inception-v4 from image to the acoustic domain, and how it learns bird sound detection by itself using attention models. The results show that it is possible to tackle state-of-the-art sound classi cation by the transfer learning of e cient pre-existing image classi cation model. This strategy can be useful to tackle other challenge without pre-segmentation.
We had not been able to let completely converge the training stage of Soundception due to the huge computation and GPU needs, however it reaches the best results in the BirdClef 2017 challenge. There is a lot to be done in this area. Our current work also explores di erent scalable optimizations to learn audio to image representations instead of pseudo multi-scale FFT spectrograms. We currently develop a model with stacked GRU at the top of the network. Acknowledgements. We thank Laura Bessone for her support in this paper. We thank XenoCanto, LifeClef team, EADM GDR CNRS MADICS, SABIOD.org and Amazon Explorama Lodges with Lucio Pando, P. Bucur and M. Trone for the co-organization of this challenge. This research is also supported by STICAmSud BRILAAM for South American bioacoustics. We thank TPM, CG83, UTLN for their support in the Captile project on soundscape analysis. We thank V. Roger for setting the GPU, and S. Paris for lending them. We are grateful to the anonymous reviewers. Antoine Sevilla has set up the experimentations and ideas in this article. 7. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, et al.: TensorFlow: Large-scale machine learning on heterogeneous systems, tensor ow.org (2015) 8. https://research.googleblog.com/2016/03/train-your-own-image-classi erwith.html