Introduction

The main objective of our approach is to create an easy-to-use pipeline of an acoustic model from an image model. We want to stay in the same framework of the state-of-the art deep learning [7] Inception-v4, and to transfer it to soundscape classification. Inception-v4 has been pre-trained on imagenet, then we adapt its inputs and learn a bird acoustic activity detector thanks to the classification outputs joint to an attention mechanism. Then we process a transfer learning from pre-trained weights on imagenet to bird classification. The paper describes our methodology to efficiently build this model, that we call 'Soundception', in few weeks a reduced GPU resources. We show that Soundception gives to the best scores of the BirdClef 2017 challenge [5]. In the last section we discuss on perspectives to increase the accuracy of Soundception.

Audio featuring

Audio to tri-channel time-frequency image

The data representation is a crucial step in any learning process. In our approach, the representation must be scalable and based on image processing including some acoustic specificities, because we take advantage of Google Inception that is fed by a RGB image. Therefore, we generate three log-spectrograms by fast Fourier transform at three scales : window size of w i∈{0,1,2} = 2 (2×i) × 128 (i.e. 128, 512, 2048). This fast computation approximates a compressed multi-scale representation of voices and chirps of birds. We think it improves usual spectral representations that deal with either temporal or frequency resolution (see usual representation in [4,3]). Next, we reshape the three spectrograms by bilinear interpolation, into an optimal dimension for Inception inputs. In sum, our audio featuring is :

1. Resample the dataset to 22050 Hz sampling rate. 2. Let min duration be the accepted minimum duration of audio sample. 3. Let min subduration be the accepted minimum duration of the subimage. 4. Let d(x) be the duration of the audio sample x. 5. While d(x) < min duration self concatenate x. 6. Compute three log-spectrogram S i (x) with window sizes w i ∈ (128, 512, 2048). 7. Remove outliers of the S i (x) distribution to avoid quantification error. 8. Resize (bilinear interpolation) S i (x) to an optimal format for Inception: height = 299 pixels for the frequency dimension, 299×min subduration/1.5 pixels for the time dimension. 9. Concatenate the three S i (x) into one 3 channels RGB multiscale image I.

Data augmentation

During the training stage, we run data augmentation. Therefore, we use standard transformations in computer vision. More precisely we run the Inception preprocessing on the spectrograms S i as random hue, contrast, brightness, and saturation, plus random crop in time and frequency, as follows : 3 Model specification

Transfer learning from Inception-v4 to Soundception

Inception-v4 is the state-of-the-art in computer vision [1]. There are several ways to adapt Inception-v4 to time-frequency analysis, as a simple average pooling on the time axis, or use recurrent layer at the top of the network or both. Here we adapt Inception-v4 to make it entirely convolutional on the time domain, the aim being to make it invariant to temporal translation, and to allow arbitrary widthsized image. Secondly we add a time and a time-frequency attention mechanisms into the branches as represented in the synopsis of Inception-v3 Fig 3. We set dI c = 15 seconds, min d = 60 sec., scale = 1.5 sec. for 299 pixels in respect to baseline Inception inputs of 299 × 299 pixels, and the available VRAM per GPU (12 Go). We do not use specific bird detection in order to avoid handcrafted detection which could weaken the complete pipeline. The detection is processed by an attention mechanism as presented in the next section. It runs on a large time window (dI c = 15sec.) to increase the probability of bird activity in the image. The main process results into one time frequency RGB image (as Fig. 1) per audio file, thus 36492 in BirdClef 2017.

Attention mechanisms in time and time-frequency

Attention mechanisms are gaining popularity. The goal is to focus attention somewhere or on something. We can learn detection from classification by adding a soft attention mechanism [2]. Thus, we add to the Inception model two attention mechanisms : a temporal attention into the auxiliary branch, and a timefrequency attention in the main branch.

Each attention mechanism is an element wise product of the detector feature map with the feature maps of the previous Inception layers. These mechanisms learn how to pass the information and thus play the role of bird activity detectors.

The first attention mechanism is the temporal attention defined by a sigmoid activation because the bird temporal activities are expected to be binomial in time. The second attention mechanism is defined by the softmax of the outputs1 , yielding to smooth neuron time-frequency activity distribution.

Next, based on the sigmoid or softmax of the outputs, we compute the element wise product between the feature maps of the signal and the feature maps of the detector.

In Fig. 4 we show how Soundception focuses in time on bird activities. In Fig. 5 we show that in time-frequency Soundception indeed focuses on the loudest formant/frequency of the bird call.

Training stage

The training of the model took several days depends of hyper-parameters, using randomly split training (90%) and (10%) validation sets as in [4]. For the transfer learning, we first train only the top layer of Soundception, then we fine tune all layers. We split training in different stages with different batch sizes, according to the available GPU memory :

1. Train the model with time window of dI c = 15sec., 2. Train last layers and detectors with mini-batch size of 8, 3. Fine tune all layers with mini-batch size of 4.

There are different options to evaluate the prediction on the development set. We could consider the audio file transform into images I as previously described with arbitrary temporal size. Here, we optimize the model according to the average score of the predictions from the subimages I c of the main images I.

Results

Fig. 6. Results of all the challengers in the BirdClef2017 international contest. The DYNI UTLN RUN1 is the best in three task categories : Soundscapes with time-code, and the two traditional classification tasks. It is third in the 'Soundscape without timecodes'. Complementary, the DYNI UTLN Run2 yields to lower scores in average but to the best score in the other task 'Soundscape without time-codes'.

We report Fig. 6 the official scores on the four tasks of each of the challenger. Our model Soundception wins this challenge in the four tasks. The DYNI UTLN RUN1 depicted in this paper is the best model in three of the tasks, with 0.714 Mean Average Precision (MAP) on the 1500 species 'traditional records' task, 0.616 MAP with 'background species' task, and 0.288 MAP on the 'Soundscapes with time-codes' task. It is third in the 'Soundscape without time-codes' task, for which our other run (DYNI UTLN Run2) which explored different parameters is first.

These results are good despite the fact that we had not time to completely train Soundception on different topologies and to develop associated preprocessings in the four weeks of the challenge.

Conclusion and future work

In this paper we show how we transferred Inception-v4 from image to the acoustic domain, and how it learns bird sound detection by itself using attention models. The results show that it is possible to tackle state-of-the-art sound classification by the transfer learning of efficient pre-existing image classification model. This strategy can be useful to tackle other challenge without pre-segmentation.

We had not been able to let completely converge the training stage of Soundception due to the huge computation and GPU needs, however it reaches the best results in the BirdClef 2017 challenge. There is a lot to be done in this area. Our current work also explores different scalable optimizations to learn audio to image representations instead of pseudo multi-scale FFT spectrograms. We currently develop a model with stacked GRU at the top of the network.