Using Triplet Loss for bird species recognition on BirdCLEF 2020

Abstract

This paper presents the approach used in the BirdCLEF 2020 Competition. The objective of the competition is to try to recognize bird species through its sings and calls among 960 species in soundscapes. We use a MultiScale CNN + Triplet Loss to learn the melspectrogram characteristics that di erentiate the species from each other. The CNN create a 128-D embedding used to classify the species. The approach achieves third place on the competition with c-mAP of 0.009752 and r-mAP of 0.008. We also present some changes on train parameters done after the competition that achieve our best evaluation with c-mAP 0.062877 and r-mAP of 0.108.

Introduction

against embeddings of new samples using a similarity by distance calculation. The Triplet Loss helps to learn which features, represented by the embedding vectors, di erentiate samples of di erent species. Related works that make use of Siamese Neural Networks have been applied in di erent contexts [ 8, 15, 13 ], and also in the context of audio recognition [ 4, 11, 14 ]. After reviewing the related works, we decided to adopt an approach in which we feed the Neural Network with Mel-spectrograms to generate a unique feature representation of each species call. 2

Preprocessing

As the competition task is to recognize species in every 5 seconds of audio in a recording, the rst preprocessing was to take all the recordings contained in the training set and split them into ve-second chunks. There were some recordings in the training set with less than ve seconds long, in these cases, the records were padded with copies of themselves until they achieved ve seconds. After this stage, we obtained a total of 623.981 segments.

Similarly to [ 11, 4, 9 ], we opted for a visual approach to model input, using Mel-scale spectrograms taken from each chunk of audio, as mentioned above. Furthermore, were also extracted the harmonic and percussive components from Mel-spectrograms [ 3, 14 ], once di erent species can has a call more expressive in one of its components. Figure 1 shows an input example of ve seconds audio belonging to the species Alder ycatcher. Due to the high computational demands of preprocessing, we decided to perform a sub sampling in which 70 thousand spectrograms were separated for training and 20 thousand for validation.

All preprocessing steps were carried out with two Python Libraries for audio processing: PyDub [ 2 ] and LIBROSA [ 1 ]. Table 1 shows the parameters used to extract the Mel-spectrograms.

Approach

Model Our approach was inspired on [ 14 ]. Our key idea is to use a Siamese Convolutional Neural Network to extract features from Mel-spectrograms that can be arranged in a Euclidean space of n dimensions represented by an embedding vector. We tested two di erent CNN architectures based on [ 14 ]. In the rst one, we reduce the number of Multiscale analysis modules from 4 to 2. In the second one, we keep the principal backbone architecture without the Dropouts layers between the two nal dense layers. These adjustments were made empirically, based on test experiments. The table 2 shows the two implemented CNNs architectures. Both of them have inputs with shape 40x200x3, representing the three Melspectrogram with shapes 40x200 mentioned above. All convolutional layers use Relu as the activation function.

The Siamese network with triplet loss aims to generate close embedding vectors when the inputs belong to the same species and, at the same time, keep them away from vectors representing other species, considering a Euclidean vector space [ 12 ]. In this way, we hope that the new vector representation would make the classes more easily separable, therefore, easier to classify using a standard classi er such as kNN (k-Nearest Neighbors).

To achieve faster training convergence, we select the semi-hard triplets, which are the triplets composed by a negative sample more distant than the positive sample from the anchor sample. However, this distance lies inside a prede ned margin . Thus, with this loss function the lower the di erence between positive and negative distances, the greater is the Loss. Equation 1 shows the loss function we want to minimize, where f (x) is the embedding vector generated by the Siamese Network for the input sample x. For the i -th triplet xia represent the anchor, xip a positive sample, and xin a negative sample.

N L = X kf (xia) i f (xip)k22 kf (xia) f (xin)k22 + + (1)

Once we convert all Mel-spectrograms into embeddings vectors, we can use them to classify each species. In our rst submission, we chose a kNN classi er with Euclidean distance and, in our other submissions, a Multilayer Perceptron (MLP) neural network. The MLP architecture has an input layer, a hidden dense layer with 256 neurons and an output layer with softmax activation and 960 neurons, equivalent to the number of species we wish to recognize. We tried various combinations of hyperparameters until we reached our results. Table 4 shows the con guration used in each submission trial. The MLP was always trained with 200 epochs, batch size of 256 and learning rate equal to 0.001. The test data has 153 soundscapes of 10 minutes each, for our submission, we divided them into segments of 5 seconds. For each segment, we build a Melspectrogram and its components. After that, we passed the spectrograms through a CNN trained to obtain a 128-dimensional embedding vectors. Once we obtained the vectors, we use this to predict, using a classi er, a possible specie present on this part of that audio.

As our model is still not very accurate, for each 5 seconds piece of audio we get the 10 rst possible species pointed out by the classi er and put them into submission le, given a higher weight to the rst one and lower weight to the last one. We also take into account, and cut o , the species that does not belong to that region where the current soundscapes were recorded. The list with species per region and the information where each soundscape were recorded was provided along with the dataset. 4

Results

To evaluate the system, the competition provides a well-known ranking metric, the Mean Average Precision (MAP). But in this case, divided into two parts. The sample-wise mean average precision (r-mAP), the classic one, and the classwise mean average precision, which takes the average precision (c-mAP) among the classes. Bellow, we have the equation for MAP, where Q is the number of test audio les and AveP(q) is an Average Precision for q le computed on equation 3:

M AP =

PQ q=0 AveP (q)

Q ; (2) and

AveP = PkN=0(P (k) rel(k)) ; (3)

num relevant documents where k is the rank in the sequence of returned species, n is the total number of returned species, P (k) is the precision at cut-o k in the list and rel(k) is an indicator function equaling 1 if the item at rank k is a relevant species.

Table 5 shows the results for the submissions made, 1 and 2 made during the competition and the others made after the competition ended. The submission 4 and 5 use the same train con guration, the di erence between them is that on submission time instead of given higher weight to the rst one and lower weight, we assign a weight for all species equal to 1. We applied a Siamese Network with Triplet Loss to the task of Bird Sounds Recognition. Our results were not the best, but we believe they can be improved. This is an initial study on this theme, from here a better study can be carried out. In this work we don't do any kind of data augmentation, a good work with augmentation techniques can be made with noise injection, as in [ 9, 7 ]. Therefore algorithms as Noise Reduction and per-channel energy normalization [ 10 ] still can be used on the preprocessing stage to make learning a simpler task.

On train stage, a better hyperparameter optimization should be done to achieve the optimal result, not only empirically choices. Furthermore, a di erent approach to the selection of the triplets can be tested, such as the dynamic triplet loss in which the margin varies according to the progress of the training [ 14 ].

The implementation made to run all the submissions mentioned here are available on https://github.com/clementino1971/BirdCLEF_2020. We would like to thank CLEF for organizing competitions of this type and encouraging research in this context of bioacoustic.

References

1. Librosa : audio and music processing in python , https://librosa.org/

2. Pydub: Manipulate audio with a simple and easy high level interface , https:// pydub.com/

3. Driedger , J. , Muller, M. , Disch , S. : Extending harmonic-percussive separation of audio signals . In: ISMIR . pp. 611 { 616 ( 2014 )

4. Honka , T. : One-shot learning with siamese networks for environmental audio ( 2019 )

5. Joly , A. , Goeau, H., Kahl , S. , Deneu , B. , Servajean , M. , Cole , E. , Picek , L. , Ruiz De Castan~eda, R., e, Lorieul, T. , Botella , C. , Glotin , H. , Champ , J. , Vellinga , W.P. , Stoter, F.R. , Dorso , A. , Bonnet , P. , Eggel , I. , Muller, H.: Overview of lifeclef 2020: a system-oriented evaluation of automated species identi cation and species distribution prediction . In: Proceedings of CLEF 2020 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2020 , Thessaloniki, Greece. ( 2020 )

6. Kahl , S. , Clapp , M. , Hopping , A. , Goeau, H., Glotin , H. , Planque , R. , Vellinga , W.P. , Joly , A. : Overview of birdclef 2020: Bird sound recognition in complex acoustic environments . In: CLEF task overview 2020 , CLEF: Conference and Labs of the Evaluation Forum , Sep. 2020 , Thessaloniki, Greece. ( 2020 )

7. Kahl , S. , Wilhelm-Stein , T. , Hussein , H. , Klinck , H. , Kowerko , D. , Ritter , M. , Eibl , M. : Large-scale bird sound classi cation using convolutional neural networks . In: CLEF (Working Notes) ( 2017 )

8. Koch , G. , Zemel , R. , Salakhutdinov , R.: Siamese neural networks for one-shot image recognition . In: ICML deep learning workshop . vol. 2 . Lille ( 2015 )

9. Lasseck , M. : Bird species identi cation in soundscapes . CLEF working notes ( 2019 )

10. Lostanlen , V. , Salamon , J. , Farnsworth , A. , Kelling , S. , Bello , J.P. : Robust sound event detection in bioacoustic sensor networks . PloS one 14 ( 10 ), e0214168 ( 2019 )

11. Manocha , P. , Badlani , R. , Kumar , A. , Shah , A. , Elizalde , B. , Raj , B. : Contentbased representations of audio using siamese neural networks . In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . pp. 3136 { 3140 . IEEE ( 2018 )

12. Schro , F. , Kalenichenko , D. , Philbin , J.: Facenet: A uni ed embedding for face recognition and clustering . In: Proceedings of the IEEE conference on computer vision and pattern recognition . pp. 815 { 823 ( 2015 )

13. Tao , R. , Gavves , E. , Smeulders , A.W. : Siamese instance search for tracking . In: Proceedings of the IEEE conference on computer vision and pattern recognition . pp. 1420 { 1429 ( 2016 )

14. Thakur , A. , Thapar , D. , Rajan , P. , Nigam , A. : Deep metric learning for bioacoustic classi cation: Overcoming training data scarcity using dynamic triplet loss . The Journal of the Acoustical Society of America 146 ( 1 ), 534 { 547 ( 2019 )

15. Wang , J. , Fang , Z. , Lang , N. , Yuan , H. , Su , M.Y. , Baldi , P. : A multi-resolution approach for spinal metastasis detection using deep siamese neural networks . Computers in biology and medicine 84 , 137 { 146 ( 2017 )