Large-scale biodiversity monitoring would profit from automated solutions supporting or complementing human experts and citizen scientists. To foster their development, the yearly BirdCLEF scientific challenge compares approaches for identifying bird species in recorded vocalizations. The solution described in this work is based on an ensemble of Convolutional Neural Networks (CNNs) processing a mel spectrogram combined with Multi-Layer Perceptrons (MLPs) processing the recording date, time and geographic location. In BirdCLEF 2018, it achieved a mean average precision of 0.705 in detecting 1,500 South American bird species (0.785 for the foreground species), the second best entry to the challenge.
Biodiversity monitoring is important to assess the impact of human actions on other species, and inform decisions on short-term projects and long-term policies. However, manually observing, classifying and counting animals is limited in scale and scope by the supply of experts, the costs of human labor and basic human needs. Automated or semi-automated approaches for recording and analyzing observations would allow monitoring more locations, over longer time spans and in forbidding terrain.
For many animals, acoustic monitoring is an interesting option – it allows to detect and classify individuals even when they are difficult to see, which is precisely the reason for some of their vocalizations. Acoustic recording devices paired with automatic classifiers could support both experts in the field in identifying species, as well as enable passive acoustic monitoring.
The CLEF (Conference and Labs of the Evaluation Forum) initiative fosters
the development of automatic classifiers for the case of bird vocalizations through
the yearly BirdCLEF challenge [
In the following, I describe my submission to BirdCLEF 2018, which reached the second place among six participating teams. Section 2 details the layout and training of a Convolutional Neural Network (CNN) processing the audio signals, Section 3 describes a Multilayer Perceptron (MLP) predicting the species from the metadata of each recording, and Section 4 explains how the predictions of multiple audio and metadata networks are combined into a final result. Section 5 compares results on an internal validation set and the official test sets. Finally, Section 6 discusses ideas that did not turn out successful, and Section 7 concludes with a summary and an outlook. 2
The foundation of my submission is an ensemble of Convolutional Neural Networks (CNNs) that process a mel spectrogram to produce occurrence probabilities for each of the 1500 possible bird species. In the following, I describe how the audio signal is preprocessed to be consumed by the network, what network architectures were chosen, how the networks were trained, and how they are applied at test time. Both for the architecture and for training I used multiple variants which were later combined in an ensemble (Section 4). 2.1
Adopted from [
The network architecture is a CNN based on my submission to a bird detection
challenge (submission sparrow in [
Figure 1 gives an overview of the architecture variants. The overarching idea is to have a fully-convolutional network – i.e., a network with only convolutional and pooling layers, no fixed-size fully-connected layers – process an input spectrogram into a sequence of local predictions for each bird species, followed by d n e t n o r
F e m i pitch t e m i pitch t s n la ito co ic L ed r p e m i logits t laob ilgn l o G op a global pooling strategy that produces a single prediction per species for the full recording. This type of architecture can be directly trained and tested on recordings of arbitrary length and does not require a hand-chosen averaging or majority voting procedure to produce recording-wise predictions at test time.
In the following, I will describe a common preprocessing frontend as well as the local prediction and global prediction stages in detail.
Frontend: The first step is a learned nonlinear magnitude transformation: y = xσ(a), where σ(a) = 1/ (1 + exp(−a)) (1) It is applied elementwise to each time-frequency bin x, enhancing low magnitudes similar to the more common application of a logarithmic function. a is initialized to zero (so y = √x) and learned as part of the network, shared over all mel bands, typically reaching values between -1.2 and -1.7 (yielding y = x0.23 to y = x0.15).
This is followed by batch normalization [
A) The first variant is based on submission sparrow in [
All layers except the last one are followed by batch normalization (the usual
variant with statistics shared over all spatial locations, but separated per
channel, and with learned scale and shift) and leaky rectification as max(x/100, x).
The last three layers are preceded by 50% dropout [
B) As a variation on this architecture, I replaced the first four convolutional layers with two residual blocks with pre-activation following [10, Fig. 1b]. Each residual block consists of four 3 3 convolutional layers, each of which is preceded × by batch normalization and linear rectification as max(x, 0) (except for the initial convolution following the preprocessing frontend, where pre-activation would be redundant). Convolution inputs are padded with a leading and trailing row and column of zeros so the output size matches the input size. Skip connections add the input of each pair of two convolutions to its output. If needed, the skip connection includes a 1 × 1 convolutional layer to adjust the number of channels. The first residual block has 64 channels throughout and is followed by 3×3 max-pooling, the second residual block has 128 channels and is followed by batch normalization, linear rectification and an unpadded convolutional layer of 128 units leaving 5 frequency bands (compared to the first CNN variant, this requires 3×22 units instead of 3×17 units since the convolutions are padded). 3×5 max-pooling and remaining layers are adopted from the first CNN variant. Global predictions: The previous stage produces a sequence of vectors of size 1500, corresponding to the 1500 possible bird species. Specifically, since the network includes two pooling operations over 3 frames each, it produces a local prediction every 9 frames,1 for the part of the input spectrogram exceeding the length of the network’s receptive field (103 frames for the first, 119 for the second variant) minus padding (none for the first, 32 frames for the second variant).
A) Treating this as a multiple-instance learning (MIL) problem [
X exp(a · yt)
T t=0
!
(2)
Here, y denotes the time series of local predictions (y0, y1, . . . , yT −1) for a single
species, and a is a hyperparameter controlling the behaviour of the function: for
a → ∞, it approximates the maximum, for a → 0, it approximates the mean.
I set a = 1 (a = 0.2 or a = 5 work just as well, I did not explore it further).
1 It would be possible to obtain a prediction for every single frame by introducing
dilated convolutions and pooling as described in [
B) As an alternative to a fixed pooling strategy that prefers larger inputs, I explored pooling with temporal attention. I let the network produce a weight αt for each time step, and compute a weighted sum:
T −1 att(y, α) = X αtyt = αT y
t=0 The αt are shared over all species, and α is produced with multi-head attention: I extend the last convolutional layer of the local prediction stage by K units, such that it produces 1500 + K time series. I split off the K additional time series C and apply a softmax over time to each one: Finally, I average the K soft-maxed time series B to obtain α:
T −1 Bk,t = exp(Ck,t)/ X exp(Ck,i)
i=0 αt = By construction, α sums to 1.0 regardless of the sequence length. Using multiple heads (I chose K = 16 or K = 64) makes it easier for the network to produce attention series with multiple local maxima. Note that there is no additional supervision needed to learn what to attend to; the global loss will be backpropagated through (3), (5), (4) to reach the last layer of the local prediction stage.
For both log-mean-exp pooling and attentional pooling, we now end up with a single vector of 1500 values. Assuming that we want to predict the occurrence of each species independently, the natural choice would be to apply the logistic sigmoid function σ(x) = 1/(1 + exp(−x)) to squash each value into the range (0, 1) and treat it as a binary prediction. However, since the training examples strongly focus on capturing a single prominent species per recording, it turns out to work slightly better to treat this as a categorical classification problem, so I apply the softmax function to produce a distribution over 1500 classes instead.
Note that when planning to train with categorical cross-entropy loss, it is
crucial to apply the softmax after the global temporal pooling operation, otherwise
the competition between classes is introduced at the wrong level. Considering
the case of global temporal max pooling, having the softmax last ensures the
loss gradient reaches the frame of maximal activation for each species. Having
the softmax first (i.e., on each local prediction) would propagate the gradient to
the frame where the correct species is most dominant over the others, and then
push down competing species for that frame only. Similar considerations apply
to log-mean-exp pooling and attentional pooling.
To deal with the fact that training recordings are usually a few minutes long,
but only sparsely populated with vocalizations of the target bird (without any
(3)
(4)
(5)
information on where in a recording the target bird is audible), several authors
employed hand-designed methods to find potential bird calls and trained on
these passages only [
Loss and targets: Optimization minimizes the cross-entropy loss between predictions y and targets t: Obviously, the network can never reach this target. However, as categorical crossentropy (Equation 6) is linear in the targets, this is equivalent to adding the cross-entropy losses against each background species and the foreground species, weighting the latter twice. Furthermore, since ADAM is invariant to the scale of the gradient, it is unimportant if we set the foreground target to 2 and background to 1, or the foreground to 1 and background to 0.5, for example. I employ three different variants for defining the target vector t for a recording.
A) The most obvious choice for a categorical classification task is to set ts to 1 for the annotated foreground species for the recording, and to 0 otherwise. Denoting the foreground species as Sf , we have:
B) To factor in the annotated background species, I set ts to 2 for the foreground species, to 1 for any of the background species, and to 0 otherwise. Denoting the set of background species as Sb, we have: ce(y, t) = −
S−1 X ts log(ys) s=0 ts = (1 if s = Sf
0 otherwise ts = 2 if s = Sf
1 if s ∈ Sb 0 otherwise (6) (7) (8)
C) As a third variant, I train a network as a Born-Again Network (BAN),
following [
Mel spectrograms are not precomputed before training to enable easy experimentation with different mel spectrogram settings (and learned mel filterbanks). STFTs are not precomputed because that would increase I/O costs (for the settings in Section 2.1, a minute-long magnitude spectrogram in 32-bit floatingpoint precision takes 8.2 MiB, while a same-length 22.05 kHz signal in 16-bit integer precision takes 2.5 MiB), cancelling out savings in computation. 2.4
Since the network can process input of arbitrary size into a vector of 1500 class probabilities, we can directly apply it to full test recordings for the first subtask of the challenge (monodirectional recordings), and to 5-second excerpts for the second subtask (long-term multidirectional recordings at 5-second intervals).
Not all 1500 bird species of the challenge are likely to occur across the whole South American continent. Similarly, some migrating birds are not likely to occur in specific seasons, and some species are more likely to be heard during dawn than at noon. To learn about and use these dependencies, I train additional networks on the metadata supplied with the training recordings, which is also available for the test recordings: the date, time, geocoordinates, and elevation. 3.1
As a first step, I cleaned up and unified the different formats for the time (e.g., “14:20”, “11am”, “morning”, “PM”) and elevation (e.g., “1400”, “300-400”, “1.270m”, “to 1100 m”) data, both of which seem to be drawn from freeform text fields. Date and geocoordinates were already unified, but not all fields are filled for each recording.
To process the metadata with a neural network, I encode it as a fixed-size vector. The numeric values for longitude, latitude and elevation are used directly. The date is converted to the day of the year, and the time to the minute of the day. To avoid a discontinuity from December 31, to January, 1 and from 23:59 to 0:00, I employ a K-dimensional circular encoding of these values using phaseshifted sinusoid functions: dˆk = √2 sin(2πd/D + πk/K), (10) where d is the value to encode (e.g., the minute of the day), D is the number of possible values (e.g., 1440), and dˆ is the resulting K-dimensional encoding of d. I experimented with two settings: K = 12 and K = 3. Missing metadata fields are represented as zeros, and 5 additional binary values indicate which fields are valid. In total, this results in a 14-dimensional vector for K = 3.
As a final step, all values are standardized to zero mean and unit variance using statistics over the training set (except for the circular-encoded values, which are approximately standardized due to the √2 factor in Equation 10, assuming a uniform distribution of days of the year and minutes of the day). 3.2
The network architecture for this part is very simple: The 14-dimensional input vector is processed by two fully-connected hidden layers of 256 and 512 leaky rectified units each, and finally classified by a fully-connected output layer of 1500 softmax units. Dropout or batch normalization were not helpful. 3.3
Optimization uses the same settings as for the audio network, except that monitoring for early stopping uses the mean average precision (MAP, see Section 5) on the validation set rather than the cross-entropy loss, and mini-epochs of 5000 instead of 1000 updates.
Training targets are chosen according to strategy A or B of Section 2.3, with strategy B turning out to be more useful for ensembling.
To obtain additional variations for ensembling, I varied training in two ways: A) I added Gaussian noise to the metadata values (before circular encoding, if applicable), either with small standard deviations (3 days, 3 minutes, 5 meters, 0.3 degrees), medium standard deviations (7 days, 10 minutes, 20 meters, 1 degree) or large standard deviations (14 days, 30 minutes, 100 meters, 3 degrees). B) I trained networks on a subset of metadata fields, either leaving out one of the fields or all but one field.
As an additional regularization, I experimented with randomly dropping single fields (handling them like missing fields, i.e., replacing with zeros and flipping the validity indicator variable), but this only worsened results. 4
Both to combine audio-based and metadata-based predictions and to benefit from multiple variations among the audio-based and metadata-based models, it is crucial to be able to merge predictions of multiple models.
In its simplest form, predictions by multiple models for the same recording can be merged by averaging them. There are different stages at which this could happen; empirically, it worked best to average predictions after global temporal pooling, but before applying the softmax output nonlinarity.
Averaging is sufficient for few similar models, but not optimal for larger sets of
diverse models, or for combining audio-based and metadata-based predictions:
in these cases, it pays off to weight the predictions of each model differently.
To automate the choice of weights, I employ hyperopt [
Challenge submissions are evaluated in terms of mean average precision (MAP), both when using only the annotated foreground species as ground truth and when including the background species as additional correct classes. Given class probabilities y and a set of annotated species S for a recording, the average precision is computed as ap(y, S) = 1
X p(y, S, rank(y, s)), |S| s∈S (11) (12) (13) where rank(y, s) is the number of values in y less than or equal to ys: rank(y, s) =
X [yi ≤ ys] , i and p(y, S, k) is the precision at k: p(y, S, k) = 1 X [rank(y, s) ≤ k] .
k
s∈S The mean average precision is the mean of the average precisions of all recordings. Note that when |S| = 1 (e.g., when only evaluating against foreground species), the average precision is equal to the reciprocal rank of the correct species, as easy to see when inserting (13) into (11).
As an additional evaluation measure, I use the classification accuracy with respect to the foreground species, and the top-5 classification accuracy (which treats a prediction as correctly classified when the foreground species is among the 5 largest class probabilities for a recording).
Table 1 lists validation results for all candidate model variants partaking in the ensemble search. In addition, it shows the ensembling weights, validation and official test results for three ensembles formed from the candidate models. We can draw several interesting insights from these results: – All else kept equal, the residual network outperforms the plain CNN (variants A and B in Section 2.2, “Local predictions”). It still pays off to combine them in an ensemble. – All else kept equal, log-mean-exp pooling tends to outperform temporal attention (variants A and B in Section 2.2, “Global predictions”). Again, it still pays off to combine them in an ensemble. – All else kept equal, including background species in the targets tends to decrease the foreground MAP against foreground species (and the classification accuracy), while increasing the MAP against background species (variants A and B in Section 2.3, “Loss and targets”). – All else kept equal, retraining a model as a Born-Again Network (BAN) tends to improve its performance (variant C in Section 2.3, “Loss and targets”). – The metadata alone suffices to classify 21% of validation recordings correctly. – Including all metadata fields gives better performance than omitting some. – When training on a single metadata field, it seems location is the most informative cue. – There is no systematic difference between 12-dimensional and 3-dimensional date/time encodings (but there is only a single pair of results with all else kept equal). – Adding Gaussian noise to metadata diminishes results. This might indicate that the models learn to exploit unwanted correlations between training and validation data, such as recordings done on the same day at the same location, rather than learning robust occurrence patterns. However, when ensembling with audio models, blurred (noisy) metadata is superior to unmodified data.
For the multidirectional task, I submitted predictions using Ensembles A, B and C applied to 5-second excerpts (Run 3 was not included since there is no information to use for filtering). They achieved c-MAP2 scores of 0.113, 0.146 and 0.097, respectively.
Note that for both tasks, ensemble A outperformed ensemble C, while the latter was superior on the internal validation set. This might indicate that the validation set results are not a reliable indicator for choosing models after all.3 6
During experimentation, I tried several ideas that did not turn out to be helpful
in the end, and were not discussed above. I will describe a few in the following.
mixup: Several authors augmented training examples by mixing them with
noise, based on a hand-designed approach for segmenting recordings into
passages containing birds and passages only containing background noise
[
weighted combinations of training examples paired with weighted combinations of their target vectors. While this seems to be especially well-suited to audio recordings, in my experiments, it slowed down convergence and lead to worse results.
Data augmentation: While data augmentation is often reported to be helpful
for audio classification (using transformations such as time stretching, pitch
shifting or equalization), in my experiments, it only diminished results.
Grouped convolution/dropout: Several authors proposed to design a
network architecture to have multiple separate computation paths processing
the same input that are merged in the end [
An easy way to implement such an architecture is to use grouped convolutions – a variation of convolutional layers that divides the input channels into non-overlapping groups to be processed separately. Multiple such layers with the same number of groups result in separated deep computation paths. Even with only two groups, this can promote the development of different feature sets (e.g., edge and color features in AlexNet [16, Fig. 3]). I experimented with different numbers of groups, keeping either the total number of feature maps or total number of parameters constant, but a single group always performed best. I also tried combining this with grouped dropout, i.e., randomly dropping complete groups at training time, to promote independence between computation paths, but to no avail.
Per-Channel Energy Normalization (PCEN): As an alternative to the
basic learned magnitude transformation in Equation 1, I tried PCEN as
proposed in [
Combining an ensemble of audio networks that individually achieve a Mean Average Precision (MAP) score of up to 0.701 with an ensemble of metadata networks that achieve up to 0.277, it is possible to reach a MAP score of 0.809 (for detecting the foreground species in an internal validation set split off from the BirdCLEF 2018 training data). Compared to an audio-only ensemble, including the metadata improved scores by 3 to 5 percent points and helped securing the second place in the official challenge results.
Results could be improved in a number of ways. For one, I hardly varied
the spectrogram settings – especially the sampling rate of 22.05 kHz (and upper
frequency bound of 10 kHz for the mel filterbank) may be a suboptimal choice.
While this was enough to win a bird detection challenge [
When judging the relevance of results for practical applications, it is important to distinguish the two subtasks. The MAP score used for the first subtask is purely rank-based (Equation 11): it rewards models giving the correct species higher scores than incorrect ones, but does not require a clear decision on which species occur in a recording. Furthermore, as the test set only contains files that include at least one of the possible 1500 bird species, models will never need to decide whether a file contains a bird (or a known bird) at all. Finally, the score takes an average over all test recordings, so species with more recordings will be given higher importance than rare species. Thus, the first subtask only captures the setting of an expert in the field interested in classification suggestions for a particular vocalization, not a passive acoustic monitoring scenario. The c-MAP score for the second subtask weights all species equally, and rewards models giving segments that contain a particular species higher scores than those that do not contain that species. This is closer to the requirements for passive monitoring, and still has a lot of room for improvements.
I would like to thank Hervé Glotin, Hervé Goëau, Willem-Pier Vellinga, and Alexis Joly for organizing this challenge, supported by Xeno-Canto, Floris’Tic, SABIOD and EADM MaDICS. This research is supported by the Vienna Science and Technology Fund (WWTF) under grants NXT17-004 and MA14-018. I also gratefully acknowledge the support of NVIDIA Corporation with the donation of two Tesla K40 GPUs and a Titan Xp GPU used for this research.