This working note summarizes our submission to the LifeCLEF 2014 Bird Task which combines the outputs from a Python and Matlab classi cation system. The features used for both systems include Mel-Frequency Cepstral Coe cients (MFCC), time-averaged spectrograms and the provided meta-data. The Python subsystem combines a large ensemble of di erent classi ers with di erent subsets of the features while the Matlab subsystem is an ensemble of the Random Forest and Linear Discriminant Analysis (LDA) classi ers using local spectral and meta features. By combining this disparate set of features and classi ers, we managed to achieve a Mean Average Precision (MAP) score that is far superior to what is possible with any single classi er.
LifeCLEF 2014 Bird Task (BirdCLEF)[
The total size of the audio database for both the MLSP and NIPS4B challenges amounts to around 1GB while the BirdCLEF data is over 20GB in size. The winning methods for the MLSP and NIPS4B challenges performed image segmentation on the audio spectrograms followed by template matching to derive class models. Based on the di erence in data size and the number of classes, it can be extrapolated that using the same methods for BirdCLEF will require at least a hundred fold increase in computation time. Given that the time between the release of the training data and the nal test submission is only around three months, it would not be e cient to try to reproduce the above methods given our limited computing resources.
All of our experiments on BirdCLEF are performed on standard 4-core desktop PCs with 8GB of RAM thus there are severe constraints on the possible machine learning methods that can be applied. Instead of attempting to build a strong learner using well-engineered features, we chose to work on building a number of weak learners with simple features and nding the optimal combination of such learners. 2
We make use of both audio and meta data for our feature front-end as they provide complementary information to the classi er. The total memory used in the computation is heavily dependent on the size of the feature used thus it is important to select a compact representation. 2.1
The MFCC features provided by the organizers consist of a single log energy coefcient and 15 cepstral coe cients, appended with the rst and second derivatives for a total of 48 dimensions per time frame. For each audio clip, we perform a simple energy-based segmentation by dropping all the time frames where the value of the log energy coe cient is below the median value. The remaining frames are further reduced by taking the maximum value for each of the 48 dimensions to arrive at a nal 48 dimension feature. 2.2
The spectral features we used are similar to the spectrum-summarizing features
described in [
In addition to the audio features described above, we also make use of the metadata represented by the following eight elds:
Latitude; Longitude; Elevation; Y ear; M onth; M onth + Day; T ime; Author The M onth + Day eld combines the two-digit M onth and Day elds into a single four-digit eld as comparing the Day elds across di erent months is illogical.
One of the challenges of BirdCLEF compared to MLSP which also provides
meta-data in the form of a numerical location index[
Although the o cial labels for the audio data only gives a single bird species as the ground truth given by ClassId, the BackgroundSpecies eld often list additional bird species. However, like the meta-data, this eld is unreliable as there is no indication if no additional species are present or identi ed, or if the author simply did not complete this particular eld. We can only assume that an empty BackgroundSpecies eld indicates that only the single species given by ClassId is present.
Another problem that arises due to the BackgroundSpecies eld is the presence of bird species that are outside of the 501 species to be classi ed. For the given training data, there are over three hundred additional species and we chose to retain the sixteen extra species that have over eleven examples each for a total of 517 target species. The remaining labels are ignored since there are insu cient examples to train proper models for them. By making use of the extra information from BackgroundSpecies, the number of examples for the 501 original target classes is also increased and the task becomes a multi-label classi cation problem. 3.1
The classi ers used in the Python subsystem are built from the scikit-learn
toolkit (sklearn)[
Two classi ers are used in the Matlab subsystem: Random Forests using the T reeBagger function and LDA using the Classif icationDiscriminant:f it function, both of which were trained in a one-vs-rest con guration. Both of these classi ers output probability estimates through the predict function and the nal output is the mean of the individual classi er outputs. The T reeBagger function was modi ed such that each tree in the forest was trained with a balanced amount of negative data, as this was found to considerably improve the speed of the classi cation without a ecting performance. In particular, each tree was trained with all of the positive class data, and a random subset of the negative data containing twenty times as much as the positive data. The number of variables to sample at each node was set to 3, while the number of trees was set to 200 for the nal evaluation run. 4
To evaluate the performance of a feature and classi er combination, 10-fold cross-validation is used on the given training data. The objective is to maximize the MAP score of the system. 4.1
From the list of feature front-ends and the selected classi ers from sklearn, combinations of feature and classi er pairs are evaluated. This step took up the bulk of our time as it involves the optimizing of feature settings, classi er settings and the combinations of the two to build the nal ensemble classi er. Out of the available classi ers, the following six were selected:
SV C in sklearn 4. AdaBoost (AdaB), AdaBoostClassif ier in sklearn 5. K-Nearest Neighbours (KNN), KN eighborsClassif ier in sklearn 6. Random Forests (RanF), RandomF orestClassif ier in sklearn
1. M eta + M F CC +
SVM 2. M eta + M F CC + Spec10030, used with AdaB and KNN 3. M eta, used with RanF
M F CC refers to the rst 16 dimensions of the described 48 dimension feature while M F CC are the next 16 dimensions (indices 17 to 32) which represents the rst derivatives or deltas. Spec10 refers to the spectral features with n = 10 and Spec10030 are the rst 30 dimensions of the spectral features with n = 100. M eta refers to the eight dimension meta features. 4.2 Only one feature con guration is used with the Matlab subsystem. Using the same terminology as above, the features can be denoted M eta + Spec40, with n = 40 dimensions used for the spectral feature concatenated with the eight dimensions of the meta features. However, instead of taking the maximum over the whole time frame, as with the Python subsystem, local spectral features are generated by taking the maximum within selected local windows. Each window is 0.5 seconds long, and the step is equivalent to the frame increment. The energy of each window is calculated simply as the average log-energy over the window, and is then used as a metric for selecting the windows for classi cation. Only those windows that are represent local maxima in the energy function are retained, with the remaining windows discarded. The idea is that the windows containing local maxima in the energy will correspond to local instances of the bird song in the clip, hence provide a complimentary source of information to the global features used in the Python subsystem. 5
Although we train the classi ers on 517 target species, the results presented only consider the actual 501 target species (MAP with k = 501). 5.1
The results shown in Table 1 give a rough breakdown of the individual components of the Python subsystem. For the sake of brevity, the other combinations of the six selected classi ers are not shown. The Audio and M eta columns show how each classi er performs when the complementary meta or audio features are removed. For all but the KNN classi er, combining both audio and meta 0.220 0.195 0.274 features is shown to improve the classi er performance. Although the KNN classi er is shown to give a better MAP score with the meta feature alone, when combined with the other classi ers in an ensemble, adding the audio features is found to result in a better score overall. It is interesting to note that using the 8-dimension meta features alone with the chosen classi ers can give a reasonable score of 0.200, highlighting the value of using meta-data for classi cation tasks.
Table 2 shows the approximate computation time for the Python subsystem. One of our objectives is to design a system with a short turnaround time as the search space for the system parameters is huge. With each new dataset, it is likely that the optimal system con guration in terms of feature and classi er choices will be di erent thus it will be necessary to retune the entire system from scratch. There is a large discrepancy in the computation time required for 5 min 15 min 1h 10 min 11 min 17 sec 8 min each of the classi ers chosen that is partly attributed to the di erent feature combinations used. The strength of using an ensemble of classi ers is the option to mix and match di erent combinations of the components to suit particular demands for performance and computation cost. The three fastest classi ers (LDA, KNN and RanF) combine to give an ensemble which runs in less than 15 minutes with a respectable score of 0.290. In contrast, removing RanF from the six classi er ensemble only saves 8 minutes but reduces the score from 0.313 to 0.274. There are no general rules determining the performance of each speci c combination, especially if the dataset is changed, thus it is usually necessary to perform an exhaustive search to nd the optimal system.
Another strength of the ensemble classi er is how the individual classi ers complement each other. For most of the classi ers tested, it is possible to improve the individual score at the cost of computation time by changing parameters such as increasing the number of iterations or the number of neighbours to query. However, these improvements do not necessarily transfer to the ensemble classi er when they are combined as the individual gains are made obsolete through the contributions of the other classi ers. Depending on the classi ers used, it is often possible to replicate a computation-intensive performance gain from a classi er. 5.2
The results shown in Table 3 give a breakdown of the individual components of the Matlab subsystem using the combined Audio + M eta features. It can be seen that the LDA classi er is performing relatively poorly, perhaps due the number of false positives in the features. These false positives arise from using local features that are extracted from local maxima in the energy, and hence may represent noise or other information that is not related to the target bird species. On the other hand, the random forest classi er is able to select the features that best separate the target classes, and this allows it to handle a large number of false positives in the labelled data. Combining the two classi ers together increases the score, giving an MAP of 0.222 on the cross-validation.
Due to time constraints, we did not manage to evaluate the performance of the combined Python and Matlab system through cross-validation. Table 4 shows the nal results for our proposed systems on the evaluation data. Compared to Table 1, the Python subsystem shows a 10% relative loss with the inclusion of BackgroundSpecies. This might be due to the use of a lower k value when evaluating the MAP score or a mismatch between the training and evaluation data. The combined system is found to improve on the Python subsystem score slightly despite the poor performance of the Matlab subsystem.
Our system was trained as a multi-label classi er and this is re ected by the lower score when the background species are removed (MAP 2). Interestingly, we are the only group that has a lower MAP 2 score than MAP 1, indicating that we are the only group that focussed only identifying the background species as well as the dominant species. 6
The challenges presented by BirdCLEF include the large amount of provided data, the large number of target species to be classi ed, incomplete or inaccurate multi-class labels and missing and unreliable meta-data. The limited time between the release of the training data and the nal submission and the limited computing resources available to our group are additional obstacles we faced. With these limitations in mind, we designed an ensemble classi er that combines the output from a number of individually weak learners using simple features that are fast to train and test. The nal system is shown to provide greatly improved results over the component classi ers and the fast turnaround time allows the setup to be quickly calibrated for new tasks and datasets.