We predict drops in electronic dance music (EDM), employing different multimodal approaches. We combine three sources of data: noisy labels collected through crowdsourcing, timed comments from SoundCloud and audio content analysis. We predict the correct labels from the noisy labels using the majority vote and Dawid-Skene methods. We also employ timed comments from SoundCloud users to count the occurrence of specific terms near the potential drop event, and, finally, we conduct an acoustic analysis of the audio excerpts. The best results are obtained, when both annotations, metadata and audio, are combined, though the differences between them are not significant.
This working notes paper describes a submission to the
CrowdSorting brave new task in the MultiMediaeval 2014
benchmark. The main aim of the task is to detect drops
in electronic music. According to the Wikipedia definition:
“Drop or climax is the point in a music track where a switch
of rhythm or bassline occurs and usually follows a
recognizable build section and break”[
We submitted four runs: three are based on annotations and other metadata, and one is based on a combination of metadata and acoustic features. Due to the social attention that drop phenomenon gets in electronic music, the task of drop detection is naturally suitable for a combined approach, using both metadata and acoustic features. The acousticonly approach is rather challenging, because there are many informal descriptions of what constitutes a drop, including rhythmic and dynamical changes, or specific patterns in the ∗ First two authors contributed equally to this work and appear in alphabetical order. bass line. Also, the presence or absence of drop in a specific case is debatable. 2.
Karthik Yadati et al. [
For each of the excerpts, three annotations from MTurk workers were provided. Fleiss’ kappa for these labels was 0.24 (calculated without songs from the fourth category, ”absent sound file”). Around 30% of the excerpts were unanimously rated by annotators. For about 60%, two of the annotators agreed. For the remaining 10% of the excerpts, all the annotators provided different answers. We mainly sought to improve the categorization of the second and especially the last categories. 3.1
The first run employs a simple majority vote. In case all the annotators categorize the segment differently, we label it as containing part of the drop.
In the second run, we use the Dawid-Skene algorithm [
As a training data, we employed the excerpts for which all the three workers agreed. There were 164 such excerpts in total, 105 for which workers indicated that the excerpt contained an entire drop, 54 for which they indicated there was no drop, and 4 for which they agreed there was part of the drop present. We decided to exclude the excerpts labeled ”part of the drop”, as it is not possible to learn to recognize it based on just four samples.
The acoustic approach was based on the fact that during a drop, there is usually a moment of silence, or sometimes the loudness level changes drastically after the drop. We analyzed the energy of the signal in non-overlapping windows of 100 ms. The obtained time-series was smoothed using the weighted moving average. The smoothed time-series was segmented on their local maximums and minimums. To predict the presence of the drop event, we used the following statistics on these events: 1. The value of the biggest local minimum in an excerpt 2. The fraction of the biggest minimum to an average minimum 3. The number of potential drop events, as detected by decrease in loudness bigger than threshold
Based on these characteristics and a ground-truth of 160 excerpts, we trained a logistic regression classifier to predict the presence of drops, and obtained 80% precision with 10fold cross validation. The model was used to predict the presence of drops for the excerpts where all three workers gave different ratings (i.e., ”drop is present”, ”part of the drop is present”, ”drop is not present”). The biggest limitation of this approach is that the model does not incorporate the ”part of the drop” category.
The evaluation metric for this task is the F1 score, calculated based on high-fidelity labels from the experts, used as a ground-truth. Though there are some differences between submissions, none of them were statistically significant on a one-sided Wilcoxon ranksum test. The majority vote scores are as usual hard to beat. Using comments from SoundCloud users results in some improvement, and using acoustic features performs similarly. Looking at the accuracy per category, we can see that the acoustic submission suffers from imprecision in the category ”part of the drop”, which is natural, because it does not model that. On the other hand, the precision of ”no drop” labels is higher than for all other submissions.
drop local maximum local minimum smoothed time series unsmoothed time series 20
30 40 Time window (index) 50 60 70 80
0.72 0.72 0.73 0.72 0.31 0.31 0.28 0.27 5.
In this task, we only achieved marginal improvement over the baseline, i.e., majority vote. Both acoustic analysis and the use of SoundCloud metadata resulted in a small but insignificant prediction improvement. This shows that in the presence of enough labels given by MTurk workers, we could not significantly improve the accuracy based on the content or social media metadata. However, they are nevertheless useful in cold start scenarios.
This publication was supported by the Dutch national program COMMIT.