This paper provides an overview of the AcousticBrainz Genre Task organized as part of the MediaEval 2018 Benchmarking Initiative for Multimedia Evaluation. The task is focused on content-based music genre recognition using genre annotations from multiple sources and large-scale music features data available in the AcousticBrainz database. The goal of our task is to explore how the same music pieces can be annotated diferently by diferent communities following diferent genre taxonomies, and how this should be addressed by content-based genre recognition systems. We present the task challenges, the employed ground-truth information and datasets, and the evaluation methodology.
Content-based music genre recognition is a popular task in
Music Information Retrieval (MIR) research [
Since MediaEval 2017 [
Participants must train model(s) using the provided development sets and then predict genre and subgenre labels for the test sets. The task includes two subtasks: • Subtask 1: Single-source Classification . This subtask explores conventional systems, each one trained on a single dataset. Participants submit predictions for the test set of each dataset separately, using their respective class spaces (genres and subgenres). These predictions will be produced by a separate system for each dataset, trained without any information from the other sources.
This subtask will serve as a baseline for Subtask 2. • Subtask 2: Multi-source Classification . This subtask explores the combination of several ground-truth sources to create a single classification system. We use the same four test sets. Participants submit predictions for each test set separately, again following each corresponding genre class space. These predictions may be produced by a single system for all datasets or by one system for each dataset. Participants are free to make their own decision about how to combine the training data from all sources. 3 3.1
We provide four datasets containing genre and subgenre
annotations extracted from four diferent online metadata sources:
• AllMusic [
We provide information about genre/subgenre tree hierarchies for every ground truth.1 3.2
We provide music features precomputed from audio for every music
recording. All features are taken from the AcousticBrainz database
and were extracted from audio using Essentia, an open-source
library for music audio analysis [
We provide four developement datasets, four validation datasets and four test datasets which will be used in the final evaluation of all submissions. The test datasets do not include any groundtruth and have been anonymized. The datasets were created by a random split of the full data ensuring that: • no recordings appear in more than one of the above sets; • no recordings in any set are from the same release groups (e.g., albums, singles, EPs) present in other sets; • the same genre and subgenre labels are present in all sets for each ground truth; • genre and subgenre labels are represented by at least 40 and 20 recordings from 6 and 3 release groups in development and validation/test sets, respectively. 1 The resulting genre metadata is licensed under CC BY-NC-SA4.0 license, except for data extracted from the AllMusic database, which is released for non-commercial scientific research purposes only. Any publication of results based on the data extracts of the AllMusic database must cite AllMusic as the source of the data. 2http://essentia.upf.edu/documentation/streaming_extractor_music.html The approximate split ratios of the datasets are 70% for training, 15% for validation, and 15% for testing. The validation dataset was previously used as the test set in the 2017 edition of the task and is now available for participants for validation as a reference for benchmarking across all current and future editions of the task3.
Table 1 provides an overview of the resulting development sets. Details on the genre/subgenre taxonomy and their distribution in the development sets in terms of number of recordings and release groups are reported online.4 Recordings are partially intersected (annotated by all four ground truths) in the development and test sets. The full intersection of all development sets contains 247,716 recording, while the intersection of the two largest sets, AllMusic and Discogs, contains 831,744 recordings.
All data are published in JSON and TSV formats; details about
format are available online.5 Each recording in the development
sets is identified by a MusicBrainz ID (MBID) 6, which can be used
by participants to gather related data. Importantly, our split allows
to avoid the “album efect” [
Participants are expected to submit predictions for both subtasks. We allow a maximum of five evaluation runs, each including both subtasks, and reporting whether they used the whole development dataset or only parts for every submission.
The evaluation is carried out for each dataset separately. We do not use hierarchical measures because the hierarchies in the Lastfm and Tagtraum datasets are not explicit. Instead, we compute precision, recall and F-score at diferent levels: • Per recording: all labels, only genre labels, only subgenre labels. • Per label, all recordings. • Per genre label, all recordings. • Per subgenre label, all recordings.
The ground truth does not necessarily contain subgenre annotations for some recordings, so we only considered recordings containing subgenres for the evaluation at the subgenre level. An example can be found online in the summaries of random baselines.7 We also provided evaluation scripts for development purposes. 5
Bringing the AcousticBrainz Genre Task to MediaEval we hope to benefit from contributions and expertise of a broader machine learning and multimedia retrieval community. We refer to the MediaEval 2018 proceedings for further details on the methods and results of teams participating in the task. 3https://multimediaeval.github.io/2017-AcousticBrainz-Genre-Task/results/ 4https://multimediaeval.github.io/2018-AcousticBrainz-Genre-Task/data_stats/ 5https://multimediaeval.github.io/2018-AcousticBrainz-Genre-Task/data/ 6https://musicbrainz.org/doc/MusicBrainz_Identifier 7https://multimediaeval.github.io/2018-AcousticBrainz-Genre-Task/baseline/
We thank all contributors to AcousticBrainz. This research has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements No 688382 (AudioCommons) and 770376-2 (TROMPA), as well as the Ministry of Economy and Competitiveness of the Spanish Government (Reference: TIN2015-69935-P). We also thank tagtraum industries for providing the Tagtraum genre annotations.