This paper provides an overview of the AcousticBrainz Genre Task organized as part of the MediaEval 2017 Benchmarking Initiative for Multimedia Evaluation. The task is focused on content-based music genre recognition using genre annotations from multiple sources and large-scale music features data available in the AcousticBrainz database. The goal of our task is to explore how the same music pieces can be annotated diferently by diferent communities following diferent genre taxonomies, and how this should be addressed by content-based genre recognition systems. We present the task challenges, the employed ground-truth information and datasets, and the evaluation methodology.
Content-based music genre recognition is a popular task in Music
Information Retrieval research [
We propose a new genre recognition task using datasets based
on AcousticBrainz for MediaEval 2017. This task is diferent from a
typical genre recognition task in the following ways:
• It allows us to explore how the same music can be annotated
diferently by diferent communities who follow diferent genre
taxonomies, and how this can be addressed when developing
and evaluating genre recognition systems.
• Genre recognition is often treated as a single category
classification problem. Our data is intrinsically multi-label, so we propose
to treat genre recognition as a multi-label classification problem.
• Previous research typically used a small number of broad genre
categories. In contrast, we consider more specific genres and
subgenres. Our data contains hundreds of subgenres.
2
The task invites participants to predict the genre and subgenre of
unknown music recordings given automatically computed music
features of those recordings. We provide four datasets of such music
features taken from the AcousticBrainz database [
Participants must train model(s) using the provided development sets and then predict genre and subgenre labels for the test sets. The task includes two subtasks: • Subtask 1: Single-source Classification . This subtask explores conventional systems, each one trained on a single dataset. Participants submit predictions for the test set of each dataset separately, using their respective class spaces (genres and subgenres). These predictions will be produced by a separate system for each dataset, trained without any information from the other sources.
This subtask will serve as a baseline for Subtask 2. • Subtask 2: Multi-source Classification . This subtask explores the combination of several ground-truth sources to create a single classification system. We use the same four test sets. Participants submit predictions for each test set separately, again following each corresponding genre class space. These predictions may be produced by a single system for all datasets or by one system for each dataset. Participants are free to make their own decision about how to combine the training data from all sources. 3 3.1
We provide four datasets containing genre and subgenre
annotations extracted from four diferent online metadata sources:
• AllMusic [
We provide information about genre/subgenre tree hierarchies for every ground truth.1 3.2
We provide music features precomputed from audio for every music
recording. All features are taken from the AcousticBrainz database
and were extracted from audio using Essentia, an open-source
library for music audio analysis [
In total we provide four development (training+validation) and four test datasets associated with the four genre ground truths. They were created by a random split of the full data ensuring that: • no recordings in the test sets are present in any of the development sets; • no recordings in the test sets are from the same release groups (e.g., albums, singles, EPs) present in the development sets; • the same genre and subgenre labels are present in both development and test sets for each ground truth; • genre and subgenre labels are represented by at least 40 and 20 recordings from 6 and 3 release groups in development and test sets, respectively. 1 The resulting genre metadata is licensed under CC BY-NC-SA4.0 license, except for data extracted from the AllMusic database, which is released for non-commercial scientific research purposes only. Any publication of results based on the data extracts of the AllMusic database must cite AllMusic as the source of the data. 2http://essentia.upf.edu/documentation/streaming_extractor_music.html The approximate split ratios are 70% for training and 15% for testing (another 15% was reserved for future evaluation purposes). Table 1 provides an overview of the resulting development sets. Details on the genre/subgenre taxonomy and their distribution in the development sets in terms of number of recordings and release groups are reported online.3 Recordings are partially intersected (annotated by all four ground truths) in the development and test sets. The full intersection of all development sets contains 247,716 recording, while the intersection of the two largest sets, AllMusic and Discogs, contains 831,744 recordings.
All data are published in JSON and TSV formats; details about
format are available online.4 Each recording in the development
sets is identified by a MusicBrainz ID (MBID) 5, which can be used
by participants to gather related data. Importantly, our split allows
to avoid the “album efect” [
Participants are expected to submit predictions for both subtasks. We allow a maximum of five evaluation runs, each including both subtasks, and reporting whether they used the whole development dataset or only parts for every submission.
The evaluation is carried out for each dataset separately. We do not use hierarchical measures because the hierarchies in the Lastfm and Tagtraum datasets are not explicit. Instead, we compute precision, recall and F-score at diferent levels: • Per recording, all labels. • Per recording, only genre labels. • Per recording, only subgenre labels. • Per label, all recordings. • Per genre label, all recordings. • Per subgenre label, all recordings.
The ground truth does not necessarily contain subgenre annotations for some recordings, so we only considered recordings containing subgenres for the evaluation at the subgenre level. An example can be found online in the summaries of random baselines.6 We also provided evaluation scripts for development purposes. 5
Bringing the AcousticBrainz Genre Task to MediaEval we hope to benefit from contributions and expertise of a broader machine learning and multimedia retrieval community. We refer to the MediaEval 2017 proceedings for further details on the methods and results of teams participating in the task. 3https://multimediaeval.github.io/2017-AcousticBrainz-Genre-Task/data_stats/ 4https://multimediaeval.github.io/2017-AcousticBrainz-Genre-Task/data/ 5https://musicbrainz.org/doc/MusicBrainz_Identifier 6https://multimediaeval.github.io/2017-AcousticBrainz-Genre-Task/baseline/
We thank all contributors to AcousticBrainz. This research has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688382 (AudioCommons). We also thank tagtraum industries for providing the Tagtraum genre annotations.