This paper provides an overview of the Emotions and Themes in Music task organized as part of the MediaEval 2020 Benchmarking Initiative for Multimedia Evaluation. The goal of this task is to automatically recognize the emotions and themes conveyed in a music recording by means of audio analysis. We provide a large dataset of audio and labels that the participants can use to train and evaluate their systems. We also ofer a baseline solution that utilizes VGG-ish architecture. This overview paper presents the task challenges, the employed ground-truth information and dataset, and the evaluation methodology.
Emotion and theme recognition is a popular task in music information retrieval relevant to music search and recommendation systems. We invite participants to try their skills at recognizing moods and themes conveyed by the audio tracks.
Before we reintroduced the task in 20191, the last emotion
recognition task in MediaEval [
While there is a diference between emotions and moods, we use the mood annotations as a proxy to understand the emotions conveyed by the music. Themes are more ambiguous, but they usually describe the concept or meaning that the artist is trying to convey with the music, or set the appropriate context for listening to the music.
Our target audience is researchers in music information retrieval, music psychology, machine learning, and music and technology enthusiasts in general.
This task involves predicting moods and themes conveyed by a music track, given an audio signal. Moods are often feelings conveyed by the music (e.g., happy, sad, dark, melancholy) and themes are associations with events or contexts where the music is suited to be played (e.g., epic, melodic, christmas, love, film, space). We do not make a distinction between moods and themes for the purpose of this task. Each track is tagged with at least one tag that serves as a ground-truth. 1https://multimediaeval.github.io/2019-Emotion-and-Theme-Recognition-in-Music-Task/ 2https://jamendo.com
Participants are expected to train a model that takes raw audio as
input and outputs the predicted tags. To solve the task, participants
can use any audio input representation they desire, be it traditional
handcrafted audio features, spectrograms, or raw audio inputs for
deep learning approaches. We also provide a handcrafted feature set
extracted by the Essentia [
We provide a dataset that is split into training, validation, and testing subsets with mood and theme labels properly balanced between subsets. The generated outputs for the test dataset will be evaluated according to typical performance metrics. 3
The dataset used for this task is the autotagging-moodtheme subset
of the MTG-Jamendo Dataset [
This subset includes 18,486 audio tracks with mood and theme annotations. There are 56 distinct tags in the dataset. All tracks have at least one tag, but many have more than one. All 56 tags are shown in the Figure 1.
As part of the pre-processing of the dataset, some tags were
merged to consolidate variant spellings and tags with the same
meaning, (e.g., “dreamy” to “dream”, “emotion” to “emotional”). The
exact mapping is available in the dataset repository.3 Also, tracks
shorter than 30 seconds were removed and tags used by less than
50 unique artists were discarded. Some tags were discarded while
generating training, validation, and testing splits to ensure the
absence of an artist and album efect [
We provide audio files in 320kbps MP3 format (152 GB) as well as compressed .npy files with pre-computed mel-spectrograms (68 GB). Scripts and instructions to download the data are located in the dataset repository. 3.1
The MTG-Jamendo dataset ofers multiple random data splits for training, validation, and testing (60-20-20%). For this challenge we 3https://github.com/MTG/mtg-jamendo-dataset
Participants should generate predictions for the test split and submit them to the task organizers together with computed evaluation metrics. We provide evaluation scripts in the GitHub repository.4
To have a better understanding of the behavior of the proposed systems, we ask participants to submit both prediction scores (probabilities or activation values) and optionally binary classifications decisions for each tag for each track in the test set. We provide a script to calculate activation thresholds and generate decisions from predictions by maximizing macro F-score. See the documentation in the evaluation scripts directory in the dataset repository for instructions on how to do this.
We will use the following metrics, both types commonly used in the evaluation of auto-tagging systems: • •
Macro PR-AUC and ROC-AUC on tag prediction scores Micro- and macro-averaged precision, recall and F-score for binary decisions.
Participants should report the obtained metric scores on the validation split and test split if they have run such a test on their own. Participants should also report whether they used the whole development dataset or only a part for each submission.
We will generate rankings of the submissions by ROC-AUC, PRAUC, and micro and macro F-score. For leaderboard purposes, we will use PR-AUC as the main metric. However, we encourage a comprehensive evaluation of the systems by using all metrics to generate more valuable insights into the proposed models when reporting evaluation results in the working notes. A maximum of ifve evaluation runs per participating team are allowed.
Note that we rely on the fairness of submissions and do not hide the ground truth for the test split. It is publicly available for benchmarking as a part of the MTG-Jamendo Dataset outside this challenge. We encourage the participants to publicly release their 4https://github.com/MTG/mtg-jamendo-dataset/tree/master/scripts/mediaeval code under an open-source/free software license on GitHub or another platform for transparency and reproducibility.
As our main baseline, we used a common VGG-ish architecture
[
The popularity baseline always predicts the most frequent tag in the training set (Table 1). For the training set of split-0 this is “happy”.
By bringing Emotions and Themes in Music to MediaEval, we hope to benefit from the contributions and expertise of a broader machine learning and multimedia retrieval community. We refer to the MediaEval 2020 proceedings for further details on the methods and results of teams participating in the task.
We are thankful to Jamendo for providing us the data and labels.
This work has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 765068.
This work was funded by the predoctoral grant MDM-2015-050217-2 from the Spanish Ministry of Economy and Competitiveness linked to the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).