Musical caption, when expressed in free-form text as opposed to more structured and limited musical tags, often encompasses the individual characteristics of the annotator, thereby injecting a degree of subjectivity into the resultant dataset. This study explores the impact of such annotator subjectivity within the MusicCaps dataset, a pioneering collection of human-annotated captions explaining 10second music audio clips. We conducted three distinct analyzes to investigate the presence of this subjectivity. This includes examining the frequency distribution of tag categories (i.e., genre, mood, or instruments) among diferent annotators, a qualitative assessment of caption embeddings through UMAP visualizations, and a quantitative analysis where we train and compare cross-modal retrieval models using an annotator-specified training split. Our findings underscore the significant annotator subjectivity inherent in the MusicCaps dataset, emphasizing the need for its consideration when collecting free-form text annotations on music or developing machine-learning models using this type of dataset.
As text-image multi-modal models such as CLIP [
However, while free-form descriptions allow a full range of creative expression, they can be influenced by the annotator’s personal characteristics. This paper analyzes the annotator subjectivity in a music caption dataset and its efect on audio-text joint embedding space. Our main contributions include: (1) identifying the subjectivity in the music caption dataset; (2) analyzing the impacts of this subjectivity in training audio-text joint embedding space.
We use the MusicCaps dataset [
The first thing we analyzed was the diference in distribution of tag categories that annotators
focus on while captioning. We calculated the distribution by counting the number of captions
that contain the list of selected tags for each category. We chose musical keywords from the
aspect list over the caption since the aspects ofer a clearer keyword representation than the
terms extracted directly from the caption. To identify the most representative keywords that
are frequently used, we first collated the top 50 of the most frequently annotated aspects per
annotator. Then we sorted the collated aspects following two criteria in order: first, by the
number of annotators that have included the aspect in their top 50, and second, by the sum
of the percentages of occurrences of the aspect. We get the top 50 aspects from the sorted
results and divide them into five categories - tempo, instrument, mood, genre, and audio quality,
referring to the explanation of the aspect list in [
Table 1 shows the counted result. We can see some extreme deviations, such as theme annotated in 94.4% of annotator 6’s samples, but none and 1.0% samples of annotators 1 and 7. Similarly, tempo was annotated in 96.5% and 95.3% samples of annotators 7 and 1, but only 4.2% among the samples annotated by annotator 4. This indicates that consideration of tag categories when captioning significantly varies by annotators, thus afecting the resulting captions’ characteristics.
We extracted feature embeddings to analyze the semantics of each music and caption quantitatively. For audio feature embeddings, we used VGGish model [14] pre-trained with the AudioSet dataset, and averaged across the time axis to get a single embedding for each 10-sec audio. We used 5,480 music examples, of which the audio source was obtainable. For text captions, we use a pre-trained BERT model [15] and extract the embeddings from the last hidden state of [CLS] token to capture the comprehensive semantics of the entire sentence. The resulting embeddings for audio and text are 128-dimensional and 768-dimensional, respectively.
We visualized 2D UMAP [16] embedding space in Figure 1. The plot for caption embedding shows clear cluster by the annotator. This means that the neighbors of each caption in the BERT embedding space are mainly decided by who wrote the caption, not by which music it described. On the other hand, audio embeddings do not show author-wise clusters, which implies that the audio samples were not assigned to the annotators by their preferences or choices.
To quantify how distinctive those embeddings are, we conducted an experiment to predict the annotator from the embedding using a random forest classifier with 100 decision trees. The experiment yielded an average F1 score of 0.76 across all annotators, underlining the influence of annotator-specific subjectivity in text embeddings. To assess whether these clusters could result from each annotator being assigned to specific music or favoring music they are familiar with, we carried out the same prediction experiment on audio embeddings. The result was an F1 score of 0.08 on average, which implies that the audio samples were randomly assigned to the annotators regardless of their musical preferences. Therefore, we can presume that the clear distinction between each annotator’s caption embedding originates from the annotator subjectivity.
We also explored the impact of the subjectivity observed within the feature embedding space
on the other applications. A common area of research in audio and text is cross-modal retrieval
tasks that involve the use of audio-text joint embedding spaces [
Following the previous works [
For this experiment, we employed an annotator-specific training/test split to investigate the generalizability of a joint embedding model across diferent annotators’ audio-caption pairs. Each separated dataset is represented as (, ), where and denote the sets of audio and text examples annotated by the annotator . We trained an annotator-specified model model with (,train, ,train) for each annotator. The baseline model for the comparison was trained with the whole train set (,train, ,train) : ∈ {0, 1, 2, ..., 9}. We evaluate the performance of the baseline model and model by computing Recall at K (R@K) of (,test, ,test) over the whole test set (,test, ,test) : ∈ {0, 1, 2, ..., 9}, for both audio-to-text and text-to-audio retrieval tasks.
The train-evaluation split provided by MusicCaps metadata, inherited from the AudioSet dataset, yields imbalanced partitions across the datasets of individual annotators. Importantly, no examples of annotators 2 and 5 were included in the MusicCaps training set. As an alternative, we employed our own partitioning strategy, maintaining the original train-evaluation ratio of 48:52 but applying it in an annotator-specific manner. Note that there exists an imbalance in the sample distribution across annotators. In particular, annotator 5 has the smallest test sample size with only 18 examples, followed by annotator 2 with 58 test samples.
The result is presented in Figure 2. As shown in the result, the annotator-specified models outperform the baseline model for the target annotator in audio-to-text retrieval, even though it used a smaller training set. Conversely, the annotator-specific models exhibit significantly reduced accuracy compared to the baseline when applied to other annotators’ data subsets. When trained to align audio with captions from a specific annotator (in domain), it does not generalize well to audios paired with captions from other annotators (out of domain). We assume that this is due to annotator subjectivity.
In contrast, the baseline model showed better results in the text-to-audio retrieval task. The annotator-specified model showed severely degraded performance even on the target annotator compared to the audio-to-text retrieval. In audio-to-text retrieval, the clear diferences in the description of the caption might help to narrow down the candidates to a specific annotator’s caption. For instance, a model trained on the captions from annotator 6—who includes theme descriptions in 94.4% and genre descriptions in only 5.6% of the captions—would prioritize retrieving captions that include theme descriptions but not genre descriptions. This can largely narrow down the retrieval candidates, thus increasing the accuracy of the target-specified model.
Nonetheless, in text-to-audio retrieval tasks, the same diference in the text caption failed to enhance the performance, as evidenced by the result. This limitation arises because the stylistic diferences of the query text are insuficient to refine the pool of audio candidates for retrieval. The enhanced accuracy achieved through mixed training sets, as opposed to annotator-specific sets, suggests that incorporating captions from multiple annotators can improve text-to-audio retrieval performance. This implies that the captions still share commonness in how they describe the audio, regardless of annotator subjectivity.
The observed asymmetry between audio-to-text and text-to-audio retrieval outcomes indicates that annotator subjectivity exerts a more pronounced impact when text serves as a retrieval candidate rather than a query.
In this paper, we have shown the clear presence of annotator subjectivity in the music caption dataset and the consequent impact on the embedding space and its applications. In our future work, we aim to figure out what leads to this subjectivity and how we might be able to lessen its efects. Its efect on other tasks, such as text-based music generation, must also be investigated, as the style of the captions can largely influence the generation result. We hope that our work helps the research community to be aware of and consider annotator subjectivity when working with music caption datasets.
This work was supported by Sogang University Research Grant of 202110035. audio, in: 2021 International Joint Conference on Neural Networks (IJCNN), IEEE, 2021, pp. 1–8. [13] S. Doh, K. Choi, J. Lee, J. Nam, Lp-musiccaps: Llm-based pseudo music captioning, arXiv preprint arXiv:2307.16372 (2023). [14] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al., Cnn architectures for large-scale audio classification, in: 2017 ieee international conference on acoustics, speech and signal processing (icassp), IEEE, 2017, pp. 131–135. [15] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [16] L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction, arXiv preprint arXiv:1802.03426 (2018). [17] A. v. d. Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748 (2018). [18] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).