The task of cover song identification (CSI) deals with the automatic matching of audio recordings by modeling musical similarity. CSI is of high relevance in the context of applications such as copyright infringement detection on online video platforms. Since online videos include metadata (eg. video titles, descriptions), one could leverage it for more efective CSI in practice. In this work, we experiment with state-of-the-art models of CSI and entity matching in a Co-Training ensemble. Our results outline slight improvements of the entity matching model. We further outline some suggestions for improvements of our approach to overcome the issue of overfitting CSI models which we observed.
Co-Training was initially proposed by Blum and Mitchell [10] and refers to the idea to leverage automatically generated pseudo labels to improve the performance of models in the ensemble. This enables training models in cases where only a small subset of the available data is labeled, which applies to many real world scenarios. Co-Training relies on the availability of multiple views which are required to fulfill the following assumptions: 1. Suficiency : Each view is suficient to address the task at hand. 2. Independence: The views are conditionally independent.
As illustrated in Figure 1, the models within the ensemble iteratively provide pseudo labels for each other. One key component is the selection of a fraction of predictions as pseudo labels, based on a constraint such as confidence thresholds [ 11, 12], limiting to a fraction of most confident samples by ranking [13, 14] or other methods [15, 16, 17, 18].
Recently, various applications of Co-Training with deep learning models have been proposed for the modalities text [13, 16], images [14, 19, 11, 12, 20, 21] and on multiple modalities [22, 15]. Lang et al. [13] improve prompt based learning by using GPT-3 output probabilities and frozen representations of openly available large language models to improve prompt-based learning. Both of their proposed methods for pseudo label selection are based on the ranking of samples. An approach by Wu et al. [16] applies Q-learning to improve the selection policy for the partition of unlabeled data to be pseudo-labeled. They demonstrate the efectiveness of reinforced Co-Training on text classification tasks. Peng et al. [ 14] use adversarial examples in an ensemble of multiple models for model diversity to improve the ensemble performance for image segmentation on medical images. To select pseudo labels, a fraction parameter is used that increases over iterations. In contrast, Yang et al. [11] apply a Co-Training framework with a fixed threshold to the task of domain adaption task. Xian and Hu [ 12] use a fixed threshold parameter for pseudo-labeling in the task of person re-identification.
Some approaches successfully make use of views arising from modalities. A multimodal approach of Hinami et al. [15] leverages multiple views of the modalities text, audio and video found in web videos to improve concept classification. Pseudo labels are selected based on a voting approach within the ensemble. Another multimodal approach with text and images from web articles of Bhattacharjee et al. [22] improves the task of fake news detection. Their pipeline includes an attention-aware step to fuse two views. The models then co-train based on sampling of hard positive samples. In the following, we present our Co-Training algorithm which is based on fixed-thresholds.
We have access to an entity matching model based on a language model and a cover song identification model based on metric learning. Both models are pretrained and achieve state-of-the-art performance for the task at hand. However, we aim to improve their performance by training these models on a labeled dataset and an unlabeled dataset . Each item within either of the datasets is a YouTube video representation which is represented by a text view (YouTube metadata) and an audio view based on audio features (cf. Section 5).
Accordingly, (, ) computes the entity matching confidence 0 < < 1 for a pair , ∈ ∪ and (, ) the musical similarity modeled as cosine similarity − 1 < < 1. A labeled item ∈ has a known clique or musical work where it belongs to represented by () ∈ . Unlabeled items ∈ have a candidate clique ˆ() ∈ . It is unknown whether belongs to this clique. However, among all possible cliques this is the most likely one, because was found with queries formulated to find items for this clique as explained in Section 4.
We argue that we can address the problem of multimodal CSI by Co-Training. Since both models are pretrained, we expect the first Co-Training assumption (cf. Section 2) to hold. We further argue that both views are conditionally independent, due to the natural split given by modalities.
In Algorithm 1, we show the Co-Training loop. We randomly sample three labeled ( = 3) and three unlabeled videos ( = 3) for two randomly selected cliques ( = 2) per iteration. We make use of hard threshold parameters. The output of is based on softmax layers on top of a language model. We therefore simply denote as the outer boundary for confidence, indicating that a pseudo label is either positive if > 1 − or negative if < . For the audio model we impose two thresholds. We observed that the CSI model we use does not output equally distributed similarity values spreading to the boundaries of the cosine similarity. Hence, we use a lower threshold to set negative pseudo labels if < lower and an upper threshold to set positive pseudo labels if < upper.
In Algorithm 2, we show one iteration of Co-Training where MAX(, 0) denotes the elementwise maximum operator applied to the matrix and 0 and MAX( ) and denotes the respective row-wise maximum operation applied to .
We first predict the entity matching confidences and musical similarities for each pair of items within all the pairs in the batch and assign those to matrices ˆ text and ˆ audio. Subsequently, the similarity square matrices are masked to retain only pairwise relationships with a known ground truth label from or with a suficiently confident pseudo label. As masking values, we select 1 to represent a indicating a positive relationship among the items (both are from the Algorithm 1 Multimodal Cover Song Co-Training Loop 1: Initialize 2: Maximum number of iterations , Number of cliques per batch , set of clique identifiers , number of labeled items per batch , number of unlabeled items per batch , outer boundary for text model , lower threshold for audio model lower, upper threshold for audio model upper, labeled dataset , unlabeled dataset , learning rate , audio model , text model 3: 4: for − ← 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: end for
to do Sample = {1, . . . , } from ´ for ∈ do:
Sample = {1, . . . , } from where () ∈
Sample = {ˆ1, . . . , ˆ} from where ˆ(ˆ) ∈ end for = ⋃︀∈ = ⋃︀∈ CoTrainIter(, , lower, upper, , , , ) same clique) and -1 to indicate the contrary. Additionally, we select 0 to mask out uncertain relationships for pairs without a ground truth label and insuficient confidence of the model generating the pseudo label. The pseudo label masks text and audio are used to sample the similarity values to use for training updates with hard triplet mining as proposed by Xuan et al. [23] and applied to train prior CSI models [24, 25]. The lowest distances of the positive relationships in DIST+ and DIST+ and the highest distances for the pairwise negative relationships DIST− and DIST− represent the components of the hard triplets that are used for the training updates.
We train the metric learning model with triplet loss which is defined as: tri = max((, +) − (, − ) + , 0), (1) where = 1 is the margin parameter, + and − are the positive and negative to anchor which are used to compute the distances (, +) and (, − ) as found in DIST+ and DIST− respectively.
Our entity matching model is based on a large language model which we train with binary cross entropy loss: Algorithm 2 Co-Training Iteration for One Batch (Triplet Loss with Hard Triplet Mining). 1: Initialize 2: Set of labeled items per batch , set of unlabeled items per batch , outer boundary for text model , lower threshold for audio model lower, upper threshold for audio model upper, learning rate , audio model , text model ⎧⎪[, ] if [, ] ̸= 0 ⎪ ⎪⎪⎨1 if ˆ [, ] > 1 − ⎪− 1 if ˆ [, ] < ⎪ ⎪⎩⎪0, otherwise if ˆ [, ] < otherwise 19: [, ] = 20: 21: Hard Triplet Mining 22: DIST+ = 1 − min(max(, 0) * ˆ ) ∈ R× 1 23: DIST− = 1 − max(max(− 1 * , 0) * ˆ ) ∈ R× 1
ˆ 24: DIST+ = 1 − min(max(, 0) * ) ∈ R× 1 25: DIST− = 1 − max(max(− 1 * , 0) * ˆ ) ∈ R× 1 26: 27: Loss Computation 28: LOSS = tri(DIST+, DIST−) 29: LOSS = ce(DIST+, DIST−) 30: 31: Update 32: ← − ∆ LOSS( ) 33: ← − ∆ LOSS( ) where ˆ is one prediction as found in in either DIST− or DIST+ and hence , ∈ {0, 1}. In the following, we outline details about our dataset, preprocessing and training implementation.
We provide an overview of the datasets used in Table 1 and CSV files containing cliques identifiers and YouTube identifiers 1. The cliques used for implementation rely on two datasets from prior research in CSI: SHS100K 2 for training, validation and testing and Da-Tacos [26] for testing.
Based on the test subset of SHS100K we formulated around 44 text queries per clique to crawl YouTube3 to find additional songs for these cliques, similarly to our prior work [ 27]. We split this crawl into two parts with 50 cliques each. One for training composed of Train-SHS (labeled dataset ) and Train-YT (unlabeled dataset ) and one for testing which is composed of Test-SHS and Test-YT. Test-SHS is a subset of songs that are represented by YouTube videos in the initial SHS100K test subset and Test-YT contains other YouTube videos found by the query procedure. We annotated these 628 crawled videos with the help of two students and up to five workers on Mechanical Turk. We only considered labels with full agreement among students and aggregated the worker labels by majority vote.4
For validation we use the validation subset of SHS100K which we denote by Val-SHS.5 We
additionally use the larger Da-Tacos dataset for testing.6
1https://github.com/progsi/datasets_shs_yt_cotraining
2cf. https://github.com/NovaFrost/SHS100K provided by Yu et al. [
For each video, we downloaded the MP3 files with a sampling rate of 22,050 Hertz 7 to extract audio features. We extract CREMA8 features and constant-Q transform features9 (CQT) with 84 frequency bins.
Furthermore, we retrieved the metadata for each video. To ensure that semantics are preserved independently of the Unicode font, we mapped various Unicode fonts to basic Latin characters using Unicodedata10.
We use the BERT -based entity matching model Ditto [28] as our text model which is publicly available on Github.11 Ditto requires fine-tuning specifically to the structure of attributes in the entities, in our case YouTube videos. We use the SHS100K-Train subset as Ditto pretraining dataset, which does not overlap with dataset any of our other datasets shown in Table 1. Following the splits applied by Li et al. [28] we created a training, validation, and test set with a ratio of 3:1:1 with each containing positive and negative pairs of YouTube videos in a 1:4 ratio. We gathered the negative pairs by randomly sampling videos from another randomly selected work. We use only the video titles and channel names as YouTube metadata representations. We additionally experimented with YouTube descriptions but preliminary results showed inferior results (F1 score of 27% against 95%) to the ones achieved by using only video titles and channels. We used all of the proposed data augmentation techniques and the best performing language model (RoBerta) as described in [28]. We applied the best model checkpoint evaluated on the test set after 50 epochs for our matching task.
We use two diferent state-of-the-art CSI models which are publicly available 12: CQTNet
[
Re-MOVE processes CREMA features which are a variant of pitch class profiles and mainly represent harmonic information. CQTNet processes constant-Q transform features (CQT), which are spectograms with a logarithmically spaced frequency axis.
Following the Co-Training approach by Yang et al. [11], we use stochastic gradient descent as optimizer with learning rate 0.01 and momentum ∈ {0, 0.9}. We validate the used audio model and Ditto every 100 iterations. Since the prediction of a square matrix is expensive with Ditto, we initialize a random subset of the validation set at the beginning of each training and use it throughout the training.
MP3s for CQTNet. Hence, we only include the subset of videos which were available on YouTube which makes up around 92% of full Da-Tacos. 7cf. https://github.com/yt-dlp/yt-dlp 8cf. https://github.com/bmcfee/crema 9cf. https://librosa.org/doc/latest/index.html 10cf. https://docs.python.org/3/library/unicodedata.html 11cf. https://github.com/megagonlabs/ditto 12We experimented with the ByteCover implementation by Orfium: https://github.com/Orfium/bytecover However, the implementation was not provided by the authors of the paper and achieves lower performance than both models we use.
We evaluate our proposed Co-Training algorithm on ensembles with Ditto [28] as paired
with one of the pretrained audio models CQTNet [
• Co-ReM: with Re-MOVE and = 0.2, upper = 0.5, lower = 0.3.
We compare the two audio models with upper ∈ {0.5, 0.6}, = 0.2 and lower = 0.3. In Figure 2 we show the triplet loss over 1,000 iterations as well as the validation mAP. The strong observable drop in mAP and loss for Re-MOVE strongly reflects an overfit. As we show in Table 2, Re-MOVE generally performs worse than CQTNet. We therefore focus on experimenting with various diferent thresholds for CQTNet. We further observe that the convergence of the loss of CQTNet is rather slow. Thus, we impose a momentum of 0.9 in the next experiments.
We experimented with diferent hyperparameter configurations: ∈ {0.1, 0.2, 0.49}, upper ∈ {0.5, 0.6, 0.7}, lower ∈ {0.2, 0.3, 0.4}.
In Figure 3 we show the loss and validation mAP of Co-CQT. We observe that CQTNet overfits, shown by the jointly decreasing loss and mAP. The triplet loss converges rather close to the margin for the triplet loss = 1. We observed this result consistently across configurations. 13cf. https://github.com/maxbachmann/RapidFuzz
However, we as well observe an increase in loss but a constant validation mAP for Ditto14. As shown in Table 2, Ditto is the only model which actually improves with the Co-Training procedure. Given these two key observations, we hypothesize that balancing the two very diferent models is a key challenge. In the closing section, we therefore outline some of the potential issues with our approach and ideas to address these.
In this paper, we applied a Co-Training algorithm for multimodal CSI using an audio-based CSI model along with an entity matching model. We slightly improved the entity matching model Ditto for our task. This might suggest that further training iterations can improve Ditto. However, both audio-based models seem to overfit quite rapidly.
In the following, we outline some ideas which might have an impact on this problem. 14Please note that sampling of a subset of 100 items of the full Val-SHS as mentioned in Section 5, can have a major impact on the validation mAP. Learning Rates. In comparison, Ditto seems to learn rather slow while the audio-based models overfit. We believe that diferent learning rates for both models could help to prevent this imbalance of model convergence. One potential improvement can be a grid search over diferent learning rates across the models as proposed by Likhosherstov et al. [29]. Alternatively, one could apply diferent learning rate schedulers like Yang et al. [ 11]. Our observations also suggest the potential continuation of the pretraining of Ditto, possibly with pseudo labels generated by the audio model. Eventually, Co-Training with both models could be done afterwards to avoid the apparent diferent starting condition of both models.
Hard Triplet Mining. We sample triplets during training based on the hard triplet mining
strategy found in metric learning. In the context of Co-Training, adversarial examples can be
used as an alternative [14, 19, 20, 21] which encourage view diference. In contrast, hard triplet
mining solely ensures that the most dificult triplets are in the batch are utilized for training.
Losses. Some state-of-the-art CSI models rely on multiloss approaches [
Batch Size. We tested diferent configurations of thresholds. However, the batch size for labeled and unlabeled items per batch was fixed for all experiments and the number of items for both input datasets was equal. We believe that the increase of unlabeled items per batch in contrast to labeled items could enforce that more interesting items are used during training. That is, due to their containment in our crawl rather than the widely used academic dataset SHS100K, which is based on the platform Secondhandsongs. The platform itself relies on manual labour by volunteers subject to policies to determine the boundaries between cover songs whereas our crawl is solely subject to the creative spectrum on YouTube.
Label Confidence Estimation. As outlined in Section 2, other label confidence estimation methods can be applied to Co-Training. In this study, we solely experimented with a threshold-based method. Ranking-based methods or possibly more sophisticated methods could further improve our proposed algorithm.
In future experiments, we plan to test the impact of the factors discussed. We hope that we can find configurations of ensembles which can efectively leverage both views to improve the task of multimodal CSI. audio and video features, in: 2017 IEEE International Conference on Multimedia and Expo (ICME), 2017, pp. 1422–1427. doi:10.1109/ICME.2017.8019444. [9] A. A. Correya, R. Hennequin, M. Arcos, Large-scale cover song detection in digital music libraries using metadata, lyrics and audio features, CoRR abs/1808.10351 (2018). URL: http://arxiv.org/abs/1808.10351. arXiv:1808.10351. [10] A. Blum, T. Mitchell, Combining labeled and unlabeled data with co-training, Association for Computing Machinery, New York, NY, USA, 1998. [11] L. Yang, Y. Wang, M. Gao, A. Shrivastava, K. Q. Weinberger, W.-L. Chao, S.-N. Lim, Deep cotraining with task decomposition for semi-supervised domain adaptation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 8906–8916. [12] Y. Xian, H. Hu, Enhanced multi-dataset transfer learning method for unsupervised person re-identification using co-training strategy, IET Computer Vision 12 (2018) 1219–1227. [13] H. Lang, M. Agrawal, Y. Kim, D. A. Sontag, Co-training improves prompt-based learning for large language models, in: International Conference on Machine Learning, 2022. [14] Deep co-training for semi-supervised image segmentation, Pattern Recognition 107 (2020) 107269. doi:https://doi.org/10.1016/j.patcog.2020.107269. [15] R. Hinami, J. Liang, S. Satoh, A. G. Hauptmann, Multimodal co-training for selecting good examples from webly labeled video, CoRR abs/1804.06057 (2018). URL: http://arxiv.org/ abs/1804.06057. arXiv:1804.06057. [16] J. Wu, L. Li, W. Y. Wang, Reinforced co-training, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 1252–1262. URL: https://aclanthology.org/N18-1113. doi:10.18653/v1/N18-1113. [17] T. Han, W. Xie, A. Zisserman, Self-supervised co-training for video representation learning, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Curran Associates Inc., Red Hook, NY, USA, 2020. [18] Semi-supervised learning combining co-training with active learning, Expert Systems with Applications 41 (2014) 2372–2378. doi:https://doi.org/10.1016/j.eswa.2013.09. 035. [19] S. Qiao, W. Shen, Z. Zhang, B. Wang, A. Yuille, Deep co-training for semi-supervised image recognition, in: V. Ferrari, M. Hebert, C. Sminchisescu, Y. Weiss (Eds.), Computer Vision – ECCV 2018, Springer International Publishing, Cham, 2018, pp. 142–159. [20] H. Xie, C. Fu, X. Zheng, Y. Zheng, C.-W. Sham, X. Wang, Adversarial co-training for semantic segmentation over medical images, Computers in biology and medicine 157 (2023) 106736. [21] Y. Wang, Y. Zhang, Y. Liu, Z. Lin, J. Tian, C. Zhong, Z. Shi, J. Fan, Z. He, Acn: Adversarial co-training network for brain tumor segmentation with missing modalities, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VII 24, Springer, 2021, pp. 410–420. [22] S. D. Bhattacharjee, J. Yuan, Multimodal co-training fornbsp;fake news identification using attention-aware fusion, Springer-Verlag, Berlin, Heidelberg, 2021. [23] H. Xuan, A. Stylianou, X. Liu, R. Pless, Hard negative examples are hard, but useful, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, pp. 126–142. [24] F. Yesiler, J. Serrà, E. Gómez, Accurate and scalable version identification using musicallymotivated embeddings, in: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 21–25. doi:10.1109/ICASSP40776. 2020.9053793. [25] F. Yesiler, J. Serrà, E. Gómez, Less is more: Faster and better music version identification with embedding distillation, in: International Society for Music Information Retrieval Conference, 2020. [26] F. Yesiler, C. J. Tralie, A. A. Correya, D. F. Silva, P. Tovstogan, E. Gómez, X. Serra, Da-tacos:
A dataset for cover song identification and understanding, in: ISMIR, 2019. [27] S. Hachmeier, R. Jäschke, H. Saadatdoorabi, Music version retrieval from youtube: How to formulate efective search queries?, in: P. Reuss, V. Eisenstadt, J. M. Schönborn, J. Schäfer (Eds.), Proceedings of the LWDA 2022 Workshops: FGWM, FGKD, and FGDB, Hildesheim (Germany), Oktober 5-7th, 2022, volume 3341 of CEUR Workshop Proceedings, CEURWS.org, 2022, pp. 213–226. URL: https://ceur-ws.org/Vol-3341/WM-LWDA_2022_CRC_ 7142.pdf. [28] Y. Li, J. Li, Y. Suhara, A. Doan, W.-C. Tan, Deep entity matching with pre-trained language models, Proceedings of the VLDB Endowment 14 (2020) 50–60. URL: https://doi.org/10. 14778%2F3421424.3421431. doi:10.14778/3421424.3421431. [29] V. Likhosherstov, A. Arnab, K. Choromanski, M. Lucic, Y. Tay, A. Weller, M. Dehghani, Polyvit: Co-training vision transformers on images, videos and audio, CoRR abs/2111.12993 (2021). URL: https://arxiv.org/abs/2111.12993. arXiv:2111.12993. [30] A. Taha, Y.-T. Chen, T. Misu, A. Shrivastava, L. Davis, Boosting standard classification architectures through a ranking regularizer, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 758–766. [31] G. Doras, G. Peeters, A prototypical triplet loss for cover detection, in: ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 3797–3801. doi:10.1109/ICASSP40776.2020.9054619.