Music Transcription using Convolutional Neural Networks and Constant-Q transform

Abstract

Automatic music transcription (AMT) is the problem of analyzing an audio recording of a musical piece and detecting notes that are being played. AMT is a challenging problem, particularly when it comes to polyphonic music. The goal of AMT is to produce a score representation of a music piece, by analyzing a sound signal containing multiple notes played simultaneously. In this work, we design a processing pipeline that can transform classical piano audio files in .wav format into a music score representation. The features from the audio signals are extracted using the constant-Q transform, and the resulting coeficients are used as an input to the convolutional neural network (CNN) model.

Keywords

• Non-negative matrix factorization (NMF) and Neural net-

1. Introduction

In music, transcription refers to the process of analyzing

an acoustic musical signal and writing down the musical information such as the pitch, onset time, duration, and the type of each sound that occurs in it. To visually represent this information, musical notations are often used. In a computational music transcription system, a MIDI music representation format is often appropriate. sented in a transcribed format is important. Transcribed music is of great help for musicians, since they do not have to memorize the whole music composition to play it. Additionally, music that has been transcribed can be shared and made accessible to other musicians.

The term ”Automatic music transcription” was first introduced by the researchers James A. Moorer, Martin Piszczalski, and Bernard Galler in their audio research in 1977.

With their audio engineering research and knowledge, they believed that a computer could be programmed to detect chord patterns and rhythmic accents in a digital recording of music [ 1 ].

AMT is a challenging problem, particularly when it comes to polyphonic music. Several factors make AMT challenging. Polyphonic music is a complex mixture of

Non-negative matrix factorization (NMF) is a numerical algorithm that decomposes a non-negative matrix into can be used for extracting meaningful patterns [2].

2.1. Non-negative Matrix Factorization

© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License two low rank approximation non-negative matrices and The first implementations of NMF in musical appli- representations and they performed hyper-parameter cations were reported in [ 2 ] and [ 3 ], where it was used tuning to improve the performance. for transcribing polyphonic music for separating various The authors also adopted the MAPS dataset [13], which sound sources. All approaches used learn representations provides MIDI-aligned recordings of numerous classiof audio segments that correspond to musical events. The cal music pieces. The authors conducted two sets of work by Arshia Cont [ 4 ] introduces a modified NMF al- experiments, which focused on types of input represengorithm with sparseness constraints to predict pitch in tations and hyper-parameter search to achieve the best real-time, so that the number of non-zero elements in possible results. The four types of input representations the solution are minimized. This modification allows for compared in the study were spectrograms with linearly an eficient and better detection of the pitch in real-time spaced bins S, spectrograms with logarithmically spaced scenarios by reducing the number of non-zero elements bins LS, spectrograms with logarithmically spaced bins, in the NMF factor matrices. Additionally, the sparse- and logarithmically scaled magnitude LM as well as the ness constraint ensures that only relevant information CQT. The authors’ analysis showed that the S type was is extracted, leading to a more precise and concise repre- the best data representation for the perceptron network sentation of the audio signal since NMF usually produces (Logistic regression), whereas the LM type was the best a sparse representation of the data. for the shallow net. The authors mentioned that the lower performance of the CQT was quite unexpected and 2.2. Neural Networks for AMT suggested further investigation.

The authors used practical training recommendations Recent advancements in deep learning technology in with Stochastic Gradient Descent (SGD) and found that pattern recognition and audio processing sparked the adaptive learning rate strategies, such as Adam, improved interest of researchers in the field of Music information the performance to some extent but did not eliminate the retrieval (MIR). There are several related works that apply need for tuning the initial learning rate and its schedule. neural networks in AMT. The current state of the art method for general pur

In [ 5 ], Zalani and Mittal proposed a polyphonic mu- pose piano transcription was proposed by Google Brain, sic transcription system based on deep learning. They by Hawthorne et al. in [14]. The approach uses two trained 88 binary classifiers capable of transcribing notes distinct architectures to detect note onsets, which is a of polyphonic music. Each classifier detects the presence convolutional layer and its output is fed to a second netof a single note in the music at each time step. The au- work which is a bidirectional LSTM with 128 units in thors proposed a Recurrent Neural Network (RNN) [ 6 ] both directions connected to a fully connected sigmoid stacked with Restricted Boltzmann Machines (RBM) [ 7 ] layer with 88 outputs to represent the 88 piano keys. The for unsupervised feature learning and trained separate paper also uses the MAPS dataset. 88 support vector machines (SVM) [ 8 ] responsible for transcribing single notes. To enhance the accuracy of their results, a Hidden Markov Model (HMM) [ 9 ] was 3. Proposed Approach also implemented.

In [10], Sigtia et al. proposed an end-to-end approach For AMT, a holistic approach is commonly used, which for transcribing a piece of piano music into music score. involves using a single model to identify and detect all The authors mentioned that their architecture is the com- the musical notes in a given piece of music (see Fig. 1). bination of both an acoustic model and a music language model. They experimented with three diferent neural 3.1. CNN architecture networks for the acoustic model, namely an artificial neural network, an RNN, and CNN, all of which were In this work, an architecture is proposed for the AMT task trained to recognize CQT coeficients. For the music lan- involving the use of CNNs and the Constant-Q transform guage model, the author experimented with three types (CQT). The features from the audio signal are extracted of neural networks - the generative RNN, neural auto- using the CQT method, and the resulting CQT coefiregressive distribution estimator (NADE) proposed by cients are used as an input to the CNN model. CNNs Larochelle and Murray et al. [11], and RNN-NDE for the have become increasingly popular, particularly in the music language model. The results demonstrate that the computer vision field [ 15]. In computer vision, an input CNN outperformed the other architectures in all evalua- image is fed into the CNN and passed through multiple tion metrics. layers of diferent nature, including convolutional, pool

Kelz et al. [12] conducted an extensive analysis of only ing, and activation layers, where non-linearities may be using an acoustic model for transcription in order to applied. The convolutional layers are made up of variexplore the limitations of simple approaches in the auto- ous filters, each capable of learning higher-level image matic music transcription task, including suitable input features. In the context of AMT, note detection can be

4.1. Pre-processing

5. Experimental Results

5.1. Metrics

In the experiment, a frame-wise accuracy measure was used to evaluate the performance of the model [17]. If the predicted labels for a specific frame completely match the true labels, then the frame is considered to be correctly predicted, resulting in a subset accuracy of 1.0. However, if there are any diferences between the predicted and true labels, the subset accuracy is 0.0.

−1 1 ( , )̂ = ∑ 1( ̂ = ) (1) =0 where ̂ is the predicted value of the -th frame and is the corresponding true value and 1() is the indicator function.

5.2. Experimental Set-up

The training experiments were performed using a workstation equipped with a 40 cores Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz processor. The workstation was also equipped with 252 GB of RAM and a Tesla V100DGXS-32GB GPU for processing.

A total of 830 piano songs, each lasting 3 minutes have been generated. Out of these, 80 percent is used for training and 20 percent is used for validation. Additionally, a dataset of 166 piano songs (the same size as the validation set) was generated with uniform distribution and used as a test set to compare the performance of the models.

5.3. Model Performance

The Figure 5 shows example model predictions from

structured models and holistic models visualized through Figure 3 depicts the evolution of the training and validation accuracy while training the holistic model. A batch size of 64 was used to train the models. From the training Model TT(minutes) IT(msec) Acc(%) plot, we can understand that convergence is achieved HHoolliissttiicc -- vv12 779230..62 666604 9828..52 after 70 epochs.

Additionally, another version of Holistic V1, referred Table 1 to as Holistic V2, was trained by removing silent frames Model result summary, including training time (TT), inference (refer 4.2). The training result of this model is depicted time (IT) and accuracy (Acc). in Fig 4.

Table 1 summarizes the performance of the models.

The models were tested on their inference time, accuracy, a piano roll representation. The green dots represent and time taken to train. The inference time is the time the false negatives, indicating the notes and chords the it takes for the models to make a prediction once they models were not able to predict correctly, and the red are given the input. The inference duration is measured dots represent false positive predictions. The dark green based on the time taken by the model to make a prediction dots and segments represent true positives where the for a 2-minutes long piano song. models make the correct predictions. Overall, the model performed well on the prediction task, as evident from 5.4. Model Prediction the high number of true positive predictions. However, as shown in the piano roll visualization, the models made some false positive and false negative predictions.

One interesting but expected finding is that most of the false positive predictions occur during the transition from one chord/note to another. This is because the frames close to the transition of notes and chords could contain the frequency of notes/chords from both sides, leading to a potential confusion and notes/chords from both side being predicted. Moreover, it is observed that these false positive predictions are often notes from the previous notes/chord that are still audible. This is expected since when we release a note on the piano, the sound might not end immediately and the note will be heard for a short period of time, leading to the prediction of the same note/chord after the note/chord ended.

6. Conclusions and Future Works

We have presented an approach to tackle the problem of AMT for polyphonic piano music using CNN and CQT, showing experimental results about the achieved accuracy and performance.

There are many ways to improve this AMT system. This work and accompanying experimentation are based on fully synthesized audio data. Extending this approach using real piano audio data for the training and/or testing phases could provide more diverse training examples for the AMT system, leading to a more realistic measurement of its achievable performance in real-world scenarios.

Now that we have baseline results using CNN, a next step would be to explore other models, such as RNN, that might capture the longer dependencies between notes and chords. By exploiting the temporal nature of music, RNNs can potentially improve the accuracy of the proposed AMT system. maximization technique occurring in the statistical analysis of probabilistic functions of markov chains, The Annals of Mathematical Statistics 41 (1970) 164–171. URL: http://www.jstor.org/stable/ 2239727. [10] S. Sigtia, E. Benetos, S. Dixon, An end-to-end neural network for polyphonic piano music transcription, https://arxiv.org/abs/1508.01774, 2015. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 5 0 8 . 0 1 7 7 4 . [11] B. Uria, M.-A. Côté, K. Gregor, I. Murray, H. Larochelle, Neural autoregressive distribution estimation, Journal of Machine Learning Research 17 (2016) 1–37. URL: http://jmlr.org/papers/v17/16-272. html. [12] R. Kelz, M. Dorfer, F. Korzeniowski, S. Böck, A. Arzt, G. Widmer, On the potential of simple framewise approaches to piano transcription, https://arxiv. org/abs/1612.05153, 2016. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 6 1 2 . 0 5 1 5 3 . [13] V. Emiya, N. Bertin, B. David, R. Badeau, Maps - a piano database for multipitch estimation and automatic transcription of music, 2010. [14] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Rafel, J. Engel, S. Oore, D. Eck, Onsets and frames: Dual-objective piano transcription, https://arxiv.org/abs/1710.11153, 2017. doi:1 0 . 4 8 5 5 0 / A R X I V . 1 7 1 0 . 1 1 1 5 3 . [15] N. Sharma, V. Jain, A. Mishra, An analysis of convolutional neural networks for image classiifcation, Procedia Computer Science 132 (2018) 377–384. URL: https://www.sciencedirect.com/ science/article/pii/S1877050918309335. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . p r o c s . 2 0 1 8 . 0 5 . 1 9 8 , international Conference on Computational Intelligence and Data Science. [16] T.-L. Cheng, S. Dixon, M. Mauch, Modelling the decay of piano sounds, International Conference on Acoustics, Speech, and Signal Processing (2015). doi:1 0 . 1 1 0 9 / i c a s s p . 2 0 1 5 . 7 1 7 8 0 3 8 . [17] scikit-learn accuracy score, https://scikit-learn. org/stable/modules/model_evaluation.html# accuracy-score, 2023. Accessed: 2023-02-30.

References

[1] W. contributors , Transcription (music) , https://en. wikipedia.org/wiki/Transcription_(music), 2022 .

[2] P. Smaragdis , J. Brown , Non-negative matrix factorization for polyphonic music transcription , 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684) ( 2003 ) 177 - 180 .

[3] S. Abdallah , M. Plumbley , Polyphonic music transcription by non-negative sparse coding of power spectra , in: International Society for Music Information Retrieval Conference , 2004 .

[4] A. Cont , Realtime multiple pitch observation using sparse non-negative constraints , in: ISMIR 2006 , 7th International Conference on Music Information Retrieval, Victoria, Canada, 8 - 12 October 2006 , Proceedings, 2006 , pp. 206 - 211 .

[5] A. Zalani , A. Mittal , Polyphonic music transcription a deep learning approach , https://cse.iitk.ac.in/users/cs365/2014/ _submissions/aniruddh/project/report.pdf, 2014 .

[6] D. E. Rumelhart , G. E. Hinton , R. J. Williams , Learning internal representations by error propagation , in: D. E. Rumelhart, J. L . Mcclelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition , Volume 1 : Foundations, MIT Press, Cambridge, MA, 1986 , pp. 318 - 362 .

[7] G. E. Hinton , A practical guide to training restricted boltzmannmachines , in: Proceedings of Neural Networks: Tricks of the Trade (2nd ed.) , 2012 , pp. 599 - 619 .

[8] V. N. Vapnik , Introduction: Four Periods in the Research of the Learning Problem , Springer New York, New York, NY, 2000 , pp. 1 - 15 . URL: https:// doi.org/10.1007/978-1- 4757 -3264- 1 _1. doi:1 0 . 1 0 0 7 / 9 7 8 - 1 - 4 7 5 7 - 3 2 6 4 - 1 _ 1 .

[9] L. E. Baum , T. Petrie , G. Soules, N. Weiss , A