This paper proposes an automatic music genre-classification system using a deep learning model. The proposed model leverages Convolutional Neural Nets(CNN) to extract local features and LSTM Sequence to Sequence Autoencoders to learn representations of time series data by taking into account their temporal dynamics. The paper also introduces Clustering Augmented Learning Method (CALM) classifier which is based on the concept of simultaneous heterogeneous clustering and classification to learn deep feature representations of the features obtained from LSTM autoencoder. Computational Experiments using GTZAN dataset resulted in an overall test accuracy of 95.4% with a precision of 91.87%.
With the increasing amount of music available online, there is automatically a growing demand for the symmetrical organization of audio files and that has increased the interest in music classification. To detect a group of the music of a similar genre is the main work of the recommendation system and playlist generators. Thus building a robust music classifier using machine learning techniques is essential to automate tagging unlabeled music and improve user’s experience of media players and music libraries. In recent years, convolutional neural networks(CNNs) have brought revolutionary changes to the computer vision community. Meanwhile, CNN’s have been widely used for music information retrieval, especially music genre classification. Recently, it became increasingly popular to combine CNNs with recurrent networks(RNNs) to process audio signals, which introduce time-sequential information to the model. In convolutional recurrent networks(C-RNNs), the CNN component is used to extract features while RNN plays the role of summarizing temporal features. The inputs of C-RNNs are soundtrack spectrograms and outputs are probabilities of each genre at each timestep. Inspired by previous literature, we propose to leverage the idea by augmenting LSTM autoencoder with CNN and use a Clustering-based classifier to predict the genre of music.
Music genre classification has been actively studied since
the early days. Tzanetakis and Cook [Tzanetakis and
Cook2002] used k-nearest neighbor classifier and Gaussian
Mixture models with a comprehensive set of features for
music classification. Those features could be summarized
into three categories: rhythm, pitch, and temporal
structure. Zhouyu Fu [Fu et al.2010] proposed a Naive Bayes
(NB) classifier framework, namely NB Nearest Neighbor
(NBNN) and NB Support Vector Machine (NBSVM) for
music genre classification. [Deshpande and Singh2001]
compared k-nearest neighbor, Gaussian Mixtures and SVM
to classify music into three genres which are rock, piano,
and jazz. In recent years, using an audio spectrogram has
become mainstream for music genre classification.
Spectrograms encode time and frequency information of given
music as a whole. Spectrograms can be considered as images
and used to train convolutional neural networks (CNNs)
In this paper, we aim to combine convolutional nets with LSTM Autoencoders to extract both spatial and temporal features of the audio signal. Instead of baseline classifiers, we propose a clustering-based classification model. In the proposed classification approach we cluster the data based on their inherent characteristics and in the process of learning the best clustering solution we optimize the hyperparameters of the classification model, thereby substantially improving the learning process. We used the mel-spectrogram as the only feature and compared the proposed model with traditional classifiers and previous literature.
Dataset In the paper, we have used the GTZAN dataset. It contains 10 music genres, each genre has 100 audio clips in .au format. The genres are - blues, classical, country, disco, hip-hop, pop, jazz, reggae, rock, metal. Each audio clips has a length 30 seconds, are 22050Hz Mono 16-bit files. The dataset incorporates samples from a variety of sources like CDs, radios, microphone recordings, etc. The training, testing and validating sets are randomly partitioned following proportion 8:1:1.
Features A popular representation of sound is the spectrogram which captures both time and frequency information. In this study, we used the Mel spectrogram as the only input to train our neural model. A mel spectrogram is a spectrogram transformed to have frequencies in the mel scale, which is logarithmic, more naturally representing how human senses different sound frequencies. To convert raw audio to Mel spectrogram, one must apply Short Time Fourier Transforms(STFT) across sliding windows of audio, around 20ms wide.
In this case, the music features are extracted using the LibROSA library in Python using 128 mel filters, frame length of 2048 samples and a hop size of 1024. We got a spectrogram of size 647 128.
The model consists of a four-layer convolutional neural network (CNN) which is followed by an LSTM Sequence to Sequence Autoencoder(AE) and ultimately consists of the proposed CALM classifier. Not only to make the network unconstrained of any handcrafted features, but the convolutional layers are also used to extract meaningful and useful features from the song. The output of the CNN is a sequence in which every timestep strongly relies on both the immediate predecessors and long term structure of the entire song. To capture both transient and overall characteristics, we use LSTM Sequence to Sequence Autoencoder and for classification, we propose CALM, which is explained in the following sections. The assumption underlying this model is that the temporal pattern can be aggregated better with LSTM Autoencoders than CNNs while relying on CNNs on input side for local feature extraction.
The CNN architecture consists of 4 convolutional layers of 64 feature maps, 3-by-3 convolution kernels and maxpooling layers of dimensions (2×2)-(3×3)-(4×4)-(4×4). In all convolutions, we pad zeros to each side of the input to keep size fixed. Dropout(0.5) is applied to all convolutional layers to increases generalization. The CNN output has a feature map size of N×1×15 (number of feature maps×frequency×time). For extracting temporal pattern we use an LSTM-based architecture. The architecture uses LSTM layers having f256,64,16g units as the encoder LSTMenc and LSTM layers having f64,256g units as the decoder LSTMdec.
Methodology To start, features are extracted from the spectrogram using convolutional layers. The output of Convolutional Neural Networks is fed to an LSTM Seq to Seq Autoencoder which collects key information about the temporal properties of the input sequence in its hidden state. The final hidden state of the LSTMenc is then passed through some layers, the output of which is used to initialize the hidden state of the LSTMdec. The function of the LSTMdec is to reconstruct the input sequence based on the information contained in its initial hidden state. The network is trained to minimize the root mean squared error between the input sequence and the reconstruction. Once the training is complete, the activation of the fully connected encoded layer is used as representations of the audio sequence and is fed as input to Clustering Augmented Learning Method Classifier. This system showed 98% accuracy at the end of the training.
Input augmentationAs in [Ghosal et al.2019], we consider a matrix of input data D and a set of cluster centers C. Since in this case study, there are 10 music genres, we keep C as 10. In this paper, we use clustering to augment input data x 2 D for better learning. To augment the input data, we add a new set of features representing either an input example belongs to a cluster or not. To distinguish input examples, we introduce an additional index h 2 f1; : : : ; jDjg representing the number of an input example (x1 is the first input example of D). We define also a vector ch composed of chl, l 2 C for each example xh 2 D. It is a one-hot representation containing zeros except for the index of the cluster it belongs to (e.g. c1 = [0; 0; 0; 1; 0; 0; 0; 0; 0; 0] means that the first input example x1 belongs to the 4th cluster out of 10 clusters). Finally, we augment input examples by concatenating the vector xh with the vector ch for each h 2 f1; : : : ; jDjg.
Cluster centers To determine the cluster centers, CALM consists of a clustering model and a Feed-Forward Neural Net(FNN) having a softmax output to classify the music genres. For the clustering model, we propose to use a Random Forest classifier to determine cluster centers. After the FNN is trained using a state-of-the-art solver for data belonging to a single cluster 2 f1; : : : ; jCjg, a Random Forest Classifier is used to find the best cluster center. Hence we repeat jCj instances of training the FNN to find the jCj centers. For any instance l of the model, we use the one-hot encoded vector of l as labels for all the input sample in that cluster. In simple words, while predicting center of 4th cluster (for example) we use [0; 0; 0; 1; 0; 0; 0; 0; 0; 0] as label for all input samples, since jCj is 10.
We propose that the input sample which has the lowest error in predicting its cluster label is considered as the center of that cluster in the subsequent iteration of the proposed approach. In such a manner, the center would be the input sample which is the most fitting representative of that cluster. As a result, the clustering process would aggregate the data having similar characteristics resulting in better learning by the FNN classification model.
We have a distance/dissimilarity measure dil between input examples i 2 D and cluster centers l 2 C. The clustering problem aims to assign each input example to a cluster such that the total distance between the elements of a cluster and its center is minimized.
In this paper we also propose a novel dissimilarity measure based on the weights of the trained FNN classifier. It uses the average of weights linked to each neuron of the input layer. Assuming that the original input (without the new clustering feature) has d dimensions (xh = [x1h; : : : ; xdh]; h 2 f1; : : : ; jDjg) and the weight linking node n of the input layer to node j 2 f1; : : : ; n1g of the following layer is wjn, the two distances measures are formulated as follows: dil =
P
avg n2f1:::dg j2f1;:::;n1g wjnjxi k
k xl j Thus the distance measure computes the distance between two examples based on how important is the contribution of each input feature to the resulting prediction. Therefore, the resulting clusters contain examples with similar potential to improve the classification results.
We propose an approach (Algorithm 1) where we iteratively train the FNN classifier, use its weights for input data clustering thus changing the input vector, train again the FNN classifier using the new input data, and so on until a stopping criterion is attained. The stopping criterion is triggered if the cluster assignment remains the same for consecutive 10 iterations, i.e., the clustering problem converges. The configuration of the proposed model is given as: where NC is the number of accurately predicted music tracks, NF is the number of falsely predicted music tracks, NM is the number of missed music tracks, totalc is the number of all accurately predicted music tracks and totalm is the number of all music tracks.
To further interpret the results, we plotted the confusion matrix(Table 3) of the proposed model. Looking more closely at our confusion matrix, we see that our proposed model managed to correctly classify 80% of rock audio as rock, labeling the others as mainly country or blues. Additionally, it incorrectly classified some country, as well as a small fraction of blues and reggae, as rock music.
Comparison with Baseline Classifiers We trained four traditional classification models on the dataset as baseline classifiers, including k-nearest neighbors, logistic regression, random forest, multilayer perceptrons, and linear support vector machine, using Mel Frequency Cepstral Coefficients(MFCCs) by flattening them into a 1-D array. Apart from baseline classifiers we also experimented by stacking a Logistic Regression classifier with the features obtained from Convolutional Net and LSTM Autoencoder to test the performance of the CALM classifier. As evident from Table 1, CALM outperforms the Logistic Regression classifier when augmented with Convolutional nets and LSTM autoencoder. Moreover, Fig.4 shows how the intra-cluster variance decreases after approximately 75 iterations and then stabilizes. To measure intra-cluster variance, we used Euclidean distance in this case study. Similarly, it is evident from Fig. 3 that testing loss starts decreasing after 80 epochs and gradually as the clustering solution converges, the accuracy begins to improve. This observation bolsters our initial assumption that clustering data based on inherent characteristics would improve the learning process of FNN. For a fair comparison, all models are trained and tested on the same dataset as the proposed model. The hyperparameters are tuned by a grid search to ensure that the best model configuration is adapted. In table 2 we have also compared our model with relevant literature and it is evident that proposed architecture performs strongly.
In this paper, we present a specially designed network for accurately recognizing the music genre. The proposed model Model aims to take full advantage of low-level information of Melspectrogram for making the classification decision. We have shown how our model is effective by comparing the stateof-art methods, including both hands crafted feature approaches and deep learning models. In this work, we use the GTZAN dataset which is a common benchmark dataset. Our proposed model has achieved an impressive accuracy of 95.4% while testing, which outperforms all other models. In the future, we will try to improve the model by improvising some new distance metric methods to compute the similarity between genres.
Blues Classical Country
Disco 0 0 82 3 0 0 0 0 0 7