-Even though algorithmic music has been around the world since the old days, it has never attracted as many researchers as in the recent years. To our knowledge it existed in Iran back in the Middle Ages and in Europe during the Age of Enlightenment. Though the form has changed and it has grown layers of complexity, the very foundations of the algorithm that generates musical compositions have not changed, i.e. most of them are based on structures of fortuity. Additionally, models that are able to learn have been discovered allowing us to imitate the music of the incredible artists throughout history. The thought alone is crazy to think of and seems to be from the sci-fi. In this paper, a research trying to find the best model of an echo state network in order to mimic the music of the legendary Wolfgang Amadeus Mozart has been carried out. As it turns out, the best models are the ones that rely on long-term dependencies.
Algorithmic music is by no means a new trend in our techy
world. In fact, three Iranian brothers collectively known as Banu
Musa were successfully devising automatic and even
programmable musical instruments back in 850 AD [
In this article we are trying to imitate classical piano music. For a quantitative rather than qualitative analysis only one composer was chosen. Mozart has been opted for his indisputable genius and some haphazardness.
Rather than working with sound signals, we chose to work with notes for several reasons. Firstly, it is a lot less intricate. Therefore, it is a lot easier for us to understand and analyse it as well as it is for the algorithm in the means of computational resources and dependency on previous notes. In the note level we are also able to compare it with the musical theory. And it helps us stay in the realm of classical music as well.
Furthermore, we are lucky enough to have the MIDI (musical instrument digital interface) protocol for a .mid is a musical file format that captures the notes, the times piano keys were pressed and released, how strong they were pressed etc. MIDI supports 128 notes whereas general pianos usually provide 88 keys.
Musical composition has been one of the long term goals of
artificial intelligence (AI) [
As music is a sequence of notes, a sequential model was
chosen to train on Mozart’s music. Markov models are not very
suitable for this task due to their monophony (although it is
possible to design a system for polyphonic music as well).
Currently, the cutting-edge approach to generative music
modelling is based on recurrent networks [
For these reasons, we chose to work with a type of recurrent neural networks – echo state network (ESN) – that have barely been researched for musical composition.
Musical data were downloaded in the format of .mid from the website http://www.piano-midi.de/. From now on MIDI and .mid will be used interchangeably meaning the same, i.e. the file format unless stated otherwise, e.g. MIDI protocol. In total, 21 pieces by Mozart were gathered (all that are found on the website).
MIDI format is a sequence of notes (and commands such as tempo change and sound perturbations) whereas the time difference is represented in ticks. A quarter note is usually 480 or 960 ticks but that depends on the resolution. Thus, a full note or, in other words, a tact is 1920 or 3840 ticks respectively.
Later on, the data had to be transformed in a format that is easier to read, maintain and process. Hence, it was read and transformed into notes as messages into a .csv (comma separated values) format.
Every message consists of information of this type: note pitch on tick off tick length
The length parameter is not in the MIDI file and had been artificially generated for the purpose of data analysis.
Table I shows the types of information as well as their ranges in a message. Note pitch ranges from 0 to 127, thus a byte is more than enough to store it. The beginning and the end of a note tick is undetermined and can grow to infinity when the data grows. Length parameter is purely the difference between on and off ticks. It may grow to a large number due to software bugs or a divergence of the algorithm, but usually it shall stay in the realm of classical music and get a value up to a full note. Info Note pitch On tick Off tick Length Type byte long long integer Range 0-127 0-infinity 1-infinity 1-full note
A message in MIDI that signifies the event of pressing a note is the note_on message. It represents an event when a note is released as well, only the velocity then is equal to zero. The iterate through the messages: --check if it is a ‘note_on’ type of message: ----if velocity > 0: ------take the time of the note that was pressed ----else if velocity equals 0: ------check if the actual note was pressed: --------release the note --------measure the length of the note ----------append the note as a message to the CSV algorithm (Fig. 1) that was applied for treatment of raw MIDI files looks as following: Fig. 1. Algorithm of raw MIDI file treatment into a CSV file
The programming language of choice was Python due to its
recognition in data science and machine learning among
scientists and developers. Also, due to the many data processing
as well as machine learning libraries although none of the
machine learning libraries were used for this work. For the
purpose of .mid processing, Mido library was chosen [
Machine learning algorithms perform better under more data. We could have just taken in all of the composers from the website full of classical MIDI files, but we chose only one for the purpose of thorough analysis. Despite the fact that our choice was only Mozart’s music and that had given us only 21 pieces of scores, this resulted in around 68 thousand notes.
Prior to the research, an analysis of the data was performed based on the distribution of note pitches as well as their lengths. As we can clearly see in Fig. 2, there are 2 maximums. One is of a higher pitch while the other is of a quite lower pitch. This is most probably due to the fact that piano is played by 2 hands and that the left hand usually wanders in the region of lower pitch notes whilst the right hand sits in the region of higher pitch notes.
These data are not so much relevant for our research, but provide us with insights such as it would make perfect sense to study the hands in more detail. We ought to bolster our research either by adding an additional dimension of the hand or by having 2 different outputs for each hand by the network. This analysis is also useful for future comparison and judgment of generated music.
Analysis of lengths (Fig. 3) provide us only one maximum, meaning both hands share the same maximum or that the note lengths of one hand are very dispersed.
Echo state networks supply an architecture and principles of
supervised learning for recurrent neural networks. The idea
behind an ESN is to drive a large, random and fixed reservoir of
neurons with the input signal (Fig. 4). Thence, inducing each
neuron within it with a nonlinear response signal. After,
combine the desirable output data by a trainable linear
combination of all of these response signals [
Echo state network may be tuned by altering the following parameters: leaking rate input scaling spectral radius
Leaking rate of the network can be regarded as the speed of the reservoir update dynamics in discrete time.
Another key parameter to optimize an ESN is the input scaling. It multiplies the input weight matrix Win by its value either strengthening the input weights or diminishing them.
Spectral radius is one of the most global parameters of an
ESN, i.e. the maximum absolute eigenvalue of the reservoir
weights matrix W. It scales the matrix W, or in alternate words,
scales the width of the distribution of its nonzero elements [
The number of neurons inside the reservoir has been opted be equal to 1000.
Programming code for ESN has been adapted from [
Research has been accomplished in a manner that can be seen in Fig. 6. First of all, music was accumulated in .mid format (hex code). As stated before, it was processed by Mido library and stored in a .csv format in a form of messages that carry the information of notes as the pitch number, on and off ticks and length.
Then the messages were read from the .csv file and quantized. Quantization was performed for the beginning and the end of the notes in the following way. A quantization unit of 60 ticks (represents a 32nd of a note) was chosen. Next, if the residual value of the tick was less than half the quantization unit, it was reduced by the residual. If the residual value was equal or higher than half of the quantization unit, i.e. 30 ticks, it was increased by the difference between the quantization unit and the residual. The lengths of notes were recalculated afterwards.
In Fig. 5 we can see the distribution of the notes after quantization. Hereby, the number of notes of the length of the quant (60 ticks) has increased. The most frequent note stayed the same (120 ticks). Also, a tiny part of the very shortest notes was quantized to zero length, thus, eliminated.
As a further step, these quantized music messages were turned into a state matrix of length that is equal to the division of the total length of the pieces by the quantization unit rounded to integer. Another dimension of the state matrix were the note pitches, that is 128 values in total. Then the value at each time step at a certain note represents its state (1 for pressed and 0 for not pressed). 80% of the data were sent to the echo state network whilst 20% were used for validation of the model, thus finding out the error. Error was calculated in the shape of root mean squared error (RMSE).
An ESN was generated according to given parameters. This ESN was then trained on input and predicted music based on its learned weights as a one time-step prediction. The training process was initialized by 300 time steps, that is by 300 quants (60 ticks).
To find out the best parameters for our echo state network, we would repeat the procedure of generating the network according to different parameters and training the new network model on the very same data. Then we predicted next notes based on the newly gained weights and found out the error by comparing with the original Mozart data. Prediction of notes was a sequel of the training process. To be more precise, the model predicted notes as a one time-step prediction. Summarizing, a grid search analysis of 4 parameters of the echo state network has been performed.
Parameters that have been investigated for tuning our network are the following. Leaking rate, input scaling, spectral radius and regularization which are the most important ESN parameters explained in Section IV. Since their ranges usually go from 0 to 1, 0 to 2, 0 to 2 and almost anything respectively, they have been tested for values in these ranges. An exhaustive grid search analysis had been performed looking for the best parameters. In addition to RMSE, mean and standard deviation were calculated. Original Mozart music had the mean of 0.04238 and standard deviation of 0.0779. Mean represents the probability of a note to played at each time step in the note spectrum. In Mozart’s case note spectrum is from the 29th to the 91st note. Standard deviation represents the mean of standard deviations of the notes in the note spectrum.
Leaking rate has been tested from 0.0025 to 1, spectral radius varied from 0.0015 to 2 in this test, input scaling from 2*10-6 to 2 and regularization from 10-6 to 105.
As we can see from the sorted by error (top 10) Table II, the lowest value of error (RMSE) is a tiny bit above 0.0307. It is clear that the best leaking rate for our model is about 0.025 while the combination of input scaling and spectral radius vary a little bit. Input scaling goes from 0.002 to 0.0002 and spectral radius from 0.01 to 0.1. We can notice that while RMSE is the lowest, the mean of the notes is about the same of the quantized original Mozart music data mean but standard deviation is quite different.
reg stand for regularization, rmse stands for RMSE and std stands for standard deviation in the tables of error (Table II, Table III, Table IV). 0.025
Having low leaking rate suggests us that the state has a lot of inertia and the change of the state is slow. Input scaling scales the Win matrix, thus, the input weights are very low and the model depends on its input just a tiny bit. Since it is lower than the spectral radius, it has a lot of memory, i.e. follows a longterm dependency. Having low spectral radius as well tells us that the models are almost linear. To summarize, the prediction function is not very complex and the model has a lot of memory.
From Table III we see that a high regularization value gives us huge errors. It has to be noted that for this particular grid search step, the maximum value of input scaling and spectral radius was 0.2. Thus, we can also deduce that high input scaling values lead to higher error. Though leaking rate is not as important, we can still see that some of its higher values lead to higher errors.
High regularization significantly reduces the mean value and standard deviation of the notes.
reg 100000 100000 100000 100000 100000 100000 100000 100000 100000 100000 mean 0.0345 0.0357 0.035 0.0352 0.0353 0.0354 0.035 0.0352
In order for us to see tendencies beyond regularization, we filtered the data for regularization below or equals 10. This brought us back to the maximum values of input scaling, spectral radius and leaking rate.
In Table IV we see that high input scaling produces high error once again. Interestingly, leaking rate stays at 0.25 for the highest error. Although spectral radius stays quite high, it is not of the highest value for the highest error. Mean is almost as with the best results. Standard deviation is higher in this case than with the best results. It is even closer to quantized Mozart’s music standard deviation than the one provided with the best results. 2 2 2 2 2 0.8 0.8 0.8
Fig. 7 shows us the minimum error dependency on leaking rate. It is worth to note that although leaking rate 0.25 yields worst results when regularization is not high, it may also yield very good results with other values of ESN parameters as can be seen in Fig. 7.
The errors were grouped by leaking rate and the minimum value of the error was taken to plot the dependency graph. In Fig. 8 we can see the most promising region of leaking rate for our echo state network.
Fig. 9 shows us the minimum error dependency on input scaling whereas Fig. 10 zooms us to the most promising region of input scaling. The best values of input scaling are 0.0002 and 0.0005. Going even lower, the values increase dramatically. Fig. 9. Minimum RMSE dependency on input scaling.
Fig. 11 shows us the minimum error dependency on spectral radius. From Fig. 12 and Fig. 13 we can see that the minimum RMSE stabilizes and reaches the minimum on spectral radius below 0.1. Then starts growing again above 0.01.
Fig. 14 and Fig. 15 implies us that the best regularization values are of the power 10-4 to 10-2.
Since it was a lot easier to find the optimal leaking rate than input scaling and spectral radius, we grouped the errors by input scaling and spectral radius taking the minimum RMSE value in Fig. 16. Regularization is an additional parameter that prevents overfitting and we have not grouped by it. It was quite easy to find as well.
As it was found out that the most optimal leaking rate is 0.025, the errors were grouped by input scaling and spectral radius once again by a set leaking rate. Now they we grouped having leaking rate set to 0.025. In Fig. 17 we can see pointy triangles in the lower region where the errors are the lowest.
It has to be taken into account that producing an even finer grid might give us even better results but this takes time. Also, it seems from all this analysed data that the reduction in error would be quite low.
We can affirm that the best value of leaking rate in our research proved to be 0.025. The best values of input scaling are 0.0005 and 0.0002 whereas the most optimal values of spectral radius and regularization vary from 0.1 to 0.01 and from 10-4 to 10-2 respectively. Having said that, the values will not produce the best results in separation, they will only produce the best results in a proper combination with other variables as it can be seen in the tables and figures.
To summarize our research, we can state that to predict Mozart’s music, one has to memorize a lot of the notes in order to predict the next note. In the terms of our echo state network, it has to follow a long-term dependency because the input scaling is lower than the spectral radius. Having low spectral radius as well implies that the prediction function ought to be quite simple because the reservoir operates in an almost linear regime.
Our main aim is to produce good music so that people would like to listen to it. To achieve this goal, we analysed the best models to replicate Mozart’s music. Lately, we have been planning to include information of piano hands into our composition model. In the future work we would like to expand the dimensions of this research since MIDI files have additional information such as the velocity of the pressed note as well as tempo changes and sound perturbations. We are also eager to expand this study for more great composers and then tune our models to not only imitate but also generate new music that people would value. If echo state networks do not prove to be deep enough, we are determined to broaden our research including deep learning models such as hierarchies of regular recurrent neural networks or long short-term memory networks and other recurrent types. We could then compare them and possibly combine the best parts of them. We are hoping that the artificial network is able to learn the rules or tendencies of music theory implicitly, at least partially. If this is not the case, we could augment it with heuristics.