Advances in machine learning have the potential to radically reshape interactions between humans and computers. Deep learning makes it possible to discover powerful representations that are capable of capturing the latent structure of highdimensional data such as music. By creating interactive latent space “palettes” of musical sequences and timbres, we demonstrate interfaces for musical creation made possible by machine learning. We introduce an interface to the intuitive, low-dimensional control spaces for high-dimensional note sequences, allowing users to explore a compositional space of melodies or drum beats in a simple 2-D grid. Furthermore, users can define 1-D trajectories in the 2-D space for autonomous, continuous morphing during improvisation. Similarly for timbre, our interface to a learned latent space of audio provides an intuitive and smooth search space for morphing between the timbres of different instruments. We remove technical and computational barriers by embedding pre-trained networks into a browser-based GPU-accelerated framework, making the systems accessible to a wide range of users while maintaining potential for creative flexibility and personalization.
Author Keywords
musical interface; latent space; variational autoencoder; deep
learning
INTRODUCTION
Music, when treated as data, is often represented in
highdimensional spaces. Digital music formats such as CD-quality
Pulse-code modulation (PCM), for example, records audible
vibrations as a discrete sequence of 44.1 thousand 16-bit
integers per second [
MILC ’18, March 11, 2018, Tokyo, Japan * Research conducted while this author was at Google Brain. of this, for example, we could represent sequences of monophonic 16th notes of equal intensity with approximately 7 bits for each note, or 112 per bar. That is far fewer dimensions than audio, but even there, exploring all possible variations of a score by flipping one bit at a time quickly becomes intractable, and further, would result in a large proportion of melodies being so musically unconventional that they would easily be perceived as being incoherent.
While the high dimensionality affords an exponential
number of possibilities, only some of these possibilities are likely
for real music, which could be seen as residing on a
lowerdimensional manifold within the space. Machine learning
techniques can learn the shape of such low-dimensional
manifolds from data, and be used to gain better understanding of
and to explore large datasets [
Autoencoders and variational autoencoders [
In this paper, we present two interfaces based on the principle of musical latent spaces:
A musical sequence explorer for 2-bar melody loops and
drum beats, using the latent space of MusicVAE [
MMorph [
The Hyperviolin [
The Morph Table [
Existing examples of AIA interfaces using latent spaces
include examples for generating faces [
METHODS
Both of the interfaces introduced in this paper employ
autoencoders, unsupervised machine learning models each composed
of three parts: an encoder, a latent vector, and a decoder
(Figure 1). The latent vector, z, is an intermediate representation
of the data examples, x, but is either of lower dimension or is
regularized to produce an information bottleneck. In the case
of variational autoencoders (VAE), z is regularized to
encourage the model to learn low-entropy latent representations of
x close to a standard multivariate Gaussian distribution. The
encoder is a neural network that produces z from the input x
and the decoder is a separate neural network that attempts to
reconstruct x from z. Gradient-based optimization is then used
to reduce the reconstruction loss (the difference between the
encoded input and decoded output) and KL divergence from
the Gaussian prior (if applicable) [
The purpose of the bottleneck is that it forces the model to distill the information content of the data to lower-entropy representations. In the process, it discovers and encodes the most important features that differentiate data examples. The model becomes specialized to produce examples like those from the training distribution, making it difficult for the decoder to produce “unrealistic” outputs that are improbable under the distribution p(x). However, when trained properly it is general enough to be able to reconstruct and generate examples that are from the same distribution as p(x) but do not appear in the train set. The resulting latent space is therefore optimized for various forms of exploration including interpolation within the latent space, which can “morph” between values in the output space, producing realistic intermediate outputs that combine features of the endpoints to varying degrees.
MusicVAE
MusicVAE is a VAE for musical sequences, such as drum beats
and melodies. It uses LSTM [
To produce good reconstructions (and interpolations), we train a VAE with a trade-off parameter that assigns a lower weight to the KL divergence, thus allowing the system to pass enough information through the bottleneck to be able to reproduce nearly any realistic input sequence in our dataset. However, models trained in this way on noisy datasets such as ours often do not produce realistic examples when sampling random points from the Gaussian prior. To produce better random samples, we train VAEs with the trade-off parameter set to a value that encourages a better KL divergence at the expense of reconstruction quality.
NSynth
With NSynth, we are able to learn latent spaces not just of
musical sequences, but of audio waveforms themselves. Like
the MusicVAE, NSynth is an autoencoder architecture (though
not variational) that can learn to encode and decode sequences
from a compressed latent space. Since the model must learn
to most efficiently use the lower-entropy representation to
represent waveforms, it learns salient acoustic features among
the training data. However, the model architecture differs
significantly from MusicVAE as it must capture correlations
over thousands of quantized waveform timesteps [
We train the NSynth model on the NSynth dataset, a collection
of ~300,000 individual 4-second notes recorded from ~1,000
separate instruments and synthesizers [
Since the model learns an expressive code for raw audio, it can be used to interpolate in this space and discover new instrument timbres that exist between pre-existing instruments. However, while the latent space is of much lower dimension than the original audio, there is no prior to sample from as was the case for MusicVAE. Therefore, there is no trivial way to randomly sample novel musical notes from the distribution, and we are limited to exploring subspaces anchored on known notes.
INTERFACES
MusicVAE Sequencer
The MusicVAE Sequencer (Figure 2) is an interface to the
latent spaces of 2-bar drum loops and monophonic melodies, as
described above. Users can toggle between these two spaces
and define the four corner sequences of a 2-D, 11x11 grid by
randomly sampling from the latent space (using the low-KL
model), selecting from a predefined list, or inputting them
manually into the sequencer at the resolution of 16th notes.
Once the corner sequences are set, they are passed through
the encoder of the VAE to determine their latent vectors. The
latent vectors are then mixed using bi-linear interpolation in
the 11 11 space, and the resulting values are decoded into
sequences (from the high-KL model). With our implementation
of the model architecture in deeplearn.js using weights learned
via the original TensorFlow [
Now that the palette is filled, the user can drag the white “puck”
around to hear the drum beat or melody loop in each square
of the grid. When the puck moves to a new square, the
sequencer immediately updates but the play-head does not reset
to the beginning, allowing the music to morph immediately
but smoothly. The sound is played via the browser using preset
audio samples with Tone.js [
The corner sequences can be changed at any time by the same methods mentioned above or by dragging one of the interior squares to a corner of the grid, which will set it as the new corner and cause the interior points to be updated. If using the sequencer as a composition tool, the user can record the MIDI or audio externally, or she can right click on any square to download a MIDI file containing that particular sequence to use as a part of an arrangement.
In the case of improvisation or live performance, the user may also use the “draw” tool to define a 1-D curve within the palette, which the puck will then move through at a rate the user controls. As the puck moves into each new square, the playback sequence is updated, just as if the user had moved the puck there by hand. In this manner, a performer can set up paths in both the drum and melody palettes that will continuously evolve at potentially different rates, introducing musically interesting phasing effects.
NSynth Instrument
The NSynth model is extremely computationally expensive
(~30 minutes of GPU synthesis to generate four seconds of
audio) and therefore presented a distinct challenge in creating
an interactive experience for interpolating within the latent
space in real-time on a laptop. Rather than generating sounds
on demand, we curated a set of original sounds ahead of time
and synthesized all of their interpolated latent representations.
To produce even finer-grained resolution during playback,
transitions are smoothed out by additionally mixing the audio in
real-time from the nearest neighbor sounds on the grid. This is
a straightforward case of trading off computation for memory.
We created a playable instrument integrated into Ableton Live
as a Max For Live device (Figure 3). We positioned real
instrument timbres at the corners of a square grid, allowing
the user to mix between all four using the latent space as a
palette. We created further “multigrid” modes, tiling many
four-instrument grids side by side. This results in an 7x7 grid
of 4x4 grids, enabling a user to explore up to 64 different
instruments by dragging across a single x-y pad. Given an x-y
point, the device finds the nearest four interpolated samples
and plays them back with volume proportional to the their
x-y distance from the point. Users can choose from 5
pregenerated grids and 3 multigrids or produce their own using
the TensorFlow code released with the NSynth model [
Finally, users are able to map all of these settings to an external
MIDI controller for ease of use in a studio or live setting.
EXPLORING THE INTERFACE
Interfaces provide mappings from the user’s actions to the
output space. In the case of the MusicVAE sequencer, the input
space consists of four user-defined basis points (the corners)
combined with a 2-D control surface (the palette), and the
output space consists of note or drum sequences. Generally,
choosing the mapping is a crucial element of designing a
user interface, with numerous trade-offs at play, and where
the goals often include carefully and deliberately shaping the
user’s cognitive model of the system [
One example of exaggeration is that we set three of the basis points (bottom-left, top-left, and top-right) to all be empty, and set the bottom-right to be entirely full (i.e. every percussion instrument playing at every 16th note). In that case, our interpolation lets us check what the system does as it goes from completely sparse (silence) to completely dense, as shown in Figure 4.
As we interpolate from empty to full, we notice that the system tends to start by populating the sequencer with percussion hits on the actual beats, then on the 8th notes, and then on the 16th notes, i.e. it progresses from more likely to less likely beats. It also progresses from sparser beat sequences to denser ones. This satisfies the requirements outlined in the introduction as follows: (1) smoothness is preserved, in that each sample along the interpolated latent path corresponds to beat sequences that are similar to (i.e. a little more or less dense than) those corresponding to its adjacent samples, and (2) by first adding beats on quarter notes, then 8th notes, etc, this suggests that the system is maintaining feasible (i.e. more likely) outputs1.
Figure 5 shows another simple experiment in which we try isolating the location (phase) of the beat in the bar. We see that 1For example, one could imagine a sequence that goes from sparse to dense, but in which the new percussion hits are added by simply choosing random cells in the grid from a uniform distribution over the grid. This could still be a smooth mapping, but it would have the problem that the intermediate beat sequences, during interpolation, would generally be very unlikely. rather than interpolating the phase (which would gradually shift the offset), there is a tendency to superpose the two rhythmic figures. This, too, is a reasonable rhythmic approach. We note that doing so in melody space would not necessarily be as natural of a choice (and in fact the system would not have that option since the melodies are constrained to be monophonic). While there is usually a lot more sophistication occurring in the interpolations aside from superposition, these results for simple cases have a natural interpretation for the user, providing landmarks on which to ground their understanding and cognitive models of the system.
In Figure 6, we explore interpolation in melody space by providing several different scales as the four basis points. As we move from top-left to top-right, we notice that at each step the output sequence is similar to the previous step, but that the combination of these smooth steps ultimately results in a very large change. Interestingly, at around a third of the way across, the output sequence includes some descending notes, even though both scales ascend. This shows that the system will sometimes move away from the basis points during the interpolation.
Finally, we note that some parameters, such as the number of grid points, were chosen in order to make the system functional, with the focus on learning an effective latent space mapping, but in future, these are design elements that would be worthwhile exploring more systematically and with user testing.
AVAILABILITY Supplementary resources including opensource code are available for both NSynth and MusicVAE interfaces in corresponding sub-directories of Magenta’s demo github repository2. CONCLUSION This work demonstrates the use of machine learning as the basis for creative musical tools. Machine learning is often seen as a method for outsourcing mundane discriminative tasks, resulting in the desire for rigid systems that perform their duties 2https://github.com/tensorflow/magenta-demos/ with high accuracy. While it is likely that such systems could be used to produce satisfying music for listeners, our work indicates directions for how we might use machine learning— specifically latent-space representations— to create UI mappings that we hope will eventually provide music creators with new and interesting creativity-supporting tools.
ACKNOWLEDGMENTS We thank Torin Blankensmith and Kyle Phillips from Google Creative Lab for implementing the MusicVAE Sequencer UI. We thank Nikhil Thorat for assistance with the deeplearn.js implementation. We thank the members of the Magenta team for helpful discussions and observations.