Here, we demonstrate Tone Transfer, an interactive web experience that enables users to use neural networks to transform any audio input into an array of several diferent musical instruments. By implementing fast and eficient neural synthesis models in TensorFlow.js (TF.js), including special kernels for numerical stability, we are able to overcome the size and latency of typical neural audio synthesis models to create a real-time and interactive web experience. Finally, Tone Transfer was designed from extensive usability studies with both musicians and novices, focusing on enhancing creativity of users across a variety of skill levels.
did this through an interactive, in-browser creative experience.
Neural audio synthesis, generating audio with
neural networks, can extend human creativity by
creating new synthesis tools that are expressive and intu- 2. User Interface Design
itive [
Tone Transfer is a musical experience powered by • Play with curated music samples. To understand Magenta’s open source DDSP library1 to model and what DDSP can do, the user could click to lismap between the characteristics of diferent musical ten to a wide range of pre-recorded samples and instruments with machine learning. The process can their machine learning transformations in other lead to creative, quirky results. For example replac- instruments. ing a capella singing with a saxophone solo, or a dog barking with a trumpet performance.
Tone Transfer was created as an invitation to novices and musicians to take part in the future of machine learning and creativity. Our focus was on cultural inclusion, increased awareness of machine learning for artists and the general public, and inspiring excitement of the future of creative work among musicians. We • Record and transform new music. We also provided options for users to record or upload new sounds and transform them into four instruments in browser. • Adjust the music. We know that control is important for the user so we allow the user to adjust the octave, loudness, and mixing of the machine learning transformations to get desired music output.
how to use Tone Transfer as well as the machine learning technology behind it. Therefore, we designed tips that guide the user through the experience and educate them on the best ways to interact with it. The user
At a technical level, the goal of our system is to be able to create a monophonic synthesizer that can take coarse user inputs of Pitch and Loudness and convert them into detailed synthesizer coeficients that produce realistic sounding outputs.
We find this is possible with a carefully designed variant of the standard Autoencoder architecture, where we train the model to: can also learn the training process of machine learning ing inference we use the SPICE model, which is faster models by clicking the "Discover more" button. and has an implementation available in TF.js (https: //tfhub.dev/google/tfjs-model/spice/2/default/1).
While the original DDSP paper used perceptually
3. Models weighted spectrograms for loudness, we find that the
root-mean-squared (RMS) power of the waveform works
well as a proxy and is less expensive to compute. We
train on 16kHz audio, with a hop size of 64 samples
(4ms) and a forward-facing (non-centered) frame size
of 1024 samples (64ms). We convert power to decibels,
and scale pitch and power to the range [
The decoder converts the encoded features ( 0, power) into synthesizer controls for each frame of audio (250Hz, • Decode: Use a network to convert pitch and loun- 4ms). As we discuss in Section 3.3, for the DDSP moddess into synthesizer controls. els in this work, the synthesizer controls are the harmonic amplitude ( ), harmonic distribution ( ), and • Synthesize: Use DDSP modules to convert syn- filtered noise magnitudes.
thesizer controls to audio. The DDSP modules are agnostic to the model architecture used and convert model outputs to desired con
We then compare the synthesized audio to the orig- trol ranges using custom nonlinearities as described
inal audio with a multi-scale spectrogram loss [10, 8, in [
To extract pitch during training (fundamental frequency,
0), we use a pretrained CREPE network [
To generate audio, we use a combination of additive
(Harmonic) and subtractive (Filtered Noise) synthesis
techniques. Inspired by the work of [
specified by the output of a neural network over dissinusoids with individually varying ( ): cies together, and run through a trainable reverb module. The resulting audio is compared against the original audio with a multi-scale spectrogram loss. Blue components represent the source audio and resynthesized audio. Yellow components are fixed components (pitch tracking, DDSP synthesizers, and loss function), green components are intermediate features (decoder inputs and synthesizer controls), and red components have trainable parameters (decoder layers and reverb impulse response). (1) (2) (3) The network outputs amplitudes
and frequenevery 4ms, which are upsampled to audio rate (16kHz) using overlapping Hann windows and linear interpolation respectively. 3.3.2. Harmonic Synthesizer Since we train on individual instruments with strong harmonic relationships of their partials, we can reparameterize the sinusoidal oscillator bank as a harmonic oscillator, with a single fundamental frequency 0, amplitude , and harmonic distribution . All the output frequencies are constrained to be harmonic (integer) multiples of a fundamental frequency (pitch), −1 =0 ( ) = ∑ ( ) sin( ( )), where ( ) is its instantaneous phase obtained by cumulative summation of the instantaneous frequency ( ) = 2 ∑ ( ), =0 ( ) = 0( )
Individual amplitudes are deterministically retrieved 3.4.1. Data by multiplying the total amplitude, ( ), with the normalized distribution over harmonic amplitudes, ( ): where ,
( ) = ( ) ( ). −1 ∑ ( ) = 1, ( ) ≥ 0 =0 (4) (5)
We train models on four instruments: Violin, Flute,
Trumpet, and Saxophone. Following [
Since DDSP models are eficient to train, for each instrument we only need to collect between 10 and 15 minutes of performance, and we ensure a that all recordings are from the same room environment to allow training a single reverb impulse response. 3.3.3. Filtered Noise Synthesizer We can model the non-periodic audio components as 3.4.2. Optimization a subtractive synthesizer, with a linear time-varying ifltered noise source. White noise is generated from a uniform distribution, which we then filter with an Finite Impulse Response (FIR) filter. Since the network outputs diferent coeficients of the frequency response in each frame, it creates an expressive time-varying filter.
We train models with the Adam optimizer [
To first approximation, room responses with fixed source and listener locations can be approximated by a single 4.1. On-device Inference with impulse response that can be applied as a FIR filter. In Magenta.js terms of neural networks, this is equivalent to a 1-D Musical interaction has strong requirements for close convolution with a very large receptive field ( ∼40k). to real-time feedback and low latency. However, maWe treat the impulse response as a learned variable, chine learning models are typically slow and compuand train a new response (jointly with the rest of the tationally expensive, requiring GPU or TPU servers model) for each dataset with a unique recording envi- to run at all. Further, large model sizes lead to long ronment. load times before execution can even begin. Running
To better disentangle the signal from the room re- models on-device, if possible, eliminates serving costs, sponse, we generate the impulse response with a fil- decreases interactive latency, and increases accessibiltered noise synthesizer as described in Section 3.3.3, ity. To create an interactive and scalable musical exand learn the transfer function coeficients to gener- perience, we optimized and converted models to be ate a desired impulse response. This prevents coherent compatible with Tensorflow.js so that they can run onimpulse responses at short time scales that can entan- device in the browser on both desktop and mobile degle the frequency response of the synthesizer with the vices. room response. At inference, we discard the expen- Even after optimization, the models are still relasive convolutional reverb component to synthesize the tively large (4mb each), so each model is only loaded "dry" signal, and apply a more eficient stock reverb ef- on demand. This ensured the user downloads only the fect. things they need, and nothing more, which resulted in a fast and responsive website. 3.4. Training The methods to extract pitches, and the four modGiven that the DDSP model described above is for mono- els that are on the website are then open sourced and phonic instruments, we collect data of individual in- 2Violin: Five pieces by John Garner (II. Double, III. Corrente, struments, and train a separate model for each dataset. IV. Double Presto, VI. Double, VIII. Double, Flute: Four pieces by Paolo Damoro (24 Etudes for Flute, Op. 15 - III. Allegro con brio in G major, 24 Etudes for Flute, Op. 15 - VI. Moderato in B minor, 3 Fantaisies for Solo Flute, Op. 38 - Fantaisie no. 1, Sonata Appassionata, Op. 140)) from https://musopen.org/music/ 13574-violin-partita-no-1-bwv-1002/ made easier for anyone to download and run their own experiences. Each model comes with a set of custom values that are manually tweaked to create a more accurate output.
These methods are added to the Magenta.js library.3
TensorFlow.js is a web ML platform that provides hardware acceleration through web APIs like WebGL and WebAssembly. DDSP relies on TensorFlow.js to speed up the model execution. To maintain accuracy of DDSP model on a variety of devices, we implemented a couple of special kernels that eliminated overflow ( ( ) > 65504) and underflow ( ( ) < 2−10) of float16 texture when running on the TensorFlow.js WebGL backend.
For example, the DDSP model uses TensorFlow Cumsum op to calculate the cumulative summation of the instantaneous frequency, then obtain the phase from those values. TensorFlow.js implements a parallel algorithm4 for cumulative sum, which requires log(n) writes of intermediate tensors to the GPU textures. The cumulative precision loss would cause a large shift on the final phase values. The solution is to register a custom Cumsum op that uses a serialized algorithm that avoids all intermediate texture writes and is incorporated with the phase computation.
Tone Transfer is an example of an interdisciplinary design, engineering, and AI research teams working together to create a User Interface Design for the next wave of AI. We leverage state-of-the-art machine learning models that are both expressive and eficient, and optimize them for client-side use to enable interactive neural audio synthesis on the web. This work demonstrates that on-device machine learning can enable interactive and creative music making experiences for novices and musicians alike. The technologies that power Tone Transfer have also been open sourced as a part of Magenta.js and provide a solid foundation for further interactive studies. Future work will hopefully allow users to train their own models based on their own instruments, and explore using new types of inputs to create multi-sensory experiences.
ddsp
algorithms
We would like to acknowledge the contributions of everyone who made Tone Transfer possible, including Lamtharn (Hanoi) Hantrakul, Doug Eck, Nida Zada, Mark Bowers, Katie Toothman, Edwin Toh, Justin Secor, Michelle Carney, and Chong Li, and many others at Google. Thank you.