Musical composition is a combinatorial art where composers extend sequences by choosing from a vast set of possible feature combinations that yield the compositions their distinctive qualities. Increasingly, composers are using generative models, such as music transformers, for crafting their pieces. Unfortunately, for composers to “steer” these models to satisfy their qualitative features typically requires retraining (which can be prohibitively expensive); further, existing models are unable to deal with arbitrary combinations of features at scale. In this paper we build on lightweight fine-tuning methods, such as prefix tuning and bias tuning, to propose a novel contrastive loss that enables us to steer music transformers over arbitrary combinations of logical features, with a relatively small number of extra parameters. We provide both quantitative and qualitative evaluations of our method which demonstrate its eficacy with respect to existing methods, as well as a case-study where our method was used to compose long-form musical pieces. Musical examples are available for listening online. 1 they would like to control, such as speed, dynamics, harmony, or texture, each of which can be decomposed into Musicians and novices are increasingly experimenting multiple sub-features. Furthermore, the relationship bewith generative models in music making [44, 19, 18]. Yet tween these features and the output is more difuse than generative models are often not trained to support human in the NLP settings: while in examples such as [25], inobjectives (such as controllability) but to maximize proxy dividual words indicate the diferent conditions being metrics that are easy to automate (such as likelihood of satisfied, in music the entire sequence of tokens are used data according to a model), making it dificult for users in evaluating a single feature (for instance, average pitch to steer these models towards expressing users' musical over a span of time depends on all tokens representing intentions [44, 19, 32]. In the language domain, recent that span). This makes compositionality even more chalresearch in making generative models (such as large lan- lenging than in the text generation setting. On the other guage models) more controllable and useful in real-world hand, music is a sequential domain where there are clear applications have focused on ways of adapting uncondi- logical features which can be examined, such as average tioned language models to perform well on conditioned dynamics or number of notes per second. This further motasks they were not initially trained on. Methods such as tivates using music as a test-bed for highly compositional ifne-tuning, side-tuning, bias-tuning, and prompt or pre- tasks involving simultaneously steering an autoregresifx tuning have emerged as lead candidates for such tasks sive model towards several particular desired attributes such as steering the sentiment of or words mentioned according to diferent classifiers. in a sentence [2, 28, 27, 8]. However, such tasks are nor- As a more immediate motivation, consider the followmally presented in a non- or minimally-compositional ing scenario: a composer wants to sample from the preapproach (possibly controlling for two orthogonal vari- trained Music Transformer [20] to complete a musical ables, such as sentiment and topic [25], but rarely more phrase. In addition to the overall musical quality of the than that). continuation, they want it to stay in key and switch to usIn contrast, in the domain of music generation, se- ing block chords (i.e. a few notes played together at once). quence continuation is inherently a highly compositional Repeatedly sampling continuations from the model and problem: the user likely has many aspects of the output cherry-picking a good sample (rejection sampling) would be very labor-intensive, but assuming they can compute binary features for “stays in key” and “uses block chords”, the composer could sample a large number of continuations and cherry-pick from the smaller set of continuations which exhibit all features (thereby delegating
Only one block chord “Uses block
chords” Steerable Transformer
Many block chords! part of the accept/reject step). While significantly less form structure, by chaining together chunks steered in labor-intensive, this solution could potentially require diferent directions (using diferent combinations of feasampling enormous amounts of continuations if, for in- tures), while maintaining long-term coherence (by leverstance, the requested features significantly difer from aging the transformers’ full self-attention receptive field). those of the priming sequence (such that it and the de- Listen to examples on this page 1 for longer steered exsired continuation form an unlikely sequence according amples, and compositions semi-automatically generated to the generative model). In other words, the pre-trained using our feature tuning approach in a musician-directed model could be a poor proposal distribution for some way. applications.
On the ML side of this work, we are interested in
developing better proposal distributions through an adaptation 2. Problem formulation
approach which can steer the generative model towards
continuations which i) are significantly more likely to Music Transformer is an autoregressive language model
exhibit the requested feature, and ii) exhibit a satisfactory which decomposes the joint probability of a sequence
musical quality. The approach should be able to accom- of tokens 1, . . . , (where ∈ , and is a set of
modate a large number of features without adding signif- categorical tokens) into
icant memory or computation overhead. We achieve this
by making features composable, making it possible to
steer features independently and also multiple features at (x = 1, . . . , ) = (1) ∏︁ ( | 1, . . . , − 1).
once. Figure 1 shows that when using a pre-trained trans- =2
former model augmented with a relatively small number (1)
of additional parameters, we are able to steer towards ar- It leverages a common modeling approach which
reprebitrary logical music features and achieve realistic music sents the conditional probabilities ( | 1, . . . , − 1)
generation simply by sampling directly from the model. using a neural network [
Our approach can be used to support human-AI co- the network maps each sequence e1, . . . , e− 1 to a probcreation, where musicians can compose on the level of ability distribution for the value of over the elements musical “features” as opposed to notes. Musicians can of . prototype the high-level “shape” of the music by specify- Sequence continuation in an autoregressive language ing how various features change throughout the piece, model works by repeatedly sampling from its distribuand in turn steer and curate music transformers to cre- tion over the next token given the previous tokens. Startatively fill in the details. With our method, a composer 1Listen to musical examples at https://storage.googleapis.com/ can control both the short-chunk features, and the long- composing-features/index.html ing from some priming sequence x = (0, . . . , ), 1. Positively: given a prime–continuation pair we first sample +1 ∼
(· | +2 ∼ end of the continued sequence x = (+1, . . . , ). 2
(· | 0, . . . , , +1), and so on, until the Many downstream tasks can be cast as sequence continuation problems, including the steerable music generation
0, . . . , ), then problem investigated in this work.
We are given a set of features Φ = continuation pair exhibits that feature (which we note as
→ {0, 1}. Each takes the value 1 if a prime(x, x) |= ), and 0 otherwise. Note that the features must take both prime and continuation sequences as input, since some continuation features may be relative to the priming sequence (e.g. “significantly higher pitch”).
Our true objective with respect to feature is to steer the model towards a distribution which maximizes
{ }=1, ∈ Ex|x ︀[ (x, x) |= ]︀ (x, x) |= , we find an adaptation of the model’s parameters that minimizes ✓ (we use the symbol ✓ to denote the fact that is computed using the correct parameters ). By using prime–continuation examples that sound musical, we ensure that the steered model stays musically grounded.
features for which (x, x) ̸|= , by maximizing × (we use the symbol × that is computed using the incorrect parameters
to denote the fact ).
The positive case corresponds to maximum-likelihood training. Additionally, we can exploit the intuition that the adapted parameters should “explain” the prime– (2) continuation pair (x, x) |
= better than (for some feature for which (x, x) ̸|= ) or (the nonadapted model parameters, with a corresponding loss while maintaining musicality.
This objective is sider the problem of composed features Φ ˆ , i.e.
In addition to the single-feature problem, we also con- loss of the form non-diferentiable because diferentiable satisfiability criterion.
(x, x) |= is a non- ∅). In other words, we can maximize the probability of choosing over and by minimizing a contrastive (x, x) |= Φ ˆ ≡ (x, x) |= ,
(3) ⋀︁ ∈Φ^ ⊆ Φ to account for scenarios where a user is interested in steering the model towards multiple features (such as in the “stays in key” and “uses block chords” scenario discussed in the introduction).
We start by describing our proposed approach in the
single-feature case and later on explain how we adapt it
to the compositional case.
3.1. Likelihood-based training
While approaches using reinforcement learning—such
as KL-regularized deep Q-learning [
(x | x) (x | x) + (x | x) + (x | x) = − log ︂(
− ✓ − ✓ + − × + − ∅
We propose a loss that interpolates between Equations 4 and 5 using an coeficient (which is treated as a hyperparameter): ℒ = · ✓ − (1− )· log ︂(
− ✓ − ✓ + − × + − ∅ ︂)
Intuitively, the maximum-likelihood setting should sufice to achieve our adaptation goals, but in practice we find that the approach benefits from the inclusion of negative cases through a contrastive loss term. We tried diferent values and found = 0.8 to work well in practice. See Figure 2 for an illustration of the training setup. 3.2. Feature-conditional adaptation Fine-tuning all model parameters can be prohibitive if the number of features is large (let alone combinatorially large in the compositional case); however, recent work provides efective and lightweight alternatives: 1. Prefix tuning
[28] works by prepending learnable task embeddings e− , . . . , e− 1 to the priming sequence embeddings e1, . . . , e . The loss
Maestro dataset Prime
Continuation
Steerable
Transformer Unconditional
Transformer
In this section we describe our experimental setup: the musical features we want to enable users to control, the procedure for setting up the training data for adapting the feature-conditional parameters, and the details of the prefix tuning and bias tuning setup.
the single-feature setting, we were unable to make it work in the compositional setting. We therefore focus our investigation on bias-tuning for the compositional domain.
In the compositional setting, a naive approach requires learning 2|Φ| − 1 model adaptations. Instead, we propose to express the adaptation for a composed feature Φ ˆ as the combination of the of its underlying features ∈ Φ ˆ . More specifically, for bias-tuning we average the adapted biases as = 1 ˆ |Φ | ∈Φ^ ∑︁ , .
(7)
Note that we do not simply use the above heuristic to compose feature adaptations post-hoc; we train the model in a compositional setting (by sampling prime– continuation pairs (x, x) |= Φ ˆ for various composed features) so that the single-feature adaptations can learn to work well in conjunction with each other. See Figure 3
Musical features To support users in controlling different aspects of music, such as harmony, texture, speed, and dynamics, we implemented eighteen features (see Appendix A for a complete list).
We included both absolute and relative features. Absolute features apply only to the continuation, while relative features describe the relationship between the prime and the continuation, such as the average pitch in the continuation being significantly higher than that in the prime or the rhythmic density in the continuation being significantly lower than that in the prime. Note that this type of model should work with any feature function which takes in sequences and returns a Boolean. In this paper, we chose examples that have clear musical meanings and are easy to implement.
Dataset features Before adapting model parameters to specific binary musical features, we may first need to “fine-tune” a model to better fit the general musical style of a given dataset (in order to support the scenario where a user brings their own dataset). We refer to these dataset-level features as . Input Concat/Average MLP
MLP Pre x tuning
I n n e r l a y e r + Average
I n n e r l a y e r + Average Bias vector Bias vector MLP
MLP
MLP
MLP
Bias tuning
Output
4.2. Training data and setup
The unconditioned model we adopt as a base for
adaptation is a pretrained music transformer 3 trained on
transcribed YouTube piano performances (where the music
was typically more melodic). To mimic the user bringing
their own dataset with a distinctive style, we use the
Maestro dataset [
To prepare the prime–continuation pairs needed for likelihood based training, we serialize the Maestro piano performances into event-based encoding [33] (the same representation that the pretrained music transformer was trained on) and then take random crops of length 200 tokens (which lasts approximately 4 to 20 seconds long). We then split each crop in half, resulting in a prime x and continuation x pair that is each 100 token long.
To prepare training sets for feature-specific adaptations, for each feature ∈ Φ we take the subset (with replacement) of all the pairs that exhibit the feature (i.e. (x, x) |= ). Note that since all prime–continuation pairs are drawn from the Maestro dataset, we implicitly assume that (x, x) |= always holds.
Single-feature setting For the non-contrastive setting, for each feature we minimize the loss introduced in Equation 4 to adapt its parameters to better fit the set of prime–continuation pairs where (x, x) |= .
https://magenta.tensorflow.org/piano-transformer.
In the contrastive setting, we minimize the contrastive loss introduced in Equation 6. For each prime– continuation pair (x, x) |= used to compute ✓, we also need to select a “negative” case for computing × . We achieve this by selecting another feature at random, and a prime–continuation pair where both (x, x) |= and (x, x) ̸|= holds.
Compositional setting Training with a contrastive loss yielded more efective models in the single-feature setting (see section 5 for details), hence we adopt the contrastive loss for all of the subsequent experiments in the compositional setting.
To prepare training examples for steering any
number of features at once, we first extend Φ so that each
feature has a corresponding “negated” feature (e.g. “has
both loud and soft pitches” vs “no contrast in dynamics”),
bringing the total number of features from |Φ | = 18 to
2 · | Φ | = 36. By definition, each prime–continuation pair
exhibits exactly 18 features. When sampling a prime–
continuation pair (x, x), we compute ✓ with respect
to a random subset of those 18 features (with size drawn
at random from [
4.3.1. Sampling eficacy (x, x) ̸|= .
At test time, what we care about is when a user requests a feature , regardless of what the prime x is, what is the probability that a tuned model can achieve it (i.e. generate continuations x^ that exhibit feature ).
Hence, we propose the following sampling eficacy (i.e. probability of achievement) as our true evaluation metric to quantify on average using rejection sampling, what proportion of a model’s generated continuations x^ exhibit feature
us to establish a baseline for “inherently” how dificult or easy it is to achieve a certain feature by priming the model and measuring how often its generated continuation carries that feature.
Figure 4 shows for each feature (blue dot) the probability of achieving it when the unconditioned music transformer was primed with cofactual primes (x-axis) versus
Ex Ex^|x ︀[ (x, x^) |= ]︀ (8) counterfactual primes (y-axis). Intuitively, we expect it where the first expectation is under the data distribu- to be more dificult for a prime that is counterfactual tion (approximated by sampling primes x from a given to a feature to produce continuations with that feature, dataset), and the second is under the model being evalu- indeed this is what we observe, i.e. in the figure all the ated (approximated by conditioning on a prime x, and blue dots are below the diagonal line. Given this consisthen using the model to generate continuations x^). tency, whenever we report results on sampling eficacy (probability of achievement) in the paper, we breakdown Steering dificulty Intuitively, given a prime x, cer- our results to compare how methods fare on the hard tain musical features would follow more musically than (counterfactual) versus the easier (cofactual) cases. others. That is, the prime can impact how dificult it is to steer a model to generate continuations x^ that exhibit a 4.4. Implementation certain feature , hence afecting the sampling eficacy of a model on that feature. Single-feature setting We use a standard prefix tun
To address this “confounding factor”, we propose to ing model as per [28], associating every feature with breakdown our analysis of sampling eficacy based on if a a prefix . As in their paper, we used the finding that feature musically follows a prime or not. To approximate a relatively low-dimensional embedding space of feathis, we check in the dataset (that a model was trained ture prefixes (200 dimensions) projected to a higherdimensional embedding (2048 dimensions) using a shared 5.1. Likelihood-based training dynamics MLP across feature prefixes worked well. The pre-trained transformer model had no parameters modified, but was Intuitively, the maximum-likelihood setting should sufchanged to accept these 2048-dimensional vectors as ifce to achieve our adaptation goals, but in practice we four 512-dimensional vectors prepended before the 512- ifnd that the approach benefits from the inclusion of negdimensional vectors representing the input tokens (as ative cases through a contrastive loss term (introduced the transformer’s latent embedding was 512 dimensions). in Equation 6), as demonstrated in Figure 5. The positive The resulting sequence of vectors were masked using case (blue line) corresponds to the maximum-likelihood causal attention in order to predict x. term, while the negative case (orange line) corresponds to the contrastive loss term. The visually “diverging” Compositional setting The loss used in the composi- training curves in the contrastive setting show that it has tional setting is almost identical to Equation 6, the main learned to “explain” features using very diferent paramediference being in the contrastive loss implementation. ters. That is, when evaluating a prime–continuation pair The features used to compute × are obtained by negating (x, x) with parameters adapted for feature that some of the features used to compute ✓, and the fraction the pair does not exhibit (x, x) ̸|= , the likelihood of negated features is used to modulate the interpola- loss becomes very high. tion coeficient as ′ = − min(, ). The intuition is that the diferences in steering should be more noticeable 5.2. Sampling eficacy when comparing feature sets with less overlap.
We first examine the single-feature setting, before going into the compositional setting. For the experiments in this section, we focus on investigating if our proposed contrastive loss (introduced in Equation 6) is beneficial.
We find that indeed it is, that training with a contrastive loss not only enables learning of parameters that better “explain” musical features in a more “discriminate” fashion, but also results in a more steerable model, i.e. when conditioned on a musical feature, more likely to generate samples that exhibit that feature.
In this section, we compare how diferent methods perform “downstream” on the objective we care about, that is when the user brings a prime and a desired feature, how likely is a model able to generate a continuation that exhibits that feature. The sampling eficacy metric (defined in in subsection 6.1) gives on average a method’s probability in achieving requested features.
Procedure In the following experiments, to compute a method’s sampling eficacy on a feature, we take 512 random primes from the feature’s validation set, and for each prime generate 10 continuations, and then check what proportion of the continuations satisfy the requested feature. We then average among all features to obtain the average sampling eficacy of a method. This is the cofactual case. While for the counterfactual case, instead
Prefix tuning Cofactual Counterfactual
Contrastive Results In Table 1, we see that prefixing tuning (es- 0.6 pecially with a contrastive loss) significantly increases y sampling eficacy over the unconditioned model, both ca in the easier cofactual cases, the harder counterfactual iffc0.4 cases. All of the improvements are of a large margin, for E example in the hard counterfactual cases, prefix tuning 0.2 Unconditional with a contrastive loss increases the probabiliy of achiev- Non-contrastive ing a feature by 70%, and within prefix tuning, the gain Contrastive from switching from a non-contrastive loss to contrastive 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 loss is 18%. Difficulty
As sampling eficacy can vary with prime and feature requested, in Figure 6 we enumerate how diferent meth- Figure 6: Prefix tuning a model (especially with a contrastive ods perform for each of the musical features (18 total). lsoinssg)leinfceraetausreess.itTshsea mcoplloinregdefidcoatcsyth(ayt-laixnies)ufporvemrtoisctaollfytchoerr1e8We see prefix tuning (especially with a contrastive loss) spond to the same feature. Each feature’s steering dificulty yields a higher sampling eficacy (y-axis) for most fea- (x-axis) is quantified as the sampling eficacy of the unconditures. As contrastive loss yielded more efective models tioned model, which gives a baseline of how dificult or easy in the single-feature setting, we adopt the contrastive it is to sample that given feature. Most of the green (nonloss for all of the subsequent experiments in the compo- contrastive) and orange (contrastive) dots are above the diagsitional setting. onal line, meaning prefix tuning increases the probability of achieving them. The vertical dotted orange lines show the improvement in the contrastive setting over the non-contrastive 6. The compositional setting setting. In the following experiments, we first illustrate how dififcult compositional steering is under an unconditioned model. Then, we show quantitatively how our biastuning method can enable compositional steering with much higher sampling eficacy, and also qualitatively how it achieves this while also improving on the musical quality of the steered examples (compared to if we were generating from the “tail” of the unconditioned model). 6.1. Sampling eficacy Unconditioned model Intuitively, we expect it would be inefective to rely on rejection sampling (on an unconditioned model) to give us samples that include a specific large number of features. Under a naive assumption of independence among features, one would expect that the eficacy of rejection sampling in achieving a total of features would roughly equal , where is the probability of a feature being satisfied.
While the above assumption is naive, we do find that in the unconditioned setting, the sampling eficacy of the unconditioned model drops substantially as the number of requested features increases. For example, when the number of requested cofactual features is 6, only 0.2% of the time were all features achieved after rejection sampling (see the unconditional line in Figure 7).
more efective overall and less brittle with respect to the number of features. We see in Figure 7 4 that even though sampling eficacy decreases for all methods when more
compositional setting (Figure 7) is much higher than that in Table 1
Difficulty vs. Efficacy
Cofactu
M U Bi Pr
Pr 1
2 3 4 Number of steere lty vs. Efficacy
Cofactual
Random features are requested as expected, bias-tuning remained rejection sampling on the unconditional model. a viable steering approach even in the 6-feature setting, with a 7.9% probability of achievement versus only 0.2% Listening study setup The listening test consists of for the unconditioned model. This meant for the user, questions posed as pairwise comparisons of musical exusing bias-tuning on average 13 samples is needed to ar- amples, where both examples started with the same prime rive at one that exhibits their requested 6 features, while as the first half of the example, while one’s continuation for the unconditioned model it would take 500 samples, was generated by the unconditional model, while another making the latter an unfeasible approach. was generated with our bias-tuned model. The ordering
Comparing the other methods, bias tuning to the Mae- within the pairs were randomized. Listeners were asked stro dataset (without tuning for specific features) im- to rate which one they thought was more musical. To proves over the unconditioned model slightly (with larger prepare the samples, we randomly picked 120 primes gains in the 2 to 3 feature case in the cofactual setting), from the test set. For steering each prime, we randomly and as expected underperforms bias tuning which is chose a three-feature set that was present in the dataset tuned to also condition on specific features. Bias tuning as the conditioning features. outperforms prefix tuning consistently in the compositional setting. 6.2. Musical quality The above experiments evaluate the efectiveness of methods in achieving the requested features. To evaluate the musical quality of these “successful” generations, we conducted a listening study with musicians to compare the steering of three features between our overall best adaptation approach (i.e. bias tuning) and the baseline of for the single-feature setting. This is due to diference in the procedure we used in the two settings. In the compositional setting, we sampled features according to how they were distributed in the datasets, while in the single-feature setting we sampled features uniformly at random. Hence, in the compositional setting, features that appear often in the dataset (perhaps typical of the dataset’s musical style) is sampled more often. The sampling eficacy is much higher possibly because these feature are also easier to achieve.
Results We asked eight musicians to each rate fifteen pairs. The results show that our bias-tuning approach was much preferred over the unconditional, and the results were statistically significant ( < 0.0003). Bias tuning won 63 of the pairwise comparisons, tied for 26 pairs, and lost for 31 pairs. This shows that our biastuning approach is not only more efective in steering features, but also produces musically more compelling results.
Discussion It may seem surprising that bias-tuning was able to produce samples that were perceived as more musical than the original expressive unconditional model.
We hypothesize that this is because we are essentially sampling from the “tail” of the unconditioned model’s distribution when using rejection sampling (i.e. only accepting samples that exhibit the requested feature).
Since the unconditional model is not trained to generate specific features, we may have to disregard for example 99.8% of the samples generated before finding a satisfac- same prime but with a diferent “vibe” by invoking a diftory sample (as seen in the 6-feature setting described ferent set of low-level features. 6 This allowed her to in subsection 6.1). This resulting distribution is very dif- combine her knowledge of non-automated algorithmic ferent from the distribution of the unconditional model composition to compose a “quasi-algorithmic" suite of without rejection sampling. variations. Algorithmic composition involves using com
We further hypothesize that, via the proxy loss of in- putational logic to choose notes; instead, in this piece, creasing the likelihood of a set of continuations with a she used computational logic to choose features. given feature, a bias-tuned model learns more likely ways to satisfy that given feature. Hence, its continuations are not only more likely to be efective at satisfying that feature, but also to do so in a musically likely manner, which is often rated by listeners as more musical too.
Composer’s reflection The composer’s findings
were that the tool enabled her on every end of the
humanmachine spectrum: as a composer who occasionally
wants to avoid using any generation tools in their final
output, as a composer who wants to co-create in order
7. Case study: Human-AI to minimize both manual coding and manual composing,
and as an algorithmic composer. As a composer trying
co-creation to improve her “manual" composition and analysis skills
within specific scenarios (e.g., fast-paced and tense music
In the previous sections, we evaluated our approach algo- with chromaticism but also a degree of stasis), and who
rithmically, and found that it is quite efective in steering is used to having to manually search for examples of
music transformers to generate continuations with multi- repertoire having such properties in order to learn from
ple requested features present. In this section, we wanted them, the ability to generate a piece of music tailored to
to understand how a generative model with such steer- a specific scenario forms a surprisingly powerful
pedaability can be useful in a human-AI co-creation setting. gogical tool which she found that she can leverage when
As a preliminary study, one asked one of our co-authors, composing by hand.
who was also a composer, to put our model to test by The workflow of co-creating music made her feel like
carrying out computer-assisted composition. she was the author (rather than the computer), but
allowed her to create piano music of a complexity and
User background and creative strategies Our com- virtuosity her piano skills do not currently aford. The
poser had a background in both purely human and algo- composer felt that this tool was even more powerful in
rithmic composition. She experimented with using the the context of algorithmic composition. Her typical
worktool in several settings. In some settings, her goal was to lfow in algorithmic composition involves using tools such
co-create music of a specific high-level “vibe", which she as SuperCollider [51] to adapt an existing algorithm to
accomplished through the inclusion of specific features generate low-level notes; here she could still use
algoand manually curating the generated samples. rithmic thinking, but at a higher level of abstraction. For
instance, to compose the suite of variations presented
• To achieve “pleasant but low energy”, she chose earlier, she wrote a python program to orchestrate which
features such as diatonicism and harmonic stasis, set of musical features are used for each variation. In
block chords, lack of extreme dynamic change, and contrast, while in the past she would invoke a Markov
a mostly low rhythmic density. Listen to this ex- chain to flesh out each variation, here she could invoke a
ample and other “vibes” online. 5 powerful generative model.
• Inspired by a given prime that had a “roller- In this case study, as the composer was also the creator
coaster"-like quality, she wanted to write a piece of this tool, she was in a unique position to explore the
with a cyclic shape which oscillated between full potential of this tool. To make the tool accessible
melodic and textural extremes. She accomplished to a broader audience, future work could include “Hello
this through the coupling and cycling of features AI”-like [
Seeing that it is possible to steer music transformer to generate specific high-level “vibes” through low-level features, our composer was inspired to compose a theme and variation where each variation would continue the
googleapis.com/composing-features/index.html#vibes.
googleapis.com/composing-features/index.html.
language-model features, has shown significant enhancements in few-shot learning, but is typically not performed compositionally [53].
Our work builds on language modeling, further exploring ways to “tune” these models for user control, through “fine-tuning” approaches such as prefix and bias tuning.
In particular, we leverage contrastive learning and compositionality to derive a lightweight augmentation for steering large transformer-based language models. Our approach enables human-AI co-creation, enabling users to compose at a higher level of abstraction by specifying features while music transformers fills in the notes. In the following, we provide a brief overview for each of the aforementioned related research areas.
Contrastive learning Contrastive learning is used in representation learning to train a network which maps “similar” (positive) inputs to nearby representations and “dissimilar” (negative) inputs far away from the positive inputs. See [26] for a theoretical framework and overview.
In generative modeling, contrastive divergence [
Human-AI co-creation in music Controllable
generative models typically does not ofer users full control (i.e.
only allows users to specify a small number of low-level
or high-level controls, or through an example), while
relying on its learned stylistic distribution and/or features
encoded from the user-specified template piece to fill in
the rest of the musical details [
We have shown that music transformers can be directed towards a specific generative “task" using some of the same methods as natural language transformers. In addition, we have studied compositionality in this domain. Compositionality (including relatively high levels of compositionality) is critical in the music domain (and other domains) if the user wants control over the output. We establish that compositionality is a hard problem, and propose adaptations of several solutions from the literature (bias-tuning and prefix-tuning) to address these challenges. We find success with bias-tuning, but not with prefix-tuning. While our results are promising, there is clearly significant room for improving on the eficacy of steering a transformer compositionally.
We have provided a preliminary demonstration of how our approach, compositional steering, enables human-AI co-creation, where musicians can compose on the level of musical “features” as opposed to notes. Musicians can prototype the high-level “shape” of the music by specifying how various features change throughout the piece, and in turn steer and curate music transformers to creatively fill in the details. We envision future work to enable end-user machine learning where users can define their own features or provide their own musical examples, and leverage our lightweight compositional bias-tuning approach to learn new controls to steer expressive music transformers compositionally. 1. “Significantly lower average pitch than prime" 2. “Significantly higher average pitch than prime" 3. “Significantly higher number of grouped attacks than prime" 4. “Significantly lower number of grouped attacks than prime" 5. “Significantly more notes per second than prime" 6. “Significantly fewer notes per second than prime" 7. “Could fit in the same key as the prime" 8. “Has 2 or more pitch classes (pitch mod 12) which the prime doesn’t have"