In this work1, we investigate how novices co-create music with a deep generative model, and what types of interactive controls are important for an efective co-creation experience. Through a needfinding study, we found that generative AI can overwhelm novices when the AI generates too much content, and can make it hard to express creative goals when outputs appear to be random. To better match co-creation needs, we built Cococo, a music editor web interface that adds interactive capabilities via a set of AIsteering tools. These tools restrict content generation to particular voices and time measures, and help to constrain non-deterministic output to specific high-level directions. We found that the tools helped users increase their control, self-eficacy, and creative ownership, and we describe how the tools afected novices' strategies for composing and managing their interaction with AI.
• Human-centered computing → Human computer interaction (HCI); User studies; Collaborative interaction.
Recent generative music models have made it conceivable for novices
to create an entire musical composition from scratch, in
partnership with a generative model. For example, the widely available
Bach Doodle [
In a need-finding study we conducted to understand the noviceAI co-creation process, we found that generative music models can sometimes be quite challenging to co-create with. Novices experienced information overload, in which they struggled to evaluate and edit the generated music because the system created too much content at once. They also struggled with the system’s nondeterministic output. While the output would typically be coherent, it would not always align with users’ musical goals at the moment. Having surfaced these challenges, this paper seeks to understand what interfaces and interactive controls for generative models are important in order to promote an efective co-creation experience.
As a step towards explicitly designing for music novices cocreating with generative models, we present Cococo (collaborative co-creation), a music editor web-interface for novice-AI co-creation that augments standard generative music interfaces with a set of AI-steering tools: 1) Voice Lanes that allow users to define for which time-steps (e.g. measure 1) and for which voices (e.g. soprano, alto, tenor, bass) the AI generates music, before any music is created, 2) an Example-based Slider for expressing that the AI-generated music should be more or less like an existing example of music, 3) Semantic Sliders that users can adjust to direct the music toward high-level directions (e.g. happier / sadder, or more conventional / more surprising), and 4) Multiple Alternatives for the user to select between a variety of AI-generated options. To implement the sliders, we developed a soft priors approach that encodes desired qualities specified by a slider into a prior distribution; this soft prior is then used to alter a model’s original sampling distribution, in turn influencing the AI’s generated output.
In a summative evaluation with 21 music novices, we found that AI-steering tools not only increased users’ trust, control, comprehension, and sense of collaboration with the AI, but also contributed to a greater sense of self-eficacy and ownership of the composition relative to the AI. We also reveal how AI-Steering tools afected novices co-creation process, such as by working with smaller, semantically-meaningful components and reducing the non-determinism in AI-generated output. Together, these findings inform the design of future human-AI interfaces for co-creation.
To understand challenges when composing music with generative
models, we conducted a 25 minute needfinding study with 11 music
composition novices. We observed novices use a tool that mirrored
conventional interfaces for composing music with deep generative
models [
Participants experienced information overload: they struggled to evaluate the generated music due to the amount of AIgenerated content. Participants struggled to identify which note was causing a discordant sound after multiple generated voices were added to their original. Participant were naturally inclined to work on the composition “bar-by-bar or part-by-part” ; however in contrast to these expectations, the generated output felt like it “skipped a couple steps” and made it dificult to follow all at once.
Participants struggled to express desired musical objectives due to the AI’s non-deterministic output. Even though what was produced sounded harmonious to the user, they felt incapable of giving feedback about their goal in order to constrain the kinds of notes the model generated. Participants likened this frustrated feeling to “rolling dice” to generate a desired sound, and instead wished to control generation based on relevant musical objectives. 3
Based on identified user needs, we developed Cococo
(collaborative co-creation), a music editor web-interface 3 for novice-AI
co-creation that augments standard generative music interfaces
3https://github.com/pair-code/cococo
with a set of AI steering tools (Figure 1). Cococo builds on top of
Coconet [
Beyond the infill mask, Cococo distinguishes itself with its AI
steering tools. In the following subsections, we describe in detail
each of the four tools. Additionally, we illustrate the co-creation
workflow enabled by these tools in Figure 1.
3.0.1 Voice Lanes. Voice Lanes allow specifying for which voice(s)
and for which time steps to generate music. With this capability,
users can control the amount of generated content they would like
to work with. This was designed to address information overload
caused by Coconet’s default capabilities to infill all remaining voices
and sections at a time. For example, a user can request the AI to
add a single accompanying bass line to their melody by
highlighting the bass (bottom) voice lane for the duration of the melody,
prior to clicking the generate button (Figure 1B). To support this
type of request, we pass a custom generation mask to the Coconet
model including only the user-selected voices and time-slices to be
generated.
3.0.2 Multiple Alternatives. Cococo provides afordances for
auditioning multiple alternatives generated by the AI. This capability
was designed based on formative feedback, in which users wanted
a way to cycle through several generated suggestions to decide
which was the most desirable. Users first choose the number of
alternatives to be generated (Figure 1C), audition each alternative
by clicking on the diefrent preview thumbnails (Figure 1F), and
listen to an alternative which is substituted within the larger musical
context (Figure 1G).
3.0.3 Example-based Slider. While prototyping the Multiple
Alternatives feature, we found that the non-determinism inherent in
Coconet could cause generated samples to be both (1) random and
unfocused, or (2) too similar to each other and lack diversity. As
a solution, we developed the example-based slider for expressing
that the AI-generated music should be more or less like an existing
example of music. Before this slider is enabled, the user must select
a reference example chunk of notes. Example-based sliders use soft
priors to guide music generation.
3.0.4 Semantic Sliders. We implemented two semantic sliders in
Cococo (Figure 1D) to constrain generated output along
meaningful dimensions: a conventional vs. surprising slider, and a major
(happy) vs. minor (sad) slider. Users can adjust how predictable vs.
unusual notes should be using the “conventional“ and “surprising”
dimensions of the slider. The conventional/surprising slider adjusts
the temperature ( ) of the sampling distribution [
adjusted (, | ) ∝ coconet (, | ) softprior (, ) where coconet (, | ) gives the sampling distribution over pitches for voice at time from Coconet given musical context ( gives the set of , positions constituting the context), softprior (, ) encodes the distribution over pitches specified by the user or AIsteering tool designer (serving as soft priors), and adjusted (, | ) gives the resulting adjusted posterior sampling distribution over pitches. The soft priors softprior (, ) are defined so that notes that should be encouraged are given a higher probability, and those discouraged are given a lower, but non-zero probability. Since none of the note probabilities are forced to zero, very probable notes in the model’s original sampling distribution can still be likely after incorporating the priors, thus making it possible for the model’s output to adhere to both the original context and the additional user-desired qualities.
The example-based and semantic sliders define a soft prior to modulate the model’s generated output. When the user sets the example-based slider to more “similar,” Cococo defines a soft prior with higher probabilities for notes in the example. Conversely, for a slider setting of more “diferent,” Cococo defines a soft prior with lower probabilities for notes in the example.
The minor/major slider uses a slightly more complicated approach to define the soft prior distribution. When the user sets the slider to happy (major), for example, Cococo defines the soft prior by asking what is the most likely major triad at each time slice within the model’s sampling distribution. The log likelihood of a triad is computed by summing the log probability of all the notes that could be part of the triad (e.g., for C major triad, this includes all the Cs, Es, and Gs in all octaves). We repeat this procedure for all possible major triads to determine which is the most likely for a time slice. We then repeat this procedure for all time slices to be generated, in order to create our soft prior for most likely major triads. 4
We conducted a within-subjects study to compare the user
experience of Cococo to that of the conventional interface. The
conventional interface is aesthetically similar to Cococo, but does not
contain the AI-steering tools. To mirror the most recent deep
generative music interfaces, the conventional interface does include
the infill-mask feature, which enables users to crop any region of
the music and request that it be filled in by the AI [
The following set of metrics sought to measure users’
compositional experience. Creative expression: Users rated “I was able
to express my creative goals in the composition made using [System
X].” Self-eficacy : Users answered two items from the Generalized
Self-Eficacy scale [
In addition, we evaluated users’ attitudes towards the AI. AI
interaction issues: Users rated the extent to which the system felt
comprehensible and controllable, two key challenges of human-AI
interaction raised in prior work on DNNs [
Results from the post-study questionnaire are shown in Figure 3. We conducted paired t-tests using Benjamani-Hochberg correction to account for the 15 planned-comparisons, using a false discovery rate = 0.05.
In regards to users perceptions of the creative process, we found Cococo significantly improved participants ability to express their creative goals, self-eficacy , perception of learning more about music, and engagement compared to the conventional interface. No significant diference was found in efort ; participants described the two systems as requiring diferent kinds of efort: While Cococo required users to think and interact with the controls, the conventional interface’s lack of controls made it efortful to express creative goals. Users perceptions of the completeness of their composition made with Cococo was significantly higher than the conventional interface; however, no significant diference was found for uniqueness.
The comparisons for users’ attitudes towards the AI were all found to be statistically significant: Cococo was more controllable, comprehensible, and collaborative than the conventional interface; participants using Cococo expressed higher trust in the AI, felt more ownership over the composition, and attributed the music to more of their own contributions relative to the AI. 6
In this section, we first report how AI-Steering tools supported novices’ composing strategies and experience, including 1) working with smaller, semantically meaningful components and 2) reducing non-determinism through testing a variety of constrained settings for generation. We then describe 3) how novices’ prior mental models shaped their interaction with AI. 6.1
AI-Steering tools allowed participants to build up the composition from smaller components, bit-by-bit. For example, one participant who used the Voice Lanes said, “I’m trying to get the bass right, then the tenor right, then soprano and alto right, and build bit-bybit” (P2). Participants who worked bit-by-bit thought about their compositions in semantically-meaningful chunks, such as melody vs. background or separate musical personas. For example, one participant gave the tenor voice an “alternating [pitch] pattern" to express indecision in the main melody, then gave other voices “mysterious... dinging sounds" as a harmonic backdrop (P4).
Working bit-by-bit helped participants feel less overwhelmed and better understand their compositions. For example, those working voice-by-voice could better handle the combination of multiple voices: “As someone who cannot be thinking about all 4 voices at the same time, it’s so helpful to generate one at a time” (P2). Participants then became familiar with their own composition during the creation process, which enabled them to more quickly identify the “cause” of problematic areas later on. For example, one participant indicated that “[because] I had built [each voice] independently and listened to them individually,” this helped them “understand what is coming from where” (P7).
Through this bit-by-bit process, participants learned how subcomponents can combine to achieve desired musical outcomes. For instance, one participant learned that “a piece can become more vivid by adding both a minor and major chord” after they applied the major/minor slider to generate two contrasting, side-by-side chunks (P12). 6.2
AI-Steering tools helped to constrain the non-deterministic output inherent in the generative model. As a result, the tools allowed users to steer generation in desired directions when composing with AI. Multiple Alternatives reduced the uncertainty that AIgenerated output would be misaligned with a user’s musical goals. Participants could simply generate a range of possibilities, audition them, and choose the one closest to their goal before continuing.
During diferent phases of the composing process, participants used the sliders to constrain the large space of possibilities that could be generated. The Semantic Sliders were sometimes used to set an initial trajectory for generated music: “Because I was able to give more inputs to [Cococo] about what my goals were, it was able to create some things that gave me a starting point” (P8). Sliders were also used to refine what the AI had already generated: “It was... not dramatic enough. Moving the slider to more surprising, and more minor added more drama at the end” (P5).
Participants constrained generation by setting the sliders to their outer limits. This enabled them to test the boundaries of AI output. For example, one participant moved a slider to the “similar” extreme, then incrementally backed it of to understand what to expect at various levels of the slider: “On the far end of similar, I got four identical generations, and now I’m almost at the middle now, and it’s making such subtle adjustments” (P18). In contrast, when using the conventional interface, participants could not as easily discern whether undesirable model outputs were due to AI limits, or a simple luck of the draw.
Participants also used the tools to consider how a specific input configuration afects the limits of AI output. For example, one participant used the Voice Lanes to generate multiple alternatives for a single-voice harmony. This enabled them to consider the limits imposed by specific voice components: “Maybe the dissonance [in the single-voice] is happening because of how I had the soprano and bass... which are limiting it... so it’s hard to find something that works” (P15). The Multiple Alternatives capability further enabled this participant to systematically infer that the specific configuration of existing voice components was unlikely to produce better results through the observation of multiple poor results generated for the single-voice. 6.3
Participants brought with them prior mental models that impacted how they interacted with the generative model. First, many participants already had a set of primitives for expressing high-level musical goals. For example, higher pitches were used to communicate a light mood, long notes to convey calmness or drawn-out emotions, and a shape of ascending pitches to communicate triumph and escalation. When participants who could not find an explicit tool that mapped to their envisioned primitive, they repurposed the tools as “proxy controls” to enact their strategy. For example, a common pattern was to set the slider to “conventional” to generate music that was “not super fast... not a strong musical intensity” (P9), and to “surprising” for generating “shorter notes... to add more interest” (P15).
In some cases, even use of the AI-steering tools did not succeed in generating the desired quality. For example, the music produced using the “similar” setting was not always similar along the userenvisioned dimension. To overcome these challenges, participants developed a strategy of “leading by example” by populating the input context with the type of content they desired from the AI. For instance, one participant manually drew an ascending pattern in the first half of the alto voice, in the hopes that the AI would continue the ascending pattern in the second half.
Second, several participants believed that the AI model was superior to their skills as novice composers. As such, when specific errors arose during the composing process, they often blamed their own eforts for these mistakes and hesitated to play an active role in the process. While we found evidence that the tools helped improve feelings of self-eficacy (See Quantitative Findings), there were also times when participants doubted their own musical abilities. Novices experienced self-doubt when poor sounding music was generated based of of their user-composed notes as the input context. For example, one user said, “All the things it’s generating sound sad, so it’s probably me because of what I generated” (P11). In cases such as this, participants seemed unable to disambiguate between AI failures and their own composing flaws, and placed the blame on themselves.
In other scenarios, novices were hesitant to interfere with the AI music generation process. For instance, some assumed that the AI’s global optimization would create better output than had they worked bit-by-bit: “Instead of doing [the voice lanes] one by one, I thought that the AI would know how to combine all these three [voices] in a way that would sound good” (P1). While editing content, others were worried that making local changes could interfere with the AI’s global optimization and possibly “mess the whole thing up” (P3). In these cases, an incomplete mental model of how the system functions seemed to discourage experimentation and their sense of self-eficacy. 7
breaking the co-creation task down into understandable chunks
and generating, auditioning, and editing these smaller pieces until
users arrived at a satisfactory result. Unexpectedly, novices quickly
became familiar with their own creations through composing
bit-bybit, which later helped them debug problematic areas. Interacting
through semantically meaningful tools also helped them learn more
about music composition and efective strategies for achieving
particular outcomes (e.g., the efect of a minor key in the composition).
Ultimately, AI-steering tools afected participants’ sense of
artistic ownership and competence as amateur composers, through an
improved ability to express creative intent. In sum, beyond
reducing information overload, tools that partition AI capabilities into
semantically-meaningful components may be fundamental to one’s
notion of being a creator, while opening the door for users to learn
efective strategies for creating in that domain.
7.0.2 Onboard Users and Divulge AI Limitations. While
participants were able to develop productive strategies using AI-steering
tools, they were sometimes hesitant to make local edits for fear of
adversely afecting the AI’s global optimization. These reactions
suggest that participants could benefit from a more accurate mental
model of the AI. Previous research suggests benefits of educating
users about the AI and its capabilities [