Music Composition as a Lens for Understanding Human-AI Collaboration

Abstract

As generative artificial intelligence (GenAI) systems gain human-like capabilities in creative tasks, they seem to blur the line between machines and users, prompting questions about how to design systems where AI and humans collaborate. Music composition with AI may ofer a lens to explore the nuances of human-AI collaboration. We review recent literature on music generation with AI, highlighting key challenges like the need for user control and context awareness, and noting a potential shift in the user's role towards curation or co-production when using AI tools. However, much of the existing research evaluates the impact of current AI tools rather than engaging in fieldwork to investigate music composition “in practice” within specific socio-cultural contexts. We then propose an ethnographic study to understand music composition as a situated practice, considering composers' personal motivations, artistic sensibilities, and the broader socio-cultural context. Preliminary findings highlight the importance of creative intentionality and meaning-making in driving compositional choices. Furthermore, music creation often involves collaboration between various human actors, raising questions about whether AI should facilitate this already present collaboration or disrupt existing dynamics.

-

CEUR ceur-ws.org

1. Introduction

The rapid evolution of Artificial Intelligence (AI), and particularly GenAI, is reshaping how humans interact with technology. We are moving beyond more traditional interaction models towards scenarios of collaboration between humans and AI, which raises critical questions about how we ought to design the interaction with systems that promise to leverage the strengths of both. GenAI systems can not only perform classification tasks but also create artifacts like music, images, and text, blurring the line between traditional tools and active collaborators with creative abilities.

In the broad research area of Human-Centered AI (HCAI) [ 1, 2, 3 ], a perspective has emerged that seeks to leverage the strengths of both humans and AI, creating synergistic systems that surpass their individual capabilities [ 3, 4, 5 ]. This concept of human-AI collaboration arguably represents a more human-centered approach compared to human-in-the-loop models, as it makes the ”best use of both human and AI capabilities, rather than the human simply being called Proceedings of the 1st International Workshop on Designing and Building Hybrid Human–AI Systems (SYNERGY 2024), CEUR Workshop Proceedings upon to do what the AI cannot yet manage in an AI led project” [ 3 ]. However, this emphasis on AI agency, which Sarkar [ 6 ] terms as an “agentistic turn”, necessitates critical consideration. While it aligns with Bruno Latour’s notion of agency [ 7 ], which extends agency across a network of human and non-human actors, showing how intricate the relationships between humans and their surroundings are, it could obscure the vast amount of “ghost work” [ 8 ] that powers AI, frequently conducted in the Global South at low wages (e.g., by data labelers) [ 6 ]. The concept of “collaboration” in human-AI interaction is certainly multifaceted, but, while some interactions with non-generative AI might exhibit collaborative aspects already [ 9 ], the emergence of GenAI presents unique challenges. These systems, capable of human-like outputs, blur the lines between tools and collaborators. This necessitates a deeper understanding of how human roles are redefined and how decision-making, creative processes, and information handling are reshaped.

To explore this further, we focus on a specific domain – music composition with AI – as an illustrative case study, to understand how collaboration and co-creation occur in practice, taking into account not only the user’s objectives, needs, and motivations, but also the social and cultural context within which humans and GenAI systems may collaborate to achieve situated goals. The collaborative process of composing music raises intriguing questions about ownership, control, and intentions, all of which are central themes in the broader discussion of human-AI collaboration.

We will begin by reviewing recent literature on human-AI music composition and then propose a study investigating the situated practices [ 10 ] of music composition.

2. AI in Music Composition

The interest in computer-based music composition has steadily increased since the 1980s, with various techniques of Algorithmic Composition like Markov Models, Generative Grammars, and Genetic Algorithms. Later, as Neural Networks became more prominent, the advancements in Deep Learning led to the adoption of established architectures from fields like Computer Vision and Natural Language Processing in music generation as well [ 11 ].

While research on interactive musical systems had already highlighted the importance of applying user-centered approaches to support composers’ processes of creation, exploration, and learning [ 12, 13, 14 ], more recent studies have delved into the specific challenges and strategies of co-creating music with AI.

Newman et al. [ 15 ] investigated through interviews how current AI tools concretely influence musical creativity. They then proposed a new model for developing ethical and productive collaborative AI tools for music. This model emphasizes the importance of clearly defined roles for AI and pays attention to how control is distributed. Their research reveals that users perceive as positive AI use cases those where they can maintain control, agency, intention, and choice throughout iterative cycles of generation and evaluation [ 15 ].

Huang et al. [ 16 ] studied the challenges and strategies of co-creating music with AI through a survey conducted during an AI song contest. The results highlight the importance of context awareness and user control, advising future AI systems to be designed to adapt to existing composers’ practices rather than forcing new AI-determined workflows.

In another study investigating the use of “steering tools” for collaborative music creation, Louie et al. [ 17 ] identified two challenges in using GenAI: information overload and nondeterministic output. Their findings suggest that steering tools can enhance the user’s sense of control, trust, and understanding of the AI system, resulting in a greater feeling of involvement in the creative process. The authors also comment that users often have pre-existing mental models about music composition and use them for tackling problems. These preconceptions should be considered so that both the AI models and the interfaces can be designed to be more intuitive, requiring less cognitive efort, and ultimately increasing the user’s sense of agency. Louie et al. [ 17 ] also argue that the AI’s role should adapt to the user’s needs and the creative context: while ceding control is welcomed during exploratory phases, where the user is in search of unexpected inspiration, maintaining control over specific details becomes critical during production. Context thus plays a crucial role in shaping the dynamics of human-AI collaboration, influencing how control and agency are perceived.

Suh et al. [ 18 ] explored how AI systems can act as a “social glue” to support human-human collaboration in compositional practices. Their findings suggest that AI can facilitate the exchange of ideas and group cohesion, potentially reducing tensions typically associated with collaboration. They advocate for an intentional design of AI to further strengthen social collaboration. However, while AI can act as a support system, they also observed a potential shift in roles: participants reported feeling more like curators or co-producers, focusing on evaluating AI-generated material rather than actively developing ideas, leading to a weaker sense of creative involvement. This observation aligns with the findings of Civit et al. [ 19 ], who noted that “the composer became more of an arranger of diferent melodies” comparing their role to that of a producer managing misbehaving musicians. While this shift is viewed as a “very creative, fruitful process” by them instead, it still highlights the need for future AI systems to be adaptive to the creative context, user needs, and the composer’s specific intentions.

This said, despite the growing interest in human-AI collaboration for music composition, current research seems to have limited scope: much of the focus is on evaluating the impact of existing generative systems, exploring strategies for composers to navigate their challenges, and integrating steering tools into current interfaces. Eforts to bridge the gap between AI music generation and the social and cultural complexities of music composition often lack engagement with field studies on compositional practices. Unlike humans, who are influenced by personal and social motivations and cultural context, AI systems currently operate solely on algorithms and predictive models, potentially limiting their efectiveness and failing to fully capture the nuances of human creativity [ 20 ]. The existing research gap highlights the importance of investigating current practices adopted by music composers, as well as their personal motivations, artistic sensibilities, the broader cultural and social context that influences their work, and the specific characteristics of various musical genres. This knowledge will be crucial for designing human-AI systems and creative workflows that efectively complement human strengths and intentions.

3. Music Composition as a Situated Practice

The existing literature emphasizes the need to better understand the complexities of music compositions in order to design human-AI collaborative music systems that account for both the individual and the socio-cultural context. While eforts like that of Hernandez-Olivan and Beltrán [ 11 ] to create generalized models of music composition are valuable for highlighting core principles, such rigid frameworks can hardly capture the dynamic and diverse nature of music creation, which is constantly evolving, shaped by genre, stylistic trends, personal choices, improvisation, and the unique socio-cultural setting where musicians operate.

To address this complexity, we propose an ethnographic approach that foregrounds the situated nature of music composition. This approach will delve into both the composers’ personal motivations (e.g., creative aspirations and career goals) and the socio-cultural context that influences their work. Ethnography allows us to explore these aspects of music creation and the lived experiences of composers, and has already been applied and discussed as a method for AI research [21, 22, 23, 24, 25], feeding into a broader discussion about the lack of, and thus need to integrate, the social sciences into AI research to mitigate the exclusive use of quantitative methods, which lack of socio-technical perspective, and their uncritical and positivist use that often leads to ignoring the context and causes that bring to a certain outcome and the ways in which it occurs [26, 21, 27]. By employing ethnography, we look at users not simply as passive recipients of technology, but as active agents who shape its context, meanings, and consequences [28]. This approach emphasizes the context-dependent nature of music composition, laying the groundwork for designing synergistic human-AI systems that are sensitive to the nuances of human creativity and to the specific settings where music is created.

Our research aims to answer the following provisional questions: • Can the use of Generative AI systems be actually considered a collaboration? • How does the specific context of music creation influence decision-making, workflows, and creative choices when working with AI? • How might collaboration with Generative AI systems redefine the roles of composers in the creative process of composition? • How do composers and AI negotiate creative control and authorship within this collaboration?

To answer these questions, we propose an ethnographic approach combining semi-structured interviews and participant observation. We will interview 18 musicians with experience in composing for diverse genres, exploring topics like: • Motivations, aspirations, creative sensibilities and personal workflow • The role of context in their music creation process • Experiences with AI tools and their perception of AI’s role • How users perceive their own role evolving in collaboration with AI

Following the interviews, we will conduct at least 60 hours of participant observation, to immerse ourselves in the composers’ practice.

In summary, we aim to shed light on the situated practice of composing music and on how the broader context influences the collaborative process. We do this to inform the design of human-AI collaborative systems that support and empower, not replace, musicians in their creative practices. Additionally, this study can ofer insights into human-AI collaboration beyond music, contributing to uncover design patterns for systems that are synergistic with human capabilities and responsive to the specific context in which they are used. 3.1. Preliminary findings Our analysis of initial interviews reveals some preliminary findings:

(a) Intentionality Shapes and Gives Meaning to Music: Composers’ creative intentions significantly impact their approach to composition. For example, whether they start with a melody, harmony, or specific sound (timbre) often depends on what they want to convey. While the existing literature acknowledges the link between intention, control, and creative agency, our study highlights the specific link between intention and a deeper meaning-making process, where composers strive to construct a “coherent discourse” through their music. How could we design human-AI collaborative systems that support and adapt to user intentions? (b) Music is already collaborative: Music composition and production are already an often collaborative process. From bandmates to collaborators, clients, and sound engineers, various stakeholders contribute to and have an interest in the final product. This raises the question of whether AI systems should be designed to enhance these existing human-human interactions, or whether they themselves should become an additional collaborator within a system in which creative control is already dynamically distributed and negotiated.

More findings will be shared during the workshop. Handbook of Artificial Intelligence for Music, Springer International Publishing, Cham, 2021, pp. 1–20. doi:10.1007/978-3-030-72116-9_1. [21] V. Marda, S. Narayan, On the importance of ethnographic methods in AI research, Nature

Machine Intelligence 3 (2021) 187–189. doi:10.1038/s42256-021-00323-0. [22] A. Christin, The ethnographer and the algorithm: Beyond the black box, Theory and

Society 49 (2020) 897–918. doi:10.1007/s11186-020-09411-3. [23] A. F. Blackwell, Ethnographic artificial intelligence, Interdisciplinary Science Reviews 46 (2021) 198–211. doi:10.1080/03080188.2020.1840226. [24] R. Van Voorst, T. Ahlin, Key points for an ethnography of AI: An approach towards crucial data, Humanities and Social Sciences Communications 11 (2024) 337. doi:10.1057/ s41599-024-02854-4. [25] N. Seaver, Algorithms as culture: Some tactics for the ethnography of algorithmic systems,

Big Data & Society 4 (2017) 205395171773810. doi:10.1177/2053951717738104. [26] M. Sloane, E. Moss, AI’s social sciences deficit, Nature Machine Intelligence 1 (2019) 330–331. doi:10.1038/s42256-019-0084-6. [27] E. Dahlin, Mind the gap! On the future of AI research, Humanities and Social Sciences

Communications 8 (2021) 1–4. doi:10.1057/s41599-021-00750-9. [28] P. Dourish, Implications for design (2006) 541–550. doi:10.1145/1124772.1124855.

References

[1] W. Xu , Toward human-centered AI: A perspective from human-computer interaction , Interactions 26 ( 2019 ) 42 - 46 . doi: 10 .1145/3328485.

[2] B. Shneiderman , Human-Centered AI , Oxford University Press, Oxford, 2022 .

[3] T. Capel , M. Brereton , What is Human-Centered about Human-Centered AI? A Map of the Research Landscape , in : Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23 , Association for Computing Machinery, Hamburg, Germany, 2023 , p. 23 . doi: 10 .1145/3544548.3580959.

[4] L. G. Terveen, Overview of human-computer collaboration, Knowledge-Based Systems 8 ( 1995 ) 67 - 81 . doi: 10 .1016/ 0950 - 7051 ( 95 ) 98369 - H .

[5] Z. Wu , D. Ji , K. Yu , X. Zeng , D. Wu , M. Shidujaman, AI Creativity and the Human-AI Co-creation Model , in: M. Kurosu (Ed.), Human-Computer Interaction . Theory, Methods and Tools, volume 12762 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2021 , pp. 171 - 190 . doi: 10 .1007/978- 3- 030 - 78462- 1_ 13 .

[6] A. Sarkar , Enough With “ Human-AI Collaboration” , in: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23 , Association for Computing Machinery, Hamburg, Germany, 2023 , p. 8 . doi: 10 .1145/3544549.3582735.

[7] B. Latour , Reassembling the Social: An Introduction to Actor-Network- Theory , Clarendon Lectures in Management Studies, Oxford University Press, Oxford, 2005 .

[8] M. L. Gray , S. Suri , Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass , Houghton Miflin Harcourt, Boston, 2019 .

[9] A. Rapp , A. Boldi , L. Curti , A. Perrucci , R. Simeoni , Collaborating with a Text-Based Chatbot: An Exploration of Real-World Collaboration Strategies Enacted during HumanChatbot Interactions , in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23 , Association for Computing Machinery, New York, NY, USA, 2023 , pp. 1 - 17 . doi: 10 .1145/3544548.3580995.

[10] L. A. Suchman , Plans and Situated Actions: The Problem of Human-Machine Communication , volume 103 , Cambridge University Press, USA, 1987 .

[11] C. Hernandez-Olivan , J. R. Beltrán , Music Composition with Deep Learning: A Review, in: A . Biswas , E. Wennekes , A. Wieczorkowska , R. H. Laskar (Eds.), Advances in Speech and Music Technology: Computational Aspects and Applications , Signals and Communication Technology, Springer International Publishing, Cham, 2023 , pp. 25 - 50 . doi: 10 .1007/978- 3- 031 - 18444- 4 _ 2 .

[12] R. Fiebrink , D. Trueman , C. Britt , M. Nagai , K. Kaczmarek , M. Early , MR. Daniel, A. Hege , P. Cook , Toward Understanding Human-Computer Interaction In Composing The Instrument , in: Proceedings of the International Computer Music Conference, International Computer Music Association, New York, 2010 , pp. 135 - 142 .

[13] Y. Wu , N. Bryan-Kinns, Supporting Non-Musicians? Creative Engagement with Musical Interfaces , in: Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition , C&C '17, Association for Computing Machinery, New York, NY, USA, 2017 , pp. 275 - 286 . doi: 10 .1145/3059454.3059457.

[14] H. Scurto , F. Bevilacqua , Appropriating Music Computing Practices Through Human-AI Collaboration, in: Journées d'Informatique Musicale (JIM 2018 ), Amiens, France, 2018 .

[15] M. Newman , L. Morris , J. H. Lee , Human-AI Music Creation: Understanding the Perceptions and Experiences of Music Creators for Ethical and Productive Collaboration , in: Proc. of the 24th Int. Society for Music Information Retrieval Conf ., International Society for Music Information Retrieval , Milan, Italy, 2023 , pp. 80 - 88 .

[16] C.-Z. A. Huang , H. V. Koops , E. Newton-Rex, AI Song Contest: Human-AI Co-creation in Songwriting , in: Proc. of the 21st Int. Society for Music Information Retrieval Conf ., International Society for Music Information Retrieval , Montréal, Canada, 2020 , pp. 708 - 716 .

[17] R. Louie , A. Coenen , C. Z. Huang , M. Terry , C. J. Cai , Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models , in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20 , Association for Computing Machinery, Honolulu, HI , USA, 2020 , pp. 1 - 13 . doi: 10 .1145/3313831.3376739.

[18] M. M. Suh , E. Youngblom, M. Terry , C. J. Cai , AI as Social Glue: Uncovering the Roles of Deep Generative AI during Social Music Composition , in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21 , ACM , Yokohama, Japan, 2021 , p. 11 . doi: 10 .1145/3411764.3445219.

[19] M. Civit , J. Civit-Masot , F. Cuadrado , M. J. Escalona , A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends , Expert Systems with Applications 209 ( 2022 ) 118 - 190 . doi: 10 .1016/j.eswa. 2022 . 118190 .

[20] O. Bown , Sociocultural and Design Perspectives on AI-Based Music Production: Why Do We Make Music and What Changes if AI Makes It for Us? , in: E. R. Miranda (Ed.),