Introduction

The rapid evolution of Artificial Intelligence (AI), and particularly GenAI, is reshaping how humans interact with technology. We are moving beyond more traditional interaction models towards scenarios of collaboration between humans and AI, which raises critical questions about how we ought to design the interaction with systems that promise to leverage the strengths of both. GenAI systems can not only perform classification tasks but also create artifacts like music, images, and text, blurring the line between traditional tools and active collaborators with creative abilities.

In the broad research area of Human-Centered AI (HCAI) [1,2,3], a perspective has emerged that seeks to leverage the strengths of both humans and AI, creating synergistic systems that surpass their individual capabilities [3,4,5]. This concept of human-AI collaboration arguably represents a more human-centered approach compared to human-in-the-loop models, as it makes the "best use of both human and AI capabilities, rather than the human simply being called Proceedings of the 1st International Workshop on Designing and Building Hybrid Human-AI Systems (SYNERGY 2024), Arenzano (Genoa), Italy, June 03, 2024. Envelope erictrngnt@gmail.com (E. Tron Gianet); luigi.dicaro@unito.it (L. Di Caro); amon.rapp@unito.it (A. Rapp) Orcid 0009-0009-3092-7089 (E. Tron Gianet); 0000-0002-7570-637X (L. Di Caro); 0000-0003-3855-9961 (A. Rapp) upon to do what the AI cannot yet manage in an AI led project" [3]. However, this emphasis on AI agency, which Sarkar [6] terms as an "agentistic turn", necessitates critical consideration. While it aligns with Bruno Latour's notion of agency [7], which extends agency across a network of human and non-human actors, showing how intricate the relationships between humans and their surroundings are, it could obscure the vast amount of "ghost work" [8] that powers AI, frequently conducted in the Global South at low wages (e.g., by data labelers) [6]. The concept of "collaboration" in human-AI interaction is certainly multifaceted, but, while some interactions with non-generative AI might exhibit collaborative aspects already [9], the emergence of GenAI presents unique challenges. These systems, capable of human-like outputs, blur the lines between tools and collaborators. This necessitates a deeper understanding of how human roles are redefined and how decision-making, creative processes, and information handling are reshaped.

To explore this further, we focus on a specific domain -music composition with AI -as an illustrative case study, to understand how collaboration and co-creation occur in practice, taking into account not only the user's objectives, needs, and motivations, but also the social and cultural context within which humans and GenAI systems may collaborate to achieve situated goals. The collaborative process of composing music raises intriguing questions about ownership, control, and intentions, all of which are central themes in the broader discussion of human-AI collaboration.

We will begin by reviewing recent literature on human-AI music composition and then propose a study investigating the situated practices [10] of music composition.

AI in Music Composition

The interest in computer-based music composition has steadily increased since the 1980s, with various techniques of Algorithmic Composition like Markov Models, Generative Grammars, and Genetic Algorithms. Later, as Neural Networks became more prominent, the advancements in Deep Learning led to the adoption of established architectures from fields like Computer Vision and Natural Language Processing in music generation as well [11].

While research on interactive musical systems had already highlighted the importance of applying user-centered approaches to support composers' processes of creation, exploration, and learning [12,13,14], more recent studies have delved into the specific challenges and strategies of co-creating music with AI.

Newman et al. [15] investigated through interviews how current AI tools concretely influence musical creativity. They then proposed a new model for developing ethical and productive collaborative AI tools for music. This model emphasizes the importance of clearly defined roles for AI and pays attention to how control is distributed. Their research reveals that users perceive as positive AI use cases those where they can maintain control, agency, intention, and choice throughout iterative cycles of generation and evaluation [15].

Huang et al. [16] studied the challenges and strategies of co-creating music with AI through a survey conducted during an AI song contest. The results highlight the importance of context awareness and user control, advising future AI systems to be designed to adapt to existing composers' practices rather than forcing new AI-determined workflows.

In another study investigating the use of "steering tools" for collaborative music creation, Louie et al. [17] identified two challenges in using GenAI: information overload and nondeterministic output. Their findings suggest that steering tools can enhance the user's sense of control, trust, and understanding of the AI system, resulting in a greater feeling of involvement in the creative process. The authors also comment that users often have pre-existing mental models about music composition and use them for tackling problems. These preconceptions should be considered so that both the AI models and the interfaces can be designed to be more intuitive, requiring less cognitive effort, and ultimately increasing the user's sense of agency. Louie et al. [17] also argue that the AI's role should adapt to the user's needs and the creative context: while ceding control is welcomed during exploratory phases, where the user is in search of unexpected inspiration, maintaining control over specific details becomes critical during production. Context thus plays a crucial role in shaping the dynamics of human-AI collaboration, influencing how control and agency are perceived.

Suh et al. [18] explored how AI systems can act as a "social glue" to support human-human collaboration in compositional practices. Their findings suggest that AI can facilitate the exchange of ideas and group cohesion, potentially reducing tensions typically associated with collaboration. They advocate for an intentional design of AI to further strengthen social collaboration. However, while AI can act as a support system, they also observed a potential shift in roles: participants reported feeling more like curators or co-producers, focusing on evaluating AI-generated material rather than actively developing ideas, leading to a weaker sense of creative involvement. This observation aligns with the findings of Civit et al. [19], who noted that "the composer became more of an arranger of different melodies" comparing their role to that of a producer managing misbehaving musicians. While this shift is viewed as a "very creative, fruitful process" by them instead, it still highlights the need for future AI systems to be adaptive to the creative context, user needs, and the composer's specific intentions.

This said, despite the growing interest in human-AI collaboration for music composition, current research seems to have limited scope: much of the focus is on evaluating the impact of existing generative systems, exploring strategies for composers to navigate their challenges, and integrating steering tools into current interfaces. Efforts to bridge the gap between AI music generation and the social and cultural complexities of music composition often lack engagement with field studies on compositional practices. Unlike humans, who are influenced by personal and social motivations and cultural context, AI systems currently operate solely on algorithms and predictive models, potentially limiting their effectiveness and failing to fully capture the nuances of human creativity [20]. The existing research gap highlights the importance of investigating current practices adopted by music composers, as well as their personal motivations, artistic sensibilities, the broader cultural and social context that influences their work, and the specific characteristics of various musical genres. This knowledge will be crucial for designing human-AI systems and creative workflows that effectively complement human strengths and intentions.

Music Composition as a Situated Practice

The existing literature emphasizes the need to better understand the complexities of music compositions in order to design human-AI collaborative music systems that account for both the individual and the socio-cultural context. While efforts like that of Hernandez-Olivan and Beltrán [11] to create generalized models of music composition are valuable for highlighting core principles, such rigid frameworks can hardly capture the dynamic and diverse nature of music creation, which is constantly evolving, shaped by genre, stylistic trends, personal choices, improvisation, and the unique socio-cultural setting where musicians operate.

To address this complexity, we propose an ethnographic approach that foregrounds the situated nature of music composition. This approach will delve into both the composers' personal motivations (e.g., creative aspirations and career goals) and the socio-cultural context that influences their work. Ethnography allows us to explore these aspects of music creation and the lived experiences of composers, and has already been applied and discussed as a method for AI research [21,22,23,24,25], feeding into a broader discussion about the lack of, and thus need to integrate, the social sciences into AI research to mitigate the exclusive use of quantitative methods, which lack of socio-technical perspective, and their uncritical and positivist use that often leads to ignoring the context and causes that bring to a certain outcome and the ways in which it occurs [26,21,27]. By employing ethnography, we look at users not simply as passive recipients of technology, but as active agents who shape its context, meanings, and consequences [28]. This approach emphasizes the context-dependent nature of music composition, laying the groundwork for designing synergistic human-AI systems that are sensitive to the nuances of human creativity and to the specific settings where music is created.

Our research aims to answer the following provisional questions:

• Can the use of Generative AI systems be actually considered a collaboration?

• How does the specific context of music creation influence decision-making, workflows, and creative choices when working with AI? • How might collaboration with Generative AI systems redefine the roles of composers in the creative process of composition? • How do composers and AI negotiate creative control and authorship within this collaboration?

To answer these questions, we propose an ethnographic approach combining semi-structured interviews and participant observation. We will interview 18 musicians with experience in composing for diverse genres, exploring topics like:

• Motivations, aspirations, creative sensibilities and personal workflow • The role of context in their music creation process • Experiences with AI tools and their perception of AI's role • How users perceive their own role evolving in collaboration with AI Following the interviews, we will conduct at least 60 hours of participant observation, to immerse ourselves in the composers' practice.

In summary, we aim to shed light on the situated practice of composing music and on how the broader context influences the collaborative process. We do this to inform the design of human-AI collaborative systems that support and empower, not replace, musicians in their creative practices. Additionally, this study can offer insights into human-AI collaboration beyond music, contributing to uncover design patterns for systems that are synergistic with human capabilities and responsive to the specific context in which they are used.

Preliminary findings

Our analysis of initial interviews reveals some preliminary findings:

(a) Intentionality Shapes and Gives Meaning to Music: Composers' creative intentions significantly impact their approach to composition. For example, whether they start with a melody, harmony, or specific sound (timbre) often depends on what they want to convey. While the existing literature acknowledges the link between intention, control, and creative agency, our study highlights the specific link between intention and a deeper meaning-making process, where composers strive to construct a "coherent discourse" through their music. How could we design human-AI collaborative systems that support and adapt to user intentions? (b) Music is already collaborative: Music composition and production are already an often collaborative process. From bandmates to collaborators, clients, and sound engineers, various stakeholders contribute to and have an interest in the final product. This raises the question of whether AI systems should be designed to enhance these existing human-human interactions, or whether they themselves should become an additional collaborator within a system in which creative control is already dynamically distributed and negotiated.

More findings will be shared during the workshop.