Constraining Control in Mixed-Initiative Musical Interfaces

Abstract

Research on musical instrument design suggests that deliberate constraints can offer new creative experiences to the performer. At times, design constraints are physically embedded in the instrument to limit the interaction possibilities; in other cases the constraints are given by delegating part of the control to the machine. In this paper we propose a case study related to the latter form of constraints: in Chimney the musician delegates control on timing to an autonomous agent. Elaborating on opinions collected from musicians interacting with Chimney, we reflect on the consequences of delegating part of the control to the machine. Without the possibility of influencing the temporal evolution of the piece, the human performer is pushed to find creative workarounds and to surrender to a more balanced collaboration with the machine.

-

Copyright ©2017 for this paper is held by the author(s). Proceedings of MICI 2017: CHI Workshop on Mixed-Initiative Creative Interfaces.

Author Keywords

digital musical instruments; creativity; constraints; interface design

ACM Classification Keywords

H.5.5 [Sound And Music Computing]: Systems

Introduction

In Mixed-Initiative Creative Interfaces humans and machines collaborate to produce creative outputs. Efforts are usually made for the two parts to dialogue, each employing their own assets and delegating their limitations to the other. However, efforts can also be made to specifically constrain human control on the creative process. Constraints are not ruthless attempt to limit human creativity; rather the opposite can be true as advocated by Margaret A. Boden in her seminal book on human creativity [ 1 ] : “People often claim that talk of ’rules’ and ’constraints’...must be irrelevant to creativity, which is an expression of human freedom. But far from being the antithesis of creativity, constraints on thinking are what make it possible...Constraints map out a territory of structural possibilities which can then be explored, and perhaps transformed to give another one” In the musical domain, physical constraints contribute to characterise the instrument expressive scope and influence its playing possibilities and the development of a personal style [ 8 ]. For instance, the constraints of the piano define its expressive character, which, in turn, defines the pianist’s style of playing and the compositions possibilites [ 4 ]. New digital musical instruments (DMIs) also sometimes integrate design constraints for creative explorations. Such constraints can limit the number of inputs and the outcomes that are mapped to those inputs. For example, Gurevich and colleagues designed a purposely over-simplified instrument, in which performer interaction is very limited [ 3 ]. Despite its limitations, or perhaps because of them, the performers exhibited a significant degree of stylistic diversity in their interaction with the instrument.

In a similar study, Zappi’s and McPherson’s Cube Instrument was designed to show the role of interaction constraints in encouraging “creative (mis)uses of technology” [ 8 ]. Several performers that used the Cube Instrument reported that constraints themselves were conducive to exploring subtle musical variations.

Another form of constraint in musical interaction is given by delegating part of the control to a machine. This is the case for conductor systems, which are essentially tools that allow musicians to control the playback of pre-recorded music. Performers only control timing, dynamics, and other expressive factors and the machine worries about getting the notes right [ 5 ]. These systems have attracted the interest of the general public (Guitar Hero1, Magic Fiddle 2 and similar applications are forms of conductor programs) by offering a simple interaction that does not require extensive training on the instrument [ 2 ].

This paper proposes another example in which part of the control is delegated to the machine. Chimney, a software instrument developed by two of the authors [ 6 ], forces the musician to delegate timing decisions to a non-responsive autonomous agent. As a consequence, the human agent has limited control on the temporal evolution of the piece. The behaviour of Chimney is in a sense the opposite kind of delegation compared to a conductor system. With conductor systems a musicians can mainly get time control; with Chimney timing is the main thing the musician lacks. The specific behaviour of the agent is secondary to the scope of this paper, which rather focuses on the consequences of divesting control on timing to an unpredictable algorithmic agent. The implications on the experience of the performer of delegating control on timing to the machine are presented in the Discussion section.

Chimney: Delegating Timing

An example of DMI in which the musician divests control on timing to the machine is offered by Chimney [ 6 ]. In this virtual DMI the control on the temporal evolution of the piece is delegated to a computer agent, an algorithmic random walker. The walker autonomously roams throughout a virtual space following a specific statistical distribution modelled on an adapted version of the Perlin Noise [ 7 ]. The musician can interact with the composition by adding to the virtual space pre-recorded sonic material. These sound sources are displayed as circles that can be resized, repositioned, or deleted in real time by the musician. When a new circle is added to the canvas, the system sets it in idle state 1https://www.guitarhero.com/uk/en/ 2https://www.smule.com/sunset/magicfiddle and mutes it. As the random walker enters a circle, the amplitude of the sound connected to that circle increases. The maximum level is reached when the walker is at the centre of the circle.

Player interaction is then reduced to deciding the sonic objects, and their likelihood of being played. Under these conditions, the musician cannot organise the temporal structure, which is entirely controlled by the algorithmic agent. An informal evaluation was conducted at a public concert in which Chimney was perfomed in a duo with a trumpet [ 6 ]. Both the musician that was controlling Chimney (the second author of this paper) and the trumpeter were interviewed at the end of the performace.

The lack of control over timing caused interesting experiences for both musicians. The performer who was controlling Chimney reported emotional reactions that usually do not belong to the palette of emotions in music. Unable to control every aspect of the music, he could only wish for the algorithm to answer in a particular way. As a consequence, he experienced hope, surprise, frustration, and resignation. Other interesting comments were collected from the trumpeter. Understanding all the details of the music played by Chimney was particularly demanding. In particular, as opposed to traditional improvisation, he did not feel free to propose new musical material. He was always answering to Chimney proposals. The only exception to such approach concerned big changes in dynamics (e.g. very intense crescendo or diminuendo). In this cases, the trumpeter explicitly communicated to the Chimney performer to add sonic material to the canvas to obtain a crescendo, and to remove some for the diminuendo. Furthermore, the trumpeter commented that the specific behaviour of Chimney of being non-responsive fostered him to find novel musical strategies. For instance, in those situations in which he would have preferred a higher complexity in the music, he compensated for this perceived deficiency by increasing the rhythm complexity or the loudness of his instrument.

Discussion

This paper reviewed and compared different ways in which constraining the interaction possibilities of a peformer can be a vector of new forms of creativity. In particular, our preliminary investigations with Chimney suggest that delegating some initiative to the machine produces a set of creative reactions and feelings that are at times similar and at times richer than simply constraining the interaction possibilies. For instance, [ 8 ] found that players who got more degrees of freedom on the Box Instrument (they could control pitch) exhibtited less unexpected techniques than those with a single degree of freedom (no control on pitch). In this case, players who could not control such an important musical feature needed to came up with original approaches to playing. Similarly, the musicians that performed with Chimney, even if they could control every other aspect of the performance, lacked control of timing, which fundamentally changed how the performance was structured.

One difference that we found is presence of a range of affective states like surpise and hope that are present in Chimney and other MICIs but seem to be absent from simple DMIs and conductor systems. The self-frustration experienced by the performer that operated Chimney is a typical feeling experienced in music - for instance by instrumentalists when they cannot play a passage right. However, in this case, the frustration was accredited to the impossibility of precisely expressing one’s personal musical initiative, no matter how hard he could try.

Finally, it should be noted that in typical MICIs, the computer acquires the status of creative agent by using forms of artificial intelligence that actively take decisions on tasks that are typically performed by humans. Although the artificial intelligence element is limited or absent in the examples presented in this paper, we believe that our reflections about the consequences of divesting control to a machine can be of interest for the disussions at the workshop.

References

[1] Margaret A Boden . 2004 . The creative mind: Myths and mechanisms . Psychology Press.

[2] Elaine Chew and Andrew McPherson. Forthcoming . Performing music: humans, computers and electronics . In The Routledge Companion to Music Cognition, R Ashley and R Timmers (Eds.). NY: Taylor and Francis.

[3] Michael Gurevich , Paul Stapleton, and Adnan Marquez-Borbon. 2010 . Style and Constraint in Electronic Musical Instruments. . In NIME . 106 - 111 .

[4] Thor Magnusson . 2010 . Designing constraints: Composing and performing with digital musical systems . Computer Music Journal 34 , 4 ( 2010 ), 62 - 73 .

[5] Max V Mathews . 1991 . The radio baton and conductor program, or: Pitch, the most important and least expressive part of music . Computer Music Journal 15 , 4 ( 1991 ), 37 - 46 .

[6] Fabio Morreale and Raul Masu . 2016 . Renegotiating Responsibilities in Human-Computer Ensembles . Proceedings of Colloquio di Informatica Musicale ( 2016 ), 41 - 45 .

[7] Ken Perlin . 1985 . An image synthesizer . ACM Siggraph Computer Graphics 19 , 3 ( 1985 ), 287 - 296 .

[8] Victor Zappi and Andrew McPherson . 2014 . Design and use of a hackable digital instrument . In NIME.