Constraining Control in Mixed-Initiative Musical Interfaces Fabio Morreale Raul Masu Abstract Centre for Digital Music interAction Lab Research on musical instrument design suggests that de- School of EECS DISI liberate constraints can offer new creative experiences to Queen Mary University of University of Trento the performer. At times, design constraints are physically London, UK Italy embedded in the instrument to limit the interaction possibil- f.morreale@qmul.ac.uk raul.masu@unitn.it ities; in other cases the constraints are given by delegating Andrew McPherson part of the control to the machine. In this paper we pro- Centre for Digital Music pose a case study related to the latter form of constraints: School of EECS Queen Mary University of in Chimney the musician delegates control on timing to an London, UK autonomous agent. Elaborating on opinions collected from a.mcpherson@qmul.ac.uk musicians interacting with Chimney, we reflect on the con- sequences of delegating part of the control to the machine. Without the possibility of influencing the temporal evolution of the piece, the human performer is pushed to find creative workarounds and to surrender to a more balanced collabo- ration with the machine. Author Keywords digital musical instruments; creativity; constraints; interface design ACM Classification Keywords Copyright ©2017 for this paper is held by the author(s). Proceedings of MICI 2017: H.5.5 [Sound And Music Computing]: Systems CHI Workshop on Mixed-Initiative Creative Interfaces. Introduction In Mixed-Initiative Creative Interfaces humans and ma- chines collaborate to produce creative outputs. Efforts are usually made for the two parts to dialogue, each employ- ported that constraints themselves were conducive to ex- ing their own assets and delegating their limitations to the ploring subtle musical variations. other. However, efforts can also be made to specifically constrain human control on the creative process. Con- Another form of constraint in musical interaction is given by straints are not ruthless attempt to limit human creativity; delegating part of the control to a machine. This is the case rather the opposite can be true as advocated by Margaret for conductor systems, which are essentially tools that allow A. Boden in her seminal book on human creativity [1] : musicians to control the playback of pre-recorded music. Performers only control timing, dynamics, and other ex- “People often claim that talk of ’rules’ and ’constraints’...must pressive factors and the machine worries about getting the be irrelevant to creativity, which is an expression of human notes right [5]. These systems have attracted the interest of freedom. But far from being the antithesis of creativity, con- the general public (Guitar Hero1 , Magic Fiddle 2 and similar straints on thinking are what make it possible...Constraints applications are forms of conductor programs) by offering map out a territory of structural possibilities which can then a simple interaction that does not require extensive training be explored, and perhaps transformed to give another one” on the instrument [2]. In the musical domain, physical constraints contribute to This paper proposes another example in which part of the characterise the instrument expressive scope and influence control is delegated to the machine. Chimney, a software its playing possibilities and the development of a personal instrument developed by two of the authors [6], forces the style [8]. For instance, the constraints of the piano define musician to delegate timing decisions to a non-responsive its expressive character, which, in turn, defines the pianist’s autonomous agent. As a consequence, the human agent style of playing and the compositions possibilites [4]. has limited control on the temporal evolution of the piece. The behaviour of Chimney is in a sense the opposite kind of New digital musical instruments (DMIs) also sometimes in- delegation compared to a conductor system. With conduc- tegrate design constraints for creative explorations. Such tor systems a musicians can mainly get time control; with constraints can limit the number of inputs and the outcomes Chimney timing is the main thing the musician lacks. The that are mapped to those inputs. For example, Gurevich specific behaviour of the agent is secondary to the scope and colleagues designed a purposely over-simplified in- of this paper, which rather focuses on the consequences strument, in which performer interaction is very limited [3]. of divesting control on timing to an unpredictable algorith- Despite its limitations, or perhaps because of them, the per- mic agent. The implications on the experience of the per- formers exhibited a significant degree of stylistic diversity in former of delegating control on timing to the machine are their interaction with the instrument. presented in the Discussion section. In a similar study, Zappi’s and McPherson’s Cube Instru- ment was designed to show the role of interaction con- straints in encouraging “creative (mis)uses of technology” [8]. Several performers that used the Cube Instrument re- Figure 1: Screenshot of Chimney. The white path is the trace left by the algorithmic walker. The circles are the sonic material added and mutes it. As the random walker enters a circle, the am- by the human performer. plitude of the sound connected to that circle increases. The maximum level is reached when the walker is at the centre of the circle. Player interaction is then reduced to deciding the sonic ob- jects, and their likelihood of being played. Under these con- ditions, the musician cannot organise the temporal struc- ture, which is entirely controlled by the algorithmic agent. An informal evaluation was conducted at a public concert in which Chimney was perfomed in a duo with a trumpet [6]. Both the musician that was controlling Chimney (the second author of this paper) and the trumpeter were interviewed at the end of the performace. The lack of control over timing caused interesting experi- ences for both musicians. The performer who was control- ling Chimney reported emotional reactions that usually do not belong to the palette of emotions in music. Unable to Chimney: Delegating Timing control every aspect of the music, he could only wish for the An example of DMI in which the musician divests control on algorithm to answer in a particular way. As a consequence, timing to the machine is offered by Chimney [6]. In this vir- he experienced hope, surprise, frustration, and resignation. tual DMI the control on the temporal evolution of the piece is delegated to a computer agent, an algorithmic random Other interesting comments were collected from the trum- walker. The walker autonomously roams throughout a vir- peter. Understanding all the details of the music played by tual space following a specific statistical distribution mod- Chimney was particularly demanding. In particular, as op- elled on an adapted version of the Perlin Noise [7]. posed to traditional improvisation, he did not feel free to propose new musical material. He was always answering to The musician can interact with the composition by adding to Chimney proposals. The only exception to such approach the virtual space pre-recorded sonic material. These sound concerned big changes in dynamics (e.g. very intense sources are displayed as circles that can be resized, reposi- crescendo or diminuendo). In this cases, the trumpeter tioned, or deleted in real time by the musician. When a new explicitly communicated to the Chimney performer to add circle is added to the canvas, the system sets it in idle state sonic material to the canvas to obtain a crescendo, and to remove some for the diminuendo. Furthermore, the trum- 1 peter commented that the specific behaviour of Chimney https://www.guitarhero.com/uk/en/ 2 https://www.smule.com/sunset/magicfiddle of being non-responsive fostered him to find novel musi- cal strategies. For instance, in those situations in which he Finally, it should be noted that in typical MICIs, the com- would have preferred a higher complexity in the music, he puter acquires the status of creative agent by using forms compensated for this perceived deficiency by increasing the of artificial intelligence that actively take decisions on tasks rhythm complexity or the loudness of his instrument. that are typically performed by humans. Although the arti- ficial intelligence element is limited or absent in the exam- Discussion ples presented in this paper, we believe that our reflections This paper reviewed and compared different ways in which about the consequences of divesting control to a machine constraining the interaction possibilities of a peformer can can be of interest for the disussions at the workshop. be a vector of new forms of creativity. In particular, our pre- liminary investigations with Chimney suggest that delegat- References ing some initiative to the machine produces a set of creative [1] Margaret A Boden. 2004. The creative mind: Myths reactions and feelings that are at times similar and at times and mechanisms. Psychology Press. richer than simply constraining the interaction possibilies. [2] Elaine Chew and Andrew McPherson. Forthcoming. Performing music: humans, computers and electron- For instance, [8] found that players who got more degrees ics. In The Routledge Companion to Music Cognition, of freedom on the Box Instrument (they could control pitch) R Ashley and R Timmers (Eds.). NY: Taylor and Fran- exhibtited less unexpected techniques than those with a cis. single degree of freedom (no control on pitch). In this case, [3] Michael Gurevich, Paul Stapleton, and Adnan players who could not control such an important musical Marquez-Borbon. 2010. Style and Constraint in Elec- feature needed to came up with original approaches to tronic Musical Instruments.. In NIME. 106–111. playing. Similarly, the musicians that performed with Chim- [4] Thor Magnusson. 2010. Designing constraints: Com- ney, even if they could control every other aspect of the per- posing and performing with digital musical systems. formance, lacked control of timing, which fundamentally Computer Music Journal 34, 4 (2010), 62–73. changed how the performance was structured. [5] Max V Mathews. 1991. The radio baton and conductor program, or: Pitch, the most important and least ex- One difference that we found is presence of a range of pressive part of music. Computer Music Journal 15, 4 affective states like surpise and hope that are present in (1991), 37–46. Chimney and other MICIs but seem to be absent from sim- [6] Fabio Morreale and Raul Masu. 2016. Renegotiating ple DMIs and conductor systems. The self-frustration expe- Responsibilities in Human-Computer Ensembles. Pro- rienced by the performer that operated Chimney is a typical ceedings of Colloquio di Informatica Musicale (2016), feeling experienced in music - for instance by instrumen- 41–45. talists when they cannot play a passage right. However, in [7] Ken Perlin. 1985. An image synthesizer. ACM Sig- this case, the frustration was accredited to the impossibility graph Computer Graphics 19, 3 (1985), 287–296. of precisely expressing one’s personal musical initiative, no [8] Victor Zappi and Andrew McPherson. 2014. Design matter how hard he could try. and use of a hackable digital instrument. In NIME.