Layering the Choreographic Process: Making Dance Work with Machine Learning Kate Sicchio, PhD Abstract New York University This paper discusses a new performance by the author Tandon School of Engineering that was made in part by machine learning algorithms. Brooklyn, NY 11237, USA Working with the t-SNE algorithm to visualize data, a sicchio@nyu.edu choreographic score can become performed through a layer of images, live coding and an improviser performer. The final performance aims to produce new possibilities for live performance through using code to traverse clusters of media that the algorithm has produced. Author Keywords Choreographic score; machine learning; dance improvisation; t-SNE ACM Classification Keywords ACM now Copyright © 2017 for this paper is held by the author(s). H.5.m. Information interfaces and presentation (e.g., Proceedings of MICI 2017: CHI Workshop on Mixed-Initiative Creative HCI): Miscellaneous Interfaces. Introduction When working with dance and technology it is common for work to be developed where the dancer and their movement is used to reflect the technology, or even demonstrate the technology. But when choreographers use machine learning techniques, there are interesting results when creating layered approaches, or mixed- initiative creative interfaces to influence the movement of these types of movements remain in the piece, in the vocabulary in new ways. form of photographs. For example, Wayne McGregor has worked with the The second layer is the camera. Time lapse Choreographic Language Agent, an AI agent used to photography was used to capture the original augment his process since 2004. The one iteration of movement. During this process a GoPro photographed this was Becoming, a tool developed with Marc Downie two photos per second. Over 3000 images from a and Nick Rothwell. The aim is to provoke new dance improvisation session were captured. Some movement creation in the dance studio, which is later photos were clear shapes, others are bizarre half shaped by McGregor to become a layer process in moments of movement. The movement is transformed creating a performance output. Both of these pieces in this layer without any machine learning just by trying rely on the computation as part of the choreographic to make it static. practice. This layered approach does not celebrate the computer as the choreographer but simply uses it in a way to inspire or create new movements that may be have been part of the choreographer or dancer’s cannon. Untitled Algorithmic Dance 2 Untitled Algorithmic Dance 2 is a dance performance where the choreography is generated through a series of interactions between humans and machines. It is a layered approach that creates a system with various places for interpretation, by both people and algorithms. This is no simple mapping of input to output, but several processes that layer to become a Figure 1: Example of clustering of images as performed by the dance. t-SNE. The first layer in this process is human. A dancer improvises to create movement. This becomes the The next layer is the actual layer of machine learning content that is layer used to produce a choreographic using Gene Kogan’s code for the t-SNE algorithm score for the performance. Some of the movement (originally by van der Maaten) in OpenFrameWorks to in material is just warming up, some is dancing for the effect cluster the images. This is an untrained machine sake of movement, some is purely gestural such as me learning algorithm used to visual large amounts of grabbing a water bottle, and some responds to the data. The algorithm finds the similarities between prompt of creating movement for a camera. For now all photos and places them into a grid. What you see right working with machine learning that are still human away is a new dance. A score made just through centered and uses this process as just a tool. recognition and reorganization. Conclusions However, one does not have to simply use this grid, and certainly using the grid as a score is only one Untitled Algorithmic Dance 2 explores the use of the t- possibility with this re-imaged plot of movement. SNE algorithm to create live choreographic scores from Another human layer becomes part of the system a collection of time lapse photos. It works with a within this piece. The navigation of the clusters of complex layering of human and machine interactions to captured movement is done live by the choreographer. create a performance in which machine learning In this first version of the piece it is very simply done influences the final piece. Through this process an through navigating the grid with a mouse. The interruption of a choreographic process via AI has choreographer controls the path of the score, the begun and will be developed further with different timing of the score and most importantly can respond initiatives such as motion capture. to the dancer performing the score. References Which brings me to the final layer of the system, 1. Kate Sicchio. 2014. Hacking Choreography: Dance another human. There is an interpreter at the end of and Live Coding. Computer Music Journal. Vol. 38, Iss. this system, which is a dancer. The dancer sees the 1 (March 2014 ) pp31 – 39. image that the choreographer has currently selected and responds. For this performance, the dancer has 2. Studio Wayne McGregor. 2004. Choreographic agency to respond however they like – they may copy Language Agent. the shape of the body in the image, or make a http://waynemcgregor.com/research/choreographic- contrapuntal shape. They may move in a direction language-agent indicated by the image or move in the opposite way. The dancer has the final say in the movement being 3. Laurens van der Maaten. 2014. Accelerating t-SNE generated. The human dancer has the ultimate control. using Tree-Based Algorithms. Journal of Machine Learning Research 15 (2014) pp 1-21. And while this layered approach works in creating a composition that we all recognize as a dance piece, it is 4. Gene Kogan. 2016. infused throughout with new tools to push the https://github.com/genekogan/ofxTSNE choreographic possibilities of a single choreographer. This resonates with McGregor in that it is an influence on the movement and not the full development of the movement. Perhaps this also demonstrates ways of