Generative methods provide rich, emergent ways to deal with many kinds of data. In this paper, we explore projects that listen to human motion, and respond through emergent generative art in ways that are inspired by dance and puppetry.
How do we respond to a body in motion? There are many things in the world that respond to a body in motion, for example, a dancer’s physical motion: dance costumes, or dance toys like fire poi, physically moved by or attached to the dancer, and subject to forces like drag, momentum, and centripetal force, depending on their materials. fields around the performer, as the dancer wades through water or smoke or tall grass, if they disturb curtains as they move a human partner, moving their body in response to their perception of their partner’s movement. An audience can sense tension, force, and connection, even if the two bodies never touch physically unattached collaborators, who, like the human dance partner, ”listen” to the movement and respond. This may be musicians or lighting directors who react collaboratively with the movements, or non-human works like interactive projections
All of these phenomena “listen” to the movement of the dancer and respond in some way. As designers of generative systems, we can build systems that operate like any of these real-world responsive systems: our systems can be costumes, fields, physically-connected agents or expressivelyconnected agents, or to have systems that combine the responsive properties of any of these examples. Observing the range of reactive systems that occur in dance practice reminds us not to limit ourselves to only one kind of ”dance partner”. In this paper, I reflect on some works where insights from dance (and other movement arts, like puppetry) inform how I can use computers to listen to movement, and respond, collaborate, or amplify that movement.
This paper is emphatically not about discrete detection
and categorization of gesture. Though we have now spent
most of a decade with moderately effective motion-tracking
(Kinect, Wii, Leapmotion, Oculus Touch), none of them
have sparked the motion-control revolution that each one
seemed to promise. In previous work,
Instead, this paper is about how we can use a variety of algorithms (some “artificial intelligence”, some not, this paper won’t quibble about the definition) to listen and respond to continuous body movement.
How can we listen to a body in motion? As mentioned earlier, artists have many sensors from which to choose. Some are relatively basic forms of sensors like accelerometers, gyroscopse, distance sensors, and bend sensors. Some sensors operate by processing image data, often via machine learning or various statistical methods, from either a single camera or multiple cameras (often assisted by invisible infrared projections), and several achieve “dead-reckoning” by combining camera, GPS, and accelerometer data.
It is easy to list the ways that we can listen to motion. But
let us instead examine what motion we listen to, and why.
Lived human experience informs us that some forms of
motion feel better than others. For example, holding arms
extended and still is wearying
One of my first Kinect projectspresented at SF Bay Area 3D Vision and Kinect Hacking, 2/1/2012 took advantage of this. In Kinect Poi, the player used their arms to swing digital fire poi, which left trails of sparks and stars as they swung them. They could then retrace their trails to collect the stars left behind on a previous swing. This had several advantages. The poi were simulated as particles, with continuous acceleration forces, so even when the Kinect sensing momentarily dropped (frequently in old models), the particle continued to move smoothly, without any of the glitching of one-to-one control. Using force-based control, rather than position based control, created a natural “anti-aliasing” effect for the motion input. Finally, the perceived weight of a player’s hand increases as they swing their arm, creating the weighted, force-based feedback that was missing from most Kinect experiences. The motion that this art used was the type of motion that felt best for an interactor, and the interaction/“game” was built around that, rather than the other way around.
Lack of haptic feedback or tactile resistance is common in motion control experiences, but this is not unavoidable. In Squishy Touchscreen 1, a user interacts with a soft spandex membrane stretched over a wooden frame. A laser projector backprojects an image onto the membrane, and a Kinect, placed under the projector, maps the deformation of the membrane into a grayscale image. This project was inspired by Kinect musical instruments (like Tim Thompson’s Space Palette 2) where the user waved their hands through the air. Few instruments, with the exception of a theremin, have no tactile resistance feedback in this way, so I wanted to create an instrument that you could feel pushing back. Spandex acts as a spring and has resistance that increases as you press harder against it. It felt good to press against it, to stroke the screen and feel the drag against your fingers. Additionally, the Kinect could see any deformation of the surface, so the user could press their palm, fingers, face, or any object into the screen, and it would change the character of the deformation.
In another early prototype touch-“screen” (circa 2005), 1(2010, https://vimeo.com/217033311 2https://spacepalette.com/ users dragged their fingers through a tabletop full of black sesame seeds (a webcam could see fingertips through the glass bottom of the table). The resistance and physical properties of the seeds provided haptic feedback of resistance, and also produced a very satisfying sound and smell when disturbed. Pressing a finger harder into the tabletop make a bigger “blob” for the image detection to track, and the size of the area of contact could also be felt by the user by the texture contrast between seed and glass.
In Kinect Poi, Squishy Touchscreen, and the Black Sesame Table, the “sensors” themselves are dance partners. Their physical properties (resistance, centrifugal force, inertia, sound, even scent) and the way they encourage interaction (through softness, texture, the pleasure of inertial movement) form a connection with the interactor even before we consider how the digital components of the systems will respond to that input.
In Mueller and Isbister’s “Movement-Based Game
Guidelines”, they encourage motion control game designers to
not focus intently on game-style interaction: “Start by
providing feedback on the movement itself, without too much
worrying about scores, multipliers etc. [..] Provide several
forms of feedback, but do not require players to engage all
of them: better to let players choose which ones to engage
based on their cognitive abilities, and shift their attention as
mastery grows.”
Fortunately, one of the major advantages and disadvantages of a thick stream of continuous motion data is that while it cannot be handled by the if-statements of traditional game logic, it does provide an excellent seed for generative methods. Often these methods need not even be complex to be engaging: they merely have to be responsive. The most successful “app” on the Squishy Touchscreen was a rainbow-remapping of the depth field, which I had made as a debug utility. As one pressed harder into the screen, the colors changed around it, like reaching one’s hand into a rainbow-colored geode. The stretchiness of the spandex also deformed around whatever was pressed into it, so a hand would become outlined in rings of hand-shaped color. The material was “responding” to the interaction, even before the algorithm got to it.
More complex responses can be designed by passing the
continuous motion stream into a pipeline of generative
methods3. Idle Hands 4 was designed as an installation in an
art festival, projected on a wall, that the users control via
a Leapmotion. Giant hands (the projection was about 10 feet
across) clenched and unclenched even when the controller
was idle. When controlled by the user, the hands
mostly3see
One interesting pattern that I discovered with Idle Hands
was the importance of flexibility of control. Like the
Kinectcontrolled poi, any motion control system has moments
where tracking drops frames, or the interactor walks away.
In these moments, a virtual agent can take over for the
interactor. This can be done to patch or smooth the motion,
but it can also be used to playfully resist the user’s control.
Is this a direct mirror, or an intelligent partner mimicking
your movements, only to break free with some
improvisation? Previous projects
My most recent motion-reactive art is on dance-reactive
puppets.5 This project was funded by the Google Creative
Lab as an experiment to use their Posenet Tensorflow
detection algorithm (Oved ). This algorithm produces
similar skeleton data to the Kinect, only instead of using
infrared dots and multiple cameras, it uses machine-learning
on normal RGB webcam data, potentially reaching a vastly
larger audience than the Kinect ever has. This project was
inspired by Nick Cave’s Sound Suits
the Sound Suits and the Muppets do with their dancers and
puppeteers. I adopted some ideas from the Spore creature
creator (Hecker ), making the bodies based on tubes, but
created more emergent and surprising forms based on the
tubes (super-ellipse cross-sections, wrinkles or oscillations
along the length of the tube). I also used Spore’s
wigglesand-jiggles system of secondary motion (and past work on
secondary motion in generative animation
At the time of development, I did not have access to the live stream of data from the webcam (that part of the technology was unreleased) so I had to create synthetic data, from a dancing virtual forward-kinematics-animated body in 3.JS, which could generate the data that we anticipated receiving from the machine-learned component. I set up the data-generating virtual body so that it could be driven via a Leapmotion (translating the finger movement into joint movement), the potential future Posenet data, music data, or some combination of all three. It also had a slider that controlled independent noise-controlled data (autonomy) versus user-provided data (mirror mode). One could imagine this slider being driven by anything, including the agent’s “boredom” with the player’s lack of movement.
The released version of Posenet yields only 2D point data, not the 3D of the Kinect, so I developed a very rudimentary system to jiggle the 3D synthetic body until it matches the 2D detected points 6 It is far from accurate, yet like much of the work discussed here, it seems that the accurate movement is far less important than continuous, reactive, responsive, and emergent movement, and it is an enjoyable “presence” to interact with.
Dance (and movement arts like puppetry) have a long and developed history of turning human movement into something pleasurable, alien, expressive, or transcendent. Movement augmentation both listens to and responds to user movement. Some patterns of listening/responding are costumes, fields, physically-connected agents or expressively-connected agents.
Both Squishy Touchscreen and the Black Sesame table were fields that the user disturbed with their motion, creating eddies and deformations in the physical interface and also in the digital response. Idle Hands is a field which the user manipulates with their fingers, but while it lacks the physically responsive interface, the seamlessly responsive interaction created an impression of physical touch. The Kinect Poi and the dance puppets are costumes: they are linked to the user’s movement, but have secondary motion that amplifies and elaborates on that emotion. Like the virtual partner Lumen.AI project, the puppet is an autonomous agent, but can move continuously between being a autonomous partner or a costume as its agency is dialed up or down. My projects do not have a strong expressively-connected agent component (I prefer more directly-reactive agent action), but this would be an avenue for exploration for either these projects or any other generative movement-reactive system, such as a musical or visual background improvisation based on some generative interpretation of user movements. These categories only begin to outline the range of how interaction in real world dance/movement arts can inspire and inform digital systems; much more exploration in the vast world of dance culture is possible.
posematch