Most brainwave-based performances adopt spectral decomposition of the EEG signal into several frequency bands to control different soundsynthesis variables. The performer is stable, usually sitting in a meditative way. It is rather difficult for the audience to determine what the performer is actually controlling and in which way. Traditionally, the audience witnesses a performance based on the telekinetic dream of brain-control but yet this is not externalized in an objective way. I believe it is possible to extend these performative limits by using a gamification paradigm and developing analytical tools that go beyond spectral analysis, allowing a more responsive and clear control of the variables involved in the performance for both the performer and the viewer. In this paper I present the qualitative results obtained using these approaches in Fragmentation - a brain-controlled performance. From the informal feedback of the audience after the performance, it appears evident how the public is able to experience several aspects of the brain of the performer: predicting his intentions, sensing effort and struggle, and spotting his mistakes. These aspects create a strong experience and involvement for the audience during the performance.
In most of the past and recent brainwave performances, the performers lie with
some EEG sensor on his their head, in a static pose, while music is played from
loudspeakers [
The way control is achieved is a fundamental aspect of every brainwave
performance. Past performances principally used the electro encephalogram (EEG) signal and
its different bands (alpha, beta, delta gamma) to control different parameters of the
sound synthesis (spectral analysis) [
Because of the complexity of brain analysis and Rosenboom's approach,
contemporary practice [
My recent research addressed the EEG's issues of direct control and performativity in order to achieve a deeper involvement of the audience during a brainwave-driven performance. In Fragmentation: a brain-controlled performance, I adopted artificialintelligence algorithms using cross-correlation to: • train the system: by storing in a set of templates the signal of brain states associated with specific thoughts, • measure the likelihood that the incoming signal of the performer during the show is matching one of the stored templates. 2
Present Approach: Fragmentation – a Brain-Controlled Performance The performance is an allegory of the modern man: exposed to aggressive stimulation and overwhelming data streams, he is daily asked to take quick decisions and be able to switch from several environments, in which he plays different roles subjected to varying rules and degrees of responsibility. The aim of the piece is to let the audience experience different degrees of mental stress, stimulation and saturation through a physiological live-scan of the performer’s brain, exposed to few extreme but common everyday situations. In this way, Fragmentation tries to bring the audience deeper in contact to the performer's brain through the exposure of his mind activity in form of individual thoughts. Throughout the piece, music and visuals are used in various ways to become a translation (a sort of visual-sonification) of the brain activity.
On a technical point of view, my approach is a continuation of Rosemboom's early experiments and Miranda's proposition. Instead of using spectral decomposition, which has proven to be difficult to rationally control by the performer, the techniques utilized in Fragmentation investigate the temporal domain of the brain signal. The main idea is to observe when a pattern reoccurs in the brain signal. If thoughts have a consistent translation into an electrical impulse, then repeating thoughts would generate electrical patterns, thus signals. Hence, analyzing pattern reoccurrence is a way of tracking reoccurring thoughts. The chain could be reversed: the performer could be trained to generate thoughts in an exact way and the system could be trained to recognize those at every instance.
The first important step of pattern recognition is the extraction of reliable patterns for matching. Patterns are calculated by asking the performer to concentrate on specific thoughts, while recording EEG inputs. From the spectral analysis of the signal, onsets are detected to isolate relevant parts. Because noise is uncorrelated by definition, while the signal is not, adding several of the signal parts creates a destructive interference of noise and strengthens the relevant parts, thereby letting the relevant signal emerge. It took several attempts to find the proper set of 'thoughts' that triggered reliable brain states; especially stable are 'kinetic thoughts': e.g., imagining moving specific parts of the body, such as left and right limbs.
Cross-correlation was used to compare how much the incoming signal matches the set of pre-stored patterns in the system. Cross-correlation is a measure of similarity between two waveforms as a function of a time lag applied to one of them. It is commonly used for searching a long-duration signal for a shorter, known feature. Considering two waveforms f and g, where f is the short stored pattern and g is a longer signal representing the real time samples of EEG, the cross-correlation at the sample n would be: (1) where n and m are sample positions and is the complex conjugate of f. a) b) c)
This pattern-recognition approach through cross-correlation demonstrated to be relatively solid but very restrictive. The performance of the system was evaluated on the recognition of three thoughts out of 100 trials. The system could correctly recognize 60% of the generated thoughts. However the algorithm was very unstable when trying to identify more than three thoughts. As a consequence, the whole system had to be conceived with direct control on only three variables; and because from a signaldetection perspective it is unclear what is a combination of two thoughts (and it is still debatable whether it is possible for humans to generate two thoughts simultaneously), the system could detect changes in only one variable at a time. It became immediately evident how important for the performance was the choice of the mapping strategy: to reliably connect three variables to generate music parameters and still achieve a reasonable degree of expressivity for the performer. 3
Mapping The mapping of the control variables was conceived to achieve both a visible and reliable control of the sound engine while allowing the performer to control several high-level parameters of the composition. I followed two directions, by splitting the signal analysis chain in two parts. The first part could sample in real time parts of the brain signal and loop them to create a direct sonification of the brain. I call this the soloist brain and it’s activated in specific parts of the structure of the piece.
The second approach aimed at achieving a control on the overall structure of different stochastic instruments. I called this the brain-conducted orchestra, accompanying the soloist brain. For this approach I used the previously described techniques of pattern recognition to reduce the possible control variables from the complex input sensor data to three variables. I then used the paradigm of terrain exploration: the performer drives an avatar through a computer-generated maze in search for the exit, in a video-game paradigm. The performer uses three thoughts to turn left, turn right and move forward the avatar. The maze is simultaneously projected on screen for the audience and onto stage for the performer so that he can physically follow his avatar while performing. The visuals projected onto the performer and stochastic music loops are triggered and modified by the avatar's position in the maze. The time and structure of the composition is thus entirely determined by the choices and concentration of the performer. Despite the distracting surroundings, the glitchy sound and flickering visuals, the performer must remain paradoxically calm in order to generate the correct states of mind that would let him navigate his avatar out of the maze.
This approach introduce a convergent mapping that brings simplicity of control, which is required to stabilize the intrinsic noisiness of the EEG. Still, in order to obtain some expressivity, I needed to map the few control parameters to the multiple synthesis and structural parameters in the music through a divergent mapping. The result is hybrid mapping in which the pattern recognition is an intermediate phase to clean the signal from noise and select few stable control parameters. The use of time analysis allows the performer to rapidly and reliably control the avatar with his brainwaves. The system is also quite rigid in nature, allowing the control of only three variables non-simultaneously. This aspect imposes heavy restrictions on the expressivity and conception of the piece. A degree of noise is present in the system, making the task of the performer more difficult. Future improvements of the hardware and software can reduce the present noise to create a solid system for brain control. The system is still quite basic in the number of variable that can be controlled and future studies are needed to investigate on how to extend such limitations. The ease of control compared to spectral approach allows a more direct display of the procedures, which generates a higher involvement of the audience. The feedback from the public demonstrates the success of such intention, as interviewed members have declared they could perceive the intentions of the performer, his mistakes, effort and struggle, thus contributing to form a stronger experience and involvement during the performance. Such effect is not only produced by the analytical tools used in the performance, but it is also conveyed through the video-game paradigm used for the performance.
Links http://www.jestern.com/