The interactions between diferent living and non-living agents and their surroundings within an environment are complex, multidimensional, and self-organizing. Within the context of electroacoustic music composition, this web of organic activity presents a fascinating potential to dictate or inform sonic gestures, textures, and the formal structure of a work. The extraction of these interactions for musical use, however, is easier said than done, and requires a method for translating a representation of an environment to a sound-mappable model. In the case of this work, the representation of the environment used is a stereo field recording, a directional sonic record of the (audible) events within an environment. A deconstruction of this model may then be mapped onto other sounds (or sound generators), ultimately creating a ”sonification of a sound”, a map from a transcription of the sonic interactions of an environment onto an entirely diferent sound world. Using methodologies at the intersection of bioacoustics and music information retrieval, the author designed and implemented a software system, AcousTrans, that facilitates such a mapping process. Segmented events within a stereo sound recording are intelligently mapped onto multi-channel sound events from another corpus of sounds using a k-nearest neighbors search of a k-dimensional tree constructed from an analysis of acoustic features of the corpus. The result is an interactive sound generator that injects the organicism of environmental soundscape recordings into the sequencing, processing, and composing of immersive electroacoustic music. This work was created within the context of bioacoustic analysis of intertidal oyster reefs, but is applicable to any environmental soundscape that may be efectively decomposed using the described method.
• Applied computing → Sound and music computing. bioacoustics, algorithmic music, music information retrieval, composing systems
IUI Workshops ’19, March 20,2019, Los Angeles, USA.
Copyright ©2019 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. 1
The history of applied musical engagements with the relationship
between sound and the environment is rich and lengthy. Classical
examples include the Ancient Greek ”Harmony of the Spheres”, a
description of the harmonious musical proportions of the planets,
a creation story in the Vedic Sanskrit texts that takes sound as
fundamental, and various other archaic ”acoustemologies” (sonic ways
of knowing the world) [
More precise, algorithmic methods include those designed by
Iannis Xenakis, or Gordon Mumma in his Mograph series, which
took as sonic material seismological data [
There exist many diferent software tools, in a variety of disciplines,
which seek to deconstruct and/or map onto sound organic
systems [
Decoding and transcribing the events within a recording is under the purview of both Music Information Retrieval (MIR) and bioacoustics. Pertinent MIR tasks include automatic transcription, track separation, and speaker diarization, each of which seek to automatically reveal structural decompositions of acousmatic sound. Within acoustic ecology and bioacoustics, techniques have been developed to assist in the decomposition of environmental soundscapes, revealing their underlying sonic components (such as in the work of sound recordist Bernie Krause and composer Hildegard Westerkamp).
The specific method that this work builds from is concatenative
sound synthesis, a synthesis technique that may be generally
described as granular synthesis driven by audio analysis, and more
specifically a process of selecting grains of sound from a file or
corpus based on their best fit to some acoustic criteria [
The goal of AcousTrans (Acousmatic Translator) is to allow a user to load in a source stereo audio file (field recording or other environmental recording) and a destination corpus of other audio files and interactively map the events, gestures, and structure of the source onto the destination. What results is a stereo or multi-channel audio ifle with gestural, rhythmic, and/or structural similarities to the source file, but with entirely diferent timbral characteristics: those of the destination corpus. 3.1
AcousTrans operates by first taking in a user-selected stereo audio ifle of a soundscape (an intertidal oyster reef stereo hydrophone recording, for example) within the segmenter module (Figure 1). This audio file is then sent through N low-pass/band-pass/highpass filters whose frequencies are tuned to the particularities of the soundscape (the specific threshold between the sub-soundscapes of wave movement, oysters, snapping shrimp, etc., within the reef, for example). Within the context of the intertidal oyster reef recordings a value of N = 4 was deemed both necessary and suficient, although the system allows for N to be variable. Each of the N spectral sub-bands of the source audio file is then segmented using one of several diferent segmentation modes: amplitude-based peak detection, spectral flux-derived segmentation, or fixed-size segmentation. After segmentation, the result is an N -channel stream of events which encode the independent activity within each subband of the source file. The events are encoded as lists of intensity (average volume), duration, stereo localization (from centered to at either left or right channels), and then a subvector of acoustic features (including fundamental frequency (F0) estimation, energy, periodicity, loudness, spectral centroid, spectral spread, spectral skewness, spectral kurtosis). This multi-channel stream of events is then passed into the playback module (Figure 2). Diferent dimensions of these events are mapped to diferent parameters of sounds generated by AcousTrans using a mapping matrix (right side of Figure 2). For example, the intensity of a source event may be used to dictate the volume of a destination event, or the stereo localization value of a source event may be used to dictate the spatialization speed of the destination event. These destination events take as sound material a user-selected corpus of audio files. Further electroacoustic abstractions may also be applied including delay, comb filtering, spectral freezing, filtering, and a probabilistic repetition (stutter) efect, the parameters of each being either set by the user or driven by diferent event dimensions.
The acoustic features embedded in each source event may be
used to select a similar sound within the user-selected destination
corpus via concatenative synthesis. Using a k-nearest neighbors
search algorithm on a k-dimensional tree constructed from the
acoustic features of segments of each audio file in the audio file
corpus, the subvector of acoustic features for a source event is
mapped to the most similar sound within the destination corpus
[
Creating Music from Environmental Soundscapes
At the time of this writing, there is no similarity thresholding procedure (i.e. producing silence when a source event does not have a highly similar destination event), so a match, however distant, will always be made. The user may customize the weighting of the acoustic features used in the search via a multislider interface (bottom left of Figure 2), which can be useful to ”tune” the system depending on the particular source and destination sounds (for example, de-emphasizing F0 estimation if only using sounds with no clear pitch center). The result of this process is an acoustic feature-driven mapping between the events in the source audio file and those generated by the system from the user-selected audio ifle corpus. Combined with the electroacoustic abstractions outlined above, this system can generate a diverse array of natural system-derived soundscapes. An overview of the functionality of AcousTrans as a system diagram may be viewed in Figure 3. 3.3
AcousTrans is implemented in Cycling ’74’s Max 8 [
The author has composed several multi-channel electroacoustic compositions using material generated by AcousTrans. These include Reef, for octophonic fixed media, presented at the 2018 Coastal
Futures Conservatory conference at University of Virginia, and No Where, for octophonic fixed media, presented at the 2018 Technosonics conference at University of Virginia, the 2019 National Student Electronic Music Event (N_SEME) at University of Virginia, and at the 2019 Society for Electroacoustic Music in the United States (SEAMUS) conference at Berklee College of Music. Examples of the system in action may be heard at www.elistine.com/software/acoustrans. 5
Future work includes expanding the segmentation algorithm to be more sensitive to dense, low dynamic range, ”lo-fi”, environmental soundscapes, expanding the type and control parameters of the acoustic features control submodule, and distributing the software so that it may be used by other soundscape composers and electroacoustic musicians.
Ultimately, AcousTrans presents a methodology for intelligently mapping a multi-dimensional stream of gestures from one environmental soundscape to an entirely diferent, multi-channel electroacoustic sound world. It harnesses techniques from both bioacoustics and MIR to facilitate the generation of electroacoustic material derived from the activity of natural systems.
Thanks to Environmental Sciences Ph.D. candidate Martin Volaric and the University of Virginia Environmental Sciences Department’s Moore Award and the Jeferson Scholars Foundation for providing financial support for this research.