-Auditory augmentation has been proposed as a specific, ambient sonification method. This paper describes an interdisciplinary workshop exploring this method by designing prototypes. Three of the prototypes are presented and discussed. Concluding on the workshop's results, the authors suggest a broader definition and deduce limitations of auditory augmentation.
of data representation tools, which unobtrusively alter the
auditory characteristics of structure-borne sounds.” One of these
authors’ examples eliciting the concept is ‘WetterReim’ [
An ordinary laptop keyboard is equipped with a contact INTERACTION SOUND microphone, recording the typing sounds. The microphone swigeantahleirs cfiolntderiteidonw.iTthhevaoryuitnpgutpaisrapmlaeyteerds tbhaactkdeinpenredalonti mthee O-BTJaEbCleT EnvPirohnysmiceanlt and fuses with the original sound to a new auditory gestalt. - Objec-tRXowomith BRIX In short: depending on the weather outside, typing on the keyboard sounds different.
The concept of auditory augmentation has been discussed
to be extended to the more general blended sonification [
For the context of SBE4 we decided to stick to the original term auditory augmentation with a new working definition that incorporates the prepared platforms and tasks (see Sec. IV-A).
Looking at a broader context, auditory augmentation is part
of Sonic Interaction Design (SID) which has been defined by
various authors with different focuses: Rocchesso et al. [
We found a few more SID systems in the literature that are
closely related to auditory augmentation, especially regarding
its focus being an ambient display. For instance, Ferguson [
Kilander and Lo¨nnqvist developed Weakly Intrusive Ambient
to “convey an intuitive sense of any graspable process” rather
than a direct information display. In a ubiquitous service
environment, individual notifications are presented with a
sound associated to the user. Playback volume and reverb are
used to convey three levels of intensity of the notification.
High intensity is mapped to a dry, loud sound, while low
intensity is saturated by reverb, giving the impression of a
far sound. Finally, a similar project has been realized by
Brazil and Fernstro¨m [
SBE4 provided a set-up for exploring auditory augmentation and defined that
Auditory augmentation is the augmentation of a physical object and/or its sound by sound which conveys additional information.
As sketched in Fig. 1, three elements are needed for auditory
augmentation:
1) Physical objects in a physical environment. In our
workshop setting, these were a table, a room, or any
sensorequipped physical object. These objects may produce
a sound, or not; users might interact physically with
them, or not. In some cases, the sound is a result of
the interaction, but does not have to be. Either of these
inputs has to be there: real-time sound input or real-time 1) REAL-TIME: The REAL-TIME data set was provided
data input from the interaction. as a real-time power measurement of five appliances at our
2) Data that are sonified. These can be real-time data or institute’s kitchen (see [
The various possibilities of auditory augmentation have been explored on three different platforms during the workshop.
1) ROOM: The ROOM is situated in the institute’s main
performance and lecture hall, equipped with a 24-channel
loudspeaker array on a half-sphere for ambisonic sound
spatialization [
2) TABLE: The TABLE is an experimentation platform
developed within an ongoing PhD project (see [
3) BRIX: Our co-organizers (see Acknowledgment)
provided their BRIX system [
In the prototyping sessions with the BRIX, the team could choose an interaction scenario with any object, equipping it with BRIX sensors and/or with microphones. Sensors that have been prepared for the workshop include accelerometer and gyroscope, as well as light, proximity, humidity, and temperature sensors.
Next to the three hardware platforms described above, we prepared three data sets of electric power consumption. The data sets are of different nature (real-time vs. recorded data) and therefore propose different tasks, i.e., real-time monitoring as opposed to data exploration. The teams had to develop scenarios that support the saving of electric energy consumption.
During the three days of the workshop, four parallel prototyping sessions took place; one for each of the three data sets and an open session on the last day where chosen prototypes were refined. An overview of all the resulting prototypes is shown on the SBE4 website4. In this paper, we only focus on three exemplary cases: sonic floor plan , writing resonances, and standby door, realized with the three different platforms respectively (see Fig. 2).
1) Sonic floor plan (ROOM): The real-time data set used in this scenario provided data of electric power consumption of different devices in one specific household. The team developed a scenario with an assumed floor plan of the household (see Fig. 2a). Feedback on energy consumption is played back in one room (e.g., a media room) on a surround sound system. A sound occurs periodically after a specified time, as well as when a person enters the room. The appliance that currently consumes the most energy defines the direction for sound spatialization. Environmental sounds from outside the building are captured by a microphone. These are played back in the room with loudness and position depending on the level of energy consumption. As only the power consumption of the appliance with the highest load is sonified, even small standby consumption may attract attention, e.g., when no major energy consumer is active.
2) Writing resonances (TABLE): Writing resonances is the
most ‘classical’ example of auditory augmentation during
the workshop, because it is based on structure-borne sounds
and therefore clearly fulfills the initial definition of
Bovermann et al. The scenario is to provide feedback of in-home
power consumption through an auditorily augmented writing
desk (see Fig. 2b), based on the system presented in [
table is augmented through additional resonances, based on a physical model. The size of the modeled plate is controlled in real time by the total amount of electric power consumption, employing the metaphor of a larger table (i.e., deeper sound) when more power is consumed. With the augmentation being only a modulation of the existing interaction sounds, the sonification only gets audible through interaction with the table; the feedback is therefore calm and unobtrusive. The primary task is writing, but also an active request of information by knocking is possible. This prototype has been extended in the fourth, open session, allowing for a modulation of individual partials in order to additionally convey information concerning the different appliances’ individual power consumption.
3) Standby door (BRIX): This scenario augmented our institute’s inside entry door (see Fig. 2c for the prototypic mockup). Most potential for saving energy lies in the reduction of standby consumption. Therefore, when the door is opened, e.g., in the morning, the standby consumption over the recent past is communicated through a simple parameter mapping sonification. The playback speed depends on the assumed stress level of the user, derived from the opening speed of the door. In the open prototyping session, this approach has been extended to be able to sonify individual appliances. For intuitive discrimination between them, the sonification is based on synthesized speech. When opening the door, an emotionless computer-voice says ‘coffee machine’, ‘microwave’, and the like, with timbre, loudness, or other parameters controlled by how much electricity this specific appliance has used over night.
We explored four main research questions by applying various analysis methods on the results of the workshop. Plenary sessions with discussions of the presented prototypes have been recorded and partially excerpted. Additional inputs are written notes, code, and demo videos resulting from the prototyping sessions. All these inputs led to fundamental considerations on auditory augmentation and how it can be used, but also to general feedback concerning sonification, workshop setting, or design issues.
Augmented reality usually refers to a mainly visual system. But if this concept is transferred to audio, why then is listening to the radio not ‘auditorily augmented reality’? Or is it? The underlying question is: what are the peculiarities of sound in the context of augmented reality?
Concerning the radio, the answer is relatively clear. In the
visual domain, augmented reality usually does not include an
overlay of a video on top of the visually perceived scene if
there is no direct connection between them [
(a) Sonic floor plan . (b) Writing resonances.
(c) Standby door.
Fig. 2. Three of the prototypes that have been developed during the workshop.
A more profound analysis of this question is without the
scope of this paper. Still, a few things came up during the
discussions in the workshop. On the one hand, sonification
is challenging because visual and auditory memory work
differently, and therefore two sonifications are more difficult
to compare than two static visualizations. These and other
challenges are well known, see for instance [
To conclude, ‘auditorily augmented reality’ clearly behaves different from its visual counterpart, and this fact deserves more systematic research.
One purpose of the workshop was to develop and test our working definition which is deliberately wide to incorporate different platforms. The question is, if this definition is more useful.
Our working definition of auditory augmentation has not been questioned within the workshop; therefore, we would like to propose it for future applications. Still, one aspect deserves more attention than has been discussed above, as came up during the final discussion round. The definition starts from an object that is being interacted with, i.e., a primary task of the user with the object is pre-assumed. “Having a concrete task helps to design” (P1), and helped the teams to elaborate their scenarios during the sessions. During the final feedback round, however, it has turned out that the task is an ill-defined entity. Does it relate only to the interaction between user and object? And if the object is augmented, and its sound conveys additional information, is there such a thing as a monitoring task? Does only a goal-oriented activity comprise a task or can it be a by-product of daily “state of being” (P7)?
We conclude that auditory augmentation always involves a primary task, may it be goal-oriented or not (e.g., writing on a desk in the writing resonances prototype or just being in the media room of the sonic floor plan ). This task should not be disturbed by the augmentation, but rather the augmentation adds a secondary task of monitoring.
In addition, we aimed at exploring the qualitative factors between object and sound in the context of auditory augmentation. Which qualities are needed to (fully) describe their relationship? For example: • Is the sound, in relation to the object, plausible? Is the mapping of data to sound intuitive/metaphoric? • Is the augmented object more useful, or more fun than the original one? Does its affordance change and is the original interaction, i.e. the primary task of the user with the object, disturbed? A central issue that has been raised throughout the workshop was in how far objects change in perception when they are auditorily augmented. This experience, “you can only get it if you interact yourself” (P1).
EITHER/OR
For comparison between the three exemplary prototypes (Sec. IV), we analyzed the qualitative behavior of input elements on auditory augmentation. Abstracting from Fig. 1, we identify four of these elements: user interaction, input sound, external data, and sound design. This more abstract concept of auditory augmentation is shown in Fig. 3. We assume that more coherent relations between user interaction, input sound, external data, and sound design lead to higher naturalness and intuitiveness of the auditory augmentation system. Multiple dependencies are possible, even though not all are needed for auditory augmentation: • User interaction may influence external data, e.g., turning on the water kettle while its electric power consumption serves as data. The task of writing by hand, however, does not directly modulate in-home electric power consumption. • Data may have a close link to sound design, utilizing either direct sonification (e.g., audification [11, p. 301324]) or a fitting metaphor, e.g., a larger desk with lower pitch representing a larger energy consumption. • User interaction may directly influence sound design, in all cases where sounds are augmented that have been produced by the interaction. • Sound input may not be stemming from the interaction but from an external source; still it may influence sound design, e.g., when using environmental sounds from outside the windows as in the sonic floor plan prototype.
In conclusion, it seems that a more natural prototype of auditory augmentation has more coherent relationships between user interaction and/or sound input on the one hand, and external data and sound design on the other hand.
As a final research question, we aimed at sketching out perceptual factors of data and sound concerning auditory augmentation systems. What is the capacity of information that can be conveyed with auditory augmentation? Which data are suitable for it? Which factors play a role for blending the augmented object in the environment: is it unobtrusive but salient enough in order to be perceived? Of course, a full analysis of perceptual factors can only be a result of evaluation that was not within the scope of the workshop. However, some ideas emerging from the final discussions may serve as a basis for future investigations.
As one participant articulated, auditory augmentation works best with low-dimensional data – “otherwise you are not augmenting the object but creating a nice sonification” (P7). The information capacity, i.e., the level of information that can be conveyed, is rather low for the examined platforms. Especially for ROOM and TABLE, only the reverberation of the room – respectively the resonances of the structureborne sounds – can be changed, with only few levels that can be differentiated perceptually. The BRIX system is more flexible in design, therefore no general conclusion can be drawn. Sonifying is difficult under these conditions, because “it boils down to the question which dimensions you chose and which ones you leave out” (P11).
In the writing resonances prototype, the developing team found a borderline for perception. Depending on the quickness of parameter changes, sounds lost their gestalt identity with the interaction, i.e., sometimes the sounds were perceived as separate auditory events played from the loudspeakers – despite measured round-trip latency below 5 ms. This example shows that perception is very sensitive and systems need to be well evaluated.
Next to finding some insights on the aforementioned research questions, the analysis of SBE4 provided some feedback on the workshop setting itself, as well as on general design issues.
A general issue of sonification is its right to exist – referring
to Supper’s thesis “about community’s struggles to have
listening to scientific data accepted as a scientific approach” [
Generally, feedback on the workshop design was positive. The interdisciplinary, hands-on sessions were “so enriching” (P1). Participants further reported that they had “really time to try out something” (P2) – even if a certain approach did not work out in the end, there was learning even by dead ends. However, it was remarked that the prepared platforms had “narrowed down” (P3) possible paths of design. Some participants articulated the wish to be more free in designing interaction scenarios independently from a platform (P3, P5), but the prepared setting was very time-efficient. Most prototypes have reached a promising state after the three hours’ sessions: “there are nine prototypes that are really worthwhile considering and working on in the future” (P4).
One participant (P1) raised the issue that designing ambient displays means designing for the background, while in the designer’s mind the sound is in the foreground: “we have an excitement for sound and sonic display”. Therefore, it would be important to cultivate a beginner’s mind – something, he stated, that has been well achieved with the interdisciplinary workshop setting for most of the participants. Another general design-issue concerns the difference between prototypes and final products. The realized prototypes are meant to be listened to for a longer period of time, but designers only hear them for a short period of time (P8). Furthermore, for the purpose of demonstration and presentation, prototypes need to exaggerate, while in final products the appropriate settings are usually less salient. Participants who worked with iconic sounds of audio recordings stated that some cartoonification is needed (e.g., through post-processing or re-synthesis of the sounds), because “for ordinary people they all sound the same” (P9).
Within this paper, we conclude on results from an interdisciplinary workshop exploring the concept of auditory augmentation. The workshop resulted in nine prototypes and, among others, recorded discussions that have been analyzed. Concluding on this material, we propose to use the term auditory augmentation with a new, broader definition: auditory augmentation is the augmentation of a physical object and/or its sound by sound which conveys additional information.
General considerations for auditory augmentation are summarized as follows.
Auditory augmentation requires a primary task of a user with an object; this task is not explorative data analysis. One reason is that data for auditory augmentation needs to be lowdimensional. Another reason is the differentiation between auditory augmentation and sonification. By augmenting an object auditorily, a secondary task of monitoring in the background turns up. This task must not interfere with the primary task.
There seems to be a quality of ‘naturalness’ (affecting also the ‘intuitivity’) of systems of auditory augmentation. The most natural systems have several coherent relationships between the four possible input factors, user interaction and/or sound input, with external data and sound design. We envisage exploring this hypothesis further.
There are borderline cases of perception, where the fusion of auditory gestalts between the original sound and the augmented one does not work anymore. The influencing factors need to be explored systematically.
Finally, the analysis of the final discussions during the workshop proved that the developed workshop setting is convincing. It establishes an interdisciplinary, playful atmosphere of research by design. The balance of possible ingenuity and well-prepared tasks, platforms, and data sets are crucial for a successful event.
We would like to thank our co-organizers from CITEC, Bielefeld University, and all participants of SBE4: Lorena Aldana Blanco, Luc Do¨bereiner, Josef Gru¨ndler, Thomas Hermann, Oliver Ho¨dl, Doon MacDonald, Norberto Naal, Andreas Pirchner, David Pirro`, Brian Joseph Questa, Stefan Reichmann, Martin Rumori, Tony Stockman, Leopoldo Vargas, Paul Vickers, and Jiajun Yang.