Sound 2121: Cross-Reality Transitions Between Real and Augmented Sound Landscapes

Abstract

Music and sounds have always been an integral part of our society since the prehistoric times. However, music listening has moved from a social experience to a more personal one with audio systems such as speakers moving ever closer to our ear canal. Based on this pattern and various futuristic

-

visions of how we are going to listen to music, one can start to imagine a situation where a chip in our auditory cortex will stream sounds directly to our brain. However, augmenting our hearing with objects around us capable of streaming sounds directly to our brains might exacerbate the already present problem of blocking the environment sounds and not paying attention to what is happening around us. In this positional paper we discuss not only “What is the future of music listening and how will we consume music a hundred years from now?” but also “How are we going to enable users to pay attention to their environment and enjoy music listening at the same time?” Put dif erently, how are we enabling users to constantly switch between the real and augmented sound landscapes and enable seamless cross-reality transitions as well as preserve a positive user experience. music listening, brain-chip interface, sound streaming, future vision 1

BACKGROUND

One area of Cross-Reality (XR) Interaction explores the shift between multiple systems on the Milgram’s reality-virtuality continuum. In this position paper, we envision a future with a chip-to-brain sound transmitting interface that allows interaction between and mixing of multiple sound realities and virtualities from the environment sound to music listening. Our position diverts from the typical visual augmented and virtual reality but rather focuses on the problems of sound and how humans transition between di erent sound sources. We intend to share a perspective that may consider also the transitions induced by visual elements, augmentations and virtual worlds as they are consumed with sound. This paper also encourages several questions such as how to retain users’ attention, task-awareness and even social communication while interacting with the environmental sound as well as streamed one (be it from digital objects in the environment or unrelated to the environment such as listening to music). 2

MUSIC LISTENING THROUGHOUT HISTORY

Music is considered to be culturally universal [ 2, 17 ] and present across all parts of the globe, reshaping the ways human live, express themselves and convey emotions [ 7, 12 ]. It is believed that music originated from naturally occurring sounds and rhythms that humans echoed by merging them in patterns, making repetitions while changing tonality using their voice[ 12, 13 ], hands clapping [ 8 ], and smacking stones, sticks and other objects around them [ 11 ]. Music has also helped humans in terms of survival, forging a sense of group identity and mutual trust [ 3 ].

We create and consume music for various purposes including: (i) dancing as a social exercise, (ii) providing a common form of personal or community entertainment, (iii) communicating ideas and emotions and (iv) having and celebrating rituals and other activities [ 12 ]. While these purposes come in handy for a variety of music activities, this positional paper is focusing on music listening only, which is present in (i) where listening is a shared experience, and in (ii) and (iv) where listening can be a shared as well as a personal experience.

In the past, human tribes gathered around res where one or several members of the group performed a music piece singing and playing instruments. This type of “community entertainment” or “celebrations” has been present for a long period of time and even nowadays people gather in live concerts, events and rituals to collectively listen to music. With recordings and radio, music has moved to people’s homes and the collective listening has been reduced to family members, friends or an individual (we do not count all the radio audience towards this group since listeners are not co-present [ 1 ]). Nevertheless, the sounds were still coming from the speaker(s) in our environment and listening to the radio was often a social event. The headphones enabled users to experience music individually and the Walkman enabled us to do it on-the-go. Smartphones and internet have expanded the instant availability of music but the consumption remained mainly personal. Looking at how listening to music has moved closer and closer to our ear canal with in-ear headphones, it is not far-fetched if we envision that listening to music will move inside our heads. 3

RE-IMAGINING MUSIC LISTENING INTERFACE

Having a chip in our brain seems like a far fetched idea but several researchers are working on enabling people with impairments to interact with computers and the world around them via implanted chips [ 16 ]. A recent startup Neuralink1 envisions all humans to have such a chip in order to interact with smart devices around us. However, for many, implanting a chip in our brain is not appealing at the moment [ 10 ]. Nevertheless, as privacy boundary with the advent of social media has risen or using mobile phones in restaurants has become a norm, what is individually and culturally accepted is rapidly changing. This shows that technologies are moving forward far quicker than societal ability to grasp all moral and legal implications they are introducing.

The vision of “the always on” streaming music (and other sounds) to our brain via a chip on the biological and psychological level adapting based on our mood on the y is not new. Such neurological interface instead of a visual or voice one has been envisioned before [ 6 ]. With researchers and futurists working on biometric recommendation systems [ 21 ] we could potentially achieve this in the near future. At the moment, the company called Weav2 is already adapting music rhythm to the heartbeat “augmenting and intensifying the listening experience at all levels”.

This envisioned streaming interface will be supported by the internet of things (IoT) vision with biological and arti cial objects around us connected [ 20 ] and having access to a superb computing power. These devices will not only be able to stream music, but also produce their own sounds, melodies, and rhythms or augment the existing ones. Besides physical objects in the physical environment, there might be a plethora of virtual objects around us as well that will communicate with us not only visually but also vocally. 1https://neuralink.com/ 2https://www.weav.io/ Sound 2121: Cross-Reality Sound Transitions Despite all these visions and developments, we still have not solved the issues related to current music consumption over the headphones. One of such issues is enabling users to hear sounds from the environment (be it from physical or virtual objects and environments) while still enjoying their music.

4 SWITCHING BETWEEN THE ENVIRONMENT

SOUND TO AUGMENTED STREAMED ONE

Today’s headphones are complex small devices that are capable of actively and passively cancelling environment noise in order to enable users to be uninterrupted while listening to the their favourite tunes. In-ear buds even mechanically block outer sounds from entering the auditory canal. While these might be desirable features on some occasions they are also problematic since the sounds from the environment provide a cue of what is happening around us and humans always relied on these sounds in order to survive. The sounds of objects around us are even nowadays so important for our safety that legislators made it obligatory for electric cars to make arti cial engine noise [ 5 ]. Even some electric go-karts have introduced gas engine noise to make the experience more immersive and believable. However, seeing a cyclist, pedestrian or a car driver with the headphones is not an unusual sight and greatly a ects users’ situational awareness [ 15 ].

How do we then switch or transition between the envisioned future “always on, always-adaptive” music streaming into our auditory cortex provided by the IoT enabled devices and the environment sound in order to increase our safety or enable social interaction?

Music is also vibration; for example, it has been noted that the part of the brain responsible for hearing, works perfectly in deaf people as well [ 14 ]. In order to “hear” music, we do not need to actually hear it but rather receive the stimulus to the hearing region of the brain. This has for example been explored in bone conduction “headphones” conducting sound to the inner ear primarily through the bones of the skull. Some consumer products already use this technology such as Google Glass and BAE helmet. The latter for example enabled a sailing crew in the 2017 America’s cup “to keep both their ears free in order to hear external sounds, whilst providing the ability to communicate clearly with crewmates in the most treacherous of sailing conditions” [ 19 ]. A recent study also revealed that bone conduction headphones do not signi cantly a ect driving performance or story comprehension and were no more distracting than regular car speakers [ 4 ]. This means that it is possible to mix digital sounds with the environment sounds. However, it remains unanswered how enjoyable is listening to music with such devices.

Based on the future IoT vision with biological and arti cial objects around able to communicate with each other and us, we can envision such IoT the environment to be constantly on the guard and aware of potential risks to the people in vicinity. Thus, such environment will be able to inform us in the same way it will stream sounds to our brain. Similar to the vision of autonomous cars communicating with each other and broadcast potential risks and road conditions [ 18 ].

Since music has helped humans throughout the history in terms of survival, forging a sense of group identity and mutual trust, and since environment sound landscape helped us being alert and safe we need to enable their roles in the future as well. Be it in physical environment or in any mix of reality and virtuality. At the same time we need to avoid the “polluted” sound landscape as in the video of the hyperreality vision of the future [ 9 ] visible on Figure 2 where adverts and services seek our attention via visuals as well as sounds. And imagine receiving these through our brainchip interface and transitioning between the reality only and hyper-reality.

How to avoid the mentioned sound “pollution” and achieve smooth transitions between the augmented and environment sound landscapes and vice versa to maintain enjoyable music listening user experience as well as user safety remains to be answered when such devices will be available.

References

[1] Tiziano Bonini . 2014 . The new role of radio and its public in the age of social network sites . First Monday ( 2014 ).

[2] Patricia Shehan Campbell . 1997 . Music, the universal language: Fact or fallacy ? International Journal of Music Education 1 ( 1997 ), 32 - 39 .

[3] Nicholas J Conard , Maria Malina, and Susanne C Münzel. 2009 . New utes document the earliest musical tradition in southwestern Germany . Nature 460 , 7256 ( 2009 ), 737 - 740 .

[4] Jasmine Granados , Matthew Hopper, and Jibo He . 2018 . A usability and safety study of bone-conduction headphones during driving while listening to audiobooks . In Proceedings of the Human Factors and Ergonomics Society Annual Meeting , Vol. 62 . SAGE Publications Sage CA: Los Angeles, CA, 1373 - 1377 .

[5] Jack Guy . July 1 , 2019 . EU requires carmakers to add fake engine noises to electric cars . https://edition.cnn.com/ 2019 /07/01/business/electricvehicles-warning -noises-scli-intl-gbr/index .html. [CNN Business Online; accessed 2020 - 10 -09].

[6] Cherie Hu . July 26 , 2019 . Hello, Brave New World! https://www.npr.org/ 2019 /07/26/745361267/hello-brave-newworld?t= 1600450381762 . [Online; accessed 2020- 10 -09].

[7] Patrik N Juslin and John A Sloboda. 2001 . Music and emotion: Theory and research . Oxford University Press.

[8] Jamie C Kassler . 1987 . The dancing chimpanzee: A study of the origin of music in relation to the vocalising and rhythmic action of apes . Musicology Australia 10 , 1 ( 1987 ), 79 - 81 .

[9] Keiichi Matsuda . May 19 , 2016 . HYPER-REALITY. https://www. youtube.com/watch?v= YJg02ivYzSs . [Online; accessed 2020- 10 -26].

[10] Rachel Metz . July 21 , 2019 . Elon Musk hopes to put a computer chip in your brain . Who wants one? https://edition.cnn.com/ 2019 /07/20/tech/ elon -musk-neuralink-brain-chip-experts/index .html. [CNN Business Online; accessed 2020 - 10 -09].

[11] Jeremy Montagu . 2014 . Horns and Trumpets of the World: An Illustrated Guide . Rowman & Little eld .

[12] Jeremy Montagu . 2017 . How music and instruments began: a brief overview of the origin and entire development of music, from its earliest stages . Frontiers in Sociology 2 ( 2017 ), 8 .

[13] Iain Morley . 2013 . The prehistory of music: human evolution, archaeology, and the origins of musicality . Oxford University Press.

[14] ABC Online . 2001 . Music - good vibrations for deaf . http://www.abc. net.au/science/articles/2001/11/28/426276.htm. Accessed: 2020 -09-10.

[15] ROSPA . June 2018. Road Safety Factsheet: Headphones as a Driving Distraction . Technical Report. The Royal Society for the Prevention of Accidents , 28 Calthorpe Road, Edgbaston, Birmingham B15 1RP. https://www.rospa.com/rospaweb/docs/advice-services/roadsafety/drivers/headphones -as-a-driver-distraction .pdf .

[16] Gopal Santhanam , Stephen I Ryu , M Yu Byron , Afsheen Afshar, and Krishna V Shenoy . 2006 . A high-performance brain-computer interface . nature 442 , 7099 ( 2006 ), 195 - 198 .

[17] Charles Seeger . 1971 . Re ections upon a given topic: Music in universal perspective . Ethnomusicology 15 , 3 ( 1971 ), 385 - 398 .

[18] Stephen Shankland . August 27 , 2019 . 5G could make self-driving cars smarter and commutes safer . https://www.cnet.com/news/5g-couldmake -self-driving-cars-smarter-commutes-safer/ . [Cnet Online; accessed 2020- 10 -09].

[19] BAE Systems. Jun 6 , 2017 . Our bone conduction technology raised the bar in Bermuda . https://www.baesystems.com/en/article/our -boneconduction-technology-is-set-to-raise-the-bar-in-bermuda . [Online; accessed 2020 - 10 -09].

[20] Robert Weisman . 2004 . The Internet of things: Start-ups jump into next big thing: tiny networked chips . The Boston Globe ( 2004 ).

[21] Jianyu Zhang and Chia Chin Chong. 2018 . System and method for biometrics-based music recommendation . US Patent 10 , 078 , 691 .