Sound 2121: Cross-Reality Transitions Between Real and Augmented Sound Landscapes Jordan Aik o Deja Nuwan T. Attygale jordan.deja@famnit.upr.si nuwan.attygalle@upr.si University of Primorska, UP FAMNIT University of Primorska, UP FAMNIT Koper, Slovenia Koper, Slovenia Klen opi Pucihar Matjaž Kljun klen.copic@famnit.upr.si matjaz.kljun@famnit.upr.si University of Primorska, UP FAMNIT University of Primorska, UP FAMNIT Koper, Slovenia Koper, Slovenia Faculty of Information Studies Faculty of Information Studies Novo mesto, Slovenija Novo mesto, Slovenija Figure 1: Concept: Envision a future where we no longer need tangible interfaces to listen to music but stream it directly into our auditory cortex. How do we enable seamless transition between environment and augmented streamed sounds in such scenario? ABSTRACT visions of how we are going to listen to music, one can Music and sounds have always been an integral part of our start to imagine a situation where a chip in our auditory society since the prehistoric times. However, music listening cortex will stream sounds directly to our brain. However, has moved from a social experience to a more personal one augmenting our hearing with objects around us capable of with audio systems such as speakers moving ever closer to streaming sounds directly to our brains might exacerbate the our ear canal. Based on this pattern and various futuristic already present problem of blocking the environment sounds and not paying attention to what is happening around us. Copyright © 2020 for this paper by its authors. Use permied under In this positional paper we discuss not only “What is the Creative Commons License Aribution 4.0 International (CC BY 4.0). future of music listening and how will we consume music Cross-Reality (XR) Interaction, ACM ISS 2020, November 8 2020, Lisbon, a hundred years from now?” but also “How are we going Portugal to enable users to pay attention to their environment and enjoy music listening at the same time?” Put dif f erently, how are we enabling users to constantly switch between the real and augmented sound landscapes and enable seamless International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal J. A. Deja, N. T. A�ygale, K. Čopič Pucihar, M. Kljun cross-reality transitions as well as preserve a positive user in live concerts, events and rituals to collectively listen to experience. music. With recordings and radio, music has moved to peo- ple’s homes and the collective listening has been reduced to KEYWORDS family members, friends or an individual (we do not count music listening, brain-chip interface, sound streaming, future all the radio audience towards this group since listeners are vision not co-present [1]). Nevertheless, the sounds were still com- ing from the speaker(s) in our environment and listening to 1 BACKGROUND the radio was often a social event. The headphones enabled One area of Cross-Reality (XR) Interaction explores the shift users to experience music individually and the Walkman en- between multiple systems on the Milgram’s reality-virtuality abled us to do it on-the-go. Smartphones and internet have continuum. In this position paper, we envision a future with expanded the instant availability of music but the consump- a chip-to-brain sound transmitting interface that allows in- tion remained mainly personal. Looking at how listening teraction between and mixing of multiple sound realities and to music has moved closer and closer to our ear canal with virtualities from the environment sound to music listening. in-ear headphones, it is not far-fetched if we envision that Our position diverts from the typical visual augmented and listening to music will move inside our heads. virtual reality but rather focuses on the problems of sound and how humans transition between di�erent sound sources. 3 RE-IMAGINING MUSIC LISTENING INTERFACE We intend to share a perspective that may consider also the Having a chip in our brain seems like a far fetched idea but transitions induced by visual elements, augmentations and several researchers are working on enabling people with im- virtual worlds as they are consumed with sound. This paper pairments to interact with computers and the world around also encourages several questions such as how to retain users’ them via implanted chips [16]. A recent startup Neuralink1 attention, task-awareness and even social communication envisions all humans to have such a chip in order to interact while interacting with the environmental sound as well as with smart devices around us. However, for many, implant- streamed one (be it from digital objects in the environment ing a chip in our brain is not appealing at the moment [10]. or unrelated to the environment such as listening to music). Nevertheless, as privacy boundary with the advent of social media has risen or using mobile phones in restaurants has 2 MUSIC LISTENING THROUGHOUT HISTORY become a norm, what is individually and culturally accepted Music is considered to be culturally universal [2, 17] and is rapidly changing. This shows that technologies are moving present across all parts of the globe, reshaping the ways forward far quicker than societal ability to grasp all moral human live, express themselves and convey emotions [7, 12]. and legal implications they are introducing. It is believed that music originated from naturally occurring The vision of “the always on” streaming music (and other sounds and rhythms that humans echoed by merging them in sounds) to our brain via a chip on the biological and psycho- patterns, making repetitions while changing tonality using logical level adapting based on our mood on the �y is not their voice[12, 13], hands clapping [8], and smacking stones, new. Such neurological interface instead of a visual or voice sticks and other objects around them [11]. Music has also one has been envisioned before [6]. With researchers and fu- helped humans in terms of survival, forging a sense of group turists working on biometric recommendation systems [21] identity and mutual trust [3]. we could potentially achieve this in the near future. At the We create and consume music for various purposes includ- moment, the company called Weav2 is already adapting mu- ing: (i) dancing as a social exercise, (ii) providing a common sic rhythm to the heartbeat “augmenting and intensifying form of personal or community entertainment, (iii) commu- the listening experience at all levels”. nicating ideas and emotions and (iv) having and celebrating This envisioned streaming interface will be supported rituals and other activities [12]. While these purposes come by the internet of things (IoT) vision with biological and in handy for a variety of music activities, this positional arti�cial objects around us connected [20] and having access paper is focusing on music listening only, which is present to a superb computing power. These devices will not only in (i) where listening is a shared experience, and in (ii) and be able to stream music, but also produce their own sounds, (iv) where listening can be a shared as well as a personal melodies, and rhythms or augment the existing ones. Besides experience. physical objects in the physical environment, there might In the past, human tribes gathered around �res where be a plethora of virtual objects around us as well that will one or several members of the group performed a music communicate with us not only visually but also vocally. piece singing and playing instruments. This type of “com- munity entertainment” or “celebrations” has been present 1 https://neuralink.com/ for a long period of time and even nowadays people gather 2 https://www.weav.io/ Sound 2121: Cross-Reality Sound Transitions International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal Despite all these visions and developments, we still have we can envision such IoT the environment to be constantly not solved the issues related to current music consumption on the guard and aware of potential risks to the people in over the headphones. One of such issues is enabling users to vicinity. Thus, such environment will be able to inform us hear sounds from the environment (be it from physical or in the same way it will stream sounds to our brain. Similar virtual objects and environments) while still enjoying their to the vision of autonomous cars communicating with each music. other and broadcast potential risks and road conditions [18]. Since music has helped humans throughout the history 4 SWITCHING BETWEEN THE ENVIRONMENT in terms of survival, forging a sense of group identity and SOUND TO AUGMENTED STREAMED ONE mutual trust, and since environment sound landscape helped Today’s headphones are complex small devices that are ca- us being alert and safe we need to enable their roles in the pable of actively and passively cancelling environment noise future as well. Be it in physical environment or in any mix in order to enable users to be uninterrupted while listening of reality and virtuality. At the same time we need to avoid to the their favourite tunes. In-ear buds even mechanically the “polluted” sound landscape as in the video of the hyper- block outer sounds from entering the auditory canal. While reality vision of the future [9] visible on Figure 2 where these might be desirable features on some occasions they adverts and services seek our attention via visuals as well are also problematic since the sounds from the environment as sounds. And imagine receiving these through our brain- provide a cue of what is happening around us and humans chip interface and transitioning between the reality only and always relied on these sounds in order to survive. The sounds hyper-reality. of objects around us are even nowadays so important for our How to avoid the mentioned sound “pollution” and achieve safety that legislators made it obligatory for electric cars to smooth transitions between the augmented and environment make arti�cial engine noise [5]. Even some electric go-karts sound landscapes and vice versa to maintain enjoyable music have introduced gas engine noise to make the experience listening user experience as well as user safety remains to more immersive and believable. However, seeing a cyclist, be answered when such devices will be available. pedestrian or a car driver with the headphones is not an unusual sight and greatly a�ects users’ situational aware- REFERENCES ness [15]. [1] Tiziano Bonini. 2014. The new role of radio and its public in the age How do we then switch or transition between the envi- of social network sites. First Monday (2014). sioned future “always on, always-adaptive” music streaming [2] Patricia Shehan Campbell. 1997. Music, the universal language: Fact into our auditory cortex provided by the IoT enabled devices or fallacy? International Journal of Music Education 1 (1997), 32–39. [3] Nicholas J Conard, Maria Malina, and Susanne C Münzel. 2009. New and the environment sound in order to increase our safety �utes document the earliest musical tradition in southwestern Ger- or enable social interaction? many. Nature 460, 7256 (2009), 737–740. Music is also vibration; for example, it has been noted that [4] Jasmine Granados, Matthew Hopper, and Jibo He. 2018. A usability the part of the brain responsible for hearing, works perfectly and safety study of bone-conduction headphones during driving while in deaf people as well [14]. In order to “hear” music, we do listening to audiobooks. In Proceedings of the Human Factors and Er- gonomics Society Annual Meeting, Vol. 62. SAGE Publications Sage CA: not need to actually hear it but rather receive the stimu- Los Angeles, CA, 1373–1377. lus to the hearing region of the brain. This has for example [5] Jack Guy. July 1, 2019. EU requires carmakers to add fake engine noises been explored in bone conduction “headphones” conducting to electric cars. https://edition.cnn.com/2019/07/01/business/electric- sound to the inner ear primarily through the bones of the vehicles-warning-noises-scli-intl-gbr/index.html. [CNN Business skull. Some consumer products already use this technology Online; accessed 2020-10-09]. [6] Cherie Hu. July 26, 2019. Hello, Brave New World! such as Google Glass and BAE helmet. The latter for example https://www.npr.org/2019/07/26/745361267/hello-brave-new- enabled a sailing crew in the 2017 America’s cup “to keep world?t=1600450381762. [Online; accessed 2020-10-09]. both their ears free in order to hear external sounds, whilst [7] Patrik N Juslin and John A Sloboda. 2001. Music and emotion: Theory providing the ability to communicate clearly with crewmates and research. Oxford University Press. in the most treacherous of sailing conditions” [19]. A recent [8] Jamie C Kassler. 1987. The dancing chimpanzee: A study of the origin of music in relation to the vocalising and rhythmic action of apes. study also revealed that bone conduction headphones do not Musicology Australia 10, 1 (1987), 79–81. signi�cantly a�ect driving performance or story comprehen- [9] Keiichi Matsuda. May 19, 2016. HYPER-REALITY. https://www. sion and were no more distracting than regular car speakers youtube.com/watch?v=YJg02ivYzSs. [Online; accessed 2020-10-26]. [4]. This means that it is possible to mix digital sounds with [10] Rachel Metz. July 21, 2019. Elon Musk hopes to put a computer chip in the environment sounds. However, it remains unanswered your brain. Who wants one? https://edition.cnn.com/2019/07/20/tech/ elon-musk-neuralink-brain-chip-experts/index.html. [CNN Business how enjoyable is listening to music with such devices. Online; accessed 2020-10-09]. Based on the future IoT vision with biological and arti�cial [11] Jeremy Montagu. 2014. Horns and Trumpets of the World: An Illustrated objects around able to communicate with each other and us, Guide. Rowman & Little�eld. International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal J. A. Deja, N. T. A�ygale, K. Čopič Pucihar, M. Kljun Figure 2: Hyper-reality vision of the future where physical and virtual objects emit a variety of sounds and provide a cacophony of attention seeking visual and sound elements of the environment [9]. [12] Jeremy Montagu. 2017. How music and instruments began: a brief [17] Charles Seeger. 1971. Re�ections upon a given topic: Music in universal overview of the origin and entire development of music, from its perspective. Ethnomusicology 15, 3 (1971), 385–398. earliest stages. Frontiers in Sociology 2 (2017), 8. [18] Stephen Shankland. August 27, 2019. 5G could make self-driving cars [13] Iain Morley. 2013. The prehistory of music: human evolution, archaeol- smarter and commutes safer. https://www.cnet.com/news/5g-could- ogy, and the origins of musicality. Oxford University Press. make-self-driving-cars-smarter-commutes-safer/. [Cnet Online; ac- [14] ABC Online. 2001. Music - good vibrations for deaf. http://www.abc. cessed 2020-10-09]. net.au/science/articles/2001/11/28/426276.htm. Accessed: 2020-09-10. [19] BAE Systems. Jun 6, 2017. Our bone conduction technology raised the [15] ROSPA. June 2018. Road Safety Factsheet: Headphones as a Driving bar in Bermuda. https://www.baesystems.com/en/article/our-bone- Distraction. Technical Report. The Royal Society for the Preven- conduction-technology-is-set-to-raise-the-bar-in-bermuda. [Online; tion of Accidents, 28 Calthorpe Road, Edgbaston, Birmingham B15 accessed 2020-10-09]. 1RP. https://www.rospa.com/rospaweb/docs/advice-services/road- [20] Robert Weisman. 2004. The Internet of things: Start-ups jump into safety/drivers/headphones-as-a-driver-distraction.pdf. next big thing: tiny networked chips. The Boston Globe (2004). [16] Gopal Santhanam, Stephen I Ryu, M Yu Byron, Afsheen Afshar, and Kr- [21] Jianyu Zhang and Chia Chin Chong. 2018. System and method for ishna V Shenoy. 2006. A high-performance brain–computer interface. biometrics-based music recommendation. US Patent 10,078,691. nature 442, 7099 (2006), 195–198.