Audio Interactions with Autonomous Vehicles: A Hearing-Enhanced Pedestrian Story

Abstract

Smart technologies embedded in autonomous vehicles (AVs) are expected to prevent accidents by reducing human error caused by drivers. However, external communication between pedestrians and AVs is required to ensure improved safety and trust. We define a hearing-enhanced pedestrian (HEP) who depends on electronic aids for everyday living that gives them augmented capabilities. Technology enhancements in modern hearing aids can receive information directly from an AV. We explore new possibilities for AV-HEP interactions using a direct audio link to the hearing aids. In order to understand such interactions, a co-design study was conducted between two researchers of this work. One codesigner has the lived experience of wearing hearing aid enhancements due to hearing impairment. Our work presents preliminary insights on designing potential audio cues to facilitate direct communications between AVs and hearing-enhanced pedestrians.

Keywords

• Hearing-Enhanced
• Hearing-enhanced pedestrians
• hearing aid users
• autonomous vehicles

1. Introduction

Autonomous vehicles (AVs) are expected to become an integral part of our society. Google and Uber 1 have been testing their cars on public roads for several years while Waymo 2 has launched the nation’s first commercial self-driving taxi service in Arizona. With the introduction of AVs, there will be several human-computer interaction challenges – specifically how a pedestrian will interact with an AV when nonverbal cues from the human driver are no longer available such as eye movements, hand gestures, etc. Previous research has been done to overcome these challenges by designing external Human-Machine Interfaces (eHMIs) on AVs. Some prior works focused on designing visual interfaces using displays [ 1 ], LED strips [ 2 ], and projections [ 3 ] to communicate the awareness and intent of an AV. Similarly, other visualization concepts

Z. Asha) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings are designed on external car displays [ 4, 5 ]- a smiling grille, a warning system, trafic light indicators, and a gesturing robotic driver. Interfaces from other modalities, auditory and haptic, have also been explored in some past studies [ 2, 6, 7 ].

As recent research mainly focuses on designing AV-pedestrian interactions for the nondisabled adult population, their findings support visual interfaces [ 4, 3, 8 ] as the most prevalent for helping pedestrians to make safe crossing decisions when they need to share the road with AVs. However, interfaces designed for nonimpaired pedestrians might not always be accessible for people with impairments. Therefore, in our research, we are interested in exploring a novel perspective: how can we design interactions to facilitate the engagement between AVs and a hearing-enhanced pedestrian (HEP) who wears hearing aids? Individuals with a hearing impairment may use electronic enhancements such as hearing aids to improve their hearing ability. The latest generation of hearing aids contains smart technologies, such as Bluetooth, that allow for connectivity to many other devices, including televisions, computers, or smartphones [ 9 ]. This allows hearing aid users to receive calls, access diferent programs, or listen to music directly through their hearing aids. Thus, hearing impaired pedestrians have access to the technology in their hearing aids to establish direct communication with AVs. They can also benefit by having interface cues implemented in their hearing aids to interact with AVs. Finally, it is essential that we include these groups of people from the beginning to design inclusive interfaces by considering their unique needs, challenges, and strengths.

We acknowledge that eHMIs are important in AV-pedestrian interactions to ensure trust and safety for human pedestrians while communicating with AVs. We focus on designing additional interfaces via audio cues (along with the visual, haptic cues proposed by prior works) to improve HEP interactions with AVs. The use of modern hearing aids ofers new forms of communication between AVs and HEPs because both entities will be connected to audio augmented technology. While we have considered only fully AVs in our research, any non-autonomous modern vehicles might include necessary technologies to communicate with the hearing aid users via audio cues. In this research, we explore the scenarios where an AV can interact with a HEP using various audio signals by directly communicating in their ears. We conducted a series of co-design sessions between the first two authors of this paper where one has the lived experience of using hearing aids. Our findings present preliminary ideas towards designing audio cues to establish communications between AVs and pedestrians using hearing aids. The main contribution of our work lies in exploring AV-HEP interfaces for various scenarios to evoke further research explorations in this specific design space.

2. Background

While designing AV-human pedestrian interactions, researchers have presented audio cues along with other visual and tactile cues. In this body of work, verbal cues, beeps, or music [ 2, 1, 4 ] help provide auditory feedback to pedestrians. Diferent audio messages such as “I am stopping, you can cross” [ 8 ], even shorter messages such as “stopping” or “starting” [ 2 ] were introduced as external cues to communicate the vehicle’s intent. While previous studies found audio cues were the most helpful for visually impaired pedestrians, none of the works explored the use cases of audio communication explicitly for hearing aid users. Additionally, most did not consider real-world factors like confusion created by noise or multiple vehicles which may introduce challenges such as asynchronous audio messages from multiple vehicles and the dificulty for the hearing aid user to process information with ambient noise from the road. Soon, multiple AVs may collaborate to act as one system with vehicle-to-everything (V2X) technology [ 8 ] which might solve the issue of sending asynchronous messages. Some research suggests reserving audio cues for emergencies in order to provide clear commands [ 2 ]. However, none of the previous works ofer ways to efectively communicate with pedestrians by augmenting audio cues in everyday non-emergency scenarios. Broadcasting audio cues to pedestrians would increase the environmental noise in a busy street and might be irritating for people living in the neighborhood [ 10 ]. Therefore, we chose to transmit directly to the individual privately and discreetly, using the hearing aid as a medium to receive audio communications from an AV.

We chose to explore the potential interaction design for a HEP who can safely communicate with an AV. People with hearing impairment can hear with the help of a hearing aid by mediating the frequency of the sounds and experience an enhanced communications ability through audio augmentation of their everyday environment. When both entities (AV and pedestrian with a hearing aid) have the technology to communicate directly, we anticipate new possibilities for HEPs to control while interacting with an AV. User customization can include preferences, such as playing important messages only and muting outside noise, making adjustments to the audio interfaces (tones, music, speech) based on listening preferences, and so on [ 9 ].

3. Co-Design Study

3.1. Study Procedure

A remote co-design study was conducted with a hearing aid user to develop preliminary ideas and concepts. As an interaction designer, designated as the first co-designer (CD1), the first author was responsible for designing the sessions and bringing insight forward from our findings. The second co-designer (CD2) played the role of a subject matter expert (second author) due to 50 years of experience wearing high-powered hearing aids and having experience as an interaction designer. Both co-designers collaborated on brainstorming and creating the audio cues for the AV interactions. Currently, CD2 is using a modern hearing aid 3 with a Bluetooth connection and other smart-technology features.

We conducted four one-hour co-design sessions. The first session focused on a general discussion and learning from CD2 who has lived experience. During the second session, we brainstormed diferent scenarios while crossing controlled or uncontrolled crosswalks, walking in the parking lot, sideways, and so on. As stated earlier, diferent visual, audio, and tactile interfaces have already been explored for non-disabled adult populations to ensure pedestrian safety during many street crossing scenarios. Our study considered specific scenarios where the vehicle would need to send additional audio signals directly to a hearing-enhanced pedestrian without broadcasting the signal to everyone. We explored scenarios that occurred only as one AV-to-one pedestrian or many AVs-to-one pedestrian interactions. Finally, in the last two sessions, we developed some low-fidelity prototypes of audio cues for the identified scenarios by separating the actions like signaling or claiming precedence, describing intentions related to street crossing or warning in the parking lot. We created a common set of audio cues for each activity. Some of the activities are- a pedestrian is waiting to cross, the pedestrian is crossing and finishes crossing, a car is reversing out or taking a spot in the parking lot, and others.

3.2. Design Scenarios

Based on the design themes that emerged from the co-design sessions, we discovered two scenarios where an AV might interact with a HEP by directly sending audio cues. While brainstorming the scenarios, we considered multiple vehicles as one system because we expect that all AVs will soon be connected via V2X technology [ 8 ]. Therefore, instead of multiple vehicles sending cues to one HEP, only one vehicle would send a single coordinated message describing the trafic situation. We detail the scenarios we have chosen to implement below: • Scenario 1: A HEP interacts with an AV while crossing a street. There are two variations to this scenario: first, a large parked vehicle visually obstructs the AV (see Figure 1); second, the AV is coming up from behind the HEP and turning around a corner. Figure 2 shows this scenario where a HEP is waiting to cross, and an AV is coming up from behind and waiting to turn right around the corner. In this case, the vehicle might want 3https://www.resound.com/en-ca/hearing-aids/resound-hearing-aids/enzo-3d

to directly communicate with the hearing aid user because he/she is trying to cross the street and would not see the upcoming vehicle. Additionally, direct messages from the AV could enhance the pedestrian’s awareness of the AV’s existence and its intended actions. • Scenario 2: Interaction between a HEP and an AV in a parking lot. Parking lot scenarios introduce complexity for pedestrians to interact with any types of vehicles (autonomous or manually driven) [ 11 ]. If a car is reversing out of the parking spot or trying to get a spot (see Figure 3), and there is a pedestrian nearby, the car should provide information about its action to the pedestrian. In such cases, an AV could send an audio cue to the HEP to indicate its intent and reduce the risk of collision.

3.3. Implemented Audio Cues

We developed some proof-of-concept low-fidelity audio cues for the above mentioned scenarios. While designing these clips, we incorporated the same cues for interaction with common traits from the scenarios (e.g., vehicle approaching, safe to cross, and warning). We generated two types of non-speech audio cues from sonification [ 12 ] such as tone and melody, to convey information in the diferent scenarios. We also designed a set of verbal cues for all of the interactions. Combining speech with non-verbal audio cues could help train people on the meanings of diferent cues and clarify messages in complex interaction sequences for intercommunication and warning (signaling awareness/intent, waiting, alerting to action, conclusion). Overall, we designed three groups of audio cues (tones, rhythms, and speech/verbal cues) which also align with the practices recommended in the guidelines of accessible pedestrian signals 4. In the tone category, the produced audio cues are a timely linear sequence of tones considered a single entity. In contrast, audio cues in the rhythm category contain beats repeated a particular number of times. Finally, the verbal cues include audio signals with speech. These cues can be utilized by directly playing to a pedestrian’s hearing aid when necessary on top of other visual road signals. Some example audio prototypes are described below for the following 3 use cases: 4http://www.apsguide.org/chapter4 .

3.3.1. Vehicle Approaching

If a pedestrian is waiting to cross the street (Figure 1(a)), the AV greets the pedestrian at the crosswalk by informing them that it is approaching. For this interaction, we implemented a progression theme to grab the attention of the pedestrian politely from a distance. The audio cue (e.g., tone 5) quietly starts so that it does not startle the pedestrian and gradually increases the volume to imply that the AV is getting closer.

3.3.2. Safe to Cross

If it is safe to cross the road, then a single audio tone 6 or verbal cue 7 is played to inform the pedestrian that they can begin to cross. The pedestrian starts crossing the street after being informed by the AV that it is now safe to cross (Figure 2, middle). In this communication, the AV plays another audio signal from one of our three categories (e.g., rhythm 8) that would continue until the pedestrian finishes crossing and then fades at the end. This last audio signal concludes the series of interactions between the AV and HEP for the first scenario of street crossing (Figure 2). 3.3.3. Warning When it is not safe for the pedestrian to cross, then a warning audio cue is played. For this interaction, we again created three alert signals: tone 9, rhythm 10,and speech 11. Our goal was to inform the pedestrian rather than startle them how a siren, alarm, or vehicle horn does. Additionally, we wanted to avoid irritating the pedestrian or implying any impatient behavior of the AV. We also used these audio cues to warn a pedestrian about a vehicle reversing out of a parking spot for the second scenario (Figure 3).

4. Future Work and Conclusion

As a future work of this research, we would like to conduct a user study with hearing aid users evaluate the usability of the proposed audio interfaces. While exploring the audio cues, we focused on allowing individual customization by providing diferent audio libraries and designing a polite interaction experience by considering volume, human voice, etc. We also suggest introducing the auditory interfaces with verbal cues as “training wheels” [ 13 ], through which pedestrians can learn the meaning of various audio tones and rhythms. In future studies, we will investigate the practicality of the explored scenarios and the scalability of the generated audio cues. We will also examine the real-world factors for the AV-HEP engagement via audio interfaces like the implications of receiving several audio notifications, environment noise, 5https://soundcloud.com/vjowenb/arrival-tones/s-XLH7mgDSW1X 6https://soundcloud.com/vjowenb/safetocross-2/s-KMdAgg7GgC8 7https://soundcloud.com/vjowenb/safetocross-1/s-bjWYHW4x25H 8https://soundcloud.com/vjowenb/crossingfading/s-2Kyp0t1qswk 9https://soundcloud.com/vjowenb/alarmtones/s-KOo0rsEkf5k 10https://soundcloud.com/vjowenb/alarm/s-CWgEeQIuyFu 11https://soundcloud.com/vjowenb/alarm-1/s-qMpXa8I9tsL multiple AVs, multiple pedestrians, etc. Overall, this work presents our early prototyping efort toward designing potential use cases of AV-HEP interaction to support better communication through established safety and trust. While researchers and academics previously presented interface designs mostly for visual cues, we highlighted the usage of auditory interfaces to empower hearing aids users to communicate with AVs in scenarios requiring direct communication. We believe our research outlines a future path forward in this novel design space of AV-hearing enhanced pedestrian interaction.

References

[1] E. Florentine , M. A. Ang , S. D. Pendleton , H. Andersen , M. H. Ang , Pedestrian notification methods in autonomous vehicles for multi-class mobility-on-demand service , in: Proceedings of the Fourth International Conference on Human Agent Interaction , HAI '16, Association for Computing Machinery, New York, NY, USA, 2016 , p. 387 - 392 . URL: https://doi.org/10.1145/2974804.2974833. doi: 10 .1145/2974804.2974833.

[2] K. Mahadevan , S. Somanath , E. Sharlin, Communicating awareness and intent in autonomous vehicle-pedestrian interaction , in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18 , Association for Computing Machinery, New York, NY, USA, 2018 , p. 1 - 12 . URL: https://doi.org/10.1145/3173574.3174003. doi: 10 .1145/3173574.3174003.

[3] T. T. Nguyen , K. Holländer , M. Hoggenmueller , C. Parker , M. Tomitsch , Designing for projection-based communication between autonomous vehicles and pedestrians , in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications , AutomotiveUI '19, Association for Computing Machinery, New York, NY, USA, 2019 , p. 284 - 294 . URL: https://doi.org/10.1145/3342197.3344543. doi: 10 .1145/3342197.3344543.

[4] K. Holländer , A. Colley , C. Mai , J. Häkkilä , F. Alt , B. Pfleging , Investigating the influence of external car displays on pedestrians' crossing behavior in virtual reality , in: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services , MobileHCI '19, Association for Computing Machinery, New York, NY, USA, 2019 . URL: https://doi.org/10.1145/3338286.3340138. doi: 10 .1145/3338286.3340138.

[5] A. Z. Asha , F. Anzum , P. Finn , E. Sharlin, M. Costa Sousa, Designing external automotive displays: Vr prototypes and analysis , in: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications , AutomotiveUI '20, Association for Computing Machinery, New York, NY, USA, 2020 , p. 74 - 82 . URL: https://doi.org/10.1145/ 3409120.3410658. doi: 10 .1145/3409120.3410658.

[6] A. Z. Asha , C. Smith , L. Oehlberg , S. Somanath , E. Sharlin, Views from the wheelchair: Understanding interaction between autonomous vehicle and pedestrians with reduced mobility, in: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems , CHI EA ' 20 , Association for Computing Machinery, New York, NY, USA, 2020 , p. 1 - 8 . URL: https://doi.org/10.1145/3334480.3383041. doi: 10 .1145/3334480.3383041.

[7] A. Z. Asha , C. Smith , G. Freeman , S. Crump , S. Somanath , L. Oehlberg , E. Sharlin, Codesigning interactions between pedestrians in wheelchairs and autonomous vehicles , in: Designing Interactive Systems Conference 2021 , DIS '21, Association for Computing Machinery, New York, NY, USA, 2021 , p. 339 - 351 . URL: https://doi.org/10.1145/ 3461778.3462068. doi: 10 .1145/3461778.3462068.

[8] M. Colley , M. Walch , J. Gugenheimer , E. Rukzio, Including people with impairments from the start: External communication of autonomous vehicles , in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, AutomotiveUI '19 , Association for Computing Machinery, New York, NY, USA, 2019 , p. 307 - 314 . URL: https://doi.org/10.1145/3349263.3351521. doi: 10 .1145/3349263.3351521.

[9] K. Kennedy , Designing for human-machine collaboration: Smart hearing aids as wearable technologies , Commun. Des. Q. Rev 5 ( 2018 ) 40 - 51 . URL: https://doi.org/10.1145/ 3188387.3188391. doi: 10 .1145/3188387.3188391.

[10] T. Imamura , Trafic signal system for blind people , 1981 . URL: https://patents.google.com/ patent/US4253083A, uS Patent 4253083A.

[11] A. Colley , J. Häkkilä , M.- T. Forsman , B. Pfleging , F. Alt , Car exterior surface displays: Exploration in a real-world context , in: Proceedings of the 7th ACM International Symposium on Pervasive Displays , PerDis '18, Association for Computing Machinery, New York, NY, USA, 2018 . URL: https://doi.org/10.1145/3205873.3205880. doi: 10 .1145/3205873.3205880.

[12] T. Hermann, Taxonomy and definitions for sonification and auditory display , in: Proceedings of the 14th International Conference on Auditory Display (ICAD) , 2008 , pp. 24 - 27 . URL: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.584.5915.

[13] K. Mahadevan , S. Somanath , E. Sharlin, Can interfaces facilitate communication in autonomous vehicle-pedestrian interaction? , in: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, HRI '18 , Association for Computing Machinery, New York, NY, USA, 2018 , p. 309 - 310 . URL: https://doi.org/10.1145/ 3173386.3176909. doi: 10 .1145/3173386.3176909.