Beyond Beeps: Designing Ambient Sound as a Take-Over Request in Automated Vehicles

Abstract

The design of take-over requests in automated vehicles traditionally focuses on safety and reaction time. We are interested in how take-over requests can be designed from a broader user experience perspective while ensuring safety. This paper proposes designs for ambient sound (i.e., soundscape) and driving noise to inform the driver of transition situations. Drivers must take-over control within the time budget, the time from the take-over request to the automation system limit. The time required for a safe transition depends on the complexity of the driving environment. In a scheduled take-over, which is not an emergency, there is an opportunity for an interaction that gradually introduces the driver into the transition process. Ambient sound is expected to lead the driver back to the loop with comfort, creating a novel transition experience as well as safety.

Keywords

• 1 Human-machine interaction
• Automated vehicles
• Sound design

1. Introduction

As automated vehicle performance becomes advanced, the need for driving monitoring will reduce [ 1 ]. In highly automated vehicles, drivers do not have to stay in the loop and do not have to supervise the driving scenario. Automotive manufacturers emphasise the ability of automated vehicles to reduce the cognitive load of driving, allowing users to engage in secondary tasks such as doing a smartphone and watching videos [ 2 ]. The critical challenge in interaction research is the design of the control transition phase, where drivers should take-over the driving control due to automation limitations. Previous studies [ 3, 4 ] have investigated user interfaces that can lead to fast and safe transitions to manual driving. Beep or language-based sound is mainly used as a notification cue or to convey urgency. However, the transition does not always take place in an urgent situation. The Operational Design Domain (ODD) refers to the specific operating conditions and environments in which automated vehicles can safely and effectively operate. In the event that the vehicle encounters a situation outside of the ODD, it will schedule a transition to a safe state where it can no longer operate autonomously. During this transition, a take-over request can be initiated, which is not an emergency but is designed to provide sufficient time and information to allow a driver to take control of the vehicle safely.

The bustling sound of a mall tells us that there are many people nearby, while the popcornpopping sound of engine noise tells us to watch for fast-speed cars. Sound is not only complementary to visual information but can also provide information about the structures of the world [ 5 ]. Drivers’ visual attention is often needed in automated vehicles' secondary tasks. Capturing omnidirectional attention is an important advantage of an auditory interface [ 6 ]. Sound is suitable for drawing the driver’s attention regardless of the direction of the visual attention [ 7 ]. In this position paper, we propose ambient sound as a novel transition interface for take-over requests.

Ambient systems provide information in the attention periphery of a user. This aims to display information without directly interfering with primary tasks [ 8 ]. Gradually changing the level of ambient stimuli stimulates human perception and allows users to become aware of a new situation. Studies of ambient light in automated vehicles have already been conducted, such as changing colours according to automation modes or providing transition information. On the other hand, there is a paucity of research on ambient sound in automated vehicles. Although explicit sounds such as beeps provide direct information about the transition scenario, ambient sound can progressively support the driver to enter the transition phase. The driver's perception of the pre-awareness of the transition situation before the final take-over request increases safety. [ 9 ] showed that auditory pulse(beep) pre-alerts that occurred well before the take-over request resulted in a safer transition situation. At the same time, the pulse beep made drivers annoyed. Consequently, there lies an opportunity to design a more elaborate sound experience for interaction scenarios that support transition situations. A sound experience that goes beyond beeps! We explore ambient sounds that have the potential to lead the driver back to the loop with comfort, creating a novel transition experience as well as safety. Further, we will discuss the design challenge.

2. Ambient sound

We propose two types of ambient sound as a take-over request: 1. designed soundscape and 2. driving noise, which is an ambient sound in a driving situation. 2.1.

Soundscape design

A soundscape is an acoustic environment perceived by listeners in contexts [ 10 ]. Soundscape exists through human perception of the acoustic environment, and it can provide context information [ 11 ]. Drivers perceive a situation inhabiting a soundscape formed during driving, and it develops into a driving experience [12]. Our case at the 'Interactive audio design' master course at the Faculty of Industrial Design Engineering provided some excellent examples of the possibility of forming soundscape as a takeover request and new interaction about how it designs driving experience. Students designed context-relevant soundscapes for automated vehicles, including take-over requests. Students received the following scenario description on which they had to base their sound design: "The driver is around 35 years old and runs a startup company. The vehicle is highly automated. The vehicle is a B-segment-sized, commonly described as a small car with an enormous volume in Europe (i.e., Toyota Yaris, Renault Clio)." Students made a persona based on the scenario description and analysed scenarios which need sound for interaction and designed a soundscape for scenarios using a sound design tool.

The deliverables showed that soundscapes could be used in various ways in highly automated vehicles, including take-over situations. Although soundscape does not directly present explicit alert, it could provide tension that the situation in which action was to be taken was approaching by controlling the elements of sound. The design brought forward the possibility that sound could inform drivers about contextual information (transition) and gradually lead them into the loop. Driver-vehicle interaction through soundscapes can provide a new way of designing user experiences for drivers. This is the use of sound in a more advanced way that enables situation awareness of the driver. 2.2.

Driving noise [ 5 ] argued that the perceptual dimensions and attributes of attention correspond to those of the sound-producing event and its environment, not to those of the sound itself. In other words, the distinction between everyday and musical listening is between experience, not sound. For example, pedestrians recognise the approach of the car by engine noise based on previous experience, and it is related to a regulation generating artificial sound in electric vehicles for safety.

Driving-noise producing event Conscious experience (everyday listening)

Driving noise is the ‘everyday listening’ in the driving scenario. It is called 'noise', and long-term exposure has a harmful effect on the driver, so vehicle designers generally aim to reduce noise. However, drivers can recognise information about vehicles or road events through noise [13]. When drivers perceive a change in noise while driving, i.e., engine ticking sound, road friction sound, or passing vehicle sound, they pay attention to the cause of the noise.

As NVH (car noise, vibration, and harshness) technology has advanced, it has become possible to control driving noise. A noise control system can actively reduce noise by emitting sound waves inverted to the incoming noise, resulting in a quieter ride. In other words, noise can be intentionally exposed. Suppose that the noise control function reduces the driving noise during autonomous driving. In that case, if vehicles gradually expose the noise (but to the extent that the driver notices the magnitude of the change) as the transition approaches, drivers may notice an approaching mode change. While most auditory user interface designs focus on creating new sounds, noise exposure amplifies the existing ambient sound, providing information about the approaching transition situation. Driving noise can provide drivers with awareness of the automated driving mode when the noise is blocked and the approach to the manual driving mode when the noise is heard. Note that this does not mean that drivers are exposed to driving noise for an extended period. It is using the noise, the source of everyday listening that drivers are already accustomed to recognising information, to draw the driver's attention to the transition situation.

3. Design consideration

An ambient sound can be used to communicate information and provide feedback to users. However, designing an ambient sound as a takeover request in automated vehicles requires careful consideration of various factors.

First, the ambient sound should be recognisable from other sounds in the vehicle, even if they are listening to music or talking to passengers. Listening to secondary tasks needs to be considered to design ambient sound. If drivers listen to music or watch videos, the solution can be to control the volume of the secondary task so that the ambient sound can be recognised. This method is already in use on the vehicle. The music volume is momentarily lowered for drivers to listen to navigator information while driving.

Further, the soundscape should be easily distinguishable to prevent the ambient sound from being confused with other sounds in the vehicle or on the road. At the same time, it should not be too loud to cause discomfort. Regarding a designed soundscape, it is necessary to validate whether the sounds suit the transition situation or cause annoyance before the evaluation phase to prevent soundscape manipulation errors. In the case of driving noise, the exposure noise level of the sounds should be recognisable so that drivers can perceive the noise level difference.

In the experiment, several human factors will be evaluated to identify the effect of ambient sounds in takeover situations. We selected situations with a sufficient time budget, as we focus on acceptance rather than only on TOR performance. Therefore, physiological factors such as gaze tracking or electrocardiogram can be measured to detect situational awareness, and subjective measurements can include trust and acceptance.

4. Conclusions

In this paper, we introduce ambient sounds, which are currently in the design phase of our study. The sounds do not necessarily have to communicate their source but rather their function and the feeling they should evoke. The sound of take-over requests has been designed to focus on the user's recognition time or reaction time due to safety or performance-related situations. Therefore, an abstract sound (a beep) has been used instead of an ambient sound. While designing an ambient sound would require careful consideration, the use of ambient sound as a takeover request in automated vehicles has great potential. By designing a sound that is easily recognisable, attention-grabbing, and pleasant, we can make the transition from automated driving to human driving smoother and safer. The success of the ambient sound will depend on the specific design and implementation of the sound, as well as evaluations to ensure its effectiveness and safety for drivers taking over control. The design of road noise as a take-over request in automated vehicles has many potential benefits, including improved safety and driving experience.

5. Acknowledgements

This research is supported by HADRIAN, funded by the European Union’s Horizon 2020 research and innovation program under grant agreement number 875597. The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the opinion of the European Union.

6. References

[12] M. Bull, Soundscapes of the car: A critical study of automobile habitation. In Car cultures. 2001: Routledge. [13] N. Gang, Don't Be Alarmed: Sonifying Autonomous Vehicle Perception to Increase Situation Awareness. Automotiveui'18: Proceedings of the 10th Acm International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 2018: p. 237-246. doi:10.1145/3239060

References

[1] SAE International , Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles . 2021 .

[2] M. L. Cunningham , M. A. Regan , T. Horberry , K. Weeratunga , V. Dixit , Public opinion about automated vehicles in Australia: Results from a large-scale national survey . Transportation Research Part aPolicy and Practice , 2019 . 129 : p. 1 - 18 . doi: https://doi.org/10.1016/j.tra. 2019 . 08 .002

[3] I. Politis , S.A. Brewster , and F.E. Pollick . Language-based multimodal displays for the handover of control in autonomous cars . In Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications . 2015 . Nottingham, UK. doi: https://doi.org/10.1145/2799250.2799262

[4] Y. Forster , F. Naujoks , and A. Neukum , Increasing anthropomorphism and trust in automated driving functions by adding speech output . 2017 28th IEEE Intelligent Vehicles Symposium (Iv 2017 ), 2017 : p. 365 - 372 . doi: 10 .1109/IVS. 2017 .7995746

[5] W. W. Gaver , What in the world do we hear?: An ecological approach to auditory event perception . Ecological psychology , 1993 . 5 ( 1 ): p. 1 - 3 . doi: https://doi.org/10.1207/s15326969eco0501_ 1

[6] D. Siwiak , and F. Jame . Designing interior audio cues for hybrid and electric vehicles . In Audio Engineering Society Conference: 36th International Conference: Automotive Audio. Audio Engineering Society . 2009 .

[7] Y.C. Liu , Comparative study of the effects of auditory, visual and multimodality displays on drivers' performance in advanced traveller information systems . Ergonomics, 2001 . 44 ( 4 ): p. 425 - 442 . doi: https://doi.org/10.1080/0014013001001136 9

[8] A. Matviienko , Towards New Ambient Light Systems: a Close Look at Existing Encodings of Ambient Light Systems. Interaction Design and Architectures , 2015 ( 26 ): p. 10 - 24 .

[9] R. M. A. van der Heiden , S.T. Iqbal, and C.P. Janssen , Priming Drivers before Handover in Semi-Autonomous Cars . Proceedings of the 2017 Acm Sigchi Conference on Human Factors in Computing Systems (Chi'17) , 2017 : p. 392 - 404 . doi: https://doi.org/10.1145/3025453.3025507

[10] R. M. Schafer , Exploring New Soundscape. Unesco Courier , 1976 ( 10 ): p. 4 - 8 .

[11] F. Aletta , J. Kang , and O. Axelsson , Soundscape descriptors and a conceptual framework for developing predictive soundscape models . Landscape and Urban Planning , 2016 . 149 : p. 65 - 74 . doi: https://doi.org/10.1016/j.landurbplan. 2016 . 0 2 . 001