By taking over parts of the driving task, automated vehicles are expected to contribute to increase of driving safety. Moreover, with the help of numerous sensors and advanced humancomputer interaction designs, automated vehicles with higher levels of automation provide valuable information to increase driver's situational awareness (knowing what is going on around you) to further assist with the driving task. However, although road safety is expected to increase with each higher level of automation, the available data suggests that human driving behaviour in conditionally automated vehicles may be a weak link in their contribution to road safety. For example, despite the clear definition and instruction that the driver has to remain engaged in the task of driving (SAE level 2 of automation) or be ready to take over control of the vehicle at any time (SAE level 3 of automation), drivers of conditionally automated vehicles seem to get easily involved in performing secondary tasks, do not pay sufficient amount of attention to the environment and sometimes neglect (accidentally or intentionally) the primary task of driving. This paper presents the development process of a head-up display intended to help towards resolving this problem, and help the driver maintain, or regain, appropriate situational awareness when operating a conditionally automated vehicle.
Driving is a dynamic task that involves multiple and often simultaneous performance of motor,
sensory and cognitive tasks. Along with the manual operation of a vehicle, driving further requires
interaction with other road participants, following traffic regulations, adapting to weather and road
conditions, etc. All of these factors can have significant effect on the driving course and have to be
considered continuously to ensure safe driving. Monitoring the environment hence represents crucial
part of driving, as it increases the driver’s situational awareness (knowing what is going on around you)
and enables the driver to make appropriate and effective decisions for a safe and comfortable travel [
To overcome this, one of the main interests in the automotive industry over the last few decades has
been focused on the development automated vehicles, which can partially takeover the task of driving
- with the ultimate goal of one day the vehicle completely taking over task of driving in the form of an
autonomous vehicle. The Society of Automotive Engineers (SAE) has defined 6 levels of automation
ranging from level 0 (L0), where the driver operates the vehicle completely manually and without any
help from the vehicle, to level 5 (L5) or a fully autonomous vehicle, where the vehicle is capable of
performing all driving functions under all conditions [
Intrigued by this effect, we were motivated to develop a head-up display (HUD), which would help
the driver maintain, or regain, appropriate situational awareness when operating a conditionally
automated vehicle. The development process started with a review of the state-of-the art, development
of four prototypes, which were followed by an exploratory study, and creation of a final version of the
HUD. This paper presents the final version of the HUD and presents the next steps for validating it. It
also provides a short summary of the first steps of the development process; the in-detail presentation
of these steps can be found in [
The development process started with a review of state-of-the-art in-vehicle information systems
(IVIS) used in conditionally automated vehicles for helping the driver maintain situational awareness.
The review revealed that most of the available literature on IVIS for conditionally automated vehicles
focuses on the design of user interfaces for communication of information only during the takeover
request period (as evident from several reviews of related work in recent years by Gabbard et al. [
The review further revealed that there are discrepancies in how to present information, and more over how much information to feature in a HUD. For example, it was revealed that presentation of information using augmented reality (AR), which enables presentation of information directly in the environment, provides higher information content and therefore better understandability and usefulness [9] compared to classic two-dimensional HUDs, which are projected on a fixed position on the windshield. AR HUDs also result in longer gaze durations on the road further enabling faster drivers’ reactions and better recognition of dangerous events [10]. However, AR HUDs, which due to their increasing complexity, require frequent changes in attentional allocation [10][11], and a poorly designed AR HUD can, due to visual clutter, contribute to information and cognitive overload. And there isn’t a clear consensus on the amount of information that should be featured in the HUD either. On one hand, it was revealed high complexity of a HUD can have a negative effect on the driver’s situation awareness [12] and, in the case of AR HUDs can even cause cognitive stress and decreased safety [10].
However, all of these studies were completed in different testing settings, used different driving environments and with different testing participants, which do not allow for a comprehensive or reliable direct comparison of results. Currano et al. [12] found that the factors that make up the complexity of the scene may have a greater negative impact on the situation awareness than the actual complexity of the HUD design. In that regard, we decided to develop four HUD prototypes, which would differ in the amount of information presented in the HUD, and in the mode of information presentation (2D or AR), and compare them in the same driving environment and testing settings.
The HUD display prototypes were designed using OpenSceneGraph toolkit (OpenGL and C++). The final interface modules are implemented as DLL files integrated to the driving simulation software SCANeR (as augmentation of the default VISUAL module). The icons used in the HUD prototypes were designed and drawn in Adobe Illustrator. Several visual elements of the designed HUD interfaces require specific modification of the terrain elements and are therefore not directly transferable among different SCANeR scenarios.
Two different HUD prototypes differing in mode of information presentation were designed: 2D HUD – a fixed size and position visual display projected on the windshield above the steering wheel, and AR HUD – visual display that is composed of a fixed size and position visual display element and other visual highlighting elements displayed anywhere within the driver’s visual field in the environment outside of the vehicle.
Additionally, two versions of both the 2D HUD and AR HUD prototypes were created, differing in the amount of information presentation: MIN HUD – displays information that are usually not featured on the dashboard, such as road signs, GPS, etc. MAX HUD – displays information that are usually featured on the dashboard and additional environment and traffic information, such as for example current speed of the ego vehicle, distance to vehicle in front, etc.
All four prototypes are presented in Figure 1.
The study was conducted in a simulated driving environment consisting of a motion-based driving simulator [13] with real car parts (seat, steering wheel and pedals) and a physical dashboard. The visuals were displayed on three 49” curved TVs ensuring a 145° field of view of the driving environment (see Figure 2). The driving scenario was developed in SCANeR Studio [14]. It lasted for 13 km (8.08 mi) and simulated a route from a suburban area to a city center. The HUDs were evaluated for driving a conditionally automated vehicle (L3). During the driving scenario, there were multiple intersections with pedestrian crossings, and other road participants. Each trial featured four requests to turn on the automated driving system (ADS) and four requests to take over control of the vehicle. The takeover requests occurred due to both critical (for example, a busy pedestrian crossing or complicated crossroad) and non-critical events (this was intended to simulate the vehicle simply losing communication to the infrastructure or vehicle sensors’ system failure). The trial would always start and end in manual mode of driving. This resulted in five manually driven intervals, and four intervals in automated mode, which lasted approximately 6.5 km (4.04 mi) each, which resulted in half of the route being driven in automated mode and the other half in manual mode.
The test participants’ main task was to safely reach the final destination. To it, they were guided to with a navigation system, which was part of the HUD interface, as presented below. During the automated drive, they were asked to play a calculus game on a mobile phone, to simulate engagement in non-driving related activities when not required to operate the vehicle.
The study had a within-subjects design, resulting in four trials, where all participants completed all four trials: 1) with the 2D MIN HUD, 2) with the 2D MAX HUD, 3) with the AR MIN HUD and 4) with the AR MAX HUD. 30 participants (16 male) participated in the study. They were aged between 23 to 55 (M=36.767, SD=8.891), and had driving experience varying from 1 up to 36 years (M=17.200, SD=8.856)
The HUDs were evaluated on the driver’s situational awareness through monitoring of the driving performance and cognitive load. In addition, scores on user experience, perceived usability, and user’s preferences and opinions were collected.
The results from the exploratory study were used to get an insight about how should the final version of the HUD should look like: which elements it should include, and how they should be displayed.
From a driving performance perspective, the analysis showed that shortest reaction times and best speed limits following were obtained when the test participants operated the (simulated) vehicle with the 2D MAX HUD version. As slower reaction times and speeding are recognized as one the most common causes for road accidents, the 2D MAX HUD can be used to evoke faster reaction times and discourage speeding, which can lead improved driving safety.
Another indicator on driving safety is the time spent looking away from the road. Looking for more than 2 seconds away can result in 50% higher chance for road accidents. From the explored HUDs, lowest time spent looking away from the road was detected for trials with the AR MAX HUD version.
Regardless of the mode of presentation, the MAX HUD versions, which include also information on distance to vehicle in front and indicate when this distance is too short, resulted in lower periods of time the test participants drove with too short time to collision compared to the MIN versions, consequently indicating that featuring this information on the HUD can further improve the driving safety.
The lateral and longitudinal control of the vehicle were also observed as indicators of driving performance. They can be related to the driving safety, but also to driving comfort as abrupt changes can cause passenger sickness, reduced trust in the vehicle and overall discomfort. In this regard, lower acceleration rates and lateral deviations were obtained in the trials with the 2D HUDs. Somewhat contradictory, lowest deceleration rates were obtained with the AR HUDs.
The study also focused on collecting self-reported data, to obtain an insight into the test participants preferences and ratings on their user experience and perceived usability of the HUD prototypes. The result analysis revealed that the test participants obtained better user experience when driving with the MAX HUD versions; which was also true for the perceived usability.
Considering all of the results, an iterated prototype of the Visual HUD was developed. It is based on the 2D MAX HUD version, and incorporates the best rated features from the AR MAX HUD version (GPS direction on the road, highlight of important participants during takeover). Additionally, it incorporates changes highlighted by the test participants, such as, for example, displaying only too short distance to vehicle in front (previously for both 3 seconds and 1.5 seconds), and display of only important road signs from the environment (previously all). This prototype further introduces longer visual notification for takeover of 15 seconds compared to only 5 in the previous versions.
The results of the exploratory study ([
The validation of the interface was conducted with a user study with 30 participants (different from the ones in the evaluation study). They were 15 females and 15 males, aged between 21 and 57 (M = 30.17, SD = 10.60), and had driving experience ranging from 2 to 39 years (M = 11.78, SD = 10.12). Due to technical difficulties, one of the participant did not complete the study, and their results were excluded from the analysis.
*Road signs rated as important in the exploratory study
This study was completed in the same testing environment as the evaluation study (see Figure 2). It had a within-subjects design, in which all participants completed two trials: 1) a baseline trial, where 15 (counting down to 0) 4000 Hz pure tone [[15]] the test participants drove without the HUD and, 2) a trial with the HUD, where in addition to the classic head-down dashboard also used in the baseline, featured the final version of the HUD.
The HUD was evaluated on its effect on driver’s situational awareness through monitoring of the driving performance and cognitive load. Additionally subjective ratings on user experience and perceived usability, acceptance of new technologies, and user’s opinions were collected.
The driving performance was monitored through multiple driving performance indicators for which the HUD provided help in maintaining situational awareness, such as, lateral and longitudinal control of the vehicle, following traffic rules, interaction with other participants (keeping appropriate safety distance to vehicles in front), accidents, takeover performance and following navigational cues.
The cognitive load was observed through changes in the user’s pupil size, which was shown to be a good indicator of driver’s cognitive load [16]. Other pupilometry was further used to evaluate attentional demand and distractions.
For the subjective ratings, standardized questionnaires for collecting information on user experience, perceived usability and acceptance of new technologies were used: User Experience Questionnaire [17], System Usability Scale [18], and Acceptance of Advanced transport telematics [19]. Additionally, the participants were asked to rate the system using a 7-point Likert scale on the following aspects: 1) acceptable driving experience, 2) comfort of use, 3) comprehensiveness of the presented information, 4) intent to use in the future, 5) trust, 6) feeling of control over the vehicle, and, 7) safety. 3.3.
The driving performance and pupilometry data are currently under the process of data cleaning. However, already the preliminary results from the study revealed better overall driving performance for the trials completed with the HUD compared to trials with only the classic head down dashboard. Biggest differences were observed in the following of speed limits, especially in areas with lower speed limits (30 km/h), suggesting that the HUD helped the test participants with their situational awareness of the speed limitations and improved control of the speed of the vehicle.
The subjective data showed that participants rated their user experience with the HUD above average for five out of its six aspects: attractiveness, perspicuity, efficiency, dependability and novelty. The participants did not find the HUD to provide any stimulation, indicating that more encouraging features could be added in the future. The UEQ scores scale ranges from -3 (horribly bad) to +3 (extremely good), however because of the calculations of means the authors of the UEQ tool point out that it is extremely unlikely to get scores above +2 or below -2. Values between -0.8 and above +0.8 are considered as neutral, and scores above +0.8 represent a positive and scores below -0.8 represent a negative evaluation. The scores for the HUD are presented in Table 2.
Positive scores were found also on all of the evaluated aspects of the HUD (see Figure 3). Highest score was found for comprehensiveness of the presented information in the HUD, confirming the appropriate selection of information and their mode of presentation in the HUD. Lowest scores were obtained for trust and intent to use the system in the future. This two aspects are indeed very related, as a low trust in the system can negatively affect the user willingness to use a product. Given the rest of the results, and based on the feedback after completing the study from the test participants, we speculate that this scores reflect the low trust towards automated vehicles and not necessarily only the HUD. Future steps will try to include trust inducing features in the HUD, which could not only help with the maintain of situational awareness, but also trust in the vehicle.
The work presented in this paper was financially supported by the Slovenian Research Agency within the project Modelling driver's situational awareness, grant no. Z2-3204, and by the European Union’s Horizon 2020 research and innovation program for the project HADRIAN, grant agreement no. 875597. This document reflects only the authors view, the Innovation and Networks Executive Agency (INEA) is not responsible for any use that may be made of the information it contains.
[8] A Study on User Experience of Automotive HUD Systems: Contexts of Information Use and UserPerceived Design Improvement Points. International Journal of Human-Computer Interaction, 35(20), 1936–1946. [9] Schömig, N., Wiedemann, K., Naujoks, F., Neukum, A., Leuchtenberg, B., & Vöhringer-Kuhnt, T. (2018, September). An augmented reality display for conditionally automated driving. In Adjunct proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications (pp. 137-141). [10] Feierle, A., Beller, D., & Bengler, K. (2019, October). Head-up displays in urban partially automated driving: Effects of using augmented reality. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 1877-1882). IEEE. [11] Riegler, A., Riener, A., & Holzmann, C. (2021). Augmented Reality for Future Mobility: Insights from a Literature Review and HCI Workshop. I-Com, 20(3), 295–318. [12] Currano, R., Park, S. Y., Moore, D. J., Lyons, K., & Sirkin, D. (2021, May). Little road driving hud: Heads-up display complexity influences drivers’ perceptions of automated vehicles.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-15). [13] Vengust, M., Kaluža, B., Stojmenova, K., & Sodnik, J. (2017, September). NERVteh compact motion based driving simulator. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct (pp. 242-243). [14] AVSimulation. SCANeR studio. Available at: https://www.avsimulation.com/scanerstudio/. [15] Stojmenova, K., Policardi, F., & Sodnik, J. (2018). On the selection of stimulus for the Auditory Variant of the Detection Response Task Method for driving experiments. Traffic injury prevention, 19(1), 23-27. [16] Čegovnik, T., Stojmenova, K., Jakus, G., & Sodnik, J. (2018). An analysis of the suitability of a low-cost eye tracker for assessing the cognitive load of drivers. Applied ergonomics, 68, 1-11. [17] Laugwitz, B., Held, T., & Schrepp, M. (2008, November). Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and usability engineering group (pp. 63-76). Springer, Berlin, Heidelberg. [18] Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594. [19] Van Der Laan, J. D., Heino, A., & De Waard, D. (1997). A simple procedure for the assessment of acceptance of advanced transport telematics. Transportation Research Part C: Emerging Technologies, 5(1), 1-10.