=Paper=
{{Paper
|id=Vol-2700/paper5
|storemode=property
|title= Using Augmented Holographic UIs to Communicate Automation Reliability in Partially Automated Driving
|pdfUrl=https://ceur-ws.org/Vol-2700/paper5.pdf
|volume=Vol-2700
|authors=Mahdi Ebnali,Razieh Fathi,Richard Lamb,Shiva Pourfalatoun,Sanaz Motamedi
|dblpUrl=https://dblp.org/rec/conf/chi/EbnaliFLPM20
}}
== Using Augmented Holographic UIs to Communicate Automation Reliability in Partially Automated Driving==
https://ceur-ws.org/Vol-2700/paper5.pdf
Using Augmented Holographic UIs to
Communicate Automation Reliability in Partially
Automated Driving
Mahdi Ebnali Shiva Pourfalatoun
University at Buffalo Colorado State University,
Amherst, NY 14260, USA Fort Collins, CO
mahdiebn@buffalo.edu shivapf@colostate.edu
Razieh Fathi Sanaz Motamedi
Rochester Institute of University of Florida,
Technology, Rochester, NY Gainesville, FL
rxfvcs@rit.edu smotamedi@ufl.edu
Richard Lamb
Neurocognition Science
Laboratory, East Carolina
University, Greenville, NC
lambr19@ecu.edu
Abstract
Drivers are assumed to actively supervise the road in
partially automated driving, but a growing body of
research shows that they become more complacent in
system operation and fail to continuously monitor the
road, which results in mode confusion. Lack of
transparent communication of automation mode and its
________________________________________________________
level of reliability has been discussed as a main
Workshop proceedings Automation Experience across Domains
underlying cause of these challenges. Our study
In conjunction with CHI'20, April 26th, 2020, Honolulu, HI, USA
Copyright © 2020 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
Website: http://everyday-automation.tech-experience.at
assessed a concept of augmented reality lane marking to overcome challenges in user interaction with such
(AR-LM) to communicate the status of automation and sophisticated systems, as unique challenges arise with
its level of reliability. In a partially automated driving increased automation. In particular, the reliance on the
simulator study, participants’ glance behavior, takeover human driver to supervise the automation and to
time in critical events, hazard detection, and manually control the car in some limited driving
automation perception were collected in two groups modes—as is the case in many commercially available
(control and AR-LM). The results indicated an effect of automated vehicles—have been associated with issues
the AR-LM UI on takeover time, gaze time on the road, related to driver states, such as erratic workload, loss
and automation trust. Our findings suggest that the AR- of situation awareness (SA), vigilance decrements and
LM concept can potentially assist drivers in maintaining automation complacency
their visual attention to the road in low-reliability and
failure conditions. However, this UI concept may also Highly automated driving is expected to be
cause lower hazard detection when automation is commercially available in the market in the near future;
running in high-reliability mode. however, vehicles equipped with partial driving
automation are available in the current market, and a
Author Keywords growing body of studies investigate opportunities to
Augmented reality; Partial driving automation; Level of improve this system. Based on the definition provided
reliability; User experience; Gaze behavior; Trust; by SAE, partially automated vehicles are equipped with
Takeover. speed controlling and lane-keeping functions but
requires that the driver continuously monitors the road
CSS Concepts and takes over the vehicle control when it surpasses its
• Human-centered computing ~Human computer operational design domain.
interaction (HCI)~Interface Paradigms;
Mixed/augmented reality Although drivers are assumed to actively supervise the
road in partially automated driving mode, they showed
Introduction to become more complacent in system operation and
Technological advancement over the past years has led failed to monitor the system continuously[5] and as a
to significant growth of driver assistance systems and result, they maintained lower situational awareness [6].
the emergence of autonomous vehicles. It is predicted Lower situation awareness in partial driving automation
that full driving automation will be commonplace on the has been associated with mode confusion, where the
roads in the future [1]. However, nowadays, automated driver doesn’t understand what mode the vehicle is
driving continues to be challenged by technical driving[7]. Beyond the unpleasant automated driving
constraints [2], ethical issues [3], and human factors experience when mode confusion occurs, the mode
considerations [4]. While vehicle automation confusion also has been reflected in previous literature
technology will continue to mature along with advances as a primary reason for incidents and accidents in
in computer vision and artificial intelligence, it is harder various domains of human-automation interaction[8].
Several reasons have been discussed in the current the road (e.g., on a high curvature), but due to
body of literature as the potential underlying factors of technical limitations, the automation may operate the
mode confusion. Reduction of driving workload in car with lower certainty.
automated driving on one hand, and having access to
several electronic devices and displays, on the other Lastly, partial driving automation is mostly designed in
hand, encourage the driver to spend more time out of a way that the driver’s inputs to the steering wheel and
the driving loop and stay engaged in non-driving pedals deactivate the automation. Although this feature
related secondary tasks (NDSTs). This insufficient helps users to take over the vehicle control easily,
monitoring behavior could lead to mode confusion. inadvertent torque inputs may also deactivate the
Moreover, drivers in such situations may not be well- automation. In this case, the driver may not realize this
prepared to regain vehicle control when a sudden transition and fail to regain control or re-activate the
change on the road ahead (e.g., missed lane marking automation properly.
or a cut-in vehicle) has prompted an emergency
takeover request. Regarding the challenges mentioned above, ensuring
that drivers have a clear understanding of the
Misunderstanding of the internal user interfaces (UIs) automation mode and remain attentive during partially
Figure 1: Driving simulator and
four AOIs specified for eye- has been mentioned as another constraint in partially automated driving is one of the most important
movement data automated driving [7]. Most of the current partially research topics which need further investigation.
automated vehicles in the market use visual warning Designing appropriate UIs to communicate automation
and or a combination of visual and auditory feedback to mode and its reliability could avoid mode confusion,
communicate automation modes. However, sometimes encourage drivers to monitor the road continuously,
these modalities are not straightforward enough to and safely take over the vehicle control when it is
communicate the status of automation, or they may not required. Augmented reality-based UIs could be a
be salient enough to capture the driver's attention [7]. potential solution to intuitively visualize the statue of
Moreover, previous studies have reported the automation, which, compared to the conventional visual
potentially confusing or startling effects of these types UIs, requires less visual attention shifting from the road
of warnings, especially when warnings are not to the instrument cluster[10]. Moreover, the level of
presented to the driver in a timely manner[9]. reliability of the automation could be projected to the
windshield to provide the driver with a more
Furthermore, most of the current UIs are binary and transparent view of automation status. Regarding these
will only present whether the automation is on or off. potential premises, the objectives of the current
Figure 2: Tobii Eye Tracking
Glasses 2 Lack of transparent communication regarding the level research were to explore the effects of the Augmented
of reliability of the automation may mislead the driver Reality-based Lane Marking (AR-LM) concept on the
and result in the occurrence of mode confusion. For driver’s glance behavior, takeover time in critical
example, the drive may expect that the automation events, hazard detection, and automation perception
reliably operates the vehicle in a particular segment of
during the level 2 automated driving in a simulated software was used to analyze the data. To make eye-
environment. movement data easier to interpret, we specified four
area-of-interests (AOIs) including road scenery, phone
Methods display, instrument cluster, and hazard perception
Method areas (Figure 1).
A total of 15 subjects, 7 males and 8 females, between
the ages of 21 to 34 (M = 26.02, SD = 4.55) UIs
participated in the study. Participants were recruited Figure 3 shows the concept of AR-LM to communicate
using online postings on public forums. All participants the statue of automation and its level of reliability in
possessed a valid driver’s license and had a normal or three conditions: a) high-reliability, b) low-reliability,
corrected-to-normal vision (determined through near and c) failure. Participants in the control group were
and far visual acuity and contrast sensitivity). All informed about automation mode in high-reliability
participants had little or no automated driving conditions with a green color UI on the instrument
Figure 3: The concept of AR-LM experience. At the beginning of each experimental cluster. In addition to this UI, participants in the AR-LM
to communicate the statue of condition, all participants received the same pre-written group were also provided with a holographic AR
automation and its level of
textual instruction about how to use the simulator. projecting a green bar on the forward road scenery
reliability in three conditions: a)
high reliability, b) low reliability, (Figure 3-a). In low-reliability modes, the visual UI on
c) failure Apparatus the instrument cluster remained in green color in both
This experiment was performed using a fixed-based groups; however, participants in the AR-LM group were
simulator, which was designed in the Unity 3D and provided with a holographic AR projecting a yellow bar
operated on Dell Optiplex 7010 (Intel Quad-Core i7- on the forward road scenery (Figure 3-b). Once the
3470 3.2GHz, 16GB RAM) workstation running Windows vehicle passed the high curvature section of the road,
10. Two widescreen displays showed the visual the holographic yellow bar turned to a holographic
simulation imagery, rendered at 60 Hz (Figure 1). The green bar indicating the high-reliability mode. In failure
simulator was able to provide two driving modes: modes (missing lane marking and obstacle ahead), an
partial driving automation and manual driving. Based auditory feedback was provided for both groups in the
on the features outlined for level 2 automation in SAE form of sequences of three tonal beeps (each beep at
J2016, the automated mode supported simultaneous 800 Hz and lasting 0.1s) with a time budget of 10
longitudinal and lateral control. Participants were able seconds. The visual UI located on the instrument
to engage and disengage automaton by pressing the cluster also turned to red. In the AR-LM group, besides
same button located on the right side of the steering these auditory and visual feedbacks, participants were
wheel. Disengagement was also possible through provided with a holographic AR projecting a red bar on
pressing the brake (> 10% of braking length) or the forward road scenery (Figure 3-c).
turning the steering wheel more than (> 7 degrees). A
Tobii Eye Tracking Glasses 2 (Figure 2) also recorded
participants’ glance behavior, and the Tobii Lab
Design of experiment 2.9 s, SD=0.95 s, p <0.05). The result of pairwise
The driving test consisted of 15 miles long (10 minutes comparisons for the type of takeover showed no
manual driving and 20 minutes automated driving) on a significant difference in takeover time between
highway was simulated while participants drove in a obstacle-ahead events in the AR-LM group (Mean=2.4
partial driving automation mode. As shown in Figure 4, s, SD=0.88 s) and control group (Mean= 2.1 s, SD=
automated driving scenarios included three types of 0.83 s, p=0.09). However, takeover time of missing
automation modes: high-reliability in straight or low lane marking events in the control group (Mean=3.1 s,
curvature sections of the road, low-reliability in high SD= 1.01 s) was significantly longer than this measure
curvature section of the road, and failure when the lane in the AR-LM group (Mean=1.9 s, SD= 0.71 s, p<0.05).
marking has faded out or an object blocked the forward
road. Each participant experienced two low-reliability Gaze behavior
modes and two failure modes. In the failure modes, Gaze time on three AOIs (road scenery, instrument
participants were responsible to take over the vehicle cluster, and phone display) in three modes of
control in a timely manner. The rest of the automated automation (high-reliability, low-reliability, and failure)
driving session was in high-reliability mode. While were investigated. In general, compared to the control
driving in automated mode, participants in both groups group, the AR-LM concept resulted in significantly
were asked to watch a video of the Our Planet series on shorter gaze time on the road scenery (control: M=
Netflix, which was displayed on the phone. They were 919.6 s; SD= 32.5 s; AR: M= 869.2 s; SD= 28.1 s,
requested to watch this video in a self-paced manner. p<0.01), the instrument cluster (control: M= 52.9s;
SD= 17 .4 s; AR: M=23 s; SD= 8.5 s, p<0.01), and
Independent variables longer gaze time on the phone display (control: M=
As the between-subject factor, visualization of 207.5s; SD= 19 s; AR: M= 235.8 s; SD= 15 s,
automation status (with and without AR information) p<0.01).
was an independent variable; and takeover time, gaze
time, hazard detection, automation perception were Moreover, as shown in Figure 5, investigating average
dependent variables. Automation perception also was gaze time for each automation mode revealed
measured after the driving tests using an 11-item significantly longer gaze time on the road scenery in
Figure 4: Driving scenarios questionnaire regarding automation trust, automaton the low-reliability mode (control: Mean=82.8 s, SD=
including three types of
acceptability, and ease of use in a scale 1 (I strongly 20.1 s, AR: Mean=112.3 s, SD=19.5 s, p<0.05) and
automation modes- high
reliability: straight road or low disagree) to 7 (I strongly agree). failure mode (control: Mean=18.4 s, SD=5.9 s, AR-LM:
curvature, low reliability: high Mean=34.6 s, SD=8.4 s, p<0.05) when participants
curvature, and failure: lane Results were provided with AR-LM support. Compared to the
marking has faded out or Takeover time control group, participants in the AR-LM group showed
obstacle ahead
The average takeover time of both failure events for longer gaze time on the phone display when the vehicle
the AR-LM group (Mean= 2.1 s, SD=0.8 s) was less was running in straight/low curvature roads (high-
than the takeover time for the control group (Mean= reliability mode). There was no significant difference in
gaze times on the instrument cluster in low-reliability reliability helped participants to react faster in takeover
mode between two groups. Participants in both groups events.
also did not show different gaze behavior on the phone
display in failure modes. In addition, compared to the control group, gaze data
revealed that participants in the AR-LM group looked at
No significant difference was observed in gaze time the phone display for a longer time. Considering this
between two types of failure events (missing lane finding and trust data, it seems AR information led to
marking and obstacle ahead) in the AR-LM group; higher automation trust, and as a result, participants
however, participants in the control group looked at the preferred to spend more time engaged in watching the
road scenery for longer time in the obstacle ahead video. To have a deeper understanding of gaze
event (Mean=9.2 s, SD=2.1 s), compared to the behavior, we also analyzed the gaze time data
missing lane markings event (Mean=6.3 s, SD=3.1 s, separated for each automation mode. Interestingly, the
p<0.05). Regarding hazard detection event, the results results showed that when the level of reliability
showed a shorter fixation time in the hazard detection decreased (in high curvature section of the road), the
AOI when participants were provided with AR-LM AR-LM UI caused participants looked at the road
support (control group: M=7.2 s, SD= 1.5 s; AR-LM scenery for a longer time. The reason for this behavior
group: Mean=3.1 s, SD=.93 s, p<0.01). might be that the yellow holographic AR concept
captured participants’ visual attention and then they
Automation Perception interpreted an association between this change and the
The results of the automation perception questionnaire high curvature section of the road. Similar benefit of AR
(Figure 6) showed a significant difference in automation information was observed in failure events. When the
trust between AR-LM and control groups (p=0.026). participants were provided with a red holographic AR,
Participants also reported slightly higher ease of use for they spent longer time looking at the road scenery and
AR intervention, though the difference was only less time on the phone display. Although these findings
marginally significant (p= 0.054). Automation could be considered as positive effects of AR-LM in
acceptability in the AR-LM group was not significantly communicating the level of uncertainty of partial
different from participants in the control group. driving automation, the application of AR-LM should be
(p=0.9). considered with the potential costs of longer
engagement in secondary tasks when automation is
Discussion running in high-reliability mode.
Our findings suggest that the holographic AR concept of
the lane marking had a significant effect on takeover The results also showed a shorter average gaze time on
time. On average, participants in the AR-LM group the hazard perception AOI in the AR-LM group. Two
Figure 5: Gaze time on road
started to take over the vehicle control 0.8 seconds participants in this group also did not look at the hazard
scenery, instrument cluster, and
phone display AOIs earlier than those in the control group. It seems AR perception AOI at all. We only considered one hazard
information regarding automation mode and its level of perception event which was appeared when the
automation was running in high-reliability mode (green acceptance [10] and reported higher ease of use when
holographic AR concept). Two possible reasons may automation modes were presented using AR concepts.
explain this finding. First, the AR-LM UI over-captured Our results, however, did not show a meaningful
drivers’ visual attention to a particular part of the road difference in acceptance data between the control and
scenery. In this case, caution must be exercised in the AR-LM groups. This finding is confusing because we
application of AR-based UIs to avoid potential found higher trust and ease of use in the AR-LM group,
Figure 6: Results of automation
perception questionnaire
distracting effects of AR information. As the second and according to the Technology Acceptance Model
reason, higher trust achieved in the AR-LM group (TAM) [13], it seems reasonable to expect higher
caused participants to stay for a longer time engaged in acceptance in this group. One explanation for this result
secondary tasks. In this case, researchers and is that although trust and acceptance are interrelated
designers need to consider the costs associated with concepts; they do not necessarily follow the same
out-of-the-loop performance problems. This potential pattern[14]. Several sources of variability such as
complacency could reduce drivers’ situational individual differences in prior experiences, intention to
awareness and impair driving performance especially in use technology, and perceived attractiveness may
critical transition tasks [6]. contribute to this result. Attractiveness and intent are
two other components of TAM, which we did not
The results partially support our assumption regarding measure in this study. Considering the importance of
the effectiveness of the AR-LM concept on automation acceptance in technology usage, more studies are
perception. Compared to the control group, participants required to investigate underlying components of
who received AR information reported higher technology acceptance and its association with trust.
automation trust after the driving test. This finding is
supported by a recent experimental study[10] and also Conclusion
a theoretical link between trust of in-vehicle technology This paper has reported on a driving simulator study
and warning system reliability described by [11]. conducted with 15 participants to investigate whether a
Reliability information provided in the AR-LM concept holographic AR prototype could be used to
may help participants to understand the system better communicate automation mode and its level of
and build higher trust in partial diving automation. In reliability. The results indicate that participants who
addition, more transparent visualization of automation were provided with AR-LM UIs looked longer at the
mode and its level of reliability supported in AR road scenery in low reliability and failure situations.
information has been associated with a lower likelihood Moreover, they were better prepared to switch to
of mode confusion and ultimately low trust [12]. manual control than participants who did not receive
AR information. However, AR-LM led to less road-
Ease of use was also marginally higher in the AR-LM monitoring behavior when automation was running in
group. Participants who received AR information were high-reliability mode. Moreover, participants in the AR-
more likely to find the system easier to use. Previous LM group were more likely to miss hazard detection
studies evaluated ease of use as a component of events. In our future work, we will investigate these
limitations with a larger sample in more critical and and design solutions. in Proceedings of the
non-critical situations. human factors and ergonomics society
annual meeting. 2012. Sage Publications
References Sage CA: Los Angeles, CA.
9. Abe, G. and J. Richardson, Alarm timing,
1. Walker, G.H. and N.A. Stanton, Human trust and driver expectation for forward
factors in automotive engineering and collision warning systems. Applied
technology. 2017: CRC Press. ergonomics, 2006. 37(5): p. 577-586.
2. Litman, T., Autonomous vehicle 10. von Sawitzky, T., et al. Increasing trust in
implementation predictions. 2017: Victoria fully automated driving: route indication on
Transport Policy Institute Victoria, Canada. an augmented reality head-up display. in
3. Faulhaber, A.K., et al., Human decisions in Proceedings of the 8th ACM International
moral dilemmas are largely described by Symposium on Pervasive Displays. 2019.
utilitarianism: Virtual car driving study 11. Lee, J.D. and K.A. See, Trust in automation:
provides guidelines for autonomous driving Designing for appropriate reliance. Human
vehicles. Science and engineering ethics, factors, 2004. 46(1): p. 50-80.
2018: p. 1-20. 12. Parasuraman, R., T.B. Sheridan, and C.D.
4. Casner, S.M., E.L. Hutchins, and D. Norman, Wickens, A model for types and levels of
The challenges of partially automated human interaction with automation. IEEE
driving. Communications of the ACM, 2016. Transactions on systems, man, and
59(5): p. 70-77. cybernetics-Part A: Systems and Humans,
5. Baee, S., et al., EyeCar: Modeling the Visual 2000. 30(3): p. 286-297.
Attention Allocation of Drivers in Semi- 13. Davis, F.D., User acceptance of information
Autonomous Vehicles. arXiv preprint technology: system characteristics, user
arXiv:1912.07773, 2019. perceptions and behavioral impacts. 1993.
6. Eriksson, A. and N.A. Stanton, Takeover 14. Koustanaï, A., et al., Simulator training with
time in highly automated vehicles: a forward collision warning system: Effects
noncritical transitions to and from manual on driver-system interactions and driver
control. Human factors, 2017. 59(4): p. trust. Human factors, 2012. 54(5): p. 709-
689-705. 721.
7. Banks, V.A., et al., Is partially automated
driving a bad idea? Observations from an
on-road study. Applied ergonomics, 2018.
68: p. 138-145.
8. Saffarian, M., J.C. de Winter, and R. Happee.
Automated driving: human-factors issues