Drivers are assumed to actively supervise the road in partially automated driving, but a growing body of research shows that they become more complacent in system operation and fail to continuously monitor the road, which results in mode confusion. Lack of transparent communication of automation mode and its level of reliability has been discussed as a main underlying cause of these challenges. Our study
________________________________________________________ Workshop proceedings Automation Experience across Domains In conjunction with CHI'20, April 26th, 2020, Honolulu, HI, USA Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Website: http://everyday-automation.tech-experience.at assessed a concept of augmented reality lane marking (AR-LM) to communicate the status of automation and its level of reliability. In a partially automated driving simulator study, participants’ glance behavior, takeover time in critical events, hazard detection, and automation perception were collected in two groups (control and AR-LM). The results indicated an effect of the AR-LM UI on takeover time, gaze time on the road, and automation trust. Our findings suggest that the ARLM concept can potentially assist drivers in maintaining their visual attention to the road in low-reliability and failure conditions. However, this UI concept may also cause lower hazard detection when automation is running in high-reliability mode.
Augmented reality; Partial driving automation; Level of reliability; User experience; Gaze behavior; Trust; Takeover.
• Human-centered computing ~Human computer interaction (HCI)~Interface Paradigms; Mixed/augmented reality
Technological advancement over the past years has led to significant growth of driver assistance systems and the emergence of autonomous vehicles. It is predicted that full driving automation will be commonplace on the roads in the future [1]. However, nowadays, automated driving continues to be challenged by technical constraints [2], ethical issues [3], and human factors considerations [4]. While vehicle automation technology will continue to mature along with advances in computer vision and artificial intelligence, it is harder to overcome challenges in user interaction with such sophisticated systems, as unique challenges arise with increased automation. In particular, the reliance on the human driver to supervise the automation and to manually control the car in some limited driving modes—as is the case in many commercially available automated vehicles—have been associated with issues related to driver states, such as erratic workload, loss of situation awareness (SA), vigilance decrements and automation complacency Highly automated driving is expected to be commercially available in the market in the near future; however, vehicles equipped with partial driving automation are available in the current market, and a growing body of studies investigate opportunities to improve this system. Based on the definition provided by SAE, partially automated vehicles are equipped with speed controlling and lane-keeping functions but requires that the driver continuously monitors the road and takes over the vehicle control when it surpasses its operational design domain.
Although drivers are assumed to actively supervise the road in partially automated driving mode, they showed to become more complacent in system operation and failed to monitor the system continuously[5] and as a result, they maintained lower situational awareness [6]. Lower situation awareness in partial driving automation has been associated with mode confusion, where the driver doesn’t understand what mode the vehicle is driving[7]. Beyond the unpleasant automated driving experience when mode confusion occurs, the mode confusion also has been reflected in previous literature as a primary reason for incidents and accidents in various domains of human-automation interaction[8]. Several reasons have been discussed in the current body of literature as the potential underlying factors of mode confusion. Reduction of driving workload in automated driving on one hand, and having access to several electronic devices and displays, on the other hand, encourage the driver to spend more time out of the driving loop and stay engaged in non-driving related secondary tasks (NDSTs). This insufficient monitoring behavior could lead to mode confusion. Moreover, drivers in such situations may not be wellprepared to regain vehicle control when a sudden change on the road ahead (e.g., missed lane marking or a cut-in vehicle) has prompted an emergency takeover request.
Misunderstanding of the internal user interfaces (UIs) has been mentioned as another constraint in partially automated driving [7]. Most of the current partially automated vehicles in the market use visual warning and or a combination of visual and auditory feedback to communicate automation modes. However, sometimes these modalities are not straightforward enough to communicate the status of automation, or they may not be salient enough to capture the driver's attention [7]. Moreover, previous studies have reported the potentially confusing or startling effects of these types of warnings, especially when warnings are not presented to the driver in a timely manner[9]. Furthermore, most of the current UIs are binary and will only present whether the automation is on or off. Lack of transparent communication regarding the level of reliability of the automation may mislead the driver and result in the occurrence of mode confusion. For example, the drive may expect that the automation reliably operates the vehicle in a particular segment of the road (e.g., on a high curvature), but due to technical limitations, the automation may operate the car with lower certainty.
Lastly, partial driving automation is mostly designed in a way that the driver’s inputs to the steering wheel and pedals deactivate the automation. Although this feature helps users to take over the vehicle control easily, inadvertent torque inputs may also deactivate the automation. In this case, the driver may not realize this transition and fail to regain control or re-activate the automation properly.
Regarding the challenges mentioned above, ensuring that drivers have a clear understanding of the automation mode and remain attentive during partially automated driving is one of the most important research topics which need further investigation. Designing appropriate UIs to communicate automation mode and its reliability could avoid mode confusion, encourage drivers to monitor the road continuously, and safely take over the vehicle control when it is required. Augmented reality-based UIs could be a potential solution to intuitively visualize the statue of automation, which, compared to the conventional visual UIs, requires less visual attention shifting from the road to the instrument cluster[10]. Moreover, the level of reliability of the automation could be projected to the windshield to provide the driver with a more transparent view of automation status. Regarding these potential premises, the objectives of the current research were to explore the effects of the Augmented Reality-based Lane Marking (AR-LM) concept on the driver’s glance behavior, takeover time in critical events, hazard detection, and automation perception during the level 2 automated driving in a simulated environment.
A total of 15 subjects, 7 males and 8 females, between the ages of 21 to 34 (M = 26.02, SD = 4.55) participated in the study. Participants were recruited using online postings on public forums. All participants possessed a valid driver’s license and had a normal or corrected-to-normal vision (determined through near and far visual acuity and contrast sensitivity). All participants had little or no automated driving experience. At the beginning of each experimental condition, all participants received the same pre-written textual instruction about how to use the simulator.
This experiment was performed using a fixed-based simulator, which was designed in the Unity 3D and operated on Dell Optiplex 7010 (Intel Quad-Core i73470 3.2GHz, 16GB RAM) workstation running Windows 10. Two widescreen displays showed the visual simulation imagery, rendered at 60 Hz (Figure 1). The simulator was able to provide two driving modes: partial driving automation and manual driving. Based on the features outlined for level 2 automation in SAE J2016, the automated mode supported simultaneous longitudinal and lateral control. Participants were able to engage and disengage automaton by pressing the same button located on the right side of the steering wheel. Disengagement was also possible through pressing the brake (> 10% of braking length) or turning the steering wheel more than (> 7 degrees). A Tobii Eye Tracking Glasses 2 (Figure 2) also recorded participants’ glance behavior, and the Tobii Lab software was used to analyze the data. To make eyemovement data easier to interpret, we specified four area-of-interests (AOIs) including road scenery, phone display, instrument cluster, and hazard perception areas (Figure 1).
UIs Figure 3 shows the concept of AR-LM to communicate the statue of automation and its level of reliability in three conditions: a) high-reliability, b) low-reliability, and c) failure. Participants in the control group were informed about automation mode in high-reliability conditions with a green color UI on the instrument cluster. In addition to this UI, participants in the AR-LM group were also provided with a holographic AR projecting a green bar on the forward road scenery (Figure 3-a). In low-reliability modes, the visual UI on the instrument cluster remained in green color in both groups; however, participants in the AR-LM group were provided with a holographic AR projecting a yellow bar on the forward road scenery (Figure 3-b). Once the vehicle passed the high curvature section of the road, the holographic yellow bar turned to a holographic green bar indicating the high-reliability mode. In failure modes (missing lane marking and obstacle ahead), an auditory feedback was provided for both groups in the form of sequences of three tonal beeps (each beep at 800 Hz and lasting 0.1s) with a time budget of 10 seconds. The visual UI located on the instrument cluster also turned to red. In the AR-LM group, besides these auditory and visual feedbacks, participants were provided with a holographic AR projecting a red bar on the forward road scenery (Figure 3-c).
The driving test consisted of 15 miles long (10 minutes manual driving and 20 minutes automated driving) on a highway was simulated while participants drove in a partial driving automation mode. As shown in Figure 4, automated driving scenarios included three types of automation modes: high-reliability in straight or low curvature sections of the road, low-reliability in high curvature section of the road, and failure when the lane marking has faded out or an object blocked the forward road. Each participant experienced two low-reliability modes and two failure modes. In the failure modes, participants were responsible to take over the vehicle control in a timely manner. The rest of the automated driving session was in high-reliability mode. While driving in automated mode, participants in both groups were asked to watch a video of the Our Planet series on Netflix, which was displayed on the phone. They were requested to watch this video in a self-paced manner.
As the between-subject factor, visualization of automation status (with and without AR information) was an independent variable; and takeover time, gaze time, hazard detection, automation perception were dependent variables. Automation perception also was measured after the driving tests using an 11-item questionnaire regarding automation trust, automaton acceptability, and ease of use in a scale 1 (I strongly disagree) to 7 (I strongly agree).
The average takeover time of both failure events for the AR-LM group (Mean= 2.1 s, SD=0.8 s) was less than the takeover time for the control group (Mean= 2.9 s, SD=0.95 s, p <0.05). The result of pairwise comparisons for the type of takeover showed no significant difference in takeover time between obstacle-ahead events in the AR-LM group (Mean=2.4 s, SD=0.88 s) and control group (Mean= 2.1 s, SD= 0.83 s, p=0.09). However, takeover time of missing lane marking events in the control group (Mean=3.1 s, SD= 1.01 s) was significantly longer than this measure in the AR-LM group (Mean=1.9 s, SD= 0.71 s, p<0.05).
Gaze time on three AOIs (road scenery, instrument cluster, and phone display) in three modes of automation (high-reliability, low-reliability, and failure) were investigated. In general, compared to the control group, the AR-LM concept resulted in significantly shorter gaze time on the road scenery (control: M= 919.6 s; SD= 32.5 s; AR: M= 869.2 s; SD= 28.1 s, p<0.01), the instrument cluster (control: M= 52.9s; SD= 17 .4 s; AR: M=23 s; SD= 8.5 s, p<0.01), and longer gaze time on the phone display (control: M= 207.5s; SD= 19 s; AR: M= 235.8 s; SD= 15 s, p<0.01).
Moreover, as shown in Figure 5, investigating average gaze time for each automation mode revealed significantly longer gaze time on the road scenery in the low-reliability mode (control: Mean=82.8 s, SD= 20.1 s, AR: Mean=112.3 s, SD=19.5 s, p<0.05) and failure mode (control: Mean=18.4 s, SD=5.9 s, AR-LM: Mean=34.6 s, SD=8.4 s, p<0.05) when participants were provided with AR-LM support. Compared to the control group, participants in the AR-LM group showed longer gaze time on the phone display when the vehicle was running in straight/low curvature roads (highreliability mode). There was no significant difference in gaze times on the instrument cluster in low-reliability mode between two groups. Participants in both groups also did not show different gaze behavior on the phone display in failure modes.
No significant difference was observed in gaze time between two types of failure events (missing lane marking and obstacle ahead) in the AR-LM group; however, participants in the control group looked at the road scenery for longer time in the obstacle ahead event (Mean=9.2 s, SD=2.1 s), compared to the missing lane markings event (Mean=6.3 s, SD=3.1 s, p<0.05). Regarding hazard detection event, the results showed a shorter fixation time in the hazard detection AOI when participants were provided with AR-LM support (control group: M=7.2 s, SD= 1.5 s; AR-LM group: Mean=3.1 s, SD=.93 s, p<0.01).
The results of the automation perception questionnaire (Figure 6) showed a significant difference in automation trust between AR-LM and control groups (p=0.026). Participants also reported slightly higher ease of use for AR intervention, though the difference was only marginally significant (p= 0.054). Automation acceptability in the AR-LM group was not significantly different from participants in the control group. (p=0.9).
Our findings suggest that the holographic AR concept of the lane marking had a significant effect on takeover time. On average, participants in the AR-LM group started to take over the vehicle control 0.8 seconds earlier than those in the control group. It seems AR information regarding automation mode and its level of reliability helped participants to react faster in takeover events.
In addition, compared to the control group, gaze data revealed that participants in the AR-LM group looked at the phone display for a longer time. Considering this finding and trust data, it seems AR information led to higher automation trust, and as a result, participants preferred to spend more time engaged in watching the video. To have a deeper understanding of gaze behavior, we also analyzed the gaze time data separated for each automation mode. Interestingly, the results showed that when the level of reliability decreased (in high curvature section of the road), the AR-LM UI caused participants looked at the road scenery for a longer time. The reason for this behavior might be that the yellow holographic AR concept captured participants’ visual attention and then they interpreted an association between this change and the high curvature section of the road. Similar benefit of AR information was observed in failure events. When the participants were provided with a red holographic AR, they spent longer time looking at the road scenery and less time on the phone display. Although these findings could be considered as positive effects of AR-LM in communicating the level of uncertainty of partial driving automation, the application of AR-LM should be considered with the potential costs of longer engagement in secondary tasks when automation is running in high-reliability mode.
The results also showed a shorter average gaze time on the hazard perception AOI in the AR-LM group. Two participants in this group also did not look at the hazard perception AOI at all. We only considered one hazard perception event which was appeared when the automation was running in high-reliability mode (green holographic AR concept). Two possible reasons may explain this finding. First, the AR-LM UI over-captured drivers’ visual attention to a particular part of the road scenery. In this case, caution must be exercised in the application of AR-based UIs to avoid potential distracting effects of AR information. As the second reason, higher trust achieved in the AR-LM group caused participants to stay for a longer time engaged in secondary tasks. In this case, researchers and designers need to consider the costs associated with out-of-the-loop performance problems. This potential complacency could reduce drivers’ situational awareness and impair driving performance especially in critical transition tasks [6].
The results partially support our assumption regarding the effectiveness of the AR-LM concept on automation perception. Compared to the control group, participants who received AR information reported higher automation trust after the driving test. This finding is supported by a recent experimental study[10] and also a theoretical link between trust of in-vehicle technology and warning system reliability described by [11]. Reliability information provided in the AR-LM concept may help participants to understand the system better and build higher trust in partial diving automation. In addition, more transparent visualization of automation mode and its level of reliability supported in AR information has been associated with a lower likelihood of mode confusion and ultimately low trust [12]. Ease of use was also marginally higher in the AR-LM group. Participants who received AR information were more likely to find the system easier to use. Previous studies evaluated ease of use as a component of acceptance [10] and reported higher ease of use when automation modes were presented using AR concepts. Our results, however, did not show a meaningful difference in acceptance data between the control and AR-LM groups. This finding is confusing because we found higher trust and ease of use in the AR-LM group, and according to the Technology Acceptance Model (TAM) [13], it seems reasonable to expect higher acceptance in this group. One explanation for this result is that although trust and acceptance are interrelated concepts; they do not necessarily follow the same pattern[14]. Several sources of variability such as individual differences in prior experiences, intention to use technology, and perceived attractiveness may contribute to this result. Attractiveness and intent are two other components of TAM, which we did not measure in this study. Considering the importance of acceptance in technology usage, more studies are required to investigate underlying components of technology acceptance and its association with trust.
This paper has reported on a driving simulator study conducted with 15 participants to investigate whether a holographic AR prototype could be used to communicate automation mode and its level of reliability. The results indicate that participants who were provided with AR-LM UIs looked longer at the road scenery in low reliability and failure situations. Moreover, they were better prepared to switch to manual control than participants who did not receive AR information. However, AR-LM led to less roadmonitoring behavior when automation was running in high-reliability mode. Moreover, participants in the ARLM group were more likely to miss hazard detection events. In our future work, we will investigate these 1. 2. 3. 4. 5. 6. 7. 8. limitations with a larger sample in more critical and non-critical situations.