In this position paper, we describe current research questions in the area of interaction with automated vehicles from the viewpoint of users within the vehicle (i.e., passengers) and from the viewpoint of parties outside the vehicle (e.g., other manual drivers or pedestrians). First, we briefly introduce the topics of Cooperation with Automated Vehicles, External Communication of Automated Vehicles, and Explainability of Automated Vehicles before posing, in total, nine research questions guiding these three areas.
Automated vehicles (AVs) are expected to change journeys and, in general, trafic
fundamentally [
Regarding the passengers, current work can be (among other areas) broadly distinguished
into take-overs [
Regarding bystanders, especially the (potential) need to replace current driver-road user
communication via external Human-Machine Interfaces (eHMI) is investigated [
In this position paper and based on our work in these areas, we describe what we see as current challenges of AVs as highly automated systems that permanently interact with other road users and their primary intended users (i.e., the passengers).
Besides the classical take-over process [
While we presented first supportive work on this [ 8, 6, 34, 35] (see Figure 1), this area of research is still under-explored.
Recently, eHMIs have become a popular topic in the automotive domain [27]. External
communication of AVs is researched to enable communication between AVs and other road users.
This includes other manual drivers [36, 37] and vulnerable road users such as bicyclists or
pedestrians. Numerous aspects have been investigated, including anthropomorphism [38],
challenges of overtrust [39], various target groups such as children [40] or people with vision
impairments [
This field is especially interesting as people that are not instructed nor did they actively consent to using AVs in any way are involved in interacting with the AV. Therefore, this topic is one of the first to fully incorporate true novice users of automation.
(a) Derived pedestrian intentions; taken from [10]. (b) Displayed semantic segmentation result; taken from [11].
Schoettle and Sivak [43] found that 75% of respondents were at least slightly concerned about a possible system failure in unexpected situations. Additionally, the reliability of AVs is a worry of users [44]. Therefore, numerous works have investigated potential visualization concepts to communicate with the user of an AV [45, 12, 14].
One primary rationale regarding the explainability of AVs is to enhance and calibrate trust. Hof and Bashier defined trust as “a variable that often determines the willingness of human operators to rely on automation” [46, p. 407]. They proposed a three-layered trust model, including dispositional, situational, and learned trust. Dispositional trust refers to the trustor’s personal background (e.g., culture, age, and personality traits). Situational trust is categorized into internal and external variability. External variability refers to alterations occurring with changed automation complexity. Internal variability describes the trustor’s mental capacity and psychological state. Learned trust was modeled in two layers: initially learned trust, that is trust based on known information about the automation) and dynamically learned trust (which is altered via interaction with the automation).
Hereby, the approaches target either initially learned trust (see Körber et al. [47]) or dynamically learned trust (e.g., [10, 11, 14, 12]; see Figure 2). Nonetheless, some significant questions remain unanswered: 8. What are the long-term efects of using AVs, and how will interaction change? 9. How can include and aid people with disabilities, for example, people with vision impairments?
With our approaches, we especially target to enhance calibrated trust by including uncertainty information into the communication (e.g., see [11]).
In this work, we briefly outline the two major research areas on automotive automation interaction: with drivers and users or with other road users such as pedestrians. For the three areas Cooperation with Automated Vehicles, External Communication of Automated Vehicles, and Explainability of Automated Vehicles, we briefly describe the previous work and define, in our view, currently relevant research questions.
This work was conducted within the projects ’Interaction between automated vehicles and vulnerable road users’ (Intuitiver) funded by the Ministry of Science, Research and Arts of the State of Baden-Württemberg, as well as ’Semulin’ and ’SituWare’ funded by the Federal Ministry for Economic Afairs and Energy (BMWi). Mark Colley was also supported by the Startup Funding B of Ulm University. Conference on Mobile Human-Computer Interaction, Association for Computing Machinery, New York, NY, USA, 2021, pp. 1–15. URL: https://doi.org/10.1145/3447526.3472025. [6] M. Walch, M. Colley, M. Weber, Cooperationcaptcha: On-the-fly object labeling for highly automated vehicles, in: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 1–6. URL: https://doi.org/10.1145/3290607.3313022. doi:10.1145/3290607. 3313022. [7] M. Walch, D. Lehr, M. Colley, M. Weber, Don’t you see them? towards gaze-based interaction adaptation for driver-vehicle cooperation, in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, AutomotiveUI ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 232–237. URL: https://doi.org/10.1145/3349263.3351338. doi:10.1145/3349263.3351338. [8] M. Colley, A. Askari, M. Walch, M. Woide, E. Rukzio, Orias: On-the-fly object identification and action selection for highly automated vehicles, in: 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Association for Computing Machinery, New York, NY, USA, 2021, p. 79–89. URL: https://doi.org/10.1145/ 3409118.3475134. [9] M. Walch, S. Li, I. Mandel, D. Goedicke, N. Friedman, W. Ju, Crosswalk cooperation: A phone-integrated driver-vehicle cooperation approach to predict the crossing intentions of pedestrians in automated driving, in: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Association for Computing Machinery, New York, NY, USA, 2020, pp. 74–77. [10] M. Colley, C. Bräuner, M. Lanzer, W. Marcel, M. Baumann, R. Rukzio, Efect of visualization of pedestrian intention recognition on trust and cognitive load, in: Proceedings of the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’20, ACM, Association for Computing Machinery, New York, NY, USA, 2020, pp. 181–191. doi:10.1145/3409120.3410648. [11] M. Colley, B. Eder, J. O. Rixen, E. Rukzio, Efects of semantic segmentation visualization on trust, situation awareness, and cognitive load in highly automated vehicles, in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, 2021, pp. 1–11. URL: https://doi.org/10.1145/ 3411764.3445351. [12] M. Colley, S. Krauß, M. Lanzer, E. Rukzio, How should automated vehicles communicate critical situations? a comparative analysis of visualization concepts, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5 (2021). URL: https://doi.org/10.1145/3478111. doi:10. 1145/3478111. [13] M. Lanzer, T. Stoll, M. Colley, M. Baumann, Intelligent mobility in the city: the influence of system and context factors on drivers’ takeover willingness and trust in automated vehicles, Frontiers in Human Dynamics (2021) 42. [14] T. Schneider, J. Hois, A. Rosenstein, S. Ghellal, D. Theofanou-Fülbier, A. R. Gerlicher, Explain yourself! transparency for positive ux in autonomous driving, in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, Association for Computing Machinery, New York, NY, USA, 2021, pp. 1–12. URL: https://doi.org/10. autonomous vehicles, in: 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Association for Computing Machinery, New York, NY, USA, 2021, p. 263–273. URL: https://doi.org/10.1145/3409118.3475133. [25] M. Colley, R. Rukzio, Towards a design space for external communication of autonomous vehicles, in: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, ACM, Association for Computing Machinery, New York, NY, USA, 2020, pp. 1–8. URL: http://dx.doi.org/10.1145/3334480.3382844. doi:10.1145/ 3334480.3382844. [26] M. Colley, M. Walch, E. Rukzio, For a better (simulated) world: Considerations for vr in external communication research, in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, AutomotiveUI ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 442–449. URL: https://doi.org/10.1145/3349263.3351523. doi:10.1145/3349263. 3351523. [27] D. Dey, A. Habibovic, A. Löcken, P. Wintersberger, B. Pfleging, A. Riener, M. Martens, J. Terken, Taming the ehmi jungle: A classification taxonomy to guide, compare, and assess the design principles of automated vehicles’ external human-machine interfaces, Transportation Research Interdisciplinary Perspectives 7 (2020) 100174. [28] M. Colley, E. Bajrovic, R. Rukzio, Efects of pedestrian behavior, time pressure, and repeated exposure on crossing decisions in front of automated vehicles equipped with external communication, in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, Association for Computing Machinery, New York, NY, USA, 2022. URL: https://doi.org/10.1145/3491102.3517571. doi:10.1145/3491102.3517571. [29] M. Lanzer, F. Babel, F. Yan, B. Zhang, F. You, J. Wang, M. Baumann, Designing communication strategies of autonomous vehicles with pedestrians: An intercultural study, in: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 122–131. URL: https://doi.org/10.1145/3409120.3410653. doi:10.1145/3409120.3410653. [30] M. Colley, M. Lanzer, J. H. Belz, M. Walch, E. Rukzio, Evaluating the impact of decals on driver stereotype perception and exploration of personalization of automated vehicles via digital decals, in: 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Association for Computing Machinery, New York, NY, USA, 2021, p. 296–306. URL: https://doi.org/10.1145/3409118.3475132. [31] M. Colley, S. Li, E. Rukzio, Increasing pedestrian safety using external communication of autonomous vehicles for signalling hazards, in: Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction, Association for Computing Machinery, New York, NY, USA, 2021, pp. 1–10. URL: https://doi.org/10.1145/3447526.3472024. [32] M. Colley, S. Mytilineos, M. Walch, J. Gugenheimer, E. Rukzio, Requirements for the interaction with highly automated construction site delivery trucks, Frontiers in Human Dynamics 4 (2022). URL: https://www.frontiersin.org/article/10.3389/fhumd.2022.794890. doi:10.3389/fhumd.2022.794890. [33] M. Walch, K. Lange, M. Baumann, M. Weber, Autonomous driving: Investigating the feasibility of car-driver handover assistance, in: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 11–18.
URL: https://doi.org/10.1145/2799250.2799268. doi:10.1145/2799250.2799268. [34] M. Woide, D. Stiegemeier, S. Pfattheicher, M. Baumann, Measuring driver-vehicle cooperation: development and validation of the human-machine-interaction-interdependence questionnaire (hmii), Transportation research part F: trafic psychology and behaviour 83 (2021) 424–439. [35] M. Walch, M. Colley, M. Weber, Driving-task-related human-machine interaction in automated driving: Towards a bigger picture, in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, AutomotiveUI ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 427–433. URL: https://doi.org/10.1145/3349263.3351527. doi:10.1145/3349263. 3351527. [36] M. Rettenmaier, M. Pietsch, J. Schmidtler, K. Bengler, Passing through the bottleneckthe potential of external human-machine interfaces, in: 2019 IEEE Intelligent Vehicles Symposium (IV), IEEE, IEEE, New York, NY, USA, 2019, pp. 1687–1692. [37] M. Rettenmaier, J. Schulze, K. Bengler, How much space is required? efect of distance, content, and color on external human–machine interface size, Information 11 (2020) 346. [38] C.-M. Chang, K. Toda, D. Sakamoto, T. Igarashi, Eyes on a car: An interface design for communication between an autonomous car and a pedestrian, in: Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 65–73. URL: https://doi.org/10.1145/3122986.3122989. doi:10.1145/3122986.3122989. [39] K. Holländer, P. Wintersberger, A. Butz, Overtrust in external cues of automated vehicles: An experimental investigation, in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 211–221. URL: https://doi.org/10.1145/3342197.3344528. doi:10.1145/3342197.3344528. [40] V. Charisi, A. Habibovic, J. Andersson, J. Li, V. Evers, Children’s views on identification and intention communication of self-driving vehicles, in: Proceedings of the 2017 Conference on Interaction Design and Children, IDC ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 399–404. URL: https://doi.org/10.1145/3078072.3084300. doi:10. 1145/3078072.3084300. [41] M. Colley, M. Walch, E. Rukzio, Towards reducing energy waste through usage of external communication of autonomous vehicles, 2020. [42] D. Moore, R. Currano, G. E. Strack, D. Sirkin, The case for implicit external human-machine interfaces for autonomous vehicles, in: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 295–307. URL: https://doi.org/10.1145/3342197.3345320. doi:10.1145/3342197.3345320. [43] B. Schoettle, M. Sivak, A survey of public opinion about autonomous and self-driving vehicles in the US, the UK, and Australia, Technical Report, University of Michigan, Ann Arbor, Transportation Research Institute, 2014. [44] M. Kyriakidis, R. Happee, J. C. de Winter, Public opinion on automated driving: Results of an international questionnaire among 5000 respondents, Transportation research part F: trafic psychology and behaviour 32 (2015) 127–140. [45] S. R. Winter, S. Rice, N. K. Ragbir, B. S. Baugh, M. N. Milner, B.-L. Lim, J. Capps, E. Anania, Assessing pedestrians’ perceptions and willingness to interact with autonomous vehicles, Technical Report, US Department of Transportation. Center for Advanced Transportation Mobility . . . , 2019. [46] K. A. Hof, M. Bashir, Trust in automation: Integrating empirical evidence on factors that influence trust, Human factors 57 (2015) 407–434. [47] M. Körber, E. Baseler, K. Bengler, Introduction matters: Manipulating trust in automation and reliance in automated driving, Applied ergonomics 66 (2018) 18–31.