Safe, Productive, and Socially Accepted Text Input in Highly Automated Driving

Abstract

As of today, in-vehicle text-based interfaces are used to enter route information, select contacts in the phone book, or search for pieces of music. They are optimized to require low cognitive load, visual attention, and motor skills. With the advent of automated driving, however, the driverpassenger will require new ways and means to enter long and more sophisticated texts, such as in typical office work. To be able to design interfaces supporting this in a safe but also attractive fashion, we explore requirements for productive text input in highly automated vehicles and illustrate a potential solution - the DAMOW assistant - with a fictional user story. In comparison with static office environments, we identify new issues to be tackled, as well as a need to discuss several socio-technical concerns. Maybe even a shift away from the classical desktop metaphor (i.e., WIMP paradigm) as a whole is required, back to command-based interfaces?

-

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).

MobileHCI, 2018 Barcelona, Spain.

Introduction

Text-entry interfaces for drivers currently mainly target composing informal messages, defining navigational goals and infotainment instructions – all required to occupy relatively low amounts of cognitive, visual, and motor-ressources (cf. “The 15-second rule”, [ 3 ]). Highly automated driving (SAE level 3, [ 6 ]), however, will allow drivers to engage more frequently in Non-Driving Related Tasks (NDRTs), effectively opening up the possibility to execute more challenging text entry tasks, like in office work. Existing research shows that engaging in business-related tasks during driving is desired, especially by commuters [ 13 ]. However, they are currently hardly supported by specialized interfaces besides in first responder vehicles [ 10 ], although statistics show that commercial traffic is prevalent (60% of the new vehicle registrations in Germany in 2016 are company cars, [ 8 ]). We assume that people will engage in productive activities during highly automated driving, whether or not suitable interfaces exist. A lack thereof presents a safety risk due to extended off-road glances, etc., similar to illegal smartphone usage in the car. This is underpinned by study results (n=1600) from insurance company Allianz, reporting that 46% of German smartphone owners admitted using their devices manually (i.e., without specialized in-car interfaces) during conventional driving (L1-2), with 24% reading and 15% writing text messages [ 9 ], implying a substantial impact on traffic accidents. In highly automated driving (L3), the driver-passenger will still need to take-over control of the vehicle upon appropriate notification in case of emergency and/or functional limitations.

Prof. Libelle and her Highly Automated Journey

To illustrate issues but also capabilities of (the lack of) productive textual interfaces for highly automated driving (L3), we narrate the situation scenario of Mrs. L. Libelle, professor for Human-Computer Interaction in a hurry, in two ways: once with the “laptop on the lap” (Option A), and once with the “mobile office package” (Option B).

Situation scenario: 7 a.m. in a suburb of Munich. After a long night in which Professor Libelle finished her research paper just in time before the deadline, she wakes up realizing that she forgot to prepare the HCI course’s exam for today. Fifteen minutes later she starts her car and sets the L3 Automated Driving System to drive her to the university “quickest possible” . However, soon after, her journey is quickly halted by a traffic jam due to construction sites ahead.

Option A: “Laptop on the Lap”

Mrs. Libelle realizes that she cannot hold the exam without starting to prepare it now. She therefore grabs her laptop and starts writing. Frustrating fiddling around with the touchpad makes her completely forget to keep an eye on the road, while her hands start to cramp due to the uncomfortable typing posture. Suddenly, her car starts beeping with a “Please Take-Over!” warning-sign blinking in the dashboard. Startled by the alarm, she tries to grab the steering wheel, which is blocked by the notebook on her lap. She throws the laptop onto the passenger seat and rashly begins to steer without actually assessing the situation. The result is a barely avoided crash with construction workers and a broken laptop screen.

Option B: “Mobile Office Package”

She realizes that she bought the ”mobile office” add-on for her car, just for cases like this one. “Hey car, open up a new text document” she says and her car’s digital assistant overlays the windshield with a new document. She fills the document using the speech-to-text functionality, adds the half-complete Fitt’s law formula using a reduced form-factor keyboard integrated into the steering wheel and selects text for formatting using a rotary knob co-located with the keyboard, when after a while the digital assistant interrupts her: "Construction site ahead, please take-over driving control!". Luckily, with the road being in her peripheral vision (due to the windshield display), it is easy for her to assess the situation and, as her hands were already on the steering wheel, also to quickly maneuver around the construction site. Minutes later she arrives at the university – just in time for the exam and relieved that she still made it.

Issues and Vision

The hypothetical example of Professor Libelle highlights the most critical added requirement to text input interfaces in driving: safety. In L3 automated driving driver-passengers still need to occasionally perform a driving task. They need to respond to Take-Over Requests (TORs) in a safe manner, which makes it essential for (non-driving related) user interfaces to support involved cognitive, visual and motor processes, and thereby counteracting the NDRT’s distracting nature. On the other hand, they further need to be attractive in order to justify additional costs (cf. the “mobile office package” ), lower the entry barrier and actually get used. We hypothesize that staying productive while being mobile is a major attractiveness factor and will be even more impacting in the near future’s socio-economical context. Effectively integrating textual interfaces in not-yet fully automated vehicles adds several points of consideration differing from the typical static office workplace, such as: Ergonomics [ 1 ] that ensure comfortable and efficient typing but also take-over motor readiness [ 16 ], cognitive workload calibration [ 19 ], motion sickness [ 2 ], and (visual) attention to the driving situation [ 11 ].

No more WIMP, no more Desktop

To account for the issues, we envision a truly multimodal interface, which we roughly illustrated in the user story’s Option B. It opposes the typical desktop metaphor with the Windows-Icons-Menus-Pointers (WIMP) paradigm and takes “a (conceptual) step back” to command-based interfaces. Our Digital Assistant for Mobile Office Work (DAMOW) includes a smart, context-aware, voice-interaction based command interface for mode changes (initiating workflows, changing text-editing modes, . . . ), utilizing the current trend to “smart voice assistants” (cf. Google Home and Amazon Alexa) and their safety benefits due to nonexisting visual attention requirements. Secondly, visual feedback with the currently written text, current interaction modus and reality augmentations, are given as semitransparent overlay on the windshield. Windshield displays were proven to mitigate take-over performance drawbacks caused by NDRTs [ 15 ] (even though possibly causing textlegibility issues), and also to increase system trust [ 18 ]. Thirdly, a haptic interface (e.g., reduced form keyboard or haptic touch display combined with a rotary knob) is used to provide an intuitive opportunity for high precision characterto-character tasks, such as text formatting or entering formulas. Lastly, important notifications, such as TORs, are always given at least bimodally (visual and auditory) and will interrupt ongoing NDRTs to reduce stress and improve safety (cf. [ 14, 17 ]). However, while conceptualizing the idea, several potential problems emerged. Increasingly complex systems could potentially increase perception time or cognitive load. Further, socio-technical issues like privacy arise which we want to discuss in the workshop.

Workshop Discussion

Considering professor Libelle’s user story, several scenarios highlighting socio-technical issues emerged: • Acceptance: What if she had a passenger who is getting annoyed by constant voice-commands, unrelated to him/her or she herself generally doesn’t like talking to a digital assistant? • Privacy: What if one of professor Libelle’s students was also in the traffic jam, able to read parts of the upcoming exam on his/her teacher’s windshield display? • Learnability: Can we just invent new multimodal text input interfaces and hope that people will be willing to learn how to use them, or should we focus on existing techniques? • Social intra-/inter-/extra-vehicle collaboration: What are essential collaboration scenarios that need to be implemented in an office-oriented automotive text input interface and how ? • User groups: How to make sure that non tech-savvy people accept and use the technology? • Changing paradigms: Do we need to reconsider the desktop metaphor? Is WIMP still suitable for a highly dynamic mobility-context? What are alternatives? What are the advantages / drawbacks of different interaction paradigms? We realize that this (incomplete) list matches, at least partly, well-established design principles and heuristics, such as defined by Grice [ 4 ], Nielsen [ 12 ] or the ISO standard 9241-110 [ 7 ], reconfirming the need to repeatedly discuss them especially for novel interfaces. Although our research considering text input in (highly) automated vehicles is in early stages, we believe to be able to contribute to a diverse range of specialized application areas (and thus opinions) for text input, but also profit from the gathered experienced researchers at the workshop. Besides that, the authors hope to be able to introduce themselves in the research community and open up channels for future collaborations. As a follow-up to the workshop, we plan to conduct a Contextual Inquiry study [ 5 ] with professionals who might benefit from in-vehicle interfaces for office work, and combine its analysis with the workshop’s experts’ discussion findings in order to build a solid foundation for future research.

Conclusion

To realize productive and safe text input interfaces for highly automated driving will be a challenging task considering the added requirements compared to conventional office environments. We conceptualize the DAMOW, a smart and context-aware multimodal Digital Assistant for Mobile Office Work and identify several socio-technical issues we hope to discuss and explore in the workshop.

Authors’ Biographies

Dipl.-Ing. Clemens Schartmüller is a PhD student and research assistant in the Human Computer Interaction Group at Technische Hochschule Ingolstadt (THI, Germany). His early scientific career consists of prototype-driven automotive user interface research with a special interest in exploring opportunities and tackling challenges for using automated vehicles as office workplace, emphasizing objective evaluation.

Prof. Dr. Andreas Riener is a professor for Human-Machine Interaction and Virtual Reality at THI and leading the humancomputer interaction group. His research interests include driving ergonomics, driver state estimation from physiological measures, human factors in driver-vehicle interfaces, and trust/acceptance/ethics in automated driving.

Acknowledgements

This work is supported under the "Innovative Hochschule" program of the German Federal Ministry of Education and Research (BMBF) under Grant No. 03IHS109A (MenschINBewegung).

References

1. Heiner Bubb , Klaus Bengler, Rainer E. Grünen , and Mark Vollrath . 2015 . Automobilergonomie. Springer International Publishing. 663 -684 pages. DOI: http://dx.doi.org/10.1007/978-3- 642 -13505-7

2. Cyriel Diels and Jelte E. Bos . 2016 . Self-driving carsickness . Applied Ergonomics 53 , October (mar 2016 ), 374 - 382 . DOI: http://dx.doi.org/10.1016/J.APERGO. 2015 . 09 .009

3. Paul Green . 1999 . The 15-Second Rule for Driver Information Systems The 15-Second Rule for Driver Information Systems . Human Factors Cd ( 1999 ), 1 - 9 .

4. H. Paul Grice . 1975 . Logic and Conversation . Syntax and Semantics 3 ( 1975 ), 45 - 47 .

5. Karen Holtzblatt , Jessamyn Burns Wendell, and Shelley Wood . 2004 . Rapid contextual design: a how-to guide to key techniques for user-centered design . Elsevier .

6. International SAE and On-Road Automated Driving (ORAD) Committee . 2014 . Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems (J3016 Ground Vehicle Standard) . SAE International 2014 - 01 ( 2014 ). https://saemobilus.sae.org/content/J3016

7. ISO. 2008 . ISO 9241-110 Ergonomics of human-systems interaction - Part 110: Dialogue principles . September ( 2008 ).

8. Kraftfahrt Bundesamt . 2017 . Jahresbilanz der Neuzulassungen 2016 . ( 2017 ).

9. Jörg Kubitzki and Wolfgang Fastenmeier . 2016 . Ablenkung durch moderne Informations-und Kommunikations- techniken und soziale Interaktion bei Autofahrern . Technical Report. Allianz AG.

https://www.allianzdeutschland.de/ die-allianz-ablenkungsstudie-2016/id

10. Andrew L Kun , Jerry Wachtel , W Thomas Miller , Patrick Son , and Martin Lavallière . 2015 . User Interfaces for First Responder Vehicles: Views from Practitioners, Industry, and Academia . In Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '15) . ACM, New York, NY, USA, 163 - 170 . DOI: http://dx.doi.org/10.1145/2799250.2799289

11. Robert E. Llaneras , Jeremy Salinger, and Charles A. Green . 2013 . Human factors issues associated with limited ability autonomous driving systems: Drivers' allocation of visual attention to the forward roadway . Proceedings of the Seventh International Driving Symposium on Human Factors in Driver Assessment , Training, and Vehicle Design (oct 2013 ), 92 - 98 . DOI: http: //dx.doi.org/10.17077/drivingassessment.1472

12. Jakob Nielsen and Rolf Molich . 1990 . Heuristic evaluation of user interfaces . In Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 . ACM Press, New York, New York, USA, 249 - 256 . DOI: http://dx.doi.org/10.1145/97243.97281

13. Nicole Perterer , Christiane Moser, Alexander Meschtscherjakov, Alina Krischkowsky, and Manfred Tscheligi . 2016 . Activities and Technology Usage while Driving . In Proceedings of the 9th Nordic Conference on Human-Computer Interaction - NordiCHI ' 16 . ACM Press, New York, New York, USA, 1 - 10 . DOI: http://dx.doi.org/10.1145/2971485.2971556

14. Sebastiaan Petermeijer , Pavlo Bazilinskyy, Klaus Bengler, and Joost De Winter. 2017 . Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop . Applied Ergonomics 62 , July 2017 ( 2017 ), 204 - 215 . DOI: http://dx.doi.org/10.1016/j.apergo. 2017 . 02 .023

15. Clemens Schartmüller , Andreas Riener, Philipp Wintersberger, and Anna-Katharina Frison . 2018 . Workaholistic : On Balancing Typing- and Handover-Performance in Automated Driving. In 20th International Conference on Human-Computer Interaction with Mobile Devices and Services , MobileHCI 2018 (In Press). ACM, Barcelona, Spain.

16. Philipp Wintersberger , Paul Green, and Andreas Riener . 2016 . Am I Driving or Are You are Or We Both? A Taxonomy For Handover and Handback in Automated Driving. Power ( 2016 ), 1 - 7 .

17. Philipp Wintersberger , Andreas Riener, Clemens Schartmüller, Anna-Katharina Frison , and Klemens Weigl . 2018 . Let Me Finish before I Take Over: Context and Attention Aware Device Integration in Highly Automated Vehicles . In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Accepted) . ACM, Toronto, Canada.

18. Philipp Wintersberger , Tamara von Sawitzky, and Andreas Riener . 2017 . Traffic Augmentation as Means to Increase Trust in Automated Driving Systems . In Proceedings of CHItaly'17 . ACM Press, Cagliari, 7 . DOI:http://dx.doi.org/10.1145/3125571.3125600

19. Mark S. Young and Neville A. Stanton . 2002 . Attention and automation: New perspectives on mental underload and performance . Theoretical Issues in Ergonomics Science 3 , 2 ( 2002 ), 178 - 194 . DOI: http://dx.doi.org/10.1080/14639220210123789