Barcelona, Spain | September 3, 2018 MobileHCI 2018 Workshop on Socio-Technical Aspects of Text Entry Safe, Productive, and Socially Accepted Text Input in Highly Automated Driving Clemens Schartmüller Abstract Technische Hochschule As of today, in-vehicle text-based interfaces are used to en- Ingolstadt (THI), Germany ter route information, select contacts in the phone book, Johannes Kepler University, Linz or search for pieces of music. They are optimized to re- clemens.schartmueller@thi.de quire low cognitive load, visual attention, and motor skills. With the advent of automated driving, however, the driver- passenger will require new ways and means to enter long and more sophisticated texts, such as in typical office work. To be able to design interfaces supporting this in a safe but Andreas Riener also attractive fashion, we explore requirements for produc- Technische Hochschule tive text input in highly automated vehicles and illustrate a Ingolstadt (THI), Germany potential solution – the DAMOW assistant – with a fictional Johannes Kepler University, Linz user story. In comparison with static office environments, andreas.riener@thi.de we identify new issues to be tackled, as well as a need to discuss several socio-technical concerns. Maybe even a shift away from the classical desktop metaphor (i.e., WIMP paradigm) as a whole is required, back to command-based interfaces? Author Keywords Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Automated Driving; Text-based Input; Office Work; Desktop; for profit or commercial advantage and that copies bear this notice and the full citation Take-Overs; WIMP. on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Copyright held by the owner/author(s). CCS Concepts MobileHCI, 2018 Barcelona, Spain. •Human-centered computing → Ubiquitous and mobile computing systems and tools; Text input. 18 Barcelona, Spain | September 3, 2018 MobileHCI 2018 Workshop on Socio-Technical Aspects of Text Entry Introduction once with the “laptop on the lap” (Option A), and once with Text-entry interfaces for drivers currently mainly target com- the “mobile office package” (Option B). posing informal messages, defining navigational goals and infotainment instructions – all required to occupy relatively Situation scenario: 7 a.m. in a suburb of Munich. After a low amounts of cognitive, visual, and motor-ressources (cf. long night in which Professor Libelle finished her research “The 15-second rule”, [3]). Highly automated driving (SAE paper just in time before the deadline, she wakes up real- level 3, [6]), however, will allow drivers to engage more fre- izing that she forgot to prepare the HCI course’s exam for quently in Non-Driving Related Tasks (NDRTs), effectively today. Fifteen minutes later she starts her car and sets the opening up the possibility to execute more challenging text L3 Automated Driving System to drive her to the univer- entry tasks, like in office work. Existing research shows sity “quickest possible” . However, soon after, her journey that engaging in business-related tasks during driving is is quickly halted by a traffic jam due to construction sites desired, especially by commuters [13]. However, they are ahead. currently hardly supported by specialized interfaces besides Option A: “Laptop on the Lap” in first responder vehicles [10], although statistics show Mrs. Libelle realizes that she cannot hold the exam with- that commercial traffic is prevalent (60% of the new vehicle out starting to prepare it now. She therefore grabs her lap- registrations in Germany in 2016 are company cars, [8]). top and starts writing. Frustrating fiddling around with the We assume that people will engage in productive activities touchpad makes her completely forget to keep an eye on during highly automated driving, whether or not suitable the road, while her hands start to cramp due to the uncom- interfaces exist. A lack thereof presents a safety risk due fortable typing posture. Suddenly, her car starts beeping to extended off-road glances, etc., similar to illegal smart- with a “Please Take-Over!” warning-sign blinking in the phone usage in the car. This is underpinned by study re- dashboard. Startled by the alarm, she tries to grab the sults (n=1600) from insurance company Allianz, reporting steering wheel, which is blocked by the notebook on her that 46% of German smartphone owners admitted using lap. She throws the laptop onto the passenger seat and their devices manually (i.e., without specialized in-car inter- rashly begins to steer without actually assessing the situa- faces) during conventional driving (L1-2), with 24% reading tion. The result is a barely avoided crash with construction and 15% writing text messages [9], implying a substantial workers and a broken laptop screen. impact on traffic accidents. In highly automated driving (L3), the driver-passenger will still need to take-over control of Option B: “Mobile Office Package” the vehicle upon appropriate notification in case of emer- She realizes that she bought the ”mobile office” add-on for gency and/or functional limitations. her car, just for cases like this one. “Hey car, open up a new text document” she says and her car’s digital assistant Prof. Libelle and her Highly Automated Journey overlays the windshield with a new document. She fills the To illustrate issues but also capabilities of (the lack of) pro- document using the speech-to-text functionality, adds the ductive textual interfaces for highly automated driving (L3), half-complete Fitt’s law formula using a reduced form-factor we narrate the situation scenario of Mrs. L. Libelle, profes- keyboard integrated into the steering wheel and selects text sor for Human-Computer Interaction in a hurry, in two ways: for formatting using a rotary knob co-located with the key- 19 Barcelona, Spain | September 3, 2018 MobileHCI 2018 Workshop on Socio-Technical Aspects of Text Entry board, when after a while the digital assistant interrupts her: the Windows-Icons-Menus-Pointers (WIMP) paradigm "Construction site ahead, please take-over driving control!". and takes “a (conceptual) step back” to command-based Luckily, with the road being in her peripheral vision (due to interfaces. Our Digital Assistant for Mobile Office Work the windshield display), it is easy for her to assess the situa- (DAMOW) includes a smart, context-aware, voice-interaction tion and, as her hands were already on the steering wheel, based command interface for mode changes (initiating also to quickly maneuver around the construction site. Min- workflows, changing text-editing modes, . . . ), utilizing the utes later she arrives at the university – just in time for the current trend to “smart voice assistants” (cf. Google Home exam and relieved that she still made it. and Amazon Alexa) and their safety benefits due to non- existing visual attention requirements. Secondly, visual Issues and Vision feedback with the currently written text, current interac- The hypothetical example of Professor Libelle highlights the tion modus and reality augmentations, are given as semi- most critical added requirement to text input interfaces in transparent overlay on the windshield. Windshield displays driving: safety. In L3 automated driving driver-passengers were proven to mitigate take-over performance drawbacks still need to occasionally perform a driving task. They need caused by NDRTs [15] (even though possibly causing text- to respond to Take-Over Requests (TORs) in a safe man- legibility issues), and also to increase system trust [18]. ner, which makes it essential for (non-driving related) user Thirdly, a haptic interface (e.g., reduced form keyboard or interfaces to support involved cognitive, visual and motor haptic touch display combined with a rotary knob) is used to processes, and thereby counteracting the NDRT’s distract- provide an intuitive opportunity for high precision character- ing nature. On the other hand, they further need to be at- to-character tasks, such as text formatting or entering for- tractive in order to justify additional costs (cf. the “mobile mulas. Lastly, important notifications, such as TORs, are office package” ), lower the entry barrier and actually get always given at least bimodally (visual and auditory) and used. We hypothesize that staying productive while being will interrupt ongoing NDRTs to reduce stress and improve mobile is a major attractiveness factor and will be even safety (cf. [14, 17]). However, while conceptualizing the more impacting in the near future’s socio-economical con- idea, several potential problems emerged. Increasingly text. Effectively integrating textual interfaces in not-yet fully complex systems could potentially increase perception time automated vehicles adds several points of consideration or cognitive load. Further, socio-technical issues like pri- differing from the typical static office workplace, such as: vacy arise which we want to discuss in the workshop. Ergonomics [1] that ensure comfortable and efficient typing but also take-over motor readiness [16], cognitive workload Workshop Discussion calibration [19], motion sickness [2], and (visual) attention to Considering professor Libelle’s user story, several scenarios the driving situation [11]. highlighting socio-technical issues emerged: No more WIMP, no more Desktop To account for the issues, we envision a truly multimodal • Acceptance: What if she had a passenger who is interface, which we roughly illustrated in the user story’s getting annoyed by constant voice-commands, unre- Option B. It opposes the typical desktop metaphor with lated to him/her or she herself generally doesn’t like talking to a digital assistant? 20 Barcelona, Spain | September 3, 2018 MobileHCI 2018 Workshop on Socio-Technical Aspects of Text Entry • Privacy: What if one of professor Libelle’s students analysis with the workshop’s experts’ discussion findings in was also in the traffic jam, able to read parts of the order to build a solid foundation for future research. upcoming exam on his/her teacher’s windshield dis- play? Conclusion • Learnability: Can we just invent new multimodal text To realize productive and safe text input interfaces for highly input interfaces and hope that people will be willing to automated driving will be a challenging task considering learn how to use them, or should we focus on existing the added requirements compared to conventional office techniques? environments. We conceptualize the DAMOW, a smart and context-aware multimodal Digital Assistant for Mobile Office • Social intra-/inter-/extra-vehicle collaboration: Work and identify several socio-technical issues we hope to What are essential collaboration scenarios that need discuss and explore in the workshop. to be implemented in an office-oriented automotive text input interface and how ? Authors’ Biographies • User groups: How to make sure that non tech-savvy Dipl.-Ing. Clemens Schartmüller is a PhD student and re- people accept and use the technology? search assistant in the Human Computer Interaction Group • Changing paradigms: Do we need to reconsider the at Technische Hochschule Ingolstadt (THI, Germany). His desktop metaphor? Is WIMP still suitable for a highly early scientific career consists of prototype-driven automo- dynamic mobility-context? What are alternatives? tive user interface research with a special interest in explor- What are the advantages / drawbacks of different ing opportunities and tackling challenges for using auto- interaction paradigms? mated vehicles as office workplace, emphasizing objective evaluation. We realize that this (incomplete) list matches, at least partly, Prof. Dr. Andreas Riener is a professor for Human-Machine well-established design principles and heuristics, such Interaction and Virtual Reality at THI and leading the human- as defined by Grice [4], Nielsen [12] or the ISO standard computer interaction group. His research interests include 9241-110 [7], reconfirming the need to repeatedly discuss driving ergonomics, driver state estimation from physiolog- them especially for novel interfaces. Although our research ical measures, human factors in driver-vehicle interfaces, considering text input in (highly) automated vehicles is in and trust/acceptance/ethics in automated driving. early stages, we believe to be able to contribute to a diverse range of specialized application areas (and thus opinions) Acknowledgements for text input, but also profit from the gathered experienced This work is supported under the "Innovative Hochschule" researchers at the workshop. Besides that, the authors program of the German Federal Ministry of Education and hope to be able to introduce themselves in the research Research (BMBF) under Grant No. 03IHS109A (MenschIN- community and open up channels for future collaborations. Bewegung). As a follow-up to the workshop, we plan to conduct a Con- textual Inquiry study [5] with professionals who might bene- fit from in-vehicle interfaces for office work, and combine its 21 Barcelona, Spain | September 3, 2018 MobileHCI 2018 Workshop on Socio-Technical Aspects of Text Entry REFERENCES Autofahrern. Technical Report. Allianz AG. 1. Heiner Bubb, Klaus Bengler, Rainer E. Grünen, and https://www.allianzdeutschland.de/ Mark Vollrath. 2015. Automobilergonomie. Springer die-allianz-ablenkungsstudie-2016/id International Publishing. 663–684 pages. DOI: 10. Andrew L Kun, Jerry Wachtel, W Thomas Miller, Patrick http://dx.doi.org/10.1007/978-3-642-13505-7 Son, and Martin Lavallière. 2015. User Interfaces for 2. Cyriel Diels and Jelte E. Bos. 2016. Self-driving First Responder Vehicles: Views from Practitioners, carsickness. Applied Ergonomics 53, October (mar Industry, and Academia. In Proceedings of the 7th 2016), 374–382. DOI: International Conference on Automotive User http://dx.doi.org/10.1016/J.APERGO.2015.09.009 Interfaces and Interactive Vehicular Applications (AutomotiveUI ’15). ACM, New York, NY, USA, 3. Paul Green. 1999. The 15-Second Rule for Driver 163–170. DOI: Information Systems The 15-Second Rule for Driver http://dx.doi.org/10.1145/2799250.2799289 Information Systems. Human Factors Cd (1999), 1–9. 11. Robert E. Llaneras, Jeremy Salinger, and Charles A. 4. H. Paul Grice. 1975. Logic and Conversation. Syntax Green. 2013. Human factors issues associated with and Semantics 3 (1975), 45–47. limited ability autonomous driving systems: Drivers’ 5. Karen Holtzblatt, Jessamyn Burns Wendell, and allocation of visual attention to the forward roadway. Shelley Wood. 2004. Rapid contextual design: a how-to Proceedings of the Seventh International Driving guide to key techniques for user-centered design. Symposium on Human Factors in Driver Assessment, Elsevier. Training, and Vehicle Design (oct 2013), 92–98. DOI: http: 6. International SAE and On-Road Automated //dx.doi.org/10.17077/drivingassessment.1472 Driving (ORAD) Committee. 2014. Taxonomy and Definitions for Terms Related to On-Road Motor 12. Jakob Nielsen and Rolf Molich. 1990. Heuristic Vehicle Automated Driving Systems (J3016 Ground evaluation of user interfaces. In Proceedings of the Vehicle Standard). SAE International 2014-01 (2014). SIGCHI conference on Human factors in computing https://saemobilus.sae.org/content/J3016 systems Empowering people - CHI ’90. ACM Press, New York, New York, USA, 249–256. DOI: 7. ISO. 2008. ISO 9241-110 Ergonomics of http://dx.doi.org/10.1145/97243.97281 human-systems interaction - Part 110: Dialogue principles. September (2008). 13. Nicole Perterer, Christiane Moser, Alexander Meschtscherjakov, Alina Krischkowsky, and Manfred 8. Kraftfahrt Bundesamt. 2017. Jahresbilanz der Tscheligi. 2016. Activities and Technology Usage while Neuzulassungen 2016. (2017). Driving. In Proceedings of the 9th Nordic Conference 9. Jörg Kubitzki and Wolfgang Fastenmeier. 2016. on Human-Computer Interaction - NordiCHI ’16. ACM Ablenkung durch moderne Informations-und Press, New York, New York, USA, 1–10. DOI: Kommunikations- techniken und soziale Interaktion bei http://dx.doi.org/10.1145/2971485.2971556 22 Barcelona, Spain | September 3, 2018 MobileHCI 2018 Workshop on Socio-Technical Aspects of Text Entry 14. Sebastiaan Petermeijer, Pavlo Bazilinskyy, Klaus 17. Philipp Wintersberger, Andreas Riener, Clemens Bengler, and Joost De Winter. 2017. Take-over again: Schartmüller, Anna-Katharina Frison, and Klemens Investigating multimodal and directional TORs to get Weigl. 2018. Let Me Finish before I Take Over: Context the driver back into the loop. Applied Ergonomics 62, and Attention Aware Device Integration in Highly July 2017 (2017), 204–215. DOI: Automated Vehicles. In Proceedings of the 10th http://dx.doi.org/10.1016/j.apergo.2017.02.023 International Conference on Automotive User Interfaces and Interactive Vehicular Applications 15. Clemens Schartmüller, Andreas Riener, Philipp (Accepted). ACM, Toronto, Canada. Wintersberger, and Anna-Katharina Frison. 2018. Workaholistic : On Balancing Typing- and 18. Philipp Wintersberger, Tamara von Sawitzky, and Handover-Performance in Automated Driving. In 20th Andreas Riener. 2017. Traffic Augmentation as Means International Conference on Human-Computer to Increase Trust in Automated Driving Systems. In Interaction with Mobile Devices and Services, Proceedings of CHItaly’17. ACM Press, Cagliari, 7. MobileHCI 2018 (In Press). ACM, Barcelona, Spain. DOI:http://dx.doi.org/10.1145/3125571.3125600 16. Philipp Wintersberger, Paul Green, and Andreas 19. Mark S. Young and Neville A. Stanton. 2002. Attention Riener. 2016. Am I Driving or Are You are Or We Both? and automation: New perspectives on mental A Taxonomy For Handover and Handback in underload and performance. Theoretical Issues in Automated Driving. Power (2016), 1–7. Ergonomics Science 3, 2 (2002), 178–194. DOI: http://dx.doi.org/10.1080/14639220210123789 23