Autonomous vehicles (AVs) are rapidly evolving as a novel and disruptive way of transportation. However, both in industry and academia, it is believed that AVs will not be able resolve every traffic situation autonomously and therefore, remote human intervention will be required. However, existing teleoperation methods are extremely challenging and thus it is evident that novel remote operation paradigms should evolve. Such a paradigm is teleassistance, which posits that remote operators (ROs) should provide high-level guidance to AVs and delegate low-level controls to automation. Our work explores how to design such a teleassistance interface. Through interviews with 14 experts in AV teleoperation, we first discover in which road scenarios AVs will need remote human assistance. Then, based on these scenarios, we devise a set of discrete high-level commands through which a remote operator will be able to resolve most road scenarios without the need to manually control the AV. Finally, we create a prototype for such an interface.
Recent technological advancements, especially in the field of artificial intelligence and machine learning, enable rapid evolvement of autonomous vehicles (AVs) as a novel way of transportation [1]. An autonomous vehicle (AV) is envisioned to be able to drive itself on its own, without any human input, using various sensors to perceive the environment identifying paths and obstacles. However, similarly to other autonomous systems, it is most likely that AVs will also require human monitoring and 2 Disengagement - a situation when the vehicle returns to manual control or the driver feels the need to take back the wheel from the AV decision system. intervention. Situations such as road construction, a malfunctioning traffic light, or a busy junction might prevent an AV from moving autonomously [2] and can cause disengagements2 [3]. Therefore, it is widely believed today both in the industry and in academia that, at least in the near and foreseeable future, AVs will not be able to resolve all ambiguous traffic situations by their own and remote human assistance will be required [4][5][6][7][8].
A promising approach to resolve these situations and provide an actionable solution for AV disengagements is Teleoperation - operation of a machine from distance. While teleoperation systems for AVs are already in use and are being developed by various automotive companies [7] [9], manually driving a vehicle remotely is an extremely challenging task [10]. For example, since the remote operator (RO) is physically disconnected from the operated AV, she cannot feel the forces that are applied on the teleoperated vehicle or hear its surroundings sounds. Another example is latency, which is caused by the fact that a lot of information should be transmitted from the AV to the RO over the network [11][12].
Currently, there are two major teleoperation paradigms: tele-driving and tele-assistance (Figure 1). In tele-driving the remote operator continuously operates the AV using a steering wheel and pedals, while in tele-assistance the cooperation between the human (RO) and the machine (AV) happens on the guidance level [13].
There are many advantages of using teleassistance. First, remote assistance has the potential to significantly shorten a teleoperation session time because a simple command such as “wait” or “progress slowly” would be much shorter to issue than manually driving a car. Second, tele-assistance might improve safety: according to the U.S. department of transportation, 94 percent of crashes in the U.S. were caused by a human error [14]. Therefore, delegating the low-level maneuvers to the AV might significantly improve AV’s safety. Third, guiding AVs using generic commands (instead of using steering and pedals) may allow ROs to control heterogeneous vehicles (with different sizes, widths, etc.) and fleets (private cars, shuttles, trucks, etc.) without the need to develop new mental models when transitioning between one teleoperated vehicle to another [10]. Finally, properly designed tele-assistance user interfaces (UIs) has the potential to reduce RO’s cognitive load over tele-driving interfaces, which are shown to require a very high level of attention [10].
Bogdoll et. al [4] performed a comprehensive analysis of recent teleoperation methods and created a taxonomy for remote human input systems of AVs. In their review, they highlight remote high-level assistance as a viable solution and one that is already being developed by several companies. In the academia, several works examined path generation as a high-level input method in which the operator “draws” the desired 2D path for the remote vehicle to follow [15][16][17]. Others, started to explore high-level interface commands that can be delegated to the AV [18][19]. However, no research work systematically examined how such a high-level command language should look like, in what cases should it be used, and what should its components consist of. Our work aims to fill this gap and build upon it by designing, implementing, and evaluating a tele-assistance UI.
To create such an interface, we conducted a qualitative study with 14 experts in AV teleoperation with the aim to unveil and categorize the various disengagement scenarios. Following the study, and using the insights gained from it, we designed and implemented an initial prototype version of a tele-assistance UI.
Since teleoperation of AVs is an emerging area of research, currently there is no uniform teleoperation terminology across industry and academia [4]. However, it is possible to divide teleoperation into two major paradigms: teledriving and tele-assistance (Figure 1). In teledriving the remote operator uses a steering wheel and pedals (or other controls such as a joystick) to continuously drive the AV, while in teleassistance the lower-level maneuvers are delegated to the AV through high-level instructions by the RO [13].
As listed earlier, we believe that teleassistance has many advantages over tele-driving, especially when envisioning a large-scale deployment of AVs on public roads. In such a scenario, several teleoperation centers, with multiple teleoperation stations each, will be deployed in a geographic region to support all the edge-case scenarios that AVs will fail to resolve autonomously. Every RO in such a center will have to deal with many disengagements in a single work shift and therefore an efficient and intuitive teleoperation user interface (UI) is essential. Our research aims at investigating how to best design such a teleassistance interface.
The Society of Automotive Engineers3 defined 6 levels of driving automation: in Level-0 there is no automation at all, while in Level-5 vehicles have full automotive technology. The levels, in between, have partial automation capabilities. In vehicles with a safety driver (Level-1 to Level3), a disengagement is a situation in which the AV returns to manual control or the driver feels the need to take back the wheel from the AV decision system. However, when discussing Level-4 and Level-5 of automation, we refer to a situation in which the AV delivers the control to a RO, who might be located miles away from the scene. Several academic works, [21]–[23] investigated the reasons for such disengagements using quantitative methods, which were applied on California’s DMV4 reports. Dixit et.al.[23], thrive to provide fundamental insights into trust and reaction times in disengagements, Favaro et.al.[22], aim to improve the testing and deployment regulations for AVs on public roads, and Lv et al. [21], try to improve automation technologies. However, none of these studies addressed remote disengagements (i.e., a disengagement without a person in the vehicle). Unlike previous studies, we take a User-Centered Design (UCD) [24] approach and use qualitative analysis to look at disengagements with AVs and the way to address them through teleassistance.
With the purpose to automate driver’s actions and deliver driving low-level controls to the AV itself, we aim to define a discrete, finite, and generic command language, which can be used by tele-assistants in cases of disengagements and when the vehicle’s decision system needs human support. The first step in doing so is to investigate in which remote use-cases AVs will fail to deal with the remote situation autonomously. To do so we conducted in-depth semi-structured interviews with 14 experts from leading automotive companies, innovation centers of well-known automotive corporations, cutting-edge start-ups in the AV teleoperation field, and academia with an average of 20.3 years of experience in the fields. 3 https://www.sae.org/ We used Thematic Analysis [25] to analyze and categorize the data and came up with eight main categories in which remote human intervention would be required. Each category included between 3 to 6 specific sub-categories. Table 1 presents these results. Based on the above interviews and findings, we conducted several brainstorming sessions within our design team and came up with a list of possible high-level commands, which might help RO’s to resolve the above scenarios by delegating low-level controls to the AV. Table 2 summarizes our suggestions:
Defining the above commands was a necessary step in the Research Through Design [28] process we follow. In addition, we performed an in-depth investigation of two additional aspects of the future interface: (1) The perspective of the video feed(s) necessary to increase RO’s situation awareness (SA) and (2) The necessary UI interactions and affordances. We have reviewed 24 interfaces of teleoperation companies and performed a competitive analysis of 10 UIs from that list. In addition, we reviewed various academic works, which focus on various interaction paradigms [29]–[33]. Following this analysis, we defined the desired perspective of the video feed, visible to the RO, to be 5 meters behind and 5 meters above the teleoperated AV (Figure 2). Additionally, we defined the following interaction paradigms to be part of the designed UI: (1) Discrete high-level commands inserted via button clicks, (2) Path plotting, (3) Adding data to unrecognized object in the remote scene, (4) Selecting AI-suggested options.
After formulating the above, we designed a high-fidelity interactive tele-assistance prototype (Figure 2), which incorporates all the above 14 https://www.cognata.com/ insights into one coherent solution. In particular, we used screen shots from Cognata’s14 simulation platform in order to imitate the AV’s environment.
After completing the tele-assistance UI design, we plan to perform usability testing and an evaluation of the interface with expert teleoperators in order to evaluate (1) General screen taxonomy, (2) Navigation flows, (3) Necessity and location of various UI elements, (4) Interaction paradigms, (5) Affordances, and (6) Importance of the video feed perspective to RO’s SA.
Next, we plan to implement the above UI with the help of the upper-mentioned simulation platform and measure the RO’s cognitive load, situation awareness, task performance and overall system’s usability, comparing it to a tele-driving interface. We believe that such quantitative measurements along with qualitative insights will help us understand whether the tele-assistance paradigm can be a substitute for tele-driving in the majority of the disengagement scenarios.
This work was supported by the Israeli Innovation Authority, IDIT PhD fellowship, The Israeli Smart Transportation Research Center, and The Israeli Ministry of Aliyah and Integration. We also wish to thank DriveU and Cognata for their collaboration and specifically, Eli Shapira for his help throughout this work. 6. References