Interviewing stakeholders on the teleoperation of last-mile delivery robots⋆ Einat Grimberg1 , Avishag Boker2 and Joel Lanir2,∗ 1 Commonwealth Scientific and Industrial Research Organisation (CSIRO), Dutton Park, Australia 2 University of Haifa, Haifa, Israel Abstract As e-commerce continues its rapid expansion, the challenges surrounding delivery are becoming more pronounced. The surge in traffic, environmental concerns, and heightened customer expectations have compounded the complexities of the delivery process. Customers now demand quicker deliveries within increasingly narrow timeframes, placing greater pressure on last-mile logistics, a pivotal yet costly aspect of the delivery chain. To address these challenges, fully autonomous last-mile delivery robots offer a promising and sustainable solution for efficient deliveries to their final destinations. Nevertheless, despite their advanced autonomous capabilities, it is widely acknowledged that, at least in the foreseeable future, autonomous robots operating in urban environments will frequently encounter situations beyond their capabilities. Factors such as road obstructions, adverse weather conditions, congested intersections, or human interactions may necessitate the intervention of a remote human operator. This work seeks to explore the specifications and design of a teleoperation interface tailored for remote human operators, enabling them to efficiently manage a multitude of delivery robots simultaneously. Keywords Last-mile Delivery,Tele-operation, Delivery Robots. 1. Introduction cracked sidewalk) that prevents the robot from proceed- ing, customers who are not responding, an unclear or un- With the growth of e-commerce, developing faster, more af- reachable final destination, unfavorable weather conditions fordable, and more sustainable last-mile delivery solutions (snow, heavy rain, black ice, etc.), road events (e.g., crashes, are needed. Autonomous Last-Mile Delivery Robots (LM- car breakdowns), too many parcels per courier, lack of park- DRs) are emerging technologies which is seen as a promis- ing required to drop off and so forth. Some of these situa- ing solution for delivery challenges in the near future [1, 2]. tions could adversely affect the operational efficiency (e.g. LMDRs are a sub-category of autonomous vehicles (AVs), if the robots can’t find the destination), the robot or other’s which refers to mobile, electrical, relatively small units ca- safety (for example, if a robot gets stuck while crossing the pable of moving autonomously or partially autonomously road), or both. The efficient operation of the LMDRs can and delivering small goods such as groceries, food, and also be at risk because of power outages or any other tech- parcels. As such, they are able to provide faster, more effi- nical problem that may prevent them from proceeding. cient, and accurate delivery [3]. the communication with To address the situations in which LMDRs cannot handle the customer is handled through a smartphone app, which independently, a remote human operator would be called enables the customer to place the order and then notifies upon for assistance. The current research seeks to explore them of the progress (distance and arrival time), unlock the how to design a teleoperation interface that would enable robot cover lid, and retrieve the goods. a remote operator to efficiently manage one or more LM- Like other AVs, LMDRs are equipped with various sen- DRs remotely. We take a human-centered design (UCD) ap- sors enabling their autonomous mobility. These sensors proach by first understanding the needs and requirements can include cameras, LIDAR, ultrasonic, and radar for sens- of the users (i.e., the remote operators) and the task. There- ing objects in the environment. They are also equipped fore, we conducted interviews with fifteen related stake- with inertial measurement units (IMU) and global position- holders from the industry with the focus of understanding ing systems (GPS) used for navigation [4, 5, 6]; Some of the main problems that might require remote assistance of them are also equipped with microphones and speakers en- LMDRs as well as the various issues that remote operators abling them to communicate with humans [5]. Most of the might have. LMDRs are on autonomy levels 3 or 4 (partial autonomous and highly autonomous respectively) which means they can autonomously detect, recognize, and respond to differ- 2. Methods ent objects on and off the roadway [4]. Despite their clear benefits, LMDRs are not flawless. We conducted 15 interviews with industry stakeholders First, there are inherent limitations like their limited deliv- in the field of autonomous delivery robots. The semi- ery radius [5]. Secondly, aside from their autonomous ca- structured interviews aimed at extracting the main issues pabilities, LMDRs routinely encounter situations they can- experienced by teleoperators. To recruit participants, we not handle independently [7, 8, 9]. Such situations include, contacted professionals through Linkedin, companies, and for example, poor infrastructure (e.g., a blocked road or via networking. Table 1 shows the list of participants. We conducted semi-structured interviews with the par- ticipants. While we aimed to ask the same questions, their Workshop Robots for Humans 2024, Advanced Visual Interfaces, Aren- different roles and associations to the field have led the zano, June 3rd, 2024 ∗ Corresponding author. interviews in slightly different directions, to allow partici- £ grimbeet@gmail.com (E. Grimberg); avishbo@gmail.com pants to elaborate on the things they are more experienced (A. Boker); ylanir@is.haifa.ac.il (J. Lanir) and familiar with. The main question that led all the inter- DZ 0000-0002-9838-5142 (J. Lanir) views was about the main challenges that a remote opera- © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Table 1 List of Participants Role Role description P1 PhD student - Project manager P1 built a prototype and conducted 3 experiments that include LMDR traveling from a starting point to a destination point The experiments were conducted in major cities across Europe. P2 Managing Director in an au- P2 has been working in automotive industry for approximately 20 years. For the past tonomous safe vehicle and 10 years P2 worked for a company that focused on indoor and outdoor robots The delivery robot services. compaby provides and maintains LMDRs for various costumers. P3 Ex Co-Founder & COO of a tele- The software of the company founded by P3 enables people to remotely operate, as- operation software company sist, and supervise vehicles from thousands of miles away including forklifts, tuggers, robots, trucks, and more. P4 Chief Product Officer at a remote The participant has worked for the past few years in a company that enables remote operation software company supervision and teleoperation of different vehicles and robots including LMDR. P5 Head of Marketing at a company Works for the past 2 years in a company that developed market-leading on-demand which deals with autonomous transportation services, navigation products, and other mobile applications. Con- robots ducted experiments usind LMDR in various countries. P6 Business Development Manager Managing projects at an asian company aiming at providing movement of people and at an autonomous robots com- goods via the use of autonomus robots. Tele-operation function has is been developed pany duo to a growing need for remote control in extreme cases. P7 Chief Operating Officer in a com- The company enables teleoperations on public roads and public networks. It provides pany that provides teleoperations both a tele-driving platform that is costumable by the client and teledrivers service. software platform P8 Co-Founder and CEO of a startup The company develops and maintains robotic systems. The company deals mainly for both autonomous and teleop- with teleoperation of heavy industrial indoor machines. In addition, it also deals with erated mobile robots remote driving of outdoor industrial machines as well as some LMD that includes a teleoperation interface due to regulatory considerations. P9 Co-Founder & CXO of human- The company has vast experience in the operation of drones and some additional guided autonomous machine sys- experience in ground robotics operation especially for millitary purposes. Recently tems firm the company began collaborating with large vehicle and delivery compenies in the fiels of LMDR. P10 Operation Manager at a robotics The company has created a robot that is a multi purpose helper to humanity. One of company its services is LMD. They operate curently in a few ereas and cmpuses in USA. P11 VP Business Development at an For the past 7 years works in product and business development roles in an AV sim- AV simulation platform company ulation platform startup. The platform includes the option to perform teledriving simulations with AVs. P12 Principal designer of a technology In the framework of the studio he was responsible for designing an autonomous LMD Design studio located in USA. robot for a leading autonomous sidewalk delivery company that develops and oper- ates robots that serve people in public spaces, mainly for the food industry. P13 Founder and CEO at a robots man- p13 founded a bootstrap company that assisted robots companies in autonomy devel- agement platform company opment. P14 Former head of Fleet Quality at a Responsible for the warehouses that maintain the delivery robots, responsible for robotic LMD service company. providing technical support for the robots and the remote drivers. P15 Open Innovation Manager at a big The job entails finding new technologies that fit the strategic needs of the company. international car company Now the company works on POC in the field of LMDR. It will soon begin experiments using a delivery ground robot in a controlled environment. tor of LMDRs encounters when performing their job. In- 3.1. Teleoperation center terviews lasted around 50 minutes. Most of the interviews Generally, teleoperation centers work like any other call were held online and one was face-to-face. Interviews were center. Requests for assistance are often managed by an recorded and transcribed for further analysis. A thematic “administrating system” that channels them to the opera- analysis was conducted to extract the main issues. The tors according to some criteria. These criteria and the way analysis focused on two main questions: the first is when incoming calls are handled may differ among companies. and why operators need to intervene and the second is the A few models were described for assigning operators to challenges they may experience while working handle requests. One of the interviewees described a sit- uation he observed (as a bystander) in which five robots 3. Initial Findings were standing in front of a crossroad but not crossing. His assumption was that this situation occurred because of the The analysis of the interviews is currently a work in availability of operators. Since the common ratio today is progress. We report here on the initial findings from the one (operator)-to-many (robots), it is possible that in this analysis of 7 of the interviews. situation there were enough operators to help all the robots to cross the road (a situation that requires one operator per robot). 3.2. User Interface situation awareness. Another reason is the quick-shifting between the control of remote robots, which requires op- The user interface for controlling the robots seems to vary erators to recalibrate their situation awareness to the new greatly between companies both in terms of the number situation and environment in hand. It takes a few seconds and size of the screens and the type and amount of infor- to look at the camera, understand what happens, and be- mation being displayed. For example, the information can come aware of the situation. The lack of continuity and be presented on a single or multiple monitors, for a sin- the need to rapidly develop an understanding of the situa- gle or multiple robots. The type of information can in- tion requires a lot of focus and attention and may demand clude any of the following (often in various combinations): a high cognitive load. front camera footage (of one or more robots), location and Operators are expected to solve problems quickly while speed of robots, meta-data from the robot (e.g., type of vehi- not always knowing what the problem is or what they need cle), mode of operation (autonomous or manual) and much to do. This can induce a lot of stress and high cognitive load. more. In essence, the information being displayed is based Some operators may deal with these stressful situations by on the priorities of each company. trying to fix the problem by addressing the symptoms and not the root cause. In other situations, they may prefer to 3.3. Problem types call the onsite technician teams. While sometimes an onsite team can be the only solution, this is often a costly solution There is a great variety of issues requiring remote interven- which preferably should be avoided. tions. The interviewees described several types of requests that can be crudely divided into three main categories: 3.6. Helping customers 1. “Go/No-Go” question when arriving at identified lo- cations; This refers to situations in which the robot Part of the role of teleoperators is to assist the customer if requests permission to proceed when getting to a he or she has any problem with the robot. One interviewee crossroad or another point that was predefined as a noted that there might be situations when the request or as- place requiring an operator’s assistance. sistance is not initiated by the robot, but rather by the cus- 2. a robot gets stuck or cannot handle the situation tomer who is facing a problem (e.g., the lid does not open). independently. Robots can get stuck for many rea- The remote operator should be able to communicate with sons, for example, their wheels might get stuck or the customer as well as be able to resolve simple problems obstacles block their pre-planned path and there is remotely. no alternative path to consider (as they are allowed to follow only certain paths). Obstacles can be fixed objects (e.g. a tree has fallen) or people or other 4. Conclusion moveable objects blocking the way. ROs can re- Last-mile delivery robots are an emerging technology that motely drive the robot or choose a command to con- is seen as a promising solution for delivery challenges in trol it. urban areas. The current work investigates the teleopera- 3. Communication problems. These often refer to loss tion of such robots. This is a first step toward the design of or reduced quality of GPS or WIFI signal. a teleoperation interface that would enable to resolve the various issues that these autonomous robots face and can- 3.4. The teleoperation challenges not solve autonomously. Following the complete analysis of the interviews, we plan to make a complete list of use A few interviewees noted that sidewalks are a more difficult cases and tasks for which an LMDR needs assistance. This operational area compared to a road environment. Roads will be followed by the design and evaluation of a user in- are more structured than sidewalks, the traffic generally terface for a remote LMDR operator. moves in clear directions, and it is less crowded. It is rel- atively easier for autonomous technology and the operator to find a path for the robot to follow. Contrarily, the traffic 5. Acknowledgments on sidewalks is less predicted, denser, and highly dynamic – a bus may stop and block the way, construction work, peo- This work was funded by the Israeli Smart Transportation ple crowding, etc, which makes it harder for the operator Research Center (ISTRC). to guide the robot through. A few interviewees noted that teleoperators often follow scripts that guide them on how to handle various situations References and instruct them on the steps that should be taken for a [1] V. Engesser, E. Rombaut, L. Vanhaverbeke, P. Lebeau, given problem. The scripts vary by the type and complexity Autonomous delivery solutions for last-mile logistics of the problem. The more complicated the problem is, the operations: A literature review and research agenda, more actions the operator is likely to be instructed to take. Sustainability 15 (2023) 2774. [2] W. A. Mohammad, Y. Nazih Diab, A. Elomri, C. Triki, 3.5. Cognitive challenges Innovative solutions in last mile delivery: concepts, practices, challenges, and future directions, in: Sup- Like with teleoperators of other vehicles, situation aware- ply Chain Forum: An International Journal, volume 24, ness (SA) is a major challenge for LMDRs teleoperators. Taylor & Francis, 2023, pp. 151–169. Based on the interviews, we observed a few types of threats [3] S. Sorooshian, S. Khademi Sharifabad, M. Parsaee, A. R. for situation awareness. First, problems in the communica- Afshari, Toward a modern last-mile delivery: Conse- tion (e.g., no video input) may be wrongly interpreted as no change in the scene/environment, which results in poor quences and obstacles of intelligent technology, Ap- plied System Innovation 5 (2022) 82. [4] M. Clamann, M. Bryson, Sharing spaces with robots: The basics of personal delivery devices, Chapel Hill, NC (2021). [5] T. Hoffmann, G. Prause, On the regulatory framework for last-mile delivery robots, Machines 6 (2018) 33. [6] C. Lemardelé, S. P. Melo, F. Cerdas, C. Herrmann, M. Estrada, Life-cycle analysis of last-mile parcel de- livery using autonomous delivery robots, Transporta- tion Research Part D: Transport and Environment 121 (2023) 103842. [7] N. J. Cooke, Human factors of remotely operated ve- hicles, in: Proceedings of the Human Factors and Er- gonomics Society Annual Meeting, volume 50, SAGE Publications Sage CA: Los Angeles, CA, 2006, pp. 166– 169. [8] A. Howell, B. Steckler, N. Larco, K. A. V. Initiative, et al., Piloting Sidewalk Delivery Robots in pitts- burgH, miami-dade county, Detroit, and San José, Technical Report, University of Oregon, 2022. URL: https://knightfoundation.org/wp-content/uploads/ 2022/09/Knight-AV-Initiative-PDD-Report-Final_ Aug-2022-1.pdf. [9] S. Srinivas, S. Ramachandiran, S. Rajendran, Au- tonomous robot-driven deliveries: A review of recent developments and future directions, Transportation re- search part E: logistics and transportation review 165 (2022) 102834.