As e-commerce continues its rapid expansion, the challenges surrounding delivery are becoming more pronounced. The surge in traffic, environmental concerns, and heightened customer expectations have compounded the complexities of the delivery process. Customers now demand quicker deliveries within increasingly narrow timeframes, placing greater pressure on last-mile logistics, a pivotal yet costly aspect of the delivery chain. To address these challenges, fully autonomous last-mile delivery robots offer a promising and sustainable solution for efficient deliveries to their final destinations. Nevertheless, despite their advanced autonomous capabilities, it is widely acknowledged that, at least in the foreseeable future, autonomous robots operating in urban environments will frequently encounter situations beyond their capabilities. Factors such as road obstructions, adverse weather conditions, congested intersections, or human interactions may necessitate the intervention of a remote human operator. This work seeks to explore the specifications and design of a teleoperation interface tailored for remote human operators, enabling them to efficiently manage a multitude of delivery robots simultaneously.
With the growth of e-commerce, developing faster, more affordable, and more sustainable last-mile delivery solutions are needed. Autonomous Last-Mile Delivery Robots (LM-DRs) are emerging technologies which is seen as a promising solution for delivery challenges in the near future [1,2]. LMDRs are a sub-category of autonomous vehicles (AVs), which refers to mobile, electrical, relatively small units capable of moving autonomously or partially autonomously and delivering small goods such as groceries, food, and parcels. As such, they are able to provide faster, more efficient, and accurate delivery [3]. the communication with the customer is handled through a smartphone app, which enables the customer to place the order and then notifies them of the progress (distance and arrival time), unlock the robot cover lid, and retrieve the goods.
Like other AVs, LMDRs are equipped with various sensors enabling their autonomous mobility. These sensors can include cameras, LIDAR, ultrasonic, and radar for sensing objects in the environment. They are also equipped with inertial measurement units (IMU) and global positioning systems (GPS) used for navigation [4,5,6]; Some of them are also equipped with microphones and speakers enabling them to communicate with humans [5]. Most of the LMDRs are on autonomy levels 3 or 4 (partial autonomous and highly autonomous respectively) which means they can autonomously detect, recognize, and respond to different objects on and off the roadway [4].
Despite their clear benefits, LMDRs are not flawless. First, there are inherent limitations like their limited delivery radius [5]. Secondly, aside from their autonomous capabilities, LMDRs routinely encounter situations they cannot handle independently [7,8,9]. Such situations include, for example, poor infrastructure (e.g., a blocked road or cracked sidewalk) that prevents the robot from proceeding, customers who are not responding, an unclear or unreachable final destination, unfavorable weather conditions (snow, heavy rain, black ice, etc.), road events (e.g., crashes, car breakdowns), too many parcels per courier, lack of parking required to drop off and so forth. Some of these situations could adversely affect the operational efficiency (e.g. if the robots can't find the destination), the robot or other's safety (for example, if a robot gets stuck while crossing the road), or both. The efficient operation of the LMDRs can also be at risk because of power outages or any other technical problem that may prevent them from proceeding.
To address the situations in which LMDRs cannot handle independently, a remote human operator would be called upon for assistance. The current research seeks to explore how to design a teleoperation interface that would enable a remote operator to efficiently manage one or more LM-DRs remotely. We take a human-centered design (UCD) approach by first understanding the needs and requirements of the users (i.e., the remote operators) and the task. Therefore, we conducted interviews with fifteen related stakeholders from the industry with the focus of understanding the main problems that might require remote assistance of LMDRs as well as the various issues that remote operators might have.
We conducted 15 interviews with industry stakeholders in the field of autonomous delivery robots. The semistructured interviews aimed at extracting the main issues experienced by teleoperators. To recruit participants, we contacted professionals through Linkedin, companies, and via networking. Table 1 shows the list of participants.
We conducted semi-structured interviews with the participants. While we aimed to ask the same questions, their different roles and associations to the field have led the interviews in slightly different directions, to allow participants to elaborate on the things they are more experienced and familiar with. The main question that led all the interviews was about the main challenges that a remote opera- The company develops and maintains robotic systems. The company deals mainly with teleoperation of heavy industrial indoor machines. In addition, it also deals with remote driving of outdoor industrial machines as well as some LMD that includes a teleoperation interface due to regulatory considerations. P9
Co-Founder & CXO of humanguided autonomous machine systems firm
The company has vast experience in the operation of drones and some additional experience in ground robotics operation especially for millitary purposes. Recently the company began collaborating with large vehicle and delivery compenies in the fiels of LMDR.
The company has created a robot that is a multi purpose helper to humanity. One of its services is LMD. They operate curently in a few ereas and cmpuses in USA.
VP Business Development at an AV simulation platform company
For the past 7 years works in product and business development roles in an AV simulation platform startup. The platform includes the option to perform teledriving simulations with AVs.
Principal designer of a technology Design studio located in USA.
In the framework of the studio he was responsible for designing an autonomous LMD robot for a leading autonomous sidewalk delivery company that develops and operates robots that serve people in public spaces, mainly for the food industry.
Founder and CEO at a robots management platform company p13 founded a bootstrap company that assisted robots companies in autonomy development.
Former head of Fleet Quality at a robotic LMD service company.
Responsible for the warehouses that maintain the delivery robots, responsible for providing technical support for the robots and the remote drivers.
Open Innovation Manager at a big international car company
The job entails finding new technologies that fit the strategic needs of the company. Now the company works on POC in the field of LMDR. It will soon begin experiments using a delivery ground robot in a controlled environment.
tor of LMDRs encounters when performing their job. Interviews lasted around 50 minutes. Most of the interviews were held online and one was face-to-face. Interviews were recorded and transcribed for further analysis. A thematic analysis was conducted to extract the main issues. The analysis focused on two main questions: the first is when and why operators need to intervene and the second is the challenges they may experience while working
The analysis of the interviews is currently a work in progress. We report here on the initial findings from the analysis of 7 of the interviews.
Generally, teleoperation centers work like any other call center. Requests for assistance are often managed by an "administrating system" that channels them to the operators according to some criteria. These criteria and the way incoming calls are handled may differ among companies. A few models were described for assigning operators to handle requests. One of the interviewees described a situation he observed (as a bystander) in which five robots were standing in front of a crossroad but not crossing. His assumption was that this situation occurred because of the availability of operators. Since the common ratio today is one (operator)-to-many (robots), it is possible that in this situation there were enough operators to help all the robots to cross the road (a situation that requires one operator per robot).
The user interface for controlling the robots seems to vary greatly between companies both in terms of the number and size of the screens and the type and amount of information being displayed. For example, the information can be presented on a single or multiple monitors, for a single or multiple robots. The type of information can include any of the following (often in various combinations): front camera footage (of one or more robots), location and speed of robots, meta-data from the robot (e.g., type of vehicle), mode of operation (autonomous or manual) and much more. In essence, the information being displayed is based on the priorities of each company.
There is a great variety of issues requiring remote interventions. The interviewees described several types of requests that can be crudely divided into three main categories:
1. "Go/No-Go" question when arriving at identified locations; This refers to situations in which the robot requests permission to proceed when getting to a crossroad or another point that was predefined as a place requiring an operator's assistance. 2. a robot gets stuck or cannot handle the situation independently. Robots can get stuck for many reasons, for example, their wheels might get stuck or obstacles block their pre-planned path and there is no alternative path to consider (as they are allowed to follow only certain paths). Obstacles can be fixed objects (e.g. a tree has fallen) or people or other moveable objects blocking the way. ROs can remotely drive the robot or choose a command to control it. 3. Communication problems. These often refer to loss or reduced quality of GPS or WIFI signal.
A few interviewees noted that sidewalks are a more difficult operational area compared to a road environment. Roads are more structured than sidewalks, the traffic generally moves in clear directions, and it is less crowded. It is relatively easier for autonomous technology and the operator to find a path for the robot to follow. Contrarily, the traffic on sidewalks is less predicted, denser, and highly dynamica bus may stop and block the way, construction work, people crowding, etc, which makes it harder for the operator to guide the robot through.
A few interviewees noted that teleoperators often follow scripts that guide them on how to handle various situations and instruct them on the steps that should be taken for a given problem. The scripts vary by the type and complexity of the problem. The more complicated the problem is, the more actions the operator is likely to be instructed to take.
Like with teleoperators of other vehicles, situation awareness (SA) is a major challenge for LMDRs teleoperators. Based on the interviews, we observed a few types of threats for situation awareness. First, problems in the communication (e.g., no video input) may be wrongly interpreted as no change in the scene/environment, which results in poor situation awareness. Another reason is the quick-shifting between the control of remote robots, which requires operators to recalibrate their situation awareness to the new situation and environment in hand. It takes a few seconds to look at the camera, understand what happens, and become aware of the situation. The lack of continuity and the need to rapidly develop an understanding of the situation requires a lot of focus and attention and may demand a high cognitive load.
Operators are expected to solve problems quickly while not always knowing what the problem is or what they need to do. This can induce a lot of stress and high cognitive load. Some operators may deal with these stressful situations by trying to fix the problem by addressing the symptoms and not the root cause. In other situations, they may prefer to call the onsite technician teams. While sometimes an onsite team can be the only solution, this is often a costly solution which preferably should be avoided.
Part of the role of teleoperators is to assist the customer if he or she has any problem with the robot. One interviewee noted that there might be situations when the request or assistance is not initiated by the robot, but rather by the customer who is facing a problem (e.g., the lid does not open). The remote operator should be able to communicate with the customer as well as be able to resolve simple problems remotely.
Last-mile delivery robots are an emerging technology that is seen as a promising solution for delivery challenges in urban areas. The current work investigates the teleoperation of such robots. This is a first step toward the design of a teleoperation interface that would enable to resolve the various issues that these autonomous robots face and cannot solve autonomously. Following the complete analysis of the interviews, we plan to make a complete list of use cases and tasks for which an LMDR needs assistance. This will be followed by the design and evaluation of a user interface for a remote LMDR operator.
This work was funded by the Israeli Smart Transportation Research Center (ISTRC).