Researchers in telepresence have long envisioned 'beyond being there' [1]. Replicating all relevant local experiences, while remote, should not be the only goal of telepresence; rather, we should also strive to create telepresence systems which can enable benefits that are not possible when the person is physically present. As such, telepresence goes from replication to augmentation. One particular instance of this vision is enabled by camera drones: our local bodies can only walk on the ground, but our remote bodies can fly.
Researchers have noted a number of social and functional issues due to the insufficient mobility of current remote robotic presence platforms [15]. With drones becoming more affordable and reliable, they hold the potential for enabling more flexible remote presence and visual inspection experiences (e.g. [2]) for the general population.
While drones offer promise for such telepresence applications, they are challenging to manually control remotely, due to numerous factors including high degrees of freedom, narrow camera field-of-views, and network delays [7]. Their control interfaces -virtual or physical joysticks for consumer drones -are also unfamiliar for many users and take extended training time to master [9].
To relieve the burden of manual piloting, autopilot techniques have been applied to drone control. Most existing drone autopilot interfaces are based on specifying a series of planned waypoints in a 2D or 3D global map (e.g. [9]). However, in a situation where a user wishes to perform a real-time inspection, setting waypoints a priori may not be efficient for producing the viewer's desired viewpoints. Some autonomous systems avoid the use of waypoints and execute higher-level plans, such as following a subject to form canonical shots [3], but they typically do not offer the flexibility for exploring remote environments.
The difficulty of drone piloting poses a significant barrier for the widespread adoption of free-flying robots. The goal of this research is to design a camera drone control interface to support efficient and flexible telepresence experience. Our work is inspired by decades of research in interactive graphics, for which many camera navigation techniques have been established (e.g. [5]). Most relevant, we build upon object-centric techniques, where zooming, panning, and orbiting occurs relative to the location of a 3D object of interest. We demonstrated the potential of this approach through our first prototype, StarHopper [6], and illustrate future research opportunities.
StarHopper is a remote object-centric camera drone navigation interface that is operated through familiar touch interactions and relies on minimal geometric information of the environment (Figure 1). It consists of an overhead camera view for context and a 3D-tracked drone's first-person view for focus (Figure 2). New objects of interest can be specified through simple touch gestures on both camera views. We combine automatic and manual control via four navigation mechanisms that can complement each other with unique strengths, to support efficient and flexible visual inspection. The system focuses on indoor environments, representative of tasks such as remote warehouse inspection [9] and museum visits [11], and where positional tracking technology is more reliable.
We base our design for remote object-centric drone navigation on a set of guidelines grounded by our review of prior literature. These guidelines are: (1) support situation awareness (2) minimize reliance on environmental information (3) combine automated and manual control (4) Support simple touch interactions (5) Respect physical constraints.
StarHopper provides a touch screen interface for the users to view the drone's live stream video and to perform drone navigations (Figure 3). The drone camera feed fills the screen. The overview camera video and two virtual joysticks are at the bottom of the interface. The user can obtain the approximate position and dimensions of an object through a simple two-step procedure, without using pre-built maps or expensive real-time 3D re-construction methods. She selects the object of interest through a drag gesture first in the overview camera view and then in the drone camera view. A computer vision algorithm triangulates the position of the object from these two regions and estimates the dimensions of a bounding cylinder of the object (see [6] for more technical details).
Inspired by camera control mechanisms in interactive graphics, we have designed three object-centric physical camera navigation mechanisms for viewing an object of focus: 360 viewpoint widget, delayed through-the-lens control, and object-centric joysticks.
The 360 viewpoint widget is a widget for quickly navigating to and focusing on an object of interest, from a userspecified viewing angle. The widget takes the shape of a semi-transparent 3D ring, surrounding the focus object (Figure 4a). A 3D arrow aimed at the ring appears upon touch, indicating the desired viewing direction. The user can drag the finger on the ring to set the desired viewpoint position (Figure 4b). Once the user releases the finger, the autopilot system moves the drone to the calculated viewpoint. The algorithm determines a reasonable default viewing distance, based on the size of the bounding cylinder.
To use this technique, the user first rests two fingers on the drone camera view to freeze the current frame (Figure 5a, next page). The user then performs a two-finger pinch-andpan gesture to transform the current frame to the desired viewpoint (Figure 5b, next page). The system then calculates a new drone position that can produce the desired viewpoint which the drone navigates towards.
We remap the axis of traditional drone control joysticks to object-centric commands and add constraints to prevent manipulation errors. More specifically, under the objectcentric constraints, the drone keeps the object of interest in its field-of-view during the pan movements (Figure 7a, next page). In object-centric zoom, the drone aims its camera at the object of interest and moves closer or further away from it (Figure 7b, next page). In response to the orbiting commands, the drone orbits around the object while aiming at its center (Figure 7c, next page).
In addition to the three object-centric navigation mechanisms, StarHopper also supports fully manual controls. This could be useful in cases where the user wishes to make slight adjustments to a viewpoint that the auto-pilot system navigated to.
The object-of-interest list on the right of the interface (3d) records the thumbnails of all previously registered objects of interest. The user can tap on the thumbnail to set it as the object-of-interest, and the drone will turn towards it. A double-tap on the thumbnail will trigger the drone to approach that object.
StarHopper consists of a set of four navigation mechanisms, ranging from fully automated to fully manual. This suite of techniques allows users to perform both flexible and efficient scene inspections by leveraging their contrasting capabilities (Table 1, page 6). We recognize the trend that a higher automation level increases efficiency but reduces flexibility. Taken together, the system offers the user both efficient and flexible navigation mechanisms (Table 1). The 360 viewpoint widget, despite its high efficiency, lacks in flexibility and can be complemented by delayed through-the-lens control, object-centric joysticks and manual controls.
To evaluate the navigation mechanisms of StarHopper, we conducted a user study consisting of a remote object inspection task with 12 volunteers (7 female, M age = 26.3, SD age = 4.4). A Ryze Tello drone was used in the study. We compared StarHopper to a baseline, consisting of conventional manual joystick controls. In each trial, the participant was instructed to fly the drone from the starting position to inspect one of the four sides (Left, Right, Front, Back) of an item (Figure 8) using one of the two control interfaces, StarHopper or manual joysticks (Manual). We recorded the completion time of each trial.
A repeated measures analysis of variance showed that it was significantly faster to complete the task with StarHopper than with egocentric manual control (F 1,11 = 23.8, p < 0.001). Overall StarHopper was 35.4% faster (StarHopper: 20.33s, Manual: 31.45s), demonstrating a substantial gain in efficiency (Figure 6).
The StarHopper prototype demonstrated the potential efficiency advantage for object-centric camera drone con-trol. More importantly, it revealed several future challenges and opportunities for better leveraging the object-centric paradigm for unconstrained telepresence.
Increasing Situation Awareness for Leveraging Greater Mobility Supporting situation awareness has long been a key theme in robot teleoperation research ( [14]). With greater mobility, drone operators face greater risk of getting lost in space ( [7]). Prior research has shown the effectiveness of a live exocentric overview for enhancing situation awareness in teleoperation (e.g. [8]). StarHopper incorporated a static live overview camera, but this setup reduced the area where the drone could fly. Future research can explore awareness mechanisms that do not sacrifice mobility. For example, a second, spatially coupled camera drone serving as the overhead camera [10].
Interactions with objects-of-interest in StarHopper were limited to specifying desirable viewpoints, as StarHopper only exploited simple geometric information. With recent advancements in image understanding, a natural next step would be enabling richer and more meaningful interactions using semantic information about objects of interest. For instance, instead of following the single rule of placing the object at the center of the camera frame by default, the system could choose a more appropriate camera framing and trajectory depending on the object and relevant context. The drone could focus on the upper body of a person in conversation, or zooming in on the console of an instrument for key readings. While tele-operated robots give the ability to control viewpoints back to remote users, they raise challenges of accu-rately interpreting remote users' actions and intentions for local users. Such challenges are exacerbated on drones as their movements and form factors can be very different from humans. Recent research proposed signaling drone motion intent with augmented reality [12]. However, future flight paths and waypoints can be insufficient for a remote user who operates the drone to establish common ground with a local user, for example, when they want to make sure that they are discussing about the same object among a number of candidates in the environment. Visualizing objects of interest can complement the above signaling method and facilitate communication.
A free-roaming viewpoint, such as a drone, raises privacy concerns about remote users intentionally or unintentionally seeing private visual information of local users. Prior research in video-mediated communication has looked extensively into privacy issues, but largely for fixed cameras (e.g. [4]). Privacy research on drones mostly studied perceptions about drones operated by strangers (e.g. [13]). Drones for telepresence, especially drones that work closely with humans, call for new privacy mechanisms. Local users can define sensitive objects or zones, which remotely operated drones should always avert.
Remotely operated camera drones hold potential for unconstrained telepresence 'beyond being there' but require careful control interface designs to realize such potential. Through prototyping and evaluating StarHopper, we showed the advantage of the object-centric control paradigm for camera drone teleoperation. We further invite the research community to consider future opportunities in applying the object-centric paradigm to develop more useful and usable camera drone control interfaces for unconstrained