Camera drones, a rapidly emerging technology, offer people the ability to remotely inspect an environment with a high degree of mobility and agility. However, manual remote piloting of a drone is prone to errors. In contrast, autopilot systems are not necessarily designed to support flexible visual inspections. We propose the object-centric control paradigm for efficient camera drone navigation, in which a user directly specifies the navigation of a drone camera relative to a specified object of interest. We demonstrated the strengths of this approach through our first prototype, StarHopper, and discuss future research opportunities.
Researchers in telepresence have long envisioned ‘beyond
being there’ [
Researchers have noted a number of social and functional
issues due to the insufficient mobility of current remote
robotic presence platforms [
To relieve the burden of manual piloting, autopilot
techniques have been applied to drone control. Most existing
drone autopilot interfaces are based on specifying a
series of planned waypoints in a 2D or 3D global map (e.g.
[
The difficulty of drone piloting poses a significant barrier for
the widespread adoption of free-flying robots. The goal of
this research is to design a camera drone control interface
to support efficient and flexible telepresence experience.
Our work is inspired by decades of research in interactive
graphics, for which many camera navigation techniques
have been established (e.g. [
Most relevant, we build upon object-centric techniques,
where zooming, panning, and orbiting occurs relative to
the location of a 3D object of interest. We demonstrated
the potential of this approach through our first prototype,
StarHopper [
StarHopper is a remote object-centric camera drone
navigation interface that is operated through familiar touch
interactions and relies on minimal geometric information of the
environment (Figure 1). It consists of an overhead camera
view for context and a 3D-tracked drone’s first-person view
for focus (Figure 2). New objects of interest can be
specified through simple touch gestures on both camera views.
We combine automatic and manual control via four
navigation mechanisms that can complement each other with
unique strengths, to support efficient and flexible visual
inspection. The system focuses on indoor environments,
representative of tasks such as remote warehouse inspection
[
We base our design for remote object-centric drone navigation on a set of guidelines grounded by our review of prior literature. These guidelines are: (1) support situation awareness (2) minimize reliance on environmental information (3) combine automated and manual control (4) Support simple touch interactions (5) Respect physical constraints.
StarHopper provides a touch screen interface for the users to view the drone’s live stream video and to perform drone navigations (Figure 3). The drone camera feed fills the screen. The overview camera video and two virtual joysticks are at the bottom of the interface.
The user can obtain the approximate position and
dimensions of an object through a simple two-step procedure,
without using pre-built maps or expensive real-time 3D
reconstruction methods. She selects the object of interest
through a drag gesture first in the overview camera view
and then in the drone camera view. A computer vision
algorithm triangulates the position of the object from these
two regions and estimates the dimensions of a bounding
cylinder of the object (see [
To use this technique, the user first rests two fingers on the drone camera view to freeze the current frame (Figure 5a, next page). The user then performs a two-finger pinch-andpan gesture to transform the current frame to the desired viewpoint (Figure 5b, next page). The system then calculates a new drone position that can produce the desired viewpoint which the drone navigates towards.
We remap the axis of traditional drone control joysticks to object-centric commands and add constraints to prevent manipulation errors. More specifically, under the objectcentric constraints, the drone keeps the object of interest in its field-of-view during the pan movements (Figure 7a, next page). In object-centric zoom, the drone aims its camera at the object of interest and moves closer or further away from it (Figure 7b, next page). In response to the orbiting commands, the drone orbits around the object while aiming at its center (Figure 7c, next page).
In addition to the three object-centric navigation mechanisms, StarHopper also supports fully manual controls. This could be useful in cases where the user wishes to make slight adjustments to a viewpoint that the auto-pilot system navigated to.
The object-of-interest list on the right of the interface (3d) records the thumbnails of all previously registered objects of interest. The user can tap on the thumbnail to set it as the object-of-interest, and the drone will turn towards it. A double-tap on the thumbnail will trigger the drone to approach that object.
StarHopper consists of a set of four navigation mechanisms, ranging from fully automated to fully manual. This suite of techniques allows users to perform both flexible and efficient scene inspections by leveraging their contrasting capabilities (Table 1, page 6).
We recognize the trend that a higher automation level increases efficiency but reduces flexibility. Taken together, the system offers the user both efficient and flexible navigation mechanisms (Table 1). The 360 viewpoint widget, despite its high efficiency, lacks in flexibility and can be complemented by delayed through-the-lens control, object-centric joysticks and manual controls.
To evaluate the navigation mechanisms of StarHopper, we conducted a user study consisting of a remote object inspection task with 12 volunteers (7 female, Mage = 26:3; SDage = 4:4). A Ryze Tello drone was used in the study. We compared StarHopper to a baseline, consisting of conventional manual joystick controls. In each trial, the participant was instructed to fly the drone from the starting position to inspect one of the four sides (Left, Right, Front, Back) of an item (Figure 8) using one of the two control interfaces, StarHopper or manual joysticks (Manual). We recorded the completion time of each trial.
A repeated measures analysis of variance showed that it was significantly faster to complete the task with StarHopper than with egocentric manual control (F1;11 = 23:8; p < 0:001). Overall StarHopper was 35.4% faster (StarHopper: 20.33s, Manual: 31.45s), demonstrating a substantial gain in efficiency (Figure 6).
The StarHopper prototype demonstrated the potential efficiency advantage for object-centric camera drone control. More importantly, it revealed several future challenges and opportunities for better leveraging the object-centric paradigm for unconstrained telepresence.
Supporting situation awareness has long been a key theme
in robot teleoperation research ([
Interactions with objects-of-interest in StarHopper were limited to specifying desirable viewpoints, as StarHopper only exploited simple geometric information. With recent advancements in image understanding, a natural next step would be enabling richer and more meaningful interactions using semantic information about objects of interest. For instance, instead of following the single rule of placing the object at the center of the camera frame by default, the system could choose a more appropriate camera framing and trajectory depending on the object and relevant context. The drone could focus on the upper body of a person in conversation, or zooming in on the console of an instrument for key readings.
While tele-operated robots give the ability to control
viewpoints back to remote users, they raise challenges of
accurately interpreting remote users’ actions and intentions for
local users. Such challenges are exacerbated on drones as
their movements and form factors can be very different from
humans. Recent research proposed signaling drone
motion intent with augmented reality [
A free-roaming viewpoint, such as a drone, raises privacy
concerns about remote users intentionally or unintentionally
seeing private visual information of local users. Prior
research in video-mediated communication has looked
extensively into privacy issues, but largely for fixed cameras (e.g.
[
Remotely operated camera drones hold potential for unconstrained telepresence ‘beyond being there’ but require careful control interface designs to realize such potential. Through prototyping and evaluating StarHopper, we showed the advantage of the object-centric control paradigm for camera drone teleoperation. We further invite the research community to consider future opportunities in applying the object-centric paradigm to develop more useful and usable camera drone control interfaces for unconstrained