Research in human-drone interactions has primarily focused on cases in which a person interacts with a single drone as an active controller, recipient of information, or a social companion; or cases in which an individual, or a team of operators interacts with a swarm of drones as they perform some coordinated flight patterns. In this position paper we explore a third scenario in which multiple humans and drones collaborate in an emergency response scenario. We discuss different types of interactions, and draw examples from current DroneResponse project.
This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution. Interdisciplinary Workshop on Human-Drone Interaction (iHDI 2020) CHI ’20 Extended Abstracts, 26 April 2020, Honolulu, HI, US © Creative Commons CC-BY 4.0 License.
Human-drone collaboration, emergency response
Small Unmanned Aerial Systems, which we refer to here as drones, can be effectively deployed to support emergency responders for diverse scenarios such as search-andrescue, accident surveillance, and flood inspections. Currently, emergency responders tend to operate drones manually or using off-the-shelf applications that allow them to preprogram sets of waypoints. However, equipping drones with onboard intelligence allows them to perform tasks autonomously and to contribute more fully to the emergency response.
In our DroneResponse project we are designing and
developing a system to deploy and coordinate the efforts of
multiple semi-autonomous drones for use in emergency
situations [
We provide examples of such interactions in the sequence diagram depicted in Figure 2. The Incident Commander first defines a search area and sends a request to the hive controller to start the search (E1). This is an example of human-to-drone interactions (H2D). The hive-controller then creates a search plan and assigns search routes to drones (E2). The coordination between drones represents droneto-drone (D2D) interaction. In the modeled sequence of actions, Dronen detects a potential drowning victim. It then notifies the human incident commander and starts streaming annotated video to the ground (E3), thereby illustrating drone-to-human communication (D2H). The part of the sequence diagram highlighted in yellow provides an example of a more complex bi-directional human-drone conversation in which the drone uses its sensing (image detection) abilities to detect a victim. It then autonomously switches to track-victim mode, raises a victim-found alert, and streams annotated video to the incident commander. Finally, the incident commander uses the information relayed by the drone, to confirm the victim sighting and to push information from the drone to a physical rescue team (E4). This final step is an example of human-to-human (H2H) interaction, triggered by the initial D2H exchange. This sequence of events illustrates the complex socio-technical aspects of emergent multi-user, multi-drone interaction spaces.
There are numerous challenges that must be addressed in
order to achieve efficient human-drone collaboration. In our
concurrently published work [
claim Confirm Victim
Found streamImage() confirmTrack() streamImage() missionCompleted()
End Mission initiateRescue (GPS Coordinates, Imagery) E4
Confirm Rescue()
Rescuer Initiate Rescue Rescue Victim
Secondly, we made the drones explain themselves upon
demand by describing their current strategies and
permissions. In the case of searching for a victim in inclement
weather, the drone might explain “flying lower than
normal at 10 meters due to low visibility” or “searching river
banks at a greater distance than normal due to high wind
gusts and moving branches.” We report outcomes from our
co-design experience, especially with respect to D2H
interactions and achieving situational awareness in our related
paper [
Achieving effective H2D communication is challenging in
systems with multiple drones and complex missions.
Several research groups have explored ways to specify drone
missions using formal commands, often embedded in
domain specific languages [
Researchers have previously explored diverse solutions
for issuing commands to drones, such as the use of
gestures and voice commands [
In our GUI, which is currently under development, users can initially select a mission type from a high-level list of missions as depicted in Figure 9. They then perform a series of configurations such as marking a search area. Each predefined mission type will have a corresponding underlying mission plan with configuration points. This plan is sufficient for allowing the mission to proceed through a series of predefined stages and tasks (e.g., search, track, return home). However, users will also need to configure or tweak the mission dynamically as it evolves, by providing additional directives.
Each of these directives must specify who, what, where,
and how a task is to be accomplished. ‘Who’ refers to whether
the command is addressed to the entire cohort or an
individual drone. In the case of the cohort, then the hive
coordination is empowered to autonomously figure out which
drones are best fit to respond. ’Who’ could also be
specified with constraints such as 3 drones or drones with
thermal cameras onboard. Finally ‘who’ could be addressed to
a specific drone if it were the case, that the Incident
Commander wished to assign a task to a specific drone. This is
more risky, as the selected drone might be unfit for service
(e.g., due to low battery or current critical service). ‘What’
refers to the specific task to be completed – for example
reconnaissance, delivery, or serving as a communication
relay if drones are communicating using onboard
communication channels such as ad-hoc wifi. ‘Where’ refers to a region
or point of interest defined by GPS coordinates. For
example, in the case of reconnaissance, the user might need to
direct the drone to a certain part of a wooded river bank
where somebody has sighted a piece of clothing; while in
the case of establishing a communication relay, the user
could either specify coordinates or allow it to dynamically
position itself so as to optimize communication between all
drones. Finally, ‘how’ enables specific directives for how the
task is to be completed. In some cases, the drones could
be given significant autonomy to complete well-defined
tasks, while in other cases more specific guidelines might
be required. We are currently working closely with several
emergency response organizations to better discover their
needs and to formally model diverse mission plans.
The GUI therefore provides a human-facing interface to an
underlying mission plan specified using a more formal
approach such as belief-desire-intent [
The discussion in this position paper has focused almost entirely on human-drone interfaces based on the use of graphical interfaces; however, drones can also be controlled using physical hand-held devices. In a multi-drone scenario, humans might need to switch between graphical and physical interfaces for several reasons including taking manual control of a malfunctioning drone or temporarily using manual controls for a specific task that is currently beyond the capabilities of a drone to perform autonomously. Our prior work has shown that misalignment of GUI’s and physical controllers can easily lead to accidents [3] (see Figure 10. For example, when control is passed from a computer to a handheld device, the physical switches on the hand-held controller must be set to stable ’flightmode’ positions, otherwise accidents, including crashlandings, could occur. Furthermore, when humans take-over control of remote drones, it can be exceedingly difficult to figure out which direction the drone is facing. Commands are interpreted relative to the drones position, which means that issuing a ‘forward’ command would cause the drone to fly forward, but without clear orientation from the remote-pilot’s perception, that could actually be in any direction. A simple design solution might be to provide a feature to autonomously reorient the drone with respect to the remote pilot so that physical and GUI controls become aligned relative to the drone’s and pilot’s positions. Given this reorientation, a forward command would then consistently send the drone away from the pilot, and a moveRight command would make it move right. For deployment in emergency situations, more though should be invested in the use of both graphical and physical interfaces, the interactions between them, and transitions of control across devices and between different operators.
This position paper has presented an informal framework
for considering human-drone interactions along the
dimensions of H2D, D2H, D2D, and H2H communication in
multiuser, multi-drone environments where drones are permitted
to operate with some degree of autonomy. We have
described some of the challenges we are facing in the design
of DroneResponse and some initial ideas for addressing
those challenges. Our prior work [
The work described in this position paper has primarily been funded by the National Science Foundation under grants CNS-1737496, CNS-1931962, CCF-1647342, and CCF-1741781. We also thank the Firefighters of South Bend for closely collaborating on the DroneResponse project. [6] Markus Funk. 2018. Human-drone interaction: Let’s get ready for flying user interfaces! Interactions 25, 3 (2018), 78–81. DOI: http://dx.doi.org/10.1145/3194317