In this work, we present a framework that allows a single operator to monitor and control the operations of multiple robots during Search & Rescue missions in an alpine environment. This work is framed in the context of the SHERPA project whose goal is to develop a mixed ground and aerial robotic platform to support search and rescue activities in alpine scenario. In this context, the human operator is not fully dedicated to the robot control, but involved in the search and rescue mission, hence only able to partially monitor and command the robotic team. In this paper, we brie y illustrate the overall framework and describe on-the- eld tests with two drones equipped with on-board cameras.
We present an architecture suitable for human multi-robot interaction for search
and rescue missions in an alpine environment. This work is framed in the context
of the SHERPA project [
The proposed architecture is depicted in Figure 1. The operator is equipped with light wearable devices to interact with the robotic system. The output of these devices is continuously sent to the Multimodal Human-Robot-Interaction (MHRI ) module in order to generate new commands. If the command provided by the human operator does not explicitly assign tasks to the robots, the Distributed Multi-Robot Task Allocation (DMRTA) module is to decompose abstract tasks into primitive action nding a valid allocation for the robotic team members; when the task is already completely speci ed and allocated this is validated with respect to resources and mission constraints. In the following we describe the associated modules.
Multimodal Interaction. The MHRI module interprets the operator commands
integrating inputs from multiple communication channels. For instance, either
speech- or gesture-based commands may be used to stop a drone, while vocal
commands in combination with deictic gestures can be used to specify
navigational commands (e.g. \go-there") to the co-located drones. We mainly focus on
commands suitable for interacting with the set of co-located drones during
navigation and search tasks. More speci cally, we are concerned with multimodal
communication with the drones suitable for the following purposes: robot
selection commands (e.g. \all wasps", \red wasp", \you wasp"); motion commands
(e.g. \go there", \land", etc.); search primitives (to scan an area with a speci c
search pattern); multimedia commands used to acquire data through on-board
sensors (e.g. pictures, video, environment mapping, etc.); nally, switch meta
level commands allow the operator to change the interaction mode (e.g.
highlevel commands, interactive control, teleoperation). We rely on the Julius
framework for continuous speech recognition. As for gesture recognition, we recognize
harm, hand, and tablet-based gestures. We exploit the Thalmic Myo Armband
to detect and distinguish several poses of the hand (from the electrical activity
of the muscles) and the arm (the band is endowed with a 9 DOF IMU that
permits motion capture). The results form these multiple inputs (gestures, voice,
hand, tablet, etc.) are combined into a unique interpretation of the operator
intention exploiting a late-fusion approach: single modalities are rst classi ed
and synchronized, then fused into a single interpretation using the con dence
value associated with each input data (see Figure 2, left).
Implicit Drone Selection. In order to simplify the interaction with the robotic
team, we permit both implicit and explicit robot selection. Namely, we propose
an approach where, each available robot can evaluate the probability to be the
one designated by the human for the execution of a command when the target
is not explicitly assigned by a selection command, but it can be inferred by the
context (see Figure 2, right). The robot evaluation process relies on a
multilayered architecture in which a Dynamic Bayesian Network is deployed to infer
the human intentions form the state of the robots and learned contextual and
geometrical information. More details can be found in [
Distributed Multi-Robot Task Allocation. The DMRTA module is responsible for
multi-robot task allocation. Speci cally, given a task to perform, this module
should nd a suitable decomposition and assignment to the robotic team. This
process takes into account di erent constraints, such as resources and capabilities
needed to perform the task (e.g. take-a-picture needs a robot equipped with a
camera), the state of the robotic system and time constraints. Moreover, the
DMRTA is also to validate the feasibility of the operator's request (see [
Multi-robot Supervision and Adaptive Attentional Interface. A suitable interface
is need to lter and adapt the information presented to the busy genius through
di erent communication channels: head-set (audio), tablet (video), band
(vibrotactile). We assume that information ltering and adaptive presentation of data
are managed by an attentive interface modeled as a supervisory attentional
system that regulates contentions among multiple bottom-up stimuli (alerts,
periodic information, warnings, state changes, etc.) depending on the mission state
and the human constraints. This attentional regulation process [
Testing Scenario
We now illustrate an initial on-the- eld experimentation with the proposed
framework. In the experimental setting the human operator is located in a
real alpine scenario (i.e. Pordoi Pass in Alps: 2200 meters altitude) endowed
with his/her wearable devices to control two drones (called textitGreen and Red
wasp), each equipped with a standard camera. In this set-up human monitoring
and the adaptive interface were not enabled. The goal of the operator is to
inspect two di erent regions of the area depicted in Figure 3 (Left ). The operator
exploits the on-board camera of the drones to visualize real-time video streaming
and collect pictures of the terrain [
Acknowledgement. The research leading to these results has been supported by the SHERPA project, which has received funding from the European Research Council under Advanced Grant agreement number 600958.