Drone technology and drone control have recently advanced rapidly, to the point that consumer drones with impressive features and capabilities are commonplace [4]. Advanced sensors and improved control algorithms have made flying drones much simpler and many drone applications have become possible (e.g. aerial surveys, mapping, aerial movies or selfie-drones). As they are used for an increasing range of tasks, interacting with drones becomes more important. To enable these interactions in everyday life, it is essential to devise a natural language generation (NLG) setup that can flexibly process a variety of data collected by the drone and convey only the important information to the user. In this paper, we propose a neural generation model (or drone assistant) that verbalizes messages from sensor data records in order to perform a controlled handover to a human drone pilot (see Figure 1). Recent data-driven methods have achieved good performance on various NLG tasks [2, 3, 6]. However, most studies focus on surface descriptions of simple record sequences, e.g. attribute-value pairs of fixed or very limited schemas, such as E2E [7] and WikiBio [5]. In contrast, in our setup there is a large variety of data records, and the content selection task is substantially harder (only critical information, not all available information, should be mentioned at handover time). Moreover, it is desirable that the system generalizes well to diverse as well as unseen environments during its operation. To this end, we argue that it is necessary to leverage intermediate content representations to achieve faithful and controllable text generation in different environments. In this paper, these intermediate representations are generated using description logic (DL) ontologies [1]. This removes the burden of logical reasoning from the neural model and allows for more flexible and high quality utterances to be produced. We study this approach on a new dataset called DroneParrot that consists of 316 data records derived from real drone footage across 8 environments, such as
A tree is in path within 0.3 meter and the battery is low. Please resume human control. Drone Pilot
Autonomous Drone Urban and Ocean. Each sensor data record is combined with a natural language utterance and an intermediate representation generated by DL reasoning. The raw sensor data includes values such as wind speed, altitude, temperature, battery level, and information about nearby objects. To select the critical information for a handover, we define a DL ontology that derives additional information about the current situation. Several high-level concepts like RiskOfPhysicalDamage are defined to identify critical situations, using axioms like
∃near.Object u ∃environment.LowVisibility v RiskOfPhysicalDamage. For a data record (i.e. an ABox) that entails RiskOfPhysicalDamage(drone), we compute all ABox justifications of this assertion. We do not include TBox axioms in these justifications since there is no time to explain the whole reasoning process during a handover situation. The union of all obtained justifications is a subset of the original data record, which forms the intermediate representation in our dataset. This shorter representation is used by the neural model to generate a focused handover message. The reduced size of the intermediate representation enables learning with fewer training samples and better generalization across different environments.
The drone ontology and an example of two videos, annotated with the objects present on these records, the DL intermediate representations and the generated utterance are available publicly 3. Since the ontology and the queries are fixed we did not use a reasoner to perform query rewriting. Instead, all the rewritings were computed manually and hard-coded in Google Apps Script to make it available already in the process of time-consuming manual video annotations.
In our experiments, we observed that the size of the raw data records varies a lot between environments, with averages from 12.56 assertions for some environments to 168.85 assertions for others. However, after computing the justifications, on average only 1.68–3.26 assertions remain. Comparing the performance of the natural language generation with several baselines as well as state-of-the-art methods, we observe that all of them benefit from the additional pre-processing step of computing the intermediate representation – with differences up to 37.36 3 https://cloud.perspicuous-computing.science/s/rPqKAQoWXiq2QSQ Logic-Guided Utterance Generation for Drones (Extended Abstract) BLEU points (measuring how close the generated utterances are to the gold standard). A manual evaluation of 100 randomly selected samples also revealed an increased quality of the generated text as well as fewer errors in terms of missing important facts or hallucinated facts. To evaluate the scalability, we also exposed the system to environments not included in the training dataset. While the performance decreases in this setting, again including DL reasoning makes a big difference, as it reduces all data records to a similar form, i.e. it includes only the facts relevant for the handover.
In future work, we would like to make this approach more automated by also learning the TBox (classifying the situations of interest) from the raw data. Additionally, including TBox axioms in the intermediate representation may enable us to generate more detailed and naturalistic explanations for situations that are not time-critical.
Acknowledgements. This work was supported by the DFG grant 389792660 (TRR 248) (see https://perspicuous-computing.science).