Speech understanding is a fundamental feature for many applications focused on human-robot interaction. Although many techniques and several services for speech recognition and natural language understanding have been developed in the last years, speci c implementation and validation on domestic service robots have not been performed. In this paper, we describe the implementation and the results of a functional benchmark for speech understanding in service robotics that has been developed and tested in the context of di erent robot competitions: RoboCup@Home, RoCKIn@Home and within the European Robotics League on Service Robots. Di erent approaches used by the teams in the competitions are presented and the evaluation results obtained in the competitions are discussed.
Robots are expected to support human activities in everyday scenarios, by interacting with di erent kinds of users. In particular, domestic robots (i.e. robots operating in our homes) have already entered the market. Examples are cleaning robots, tele-presence robots and assistive robots for elderly care. In these contexts, the interaction with the user plays a key role. For this reason, the importance of enabling untrained users to interact with personal robots has increased. The goal of the research in Human-Robot Interaction (HRI) is to realize robotic systems that exhibit a natural and e ective interaction with users. Therefore, robots should be provided with sensory systems able to understand and replicate human communication, such as speech, gestures, voice intonation, pragmatic interpretation, and any other non-verbal interaction. Interaction, by de nition, requires communication. Humans usually communicate by means of natural language, which can be considered one of the most e ective vehicles of interaction. In this respect, the aim of Human-Robot Interaction in Natural Language is to develop robots that are able to solve human language references in the application context they belong. Competition is how humans improve themselves and push forward their abilities. Recently, the robotic research is facing the problem of sharing a common platform and common methodologies to quantitatively compare the di erent approaches of a particular task.
This paper addresses the problem of benchmarking robot speech understanding through scienti c competitions. In particular, we will focus on service robots operating in a home environment and on service robot competitions, i.e., RoboCup@Home4 and other ones derived by it. The goal of such a benchmark is to measure and evaluate the performances of the speech understanding capability of a general robotic platform, as well as to create a common workspace of discussion on the topic. More speci cally, we will describe the Functional Benchmark on Speech Understanding (FBM3), performed (in di erent forms) during the RoCKIn, RoboCup@Home, and European Robotics League Service Robots (ERL-SR) competitions, along with results and con gurations of the recent benchmark that took place in the ERL-SR Local Tournament5 in Peccioli on January 2017. Three teams participated: (i) SPQReL team, joint team between Sapienza University of Rome (Italy) and University of Lincoln (UK), (ii) b-it-bots@home team at Bonn-Rhein-Sieg University of Applied Sciences (Germany), and (iii) homer@uniKoblenz team at University of Koblenz and Landau (Germany).
The paper is structured as follows. In Section 2 we introduce the FBM3, along with the used performance metrics. Section 3 focuses on the di erent solutions compared in the competition and on the corresponding results. Finally, in Section 4 we analyze these results and draw some conclusions. 2
A Functional benchmark for speech understanding in
Service Robotics
This functional benchmark aims at evaluating the ability of a robot to
understand speech commands that a user gives in a home environment. A list of
commands are selected among the set of prede ned recognizable commands,
i.e., commands that the robot should be able to recognize within the tasks of
the competition or in similar situations. For this competition, the audio les
have been randomly extracted from the HuRIC corpus [
For the generation of the output, teams are provided with a knowledge base
(Frame Knowledge Base, FKB) containing a set of semantic frames, in line
with [
Goal: The nal position in the space to be occupied at the end of the motion action.
Path: The trajectory followed while performing the motion towards the Goal. { Searching : The action of inspecting an environment or a general location, with the aim of nding a speci c entity.
Theme: The entity (most of the time an object) to be searched during the searching action.
Ground: The environment or the general location in the space where to search for the Theme. { Taking : The action of removing an entity from one place, so that the entity is in robot possession.
Theme: The entity (typically an object) taken through the action. Source: The location occupied by the Theme before the action is performed and from which the Theme is removed. { Bringing : The action of changing the position of an entity in the space from a location to another.
Theme: The entity (typically an object), being carried during the bringing action.
Goal: The endpoint of the path along which the carrier (e.g. the robot - and thus the Theme) travels Source: The beginning of the path along which the carrier (e.g. the robot - and thus the Theme) travels
Composition of actions is also possible in the CFR, corresponding to more complex action as the Pick and place action, represented by a sequence of Taking frame followed by a Bringing frame (e.g. for the command \take the box and bring it to the kitchen"). 2.2
During the functional benchmark, di erent aspects of the speech understanding process will be assessed: 1. The Word Error Rate and the Speech Recognition Accuracy on the transcription of the user utterances, in order to evaluate the performance of the speech recognition process. While the former counts the errors made in transcribing the speech signal, in terms of words, the latter focuses on the transcription of the whole command. 2. For the generated CFR, the performance of the system is evaluated against the provided gold standard version of the CFR, that is conveniently paired with the transcription. Two di erent performances will be evaluated at this step. One measuring the ability of the system in recognizing the main action, called Action Classi cation (AC), and one related to the recognition of the full command, that is Full Command Recognition (FCR). AC is carried out in term of Precision, Recall and F-Measure, while FCR is measured through Accuracy. For the AC these measures are de ned as follow: { Precision: the percentage of correctly tagged frames among all the frames tagged by the system; { Recall : the percentage of correctly tagged frames with respect to all the gold standard frames; { F-Measure: the harmonic mean between Precision and Recall. For the FCR, the accuracy is the percentage of correctly interpreted commands.
The nal rank of the teams is evaluated considering the FCR. If this score will be the same for two or more teams, the WER will be used as penalty to evaluate the nal ranking. 3
Di erent approaches The task of robotic spoken commands understanding involves two main subtasks: the speech recognition step, in which a speech audio signal is processed and transcribed, and the natural language understanding phase, in which the actual interpretation of the sentence is extracted. Both of them can be carried out jointly, by following orthogonal approaches or relying on o -the-shelf tools. 3.1
The robustness of Automatic Speech Recognition in domain-speci c settings has
been addressed in several works. For example, in [
There are several works in which the combination of free-form ASR engines
and grammar based systems are exploited. In [
For understanding a robotic command, one of the simplest solution is to rely
on keywords or templates that aim at catching the semantic elements for the
targeted task [
The results have been obtained by spoken audio acquired during robot competitions, like RoboCup@Home, from di erent people and in di erent environmental conditions. These data thus contains all the typical noises a ecting robot competitions on service robots and real scenarios. Consequently, the results provide a realistic assessment of typical performance in this task. The presented benchmark for speech understanding in service robotics has been a useful tool for such an evaluation and is available for further comparisons.
The results presented in this paper outline two interesting observations. First, free-form automatic speech recognition systems signi cantly outperform grammar-based approaches. This is probably due to the di culty of speech grammars to cover all the possible linguistic phenomena, in terms of lexicon and syntactic rules. In fact, specially when dealing with spoken language, the sentences' structure is often unpredictable. Second, machine learning-based methods for command understanding seem to be more robust and reliable, as they are able to further generalize the lexicon and to cope with possible minor transcription errors. More speci cally, the combination between Google ASR and LU4R consistently provided for the best results in several di erent runs, that are encouraging for its deployment in real situations.
Although results are generally positive, we are still far away from a full understanding of the commands. Future work will include additional studies on this topic in order to further improve the performance and, to this end, we believe that robot competitions and benchmarks and the joint e ort of di erent research groups will signi cantly contribute to achieve this goal. We want to thank Nuance Communications for sponsoring academic licenses that have been used for the experiments.