This paper presents the Concept-Action Grammar Generator (CAGG) system, which is an open source framework for natural speechbased interaction to be used in the semantic analysis of sentences. The goal of our framework is two-fold: i) to deliver a formalism associating a context-aware semantics to the words in a sentence; ii) to provide a common framework to benchmark different implementations of evaluators with the aim of improving speech-based interaction. To this aim, CAGG includes a specifically designed formalism for grammars definition as well as parsing mechanisms to automatically create and evaluate their structure. CAGG implements a deterministic evaluator which provisional performance related to a number of natural queries is discussed in terms of correct semantic associations and computational time.
We make, as human beings, a large use of speech-based interaction in organising our daily activities, teaching to someone else how to do something, understanding other people’s interests, or asking for help in difficult situations. Therefore, a natural and context-aware speech-based interaction if fundamental for humanrobot interaction and collaboration as well.
There is no shortage of advanced speech-to-text systems, both as open source,
e.g., CMU Sphinx [
| <R2> {@T;@bb;}; <R1> : (hi | hello); <R2> : !optional(good{@!T;}) bye | !repeat(bye{@!!;},1,2); @«R1\hello», hello @«R1\hi», hi @«bb», good @«T\bb», bye x-or x-or or x-or and x-or
An open source framework to generate grammars and to easily prototype
and benchmark sentence evaluators is of the utmost importance. In this paper,
we propose the Concept-Action Grammar Generator (CAGG) framework, which
allows for: i) the context-aware semantic tagging of words in a sentence based on
a specified grammar, and ii) developing and validating different evaluators.
Semantic tagging is the approach we adopt to combine different knowledge sources,
e.g., the knowledge a robot may have about its environment and the context.
CAGG comes with a deterministic multi-thread implementation of a grammar
evaluator for a basic speech-based interaction, and it is available open source1.
2 System’s Architecture and Information Flow
The CAGG framework adopts an ad hoc language based on the Backus-Naur
Form syntax [
Figure 1 shows a simple example of the CAGG syntax, which defines a grammar having at least a solution containing the tags: R1 if the input is “hi” or “hello”, as well as «T\bb» if “bye” is given or «bb,T\bb» for “good bye”, whereas “bye bye” returns «». Accordingly, a tree representation is sketched in Figure 2.
Figure 3 shows the CAGG architecture. In order to build a grammar
evaluator, CAGG implements a parsing procedure (based on ANTLR [
ETs Grammar
Generator
Tag Commands List Activator
ST
Formatter Speech-to-Text
Evaluator
STIs generate an Abstract Syntax Tree (AST) for each rule. Each AST is transformed in a binary Expression Tree (ET) via syntax manipulation and, finally, the Grammar Generator module builds a unique Semantic Tree (ST) (e.g., Figure 2) by associating semantic tags to leaves (i.e., words) and all composition of rules to tree paths. The process can be supervised through a Graphical User Interface and stored in memory.
When audio information is given, a speech-to-text interface (in the current
system we use the Google Speech API [
CAGG proposes a framework for the analysis and evaluation of different approaches to grammar definition and evaluation, and provides a method to event-based action generation which employs the analysis of sentences, by means of the tags array. It allows designers and engineers to implement new Formatter or Evaluator modules based on the ST-based representation. 3
We discuss here a simplified grammar to analyse questions posed in English (Figure 4). Table 1 contains a few examples, the corresponding computation time and the resulting sentence analysis. We see how the same word can be
! start <MAIN>; <MAIN> : !optional(<WH>) <QUERY> !optional(<OBJ>{@Obj ;});
<WH> : what | where; <QUERY> : <AUX>{@? ;} <OBJ>{@Subj ;} <VERB>{@? ;}
| <VERB>{@? ;} <OBJ>{@Subj ;}; <BE> : am | are | is; <HAVE> : have | has; <BE-HAVE> : <HAVE> | <BE>; <AUX> : do | does | <HAVE>; <VERB> : <BE-HAVE> | seen; <OBJ> : <PRONOUN> | <THING>{@?;} | <CONCEPT>{@?;}; <THING> : robot | door; <CONCEPT> : name | surname; <PRONOUN> : I | you; Do you have a name? 0.463 sec «AUX\do;Subj\you;VERB\Have;Obj\CONCEPT\name» What have you seen? 0.238 sec «AUX\have;Subj\you;VERB\seen» Are you a robot? 1.161 sec «VERB\are;Subj\you;Obj\THING\robot» Are you a machine? 0.402 sec «VERB\are;Subj\you» You are smart. 1.520 sec «» interpreted according to different semantics on the basis of the question and where it is in the parsing tree. Moreover, CAGG can deal with unknown words, which can be identified for further assessments. It is possible to characterise the worst case computation time, corresponding to a full parsing. This is very important for real-world human-machine interaction scenarios.
It is noteworthy that the Evaluator module may find also other solutions based on the grammar’s definition. For example, in the grammar shown in Figure 1, the input “bye” could generate solutions without any semantic tags (in place of «bb»). This can happen when paths to such a term are verified. As an example it would not happen if the directive !repeat(bye{@!!;},2,2) were used, e.g., Figure 2. An idea to overcome this is to enable the Command Activator module to improve sentences analysis over time, where the improvement is related to the number of identified tags. 4
We propose the Concept-Action Grammar Generator (CAGG) system, an open
source framework for natural speech-based interaction providing an efficient
semantic sentences analysis. CAGG provides an open infrastructure to easily
prototype and implement grammar evaluators for benchmarking purposes. Current
work is focused on the integration with a system for spatial reasoning in
humanrobot interaction scenarios [