Social robots and artificial agents should be able to interact with the user in the most natural way possible. This work describes the basic principles of a conversation system designed for social robots and artificial agents, which relies on knowledge encoded in the form of an Ontology. Given the knowledge-driven approach, the possibility of expanding the Ontology in run-time, during the verbal interaction with the users is of the utmost importance: this paper also deals with the implementation of a system for the run-time expansion of the knowledge base, thanks to a crowdsourcing approach.
Achieving a natural interaction between a human and a robot is a very complex task. A list of ten
desired features that a conversational robot should have has been presented in [
The recent EU-Japan project CARESSES has dealt with some of these aspects [
However, although significant progress has been achieved during the last years, assistive robots and
chatbots still have many limitations. Some of the most common limitations are (i) failing to answer,
(ii) not understanding the local language of the user, (iii) not giving the proper answer if there is
some spelling mistake or some slang, (iv) having a limited knowledge, which can result in a repetitive
conversation, and (v) not being coherent when answering to what the user says [
The work proposes an approach that mainly deals with (iv), relying on the framework for autonomous conversation developed within the CARESSES project but suggesting a crowdsourcing approach for adding knowledge as a consequence of the interaction with the users.
Usually, knowledge-grounded conversation frameworks generate appropriate responses that reflect
the acquired knowledge by relying on data-driven conversation models [
The Dialogue Tree is built starting from the Ontology structure, and the relation between topics is borrowed from the structure of the Ontology: specifically, Object Properties, Data Properties, and the hierarchical relationships among instances, are analyzed to define the branches of the Dialogue Tree.
Based on the Dialogue Tree, the policies for knowledge-driven conversation can be briefly
summarized as follows [
The possibility of acquiring knowledge in a systematic manner by relying on networked interactions
with human users has been recently explored in diferent domains. For example, some museum
collections have used a “social tagging” approach to enhance curatorial documentation [
This work proposes the usage of crowdsourcing mechanisms for a run-time expansion of the knowledge base used for building the Dialogue Tree. A three-step approach has been implemented: 1. Recognition of relevant concepts in the user’s sentence; 2. Insertion of the concepts in the Ontology; 3. Validation of the concepts.
Concerning 1, a Dialogflow Agent [
To collect appropriate training phrases for each Intent in such a way that Dialogflow can correctly perform the Intent matching, a vocal questionnaire has been created, where participants are required to answer, using their microphone, to 20 questions (5 for each Intent).
Regarding 2, diferent techniques for the insertion of new concepts (and related sentences) in the right place of the Ontology have been implemented. Such techniques involve again the usage of NLP tools for detecting the category of the user’s sentence and the type of the recognized entity, and they are currently under evaluation. Finally, a procedure for validating the concepts inserted in run-time has also been developed (3). It is based on a revision process that indirectly asks users to independently revise others’ information, to reach a consensual version. This peer-review approach, complemented by an external moderation, is also being evaluated.
As mentioned before, at the actual state, experimental tests with users have been focused on step 1. of the process, i.e., the recognition of relevant concepts in the user’s sentence. A total of 30 participants have been recruited for the experiments: 20 questionnaires have been used to train the agent, while the answers of the remaining 10 questionnaires have been set aside to use them to test the Agent. To validate the approach, two independent taggers have tagged the 20 questionnaires of the training set, while a diferent person has tagged the evaluation set. It is worth saying that the Agent has been trained with pieces of the sentences (split by “and"). For this reason, to assess the performances of the Agent, the test answers have been split according to the same rules used for the training answers. The splitting also increases the probability of extracting at least one meaningful concept in the whole answer, which is what matters to expand the knowledge base.
Each piece of the test answers has been classified as: • True Positive (TP): if Dialogflow correctly recognized the concept that was manually tagged; • False Positive (FP): if Dialogflow recognized something that was not tagged; • True Negative (TN): if Dialogflow did not recognize anything and nothing was tagged; • False Negative (FN): if Dialogflow did not recognize anything but something was tagged.
The results of the classification of the pieces of each answer are summarized in the Confusion Matrix shown in Figure 1 (left). By looking at the tables, it is immediately clear that almost every time Dialogflow recognized something it was a tagged concept. However, in many cases, it did not recognize what had been tagged: as it will be clear in the following, this is not an issue if we reason in terms of answers and not in terms of pieces of answers. Starting from the Confusion Matrix, the most common parameters for a binary classifier have been computed and reported in Figure 1 (right). The same analysis has been carried out (Figure 2) considering the sentence as a whole, and by using this rationale, which allows to analyze if at least one concept has been correctly extracted from the whole sentence: • If the answer contains at least one sentence classified as TP, the answer is considered as TP; (a) Confusion Matrix (b) Parameters table
From the analysis of the collected data, it may be observed that, working on the pieces of each answer, the system achieves a high Precision, which means that almost every recognized concept would have also been tagged manually. On the other side, the Accuracy (how often is the classifier correct?), the Sensitivity (when something is tagged, how often does it recognizes it?), and the Specificity (when something is not tagged, how often does it not recognize anything?) may be improved. However, when considering the whole answers to assess the capability of the system to extract at least one meaningful concept in each of them, all performance indicators are greater than 0.8, except the Speciifcity, due to a high number of FP. These results give positive insights of the reliability of the proposed approach aimed at recognizing relevant concepts in the user’s sentence for a run-time expansion of the knowledge-grounded conversation system.