The paper discusses the development of hybrid chatbot system combining Q&A and knowledgebase approaches. The machine learning methods are used for answering question & answer pairs. The knowledgebase processing uses a metagraph model which requires hybridization of two approaches. The proposed approach of Q&A and knowledgebase hybridization is discussed.
Nowadays chatbot systems became more and more popular. It is almost
impossible to meet the company's website without chatbot. More and more e orts are
being made to improve the quality of chatbots [
The rst function (function I) is question answering (Q&A). The most
common case of Q&A is F.A.Q. (frequently asked questions). This is a list of question
and answer pairs. If the user asks a question similar to the question in the list,
then the corresponding answer from the list is returned to the user. There are
many services implementing this function, for example [
The function II that is very close to function I is answering common phrases.
A list of pairs is also used as the data model, di ering in that in this case, there
are no question and answer pairs but stimulus and response pairs. If the user
sends phrase similar to stimulus in the list, then the corresponding response
from the list is returned to the user. This function is implemented for example
in chatterbot framework [
The function III is dialog scripts. Usually, dialog script is created in the form
of the graph. The objective of chatbot is to route the user from the initial to the
terminal vertex of the graph. The vertex of the graph is associated with some
action. The action may be asking and answering user, calling web-service, etc.
Usually, the data from user input is parsed and stored in variables. These
variables are substituted for the answering templates. There are also many services
implementing this function, for example, owxo [
The function IV is knowledgebase answering. In this case, dialog scripts are created dynamically based on user questions and data stored in the knowledgebase. The history of questions and answers is stored in the session. As far as we know, information systems implementing this function are in experimental phase.
The chatbot implementing function IV is the most general case including functions I, II and III. Indeed, if we can create scripts dynamically, it is unnecessary to draw static dialog script graph as in function III. The functions I and II may also be replaced with dynamical generation on the basis of knowledgebase.
From the business user point of view, the function IV is the most signi cant. Most users can provide price lists as a knowledgebase for questions answering. But function I also should not be abandoned because most users can also provide exact Q&A pairs.
The main goal of this article is to discuss the development of hybrid chatbot system implementing functions I for Q&A pairs answering and function IV for price list answering. 2
First, the question is preprocessed using the question preprocessing module. The results of the preprocessing are passed to the Q&A and knowledgebase processing modules that are executed simultaneously. Then the hybridization and answer generation module integrates the answers of two processing modules and returns the nal answer to the user. Each module uses its own storage, and all modules have access to the user session storage. The storages are not shown in Fig. 1 in order not to confuse visualization of the gure.
The system is implemented in Python 3. The MongoDB database is used as storage backend. The mongoengine document-object mapper is used to simplify the work with MongoDB.
The principles of the operation of the modules will be discussed in details in the following sections of the article. 3
The question preprocessing module prepares question text for Q&A and
knowledgebase processing. The main library used in this module is NLTK. The Q&A
preprocessing includes:
{ tokenization;
{ lemmatization;
{ vectorization for TF-IDF and Doc2Vec [
The vectorized question is passed to the Q&A processing module.
The main preprocessing task for knowledgebase processing module is concept recognition. The idea of concept recognition is based on NLTK sentence parsing. If all the words in the concept are in the same subtree of the parse tree, then the concept is considered to be recognized in the sentence. The set of recognized concepts is passed to the knowledgebase processing module. 4
The input value of Q&A processing module is user question that is vectorized using TF-IDF and Doc2Vec processing. The module tries to nd the closest question in the Q&A pairs list. Experiments were carried out with TF-IDF and Doc2Vec processing and with cosine and Euclidean distances.
Technically, there are two approaches to solve this task. The rst one is to create a huge matrix T (N M ), where N is an amount of the questions in the database, M is the length of the dictionary. Since most of the questions won't contain that many words, the matrix will be too sparse. The advantage of this approach is its ability to use highly e cient linear algebra libraries. But there is also a very serious practical disadvantage: each time administrator adds new questions we need to check whether or not there are new words in them. And if there are, we need to rebuild the whole matrix.
The second approach is way agiler. Since we use the resulting vectors to compare between each others using metrics like cosine or Euclidean - we don't need to compare zeroes, so we don't need to vectorize all the questions in one space (with same dimensions). Actually we could create small distinct spaces for each question in the database. Each vector space would have a size of jqi [ qaj, where qi is a question from the database, and qa is the asked question. In this case, we don't need to create any matrices; we just compare asked question with the database questions one at a time. If the new questions are being added to the database, the only thing needed is to adjust IDF weights of the given words, no database transformation needed at all.
There are a lot of algorithm metrics which are matter for the practical
use of the system such as accuracy, train and evaluation time. Below one can
see experiments carried out with a goal to maximize accuracy and minimize
train/evaluation time. Corpora used in the experiments was obtained from the
Wikipedia Monolingual Corpora [
As one can see in Fig. 2 and 3 and Table 1 the optimal corpora size for given dataset is 10000 sentences. The later enlargement makes the algorithm slower and doesn't improve accuracy.
The results of the experiments showed that the best quality is achieved when using cosine distance and ensemble model with TF-IDF (weight is 0.8) and Doc2Vec (weight is 0.2).
The Q&A processing module returns the corresponding answer for the closest question and the classi er con dence level. 5 5.1
The main idea of this module is answering price list questions. We assume that typical representation for price list is a denormalized table. The rows of the table are objects, and the columns of the table are attributes.
The class of machine learning algorithms that is most relevant for this task is
recurrent neural networks (RNNs). LSTM [
It is especially important to note the DeepPavlov library [
Thus, although machine learning algorithms are well suited for Q&A processing, they are not suitable enough for price list questions answering.
Therefore, to solve price list questions answering task, we propose to use methods based on knowledge and its processing.
Though the denormalized table is a good logical data model for knowledge
representation, it is not good enough as a physical model because it does not
allow to store and process data e ectively. The rst experiments were carried
out with RDF storages. But the main problem of RDF model is poor N-ary
relationship representation [
A metagraph is a kind of complex graph model aimed for hierarchical graph description. The metagraph model simpli es N-ary relationship representation and complex contexts description.
According to [
The metagraph vertex is described by set of attributes: vi = fatrkg; vi 2 V , where vi { metagraph vertex and atrk { attribute.
The metagraph edge is described by set of attributes, the source and destination vertices (or metavertices): ei = hvS ; vE ; fatrkgi; ei 2 E, where ei { metagraph edge; vS { source vertex (metavertex) of the edge; vE { destination vertex (metavertex) of the edge; atrk { attribute.
The metagraph fragment is de ned as M Gf = fevj g; evj 2 (V [ E [ M V ), where M Gf { metagraph fragment; evj { an element that belongs to the union of vertices, edges and metavertices.
The metagraph metavertex: mvi = hfatrkg; M Gf i; mvi 2 M V , where mvi { metagraph metavertex; atrk { attribute, M Gf { metagraph fragment.
The example of metagraph representation is given in Fig. 4.
The example contains three metavertices: mv1, mv2, and mv3. Metavertex mv1 contains vertices v1, v2, v3 and connecting them edges e1, e2, e3. Metavertex mv2 contains vertices v4, v5 and connecting them edge e6. Edges e4, e5 are examples of edges connecting vertices v2{v4 and v3-v5 are contained in di erent metavertices mv1 and mv2. Edge e7 is an example of edge connecting metavertices mv1 and mv2. Edge e8 is an example of the edge connecting vertex v2 and metavertex mv2. Metavertex mv3 contains metavertex mv2, vertices v2, v3 and edge e2 from metavertex mv1 and also edges e4; e5; e8 showing emergent nature of metagraph structure.
The metagraph itself is not more than a complex data structure. To process and transform metagraph data the metagraph agents are used.
The metagraph agent uses rule-based approach: agR = hM G; R; AGST i; R = frig; ri : M Gj ! OP MG; where agR { metagraph rule agent; M G { working metagraph, a metagraph on the basis of which the rules of agent are performed; R { set of rules ri; AGST { start condition (metagraph fragment for start rule check or start rule); M Gj { a metagraph fragment on the basis of which the rule is performed; OP MG { set of actions performed on metagraph.
The antecedent of a rule is a condition over metagraph fragment, the consequent of rule is a set of actions performed on metagraph. Rules can be divided into open and closed. If the agent contains only open rules it is called open agent. If the agent contains only closed rules it is called closed agent.
The consequent of an open rule is not permitted to change metagraph fragment occurring in rule antecedent. In this case, the input and output metagraph fragments may be separated. The open rule is similar to the template that generates the output metagraph based on the input metagraph.
The consequent of closed rule is permitted to change metagraph fragment occurring in rule antecedent. The metagraph fragment changing in rule consequent cause to trigger the antecedents of other rules bound to the same metagraph fragment. But incorrectly designed closed rules system can cause an in nite loop of metagraph rule agent.
Thus metagraph agent can generate the output metagraph based on the input metagraph (using open rules) or can modify the single metagraph (using closed rules).
The distinguishing feature of the metagraph agent is its homoiconicity which means that it can be a data structure for itself. This is due to the fact that according to de nition metagraph agent may be represented as a set of metagraph fragments and this set can be combined in a single metagraph. Thus, the metagraph agent can change the structure of other metagraph agents.
The example of metagraph agent is shown in Fig. 5. The metagraph agent \metagraph rule agent 1" is represented as metagraph metavertex. According to de nition, it is bound to the working metagraph M G1, which is shown with edge e4.
The metagraph agent description contains inner metavertices corresponds to agent rules (rule 1 . . . rule N). Each rule metavertex contains antecedent and consequent inner vertices. In given example mv2 metavertex bound with antecedent which is shown with edge e2 and mv3 metavertex bound with consequent which is shown with edge e3. Antecedent conditions and consequent actions are de ned in form of attributes bound to antecedent and consequent corresponding vertices.
The start condition is given in form of attribute \start=true". If the start condition is de ned as a start metagraph fragment then the edge bound start metagraph fragment to agent metavertex (edge e1 in given example) is annotated with attribute \start=true". If the start condition is de ned as a start rule then the rule metavertex is annotated with attribute \start=true" (rule 1 in given example), Fig. 5 shows both cases corresponding to the start metagraph fragment and to the start rule.
Thus, the metagraph approach is a good basis for complex graph structures and their transformations representation. Now we can describe the knowledgebase in terms of metagraph approach. 5.3
The knowledgebase representation using the metagraph approach The knowledgebase fragment in the form of metagraph is represented in Fig. 6. This example focuses on sales of o ce supplies. The price list is a table, each line of which contains the goods, and each column contains the characteristic of the goods.
The knowledgebase processing module solves two main tasks: questions answering and an active dialogue with the user. Each solution is represented using metagraph rule agent.
The questions answering agent uses the following rules: { If the user's question contains concepts that match the data in the table, the corresponding column name is searched for each of these concepts, and the pair "column name: data" is saved in the user's session. If the session contained a pair corresponding to a column name, then this pair is deleted before insertion of a new pair. { If the user's question contains concepts that correspond to the column name, a set of values for that column are displayed based on the session data. For example, if the user's question contains the concept of "products", the system will display a list of products in response: "red pencil", "green pencil". But if the session contains a pair fproduct: green pencilg, the user will be given a message that he orders green pencil.
An active dialogue agent is used to help the user choose a product. Then the main goal of active dialogue is to lter table in order to choose a single row. An active dialogue agent uses the following rules: { After processing the user's question and saving the recognized concepts in the session, table rows are ltered based on the session data, and the number of ltered rows is estimated. { If the single row is found, the goal is considered to be completed, and the user is shown information about target columns of the single row. { If more than one row is found, then the system implements an active dialogue with the user and tries to complete the goal by asking auxiliary questions.
For this, a column with minimal diversity is de ned, that is, it determines which column contains the minimum number of possible concepts, taking into account the lter based on the data of the current session. The active dialog will continue until the goal is completed, that is until the single row of the table is found. { If the user speci es an incompatible set of concepts during the dialog process that leads to zero rows as a result of the ltering, a message is displayed indicating that no data has been found for incompatible set of session parameters and the user session is cleared.
The described process can be considered as a special kind of forward chaining based on table data. The usage of metagraph approach for knowledgebase representation allows using metavertices both as data elements used for question answering and information elements used for active dialog implementation.
It should be noted that it is not correct to talk about the accuracy of the knowledgebase processing module, because it does not use machine learning algorithms. The correctness of the module operation depends entirely on the accuracy of concept recognition. 6
Actually, the Q&A and knowledgebase processing modules are two di erent chatbot systems, and we need to implement their seamless integration. The main idea of integration is that the Q&A processing module is aimed to answer general purpose question while the knowledgebase processing module is aimed to answer speci c questions based on knowledgebase data.
After the preprocessing phase, the Q&A and knowledgebase processing modules are called simultaneously in order to reduce the response time of the system. However, when analyzing module answers, the Q&A module responsible for general purpose questions has a higher priority.
If the text of the question is found in Q&A pairs with a high degree of conformity, then we recognize that question is general purpose and the corresponding answer from the Q&A processing module is returned to the user. A high degree of conformity means that the con dence level of the Q&A classi er exceeds the threshold. The value of con dence level threshold based on the results of experiments is about 85%.
If the con dence level of the Q&A classi er does not exceed the threshold, then we apologize that question is knowledgebase speci c, and the answer from the knowledgebase processing module should be returned to the user. If the set of recognized concepts passed to the knowledgebase processing module is empty or the knowledgebase processing module returned empty answer then standard answer "Please, clarify your question" is returned to the user. 7
Comment: The concept 'green pencil' is recognized and stored in the session because words 'green' and 'pencil' are in the same subtree of the NLTK parse tree. The knowledgebase contains several rows with this concept. The system tries to o er the brand for the selected product.
Bot: We have green pencils of the following brands: 'Crayola' and 'Kores'. User: What are the advantages of Crayola pencils?
Comment: The concept 'Crayola' is recognized. But the con dence level of the Q&A classi er exceeds the threshold, and the corresponding answer from the Q&A processing module is returned to the user.
Bot: The advantages of Crayola pencils ... (answer from Q&A). User: Then I want to buy a Crayola pencil.
Comment: The concept 'Crayola' is recognized and stored in the session. The con dence level of the Q&A classi er does not exceed the threshold. The knowledgebase contains only one row corresponding to the session concepts. Then the value of the target column (price) is returned to the user.
Bot: It costs 120 price units. 8
The main result of our development is the chatbot system that integrates Q&A and knowledgebase processing functions.
The developed system can be considered as an automated consultant who provides information about the goods, answers detailed questions about the characteristics of the goods and helps to choose the goods.
The Q&A processing module uses machine learning methods while the knowledgebase processing module uses metagraph approach.
The Q&A processing module nding the closest question in the Q&A pairs list. The best quality is achieved when using cosine distance and ensemble model with TF-IDF and Doc2Vec.
The knowledgebase processing module uses metagraph model for a denormalized table data storage. The metagraph rule agents are used for knowledgebase questions answering and an active dialogue with the user.
As the future work, we plan to improve the quality of Q&A processing and add more question answering features to the knowledgebase processing module.