English. We provide our view on the components needed for both the development and further improvement of robust and effective chatbots. We focus on why Pragmatics is important in developing next generation chatbots by bringing a few generalizable examples. We report our current experience on the design and implementation of a task-oriented textual chatbot for a closed-domain Question Answering system, which tackles problems in Pragmatics.
Why Pragmatics Matters in Chatbots
Chatbot, chatterbot, natural-language interface,
dialogue system are some of the terms used to
refer to softwares that aim to carry on conversations
with humans
Chatbots and Intent Understanding. The
goal of an intelligent chatbot is to understand the
user’s intent
Natural Language Processing (NLP): much of the intelligence needed to understand human intent lies in the processing of human language. Hence, the development and improvement of NLP algorithms is a necessary prerequisite for the creation of intelligent chatbots.
Machine Learning (ML): chatbot design should
rely on ML for learning and automatically
consolidating NLP rules by means of observation
of past experience - i.e., past conversations and
their outcomes
Context and State Awareness: depending on the
purpose of the chatbot, the component responsible
for the managing of the conversation (Dialogue
Manager System - DMS) should take into account
both context and states variables
Natural Language Generation (NLG): NLG
concerns what information and in what form it
should be delivered (Breen, 2014). Dealing with
"real" conversation requires being both proactive
(e.g. suggest the best option; drive along the
compilation of a form; remind planned activities;
...)
We argue here that, in addition to the above
mentioned, moving from Semantics to Pragmatics
plays a crucial role in building chatbots. This
is because a lot of the knowledge human beings
share during a conversation gets constructed along
the conversation itself
In this interaction, the sentence “I would like to make a transfer” instantiates two different intents: informative intent, at first (H is looking for information about the transfer service); follow up intent, then (once H is satisfied with transfer service conditions, he/she wants to proceed with the transfer). Such pragmatic disambiguation involves taking knowledge from the conversational context into account, which is one of the most difficult tasks for a chatbot. We report below how we deal with this task in our task-oriented closeddomain chatbot. 2 Intent Understanding in Practice Understanding intents implies handling both semantic meaning and pragmatic meaning. Roughly speaking, while semantics concerns the meaning of a sentence from the linguistic point of view, Pragmatics concerns the interpretation of the same sentence depending on extralinguistic knowledge (Grice, 1975). As mentioned before, a sentence can be ambiguous from the intent point of view. As for classifying intents, there seems to be no comprehensive literature about it, yet - not from the chatbot perspective, at least. However, based on our business experience, we would arrange intents as follows:
Informative Intent: the user is looking for information; e.g. Question and Answer (QA), FAQ browsing typically instantiate this intent.
Follow Up Intent: as in regular conversations,
the user wants to "do things with words"
Dialogic Intent: the user uses discourse markers to connect, organize, manage the conversation; e.g.: greetings, farewells, turns markers, ...
Regular expressions, pattern matching and keyword recognition typically are not enough to achieve real intent understanding. This is because the more the interaction is human-like, the more complicated it becomes to figure out what the human really wants. Among business intent, real life cases we faced are: Onboarding, Question Answering and Education. In our applications, we break down the understanding process into subtasks. Namely: intent classification (e.g. “booking a flight”); slot filling, i.e. enriching the intent with more detailed information (such as “destination” and “departure time”); context modeling, i.e. keeping track of context to get to the correct meaning (“time” might refer to “flight departure time”, “flight arrival time”, “dinner time”, etc...).
Our task-oriented closed-domain financial textual
chatbot, Financial QA Chatbot, aims to provide
users with answers concerning banks and
insurances, through a conversation in Italian. The type
of answers that a user can obtain are similar to the
ones found on a financial platform website1: this
portal provides a search engine and FAQ section
to satisfy the information need. Therefore, it is
mainly a QA chatbot, although some additional
follow up actions are available on top of providing
an answer to questions, such as redirecting to
specific websites or services. Financial QA is
provided with a proprietary scoring algorithm to
match the current user’s questions to answers in
a database A. In line with previous work
Dialogue Scenario: a QA session consists of actions that can be performed by the user or by the automated system, according to Dialogue Management logic.
User actions: greet, quit, ask a question q, acknowledge the previous utterance, ask for help/suggestions, browse the navigation menu. System actions: greet, quit, present answer a, acknowledge the previous utterance, ask for clarifications, propose a follow up (question/action), reprimand for using swearwords, suggest questions, present or hide the navigation menu.
User’s action classification: each user’s
utterance is classified into one of five action classes:
greet, quit, ask a question, acknowledge, ask
for help. This is accomplished using predefined
dictionaries and automatic classifiers, which also
consider discourse markers and disfluencies.
Although there is promising work done on dialog
1gooruf.com
2www.celi.it
act detection with multi-level information
Dialogue Management: the conversation proceeds along these logics.
1. An initial greeting (greet action), a request for help (ask for help) or a direct question q (ask a question) from the user. 2. The system, if asked for help, presents the user with a navigation menu, based on current context and on the given hierarchical classification of contexts or topics (see Context Management below). This menu can be browsed until a terminal node in the classification is reached, and, at that point, a predefined set of questions related to that topic is suggested. The user can select a question q from that list. 3. q is analyzed to detect wh-type (Huang et al., 2008) and whether it is elliptic or anaphoric. This information is passed along with q and the current context to the subsequent QA component. 4. The QA component searches for matches of the query according to the QA Algorithm. Each matching answer ak is accompanied by a relevance score rk, rk 2 (0; 1]. If at least one match has relevance more than a fixed threshold T , only the best match (highest relevance) is returned. Otherwise, up to the top Nr highest results are returned by the QA component. In Financial QA’s basic settings, T = 0:75 and Nr = 5. 5. The QA component results are processed: they can be a single answer or, because of low relevance scores, a list of answers. If a single answer is provided by the QA component, it is returned to the user (answer action). In the case of a list, the user is asked for clarifications, and a single answer is selected based on her additional input (ask for clarifications action, then answer action). After an answer is provided to the user, context is updated accordingly. 6. The system inquires whether the user is interested in a follow up session; if this is the case, the user can enter a question again. Else, the system acknowledges.
Ask me a question or choose one of the following topics: 7. Whenever the user wants to terminate the interaction, a final greeting is exchanged (quit action).
Context Management: intuitively, all the answers a in the knowledge base are grouped in disjoint topics of maximum granularity, which are then organized in a hierarchical structure, used to model context in this QA task.
Managing topic hierarchies can improve performance in a query matching system (Domingues et al., 2014). Formally, context elements are topics of conversation belonging to the finite set C = fC1; : : : ; CN g. Topics are arranged in a hierarchical classification structure, which can be represented as a tree T = (C; E), where C is the set of nodes. Edges E express the "Ci has subclass Cj " relation. A context X is, in general, an arbitrarily ordered sequence of topics.
In our current implementation of Financial QA, we support only contexts of length 1, therefore the context Xs at step s of the conversation is the position Cs in T. We assume all interactions start at the root node C0. Xs is meant to represent the current topic of conversation at step s, according to the last answer provided or the latest click on the navigation menu. By supporting contexts of length > 1, it is also possible to keep track of previous topics of conversation.
Each node Ci has a corresponding nonempty set of topic-related keywords Wi.
An important distinction is drawn between terminal nodes C and nonterminal nodes C n C . Each terminal node Cj has a (potentially empty) set of answers Aj corresponding to it. All the Aj sets are disjointed. Let A be the set of all answers: A = [j21;:::;!Aj .
In our current implementation, there are 35 classification nodes arranged on 3 levels, 25 of them are terminal ones; the number of answers in the knowledge base is 440, and growing In the Financial QA chatbot, two types of moves between contexts in C are allowed: 1. To children nodes or root node: using the interactive navigation menu. Context is updated automatically according to the user’s selection.
Example: You are in the “People” section. (a) members (b) influencers (c) contact us (d) return to main menu 2. To any terminal node: after answer ak is provided to the user by the system, new context becomes Cj , where Aj contains ak. Example: after providing the answer COST_OF_GOORUF = "Gooruf is free, only Premium Providers are required to pay", context is changed to ROOT ! services ! info_about_gooruf, info_about_gooruf being the terminal topic containing answer COST_OF_GOORUF.
QA Algorithm Design: question q, its wh-type h, and its current context Cs are passed to the algorithm. Keywords wq are extracted from q. If q is anaphoric or elliptic, the algorithm evaluates whether to expand Wq to w_ q by using keywords Ws corresponding to Cs. The final representation of q is:
R(q) = (Cs; h; w_ q): Answers a 2 A are described by the following feature vector:
F (a) = (Ca ; Ha; Wa) where Ca is the classification terminal node corresponding to a, Wa the corresponding keywords and Ha a set of related wh-type (for example a user might inquire about Gooruf by referring to it as a what or a who).
Relevance rk for each answer ak is computed, by considering the classification structure T as well, therefore:
rk(q) = (R(q); F (ak); T ):
To compute , scores are calculated separately
by comparing contexts (using proximity in T
between Cs and Ca in T ), wh-types (h and Ha)
and a dense semantic representation of keywords
(w_ q and Wa) obtained using a Word2Vec model
for Italian language
Example: we provide below an example of Financial QA interaction which shows how managing hierarchical context helps in accomplishing the question answering task.
A subtree taxes of T models Italian taxes-related topics: taxes ! city_taxes taxes ! income_taxes – TARI – TASI – IRPEF – IRAP Individual taxes are represented as terminal nodes in T . Let Ctaxes = fIRPEF, IRAP, TARI, TASIg. Each t 2 Ctaxes has associated answers: "HOW_TO_PAY t", "WHERE_TO_PAY t", "AMOUNT_TO_PAY t".
The interaction could go as follows: H: How can I pay city taxes? QA Algorithm detects wh-type how, keywords matching the city_taxes node, finds two relevant answers in children nodes and Chatbot asks for clarification.
C: Did you mean TARI or TASI? H: the second one Chatbot presents answer "HOW_TO_PAY TASI" Context is now TASI.
H: and where can I pay it? QA Algorithm detects wh-type where and completes the question with context knowledge. Chatbot finds a single relevant answer and presents answer "WHERE_TO_PAY TASI". H: how much IRPEF should I pay? Chatbot presents "AMOUNT_TO_PAY IRPEF". Context is now IRPEF.
H: where can I pay it? Chatbot presents "WHERE_TO_PAY IRPEF". 4
We are currently in the process of evaluating
Financial QA according to a framework based
on PARADISE
NLP is crucial for the development of robust chatbots; since extra-linguistic elements are potentially very important in intent understanding, moving from semantics to pragmatics is a necessary step to develop next-generation chatbots. We have shown how Dialogue Management can support a more robust handling of context, at least in closeddomain QA tasks.
Further work is required to handle more business
cases and a broader definition of context, such as
history of activities conducted by the same user,
which can be especially useful in chatbots with
recommender functions
Antoine Bordes and Jason
End-to-End Goal-Oriented arXiv:1605.07683 [cs.CL].
Weston.
Dialog.
Learning 2016.
Benatallah Boualem, Fabio Casati, and Farouk Toumani. 2004. Web service conversation modeling: A cornerstone for e-business automation. IEEE Internet computing, 8(1):46–54.
Andrew Breen 2014. Creating Expressive TTS Voices for Conversation Agent Applications. International Conference on Speech and Computer. Springer. Marcos Aurélio Domingues, Marcelo Garcia Manzato, Ricardo Marcondes Marcacini, Camila Vaccari Sundermann and Solange Oliveira Rezende. 2014. Using contextual information from topic hierarchies to improve context-aware recommender systems. Pattern Recognition (ICPR), 2014 22nd International Conference on. IEEE, 3606–3611.
Paul Grice. 1975. Logic and conversation. New York
Academic Press, 41–58.
Zhiheng Huang, Marcus Thint and Zengchang Qin. 2008. Question Classification Using Head Words and Their Hypernyms. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
James Lester, Karl Branting and Bradford Mott. 2004.
Conversational agents. The Practical Handbook of Internet Computing, 220–240. of the eighth conference on European chapter of the Association for Computational Linguistics. Association for Computational Linguistics.