Natural language processing (NLP) is the branch of Articial Intelligence (AI) studies that will shape the future of computing and Human Computer Interaction (HCI). Operational complexity of a conversational intelligent agent (Chat-bot) stems from human-related, linguistic and computational aspects. In this study, we de ne an architectural description of a chat-bot, address the NLP problems which needs to be solved and provide our proposed solutions for a functional conversational intelligent agent. The importance of Bayesian Statistics and Data mining of domain speci c text and expected expansion areas are discussed for future research in the conversational AI development.
1 Softtech A.S.
Ar-Ge Merkezi Tuzla Piri Reis Cad. 62, 34947 I_stanbul, Turkiye
2 Sabanc Universitesi Orta Mahalle, 34956 Tuzla, I_stanbul, Turkiye
cagri.calpur@softtech.com.tr O zet. Dogal dil isleme, yapay zeka cal smalar n n insan bilgisayar etkilesiminin gelecegini sekillendirecek bir alt dal d r. Sohbet robotu sistemlerinin operasyonel karmas kl g insan etkilesiminin cesitliliginden kaynaklanan dilbilimsel ve islemsel zorluklara dayan r. Bu cal smada, bir sohbet robotu sisteminin mimari tan m n yap yoruz, alana ozgu bir sistemin gerceklenebilmesi icin cozulmesi gereken dogal dil isleme problemlerinden bahsediyoruz ve islevsel bir sohbet robotu icin cozum onerilerimizi bildiriyoruz. Alana ozgu dialog bazl ak ll ajanlar n gelistirilmesinde bayesci istatistiksel yontemlerin ve veri madenciligi tekniklerinin oneminden ve gelecek arast rma alanlar ndan bahsediyoruz.
Anahtar Kelimeler: Tan mlay c I_statistik Makine Ogrenmesi Metin Madenciligi Ontoloji Sartl Rastgele Alanlar Dialog Bazl Ak ll Ajanlar Sohbet robotlar Dogal Dil I_sleme.
The imitation of human intuition for language understanding is the motivation behind NLP studies in the industry and academia. Each year new iterations of conversational arti cial intelligence agents are entering the Loebner Prize competition. Creation of a generalized intelligent agent which can handle any conversational interaction is infeasible in todays maturity level of NLP research, but a domain speci c solution is certainly a reachable target.
Conversational AI (CAI) is replacing current user interfaces and search tools [2,3]. In the coming years, the human computer interaction will be based on text and speech based conversational agents. Such a paradigm shift is limited by the current ability of algorithms powering AI, in this paper we brie y introduce the NLP challenges and in the later sections provide our novel contribution to some of the challenges for our conversational AI platform.
The core of a domain speci c conversational AI is a combination of multidisciplinary research domain. The platform needs a solid software engineering foundation for core components and services. The amount of data and statistical computation requires implementation of e cient algorithms and intuitive design. On the other hand the NLP core is a multi-level process, which starts from acquisition of user input, preprocessing of acquired data, classi cation and understanding according to the domain concept and producing the response. 2
Natural language processing and understanding is one of the more complex contents in AI research. Human beings are naturally inclined to learn sounds, words, patterns and their associations starting from early infancy to their adulthood. The cognitive abilities of a human is not comparable to a machine which needs to be taught how to interpret language. Furthermore this e ort of teaching language to machines does not evolve into a thought process for an automaton (machine). The cognitive abilities, intuitive understanding and overwhelmingly fast relational inference capabilities of the human brain enables a human being to understand language even if the speech or text that is being processed is erroneous.
A Corpus is a large collection of text or speech data, which is used as a basis for NLP studies. Corpora are usually compiled with manual e ort, which can be a dictionary of words, sample sentences or linguistically annotated (morphological decomposition, part-of-speech, named entity information, etc.) text. 2.1
Natural language understanding (NLU) starts with unstructured data. At the start of the NLU process the text is a combination of alphanumeric and punctuation characters. Rigorous preprocessing is required to normalize the data [8]. The unstructured data is converted to a standardized format so that NLP algorithms can be developed and the data could be analyzed.
Sentence and Word Decomposition Sentence detection and decomposing a paragraph or unstructured corpus into sentences is important for sequentially processing the corpus to understand the narrative of the text. Although regular structure of a sentence includes punctuation to stop a sentence, fringe cases exist for compound sentences and abbreviations.
Tokenization is separating a sentence into its words with respect to white space characters and punctuation. Decomposing a text into word level is a milestone for applying more advanced NLP algorithms, but a word is not the smallest atomic unit for NLP. A word can be dissected into morphemes.
Stem is the prime component of a word, which is the base meaning. a xes are annexed to stems to modify the meaning of the word or form completely di erent words.
Turkish is a member of Altaic language family and a series of su xes are added to stems to produce words, which is called Agglutination . Lemmatization is used for determining words that have the same root, despite the mutations and di erences in words that originate from the same root. Agglutination is one of the causes of mutation in words, where consonant devoicing occurs when a vowel is appended to the end of a word which ends with an obstruent consonant. Stemming is a crude form of lemmatization where the stem of the word is extracted but it may not be the actual root of the word under observation. This case is one of the many examples of ambiguity in NLP. Error Correction and Spell Checking A conversational AI agent's main task is interaction with a human being, therefore the input to the agent is prone to human error. In case of text based communication, there will be misspellings in the form of insertions, deletions, substitutions or transposition of adjacent characters. Damerau [4] claims that %80 of the time the error is one the four error types previously mentioned. Error correction is very important, since errors in the processed text will disrupt the rest of the NLP e ort. 2.2
Regular expressions (RE) provide a solid tool for completing various NLP tasks. The normalization process becomes much easier and most of the basic NLP tasks can be performed with the help of REs. Unstructured text includes data in various forms and by employing REs, automation of search for recurring data such as numbers, dates or text in special formats becomes much easier. The complexity caused by highly varying data such as numbers can be reduced by substituting numbers with class labels such as decimal, date-US-format. This kind of substitution is a good way of feature set reduction for NLP (Table 2). REs can also be used for simple named entity extraction or part of speech tagging duties. In Turkish there are ve types for parts of speech (POS), which are noun, verb, adjective, adverb and conjunction. POS tagging sentences gives us precious information about the words, which is useful for morphological parsing of a word or named entity recognition by examining locations and combinations of POS in a sentence (Table 3). Verbs are generally the terminating POS in a sentence, while the starting POS is usually the subject(noun). Lemmatization and a x determination is useful for POS tagging, since the agglutination property of the language makes conversions such as noun-to-verb, verb-to-noun, etc. possible. 2.4
A domain speci c conversational AI is required to distinguish between regular words and named entities. The scope of named entities changes according to application domain, but usually people, places, organizations, time, date, numerical values, currency can be considered as named entities. The conversation evolves around such special words since named entities are the anchors for context tracking in progressive conversation. Time, space, quantity and related objects and people have attributes and modi ers, so extracted named entities constitute the conceptual existence of the objects in the conversation. Intent is the key point or ultimate goal of a sentence in a conversation. A CAI must implement a learning method for determining the intent of a sentence, in order to perform necessary actions and generate appropriate response. Actions and related data are the named entities extracted from the text. The information gathered about the intent is sorted into their respective information slots for completing the required action or generating appropriate response (Table 4). The domain speci c conversational AI project supports the digital and arti cial intelligence transition vision of Softtech A.S. The NLP research is implemented as a chat-bot and the solution is deployed at a number of in-house and customer systems as the intelligent information retrieval and query interface. The CAI system has multi-layered architecture (Figure 1). The Classi cation Component is used for both classi cation of the input text and training the classi cation system. Model versioning is implemented and all trained classi cation models are stored in the data store, so the system adapts to newly available data and roll back to a previous accurate model is possible. Domain speci c corpora are also stored in the data store, which is used to initialize the NLP Engine component during start-up of the system. NLP Engine implements various algorithms to provide Named-Entity-Recognition, Spell Checking and Intent Slot Filling.
Classification Component Activated Model Multi-Layer Perceptron (MLP) technique is utilized for training and prediction. Training procedure includes 10-fold cross-validation with 80-20 distribution of training and test sets. Con guration of the MLP is 2000 iterations with 0.05 learning rate and hyperbolic tangent activation function. Our test domains included 323 classes and the F1-score for the accuracy is %93 (Table 5). A domain speci c conversational AI requires expert guidance for the compilation of the classi cation classes and corpora. The data is usually created by domain experts. Another method is employing market research professionals to gather data about the domain. For our systems, we use a combination of domain expert generated data and data generated by a market research company. The company owns a mobile crowd sourcing platform and queried 500 people about the domain with this mobile application. Each user prepares 3 sentences for the research. This kind of data generation method could be called the Wisdom of the Crowds. Domain experts also checked these results and made necessary modi cations and additions. 3.4
Data mining e ort on the corpus data generates insight about the domain language model. The data can be used to improve the CAI capabilities, such as spell checking. Damerau and Levenshtein algorithms are widely used for tasks in many domains that require a metric to de ne the di erence of sequences of data, such as bioinformatics. But these algorithms are prone to errors [4,7].
of Bayesian methods on text mining and NLP, led us to an augmentation of the Damerau-Levenshtein word distance metric. All sentences in the domain corpus is processed to generate word windows of a parametric length (Window lengths of 3 & 5 have been tested). A word which will be spell checked stays in the center of the word window. Since any word of a sentence may have errors, the word window requires padding with No-operation (noop) labeled null strings for the rst and last words of a sentence (Figure 6).
The word window is a combination of speci c words, each window has an
occurrence frequency. Normalized-Levenshtein similarity is calculated from the
Turkish dictionary for the misspelled word and the metric is reinforced with
the frequency of matching word windows for the misspelled word. Word
windows are generated from an input text with misspelled word and the pattern
is searched in the collection of all word windows generated from the corpus.
The middle word of the word window with the highest (frequency *
similarity) multiplication value is suggested as the correction. Our proposed
algorithm (
Algorithm 1: Misspelling correction based on Domain Speci c Bayesian Methods
Data: Domain Corpus, Dictionary Result: Bayesian Word Window (WW) Based Correct Word Suggestion C = Word Window Collection with frequencies from Corpus; foreach word 2= Dictionary do
S Damerau Levenshtein(word); if word = middleword W W patternmatch in C then
P airSet < word; f requency > S W W f requency; return word
M axF requency(P airSet < word; f requency >); 4
In this paper we have given brief description of some of the NLP topics that require immediate solution for implementing a Domain Speci c Conversational AI. Also we have discussed our approach to development of such a system and suggested an improved algorithm to a fundamental problem that a ects whole NLP process.
A Domain speci c conversational AI is a miniaturized sub-problem of generalized conversational intelligence. Deep learning techniques are taking over all areas of AI and machine learning research, but the curse of dimensionality [5] is an issue for the size of the problem at hand. Application of Bayesian methods to this problem looks promising for areas such as unsupervised morphological inference of a xes [6] or text summarization problem [9]. 4. Damerau, F.J.: A technique for computer detection and correction of spelling errors.
Commun. ACM 7(
Prentice-Hall, Inc., Upper Saddle River, NJ, USA (2009) 9. Takamura, H., Okumura, M.: Learning to generate summary as structured output. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management. pp. 1437{1440. CIKM '10, ACM, New York, NY, USA (2010). https://doi.org/10.1145/1871437.1871641, http://doi.acm.org/10.1145/1871437.1871641