Command understanding, the basis of dialogue system and human-computer interaction, roughly involves two sub tasks: intent detection and slot lling. Traditional methods conduct these two tasks in a pipeline fashion, su ering from error propagation. To overcome this problem, this paper proposes a method to solve intent detection and slot lling jointly by back introducing the result of slot lling into intent detection step. Moreover, rules extracted on the training set are used to ease the noise and imbalance problems. The nal result are generated by fusing the results of rule-based method and model-based method. Experiments show that our method is e ective and achieves best performance among all teams.
Dialogue system is a signi cant research direction in natural language processing,
whose main purpose is to allow the computer to execute commands or answer
questions. Command understanding, the basis of dialogue system, involves two
sub tasks: intent detection and slot lling. Intent detection task is generally
regarded as a sentence classi cation problem. Recently, the most popular way
to accomplish this task is to use the deep learning framework[
Existing models follow a pipeline to conduct these two tasks. Intent detection is rst conducted on natural language utterances. Only ones classi ed as positive are delivered to extracted slots. The error of intent detection will propagate to slot lling and harm the nal performance. Actually, the result of slot lling can also help to improve the performance of intent detection task. For example, even given that "陈周" is a singer, this example "陈周的怎么说？" could be classi ed into negative. But the truth that "怎么说" is a song may be uncovered by the
Command Time Objective s 2 来首刘德华的歌 2017-10-17 19:41:53 s 1 你叫什么名字 2017-10-17 19:42:51 s0 请播放周杰伦的稻香 2017-10-17 19:43:53 f"song":"稻香","artist":"周杰伦"g slot lling model. If been told that, its intent can be determined more easily and correctly.
To ease the error propagation problem of pipeline model, this paper adopts a piggybacking step which feeds the results of slot extraction into intent classi cation model. Moreover, there have challenges of lacking labeled data and unbalance as Figure 1 says. To cope with the these problems, we also extract rules adopting statistical technique for slot lling task. Finally, the results of rule-based and model-based slot extraction methods are fused to generate the nal result.
This paper proceeds as follows. Section 2 presents the problem restatement in this task formally. Section 3 proposes our model framework for intent detection and slot lling. Section 4 covers the experimental studies. Section 5 concludes the work. 2
As Table 1 illustrates, given an utterance x = (s 2; s 1; s0) where s0 is current command and s 2; s 1 are related histories. The objectives of this task are (1) detecting the intention of s0. (2) extracting the values of probable slots of s0 if it is considered as positive. 3
Classi er Intuitively, intent detection can be solved as a classi cation
problem. In this paper, it is viewed as a multi-class classi cation problem. There are
three coarse classes (No-music, random and other ) instead of 9 classes(9 kinds
of intentions) for the unbalance problem in the given data. In addition, a
hybrid model of Convolutional Neural Networks(CNNs)[
Tagger Slot lling is typically considered as a sequence labeling problem. This paper adopts conditional random elds(CRF), a state-of-the-art model solution for labeling, to extract the values of song, artist and album. We use the pythoncrfsuite3 library to train the CRF model. Besides, We utilize character itself, character shape and some information from nearby characters as features. Some mention information such as artist, album and song in the utterance is also used. More details of features are shown in Table 2. 3.2
All rules are extracted from labeled training set. Considering that there has label noise in the training set, we also check the rules manually. In this paper, song, artist and album are considered as open slots whose value domain is open. The 3 https://github.com/scrapinghub/python-crfsuite other 5 slots are treated as closed slot, assuming that their probable values all be presented in the training set. Two kinds of rules are extracted, rules for open slots and rules for closed slots respectively.
Rules for open slots To extract rules for open slots, all sentences in training set are generalized by replacing matched values with its slot type, as Figure 2 shows. Rules are exactly the generalized sentences and their types.
The con dence of rules s = (Rule:r; Rule:v) are de ned as
Conf =
F reqP (s) F reqP (s) + F reqN (s) (1) where F reqP is the frequency of s be positive and F reqN is the frequency of s be negative. The rules whose Conf is lower than 0.5 are discarded. Some of rules are shown in Table 3.
Rules for closed slots To extract rules for closed slots, we start with its values which are treated as key words, sentences contain these key words are collected. Then, related words or phrases are categorized into positive and negative as Table 4 shows. When prediction, an utterance contains no negative phrases but positive phrases is assigned with corresponding slots and values. 4 4.1
Datasets The datasets is mainly from the real user's utterance in the music eld and the non-music eld of the human-machine dialogue system. The training set consists of 12,000 samples, each sample consists of the three consecutive user utterance and the label of the sample. The testing set consists of 3,000 unlabeled samples. Evaluation metrics The intent detection is a classi cation task. Precision, Recall, and F1-score(F1l) are utilized as evaluation metrics. The slot lling is a sequence labeling task. Two types of performance evaluation are discussed: for the utterances containing entities, Precision, Recall, and F1-score(F1E ) are used as evaluation metrics; for the utterances without entity, Accuracy(ACC) is used as the evaluation metric. Overall, we evaluate the quality of our method as follows:
numl F 1l + numE F 1E + numa ACC Score = (2) numt Where numE represents the total number of utterances which need to judge the intent, numE represents the total number of utterances containing the entities, numa represents the total number of utterances without entity, numt represents the total number of utterances in dataset. 4.2
This paper proposes a solution solving intent detection and slot lling jointly to overcome the error propagation problem of pipeline mechanism. Moreover, rules are extracted from the training set and used to help to produce better result. Experiments show that joint learning signi cantly improves performances. Besides, rules help to produce better results. The adopted strategies of joint learning and rules fusing are naive. In future, more advanced and e ective mechanisms for joint intent detection and slot lling are worth of study.