Two major tasks in spoken language understanding (SLU) are intent detection and slot filling. To address these two tasks, in this work, we proposed to employ Conditonal Random Filed(CRFs) with gazetteers features for the slot filling problem and use Naïve Bayes model with N-gram feature for intent detection based on a new shared task dataset released in CCKS[5] . The experiments shows that CRFs with gazetteers features obtains 90.8% F1 score outperforms some deep learning methods. The joint result of slot filling and intent detection help us archive the ranks 1 in the Preliminary round.
Feature Names Single word feature in gazetteers
In this work, we treat slot filling as a sequence labelling problem work as many works have done. In sequence labeling problem, we want to learn a function: : → given the training example of *+(), ()2, = 1,2, … . . 8. Many methods have employed to sequence labelling such as: CRFs、BiLSTM-CRFs、Attention based BiLSTM-CRFs etc. We explore the importance of gazetteers using CRFs. CRFs are a type of discriminative undirected probabilistic graphical model which can take context into account. CRFs can archive better result with properly defined features. In this work we using singers and albums as new features. We have collected about 1million records about singer, the song names and the albums from the internet using crawler. The custom defined features as described in the table blew. C: current word, N: next word, P: previous word CP: cur and previous word, FEATURE_IN_GAZ means the FEATURE in GAZ. FEATURE_PREFIX means whether the FEATURE is the PREFIX of the word in gazetteers, FEATURE_SUFFIX means whether the FEATURE is the suffix of word in gazetteers.
Table 1. Custom Gazetteer features
Features
C_IN_GAZ,P_IN_GAZ,N_IN_GAZ, Double word feature in CP_IN_GAZ,CN_IN_GAZ gazetteers Double word is prefix CP_PREFIX,CN_PREFIX Double word is suffix CP_SUFFIX,CN_SUFFIX Third word feature CPN_IN_GAZ,CPN_PREFIX,CPN_SUFFIX
Besides the custom features defined in the table 1, we also using other features which are frequently used in many papers such as: n-gram features, word distribution, people names etc.
Navies Bayes is a conditional probability model based on the bayes’theorem. In this task, we apply Naïve Bayes with N-gram features for intent detection task. From the experiment we found that this simple model can archive 84% f1 score for detecting the utterance when there is no music in the utterance.
The training dataset contains 1200 utterance，we divide the train dataset into two parts according to each single task. As for slot filling task, the dataset is divided into three parts: 70% as training dataset, 15% as valid dataset and the rest 15% for testing. The dataset contains 19 slots and we use BIOE annotation scheme. In our work, we treat the intent detection problem as three class classification problem, The classes are: No-music、Random、Music.
As for slot filling task, we using the Stanford CoreNLP tools to train our CRFs
model. Stanford CoreNLP is a widely used, integrated NLP toolkit. We using the
basic feature as they do which get the remarkable result on CONLL 2013 dataset. We
have compare our model with BiLSTM-CRFs[
model F1
CRF 0.867 Bilstm-CRF 0.854 Lattice LSTM 0.892
CRF Gazetteers 0.908
As for intent detection task, we have tried the Logistic Regression and Multinomial Naïve Bayes and Random Forest, we found that the Navies Bayes archive the best result. We treat the intent detection problem as classification problem as described above.
F1 0.86 0.69 0.79
In this work, we simply joint the result from two task by setting a threshold 0.95 to using the classification result when the model identify the text as No-music. The result in this shared task are described in the following table.
Competition stage Preliminary Round Finals score 1.38337 (rank 1) 1.31264 (rank 3)
In this work, we apply CRFs with Gazetteers features for slot filling and Naïve
bayes with N-gram features for intent detection. The result shows the useful of the
gazetteers features. In the future we can explore how to apply the gazetteers features
with deep learning methods like Lattice LSTM[