The problem of question pairs matching aims to seek whether the underlying semantics of two questions are equivalent. For WeBank Chinese question pairs which are collected from real-world intelligent customer service questions, the goal is to identify question pairs that have the same intent. In this paper, we propose an ensemble model which based on both word and character level neural networks such as the convolutional neural network (CNN), and the long short-term memory network (LSTM) for modeling semantic similarity. And we adopt an enhanced deep semantic model (R-ESIM) which is proved to be more e ective for sentence modeling. Our model takes 10-fold cross-validation into account to improve the generalization ability. In the evaluation of the CCKS 2018 shared task three, our model achieves the F1 score of 0.85085 for the opening test data which ranks the second.
Semantic textual similarity plays an important role in natural language processing (NLP). It is the basis of many NLP tasks such as question answering and information retrieval. In recent years, there are more and more English semantic similarity tasks such as Quora Question Pairs in Kaggle and Semantic Textual Similarity (STS) in SemEval. The CCKS 2018 WeBank intelligent customer service question pairs matching task provides a Chinese dataset similar to the Quora question pairs. We need to assess the degree of underlying semantic similarity between two questions and identify whether the two questions have the same intent. Similar question pairs are labeled as 1 and 0 otherwise. The question matching task is challenging not only due to the short text with less semantic information but also due to many typos from real users.
There have been numerous methods to solve the problem of similarity. LSI
[
In this paper, we propose an ensemble method combining the k-fold crossvalidation results of various word-level and character-level deep learning models to catch more semantic information for the WeBank task. Our approach achieves the F1 score of 0.85085 which ranks the second in the nal evaluation. 2
In this section, we describe the proposed deep neural networks to solve the WeBank question matching task. We rst model the questions by a certain siamese base network to translate natural language into mathematical representations. Then we treat the task as a binary classi cation problem and compute the matching score of semantic representations for question pairs with a multi-layer perceptron (MLP). Finally, we nd optimal ensemble strategy based on the results of validation set and average the output of di erent single models. 2.1
Word embedding and character embedding are pre-trained by word2vec [
We use siamese architectures due to the input of question pairs. Namely, two networks with the same structure and the same weight each process one question in a pair. We have tried multiple deep siamese models for sentence modeling and classi cation in the task. The input of them is a sequence of words or characters de ned as x = (x1; x2; ; xn) (n is the xed length). The following is a detailed description of these networks.
CNN Siamese Kim [
convolutional weight, b is the bias, and is the activation function (we use ReLU in this task). Then we apply 1-max pooling which means taking the highest value for each feature map to capture the most important feature. All the features generated by lters can be concatenated to represent the input sentences. Then we concatenate two sentence vectors and feed it into two fully-connected layers to obtain the matching score.
BiLSTM Siamese LSTM [
CNN-BiLSTM Siamese The CNN-BiLSTM model contains both CNN and BiLSTM network. CNN can extract n-gram features and LSTM is able to operate over sequence input. Both of these two mainstream models have their own advantages. So we combine the strengths of them. We take advantage of CNN to extract features and use LSTM to encode them to a semantic representation. R-ESIM Siamese ESIM is a natural language inference (NLI) model, but the main idea is still the sentence modeling and classi cation. Thus, we can modify and re ne the ESIM network to t the sentence matching task. There are mainly six steps: 1) Input question pairs are encoded to hidden vectors (a = fa1; :::; ang and b = fb1; :::; bng) by BiLSTM; 2) We compute the similarity of each two hidden vectors from given pairs as attention weight (eij = aiT bj); 3) New vectors that can re ect local relevance between question pairs are obtained by weighted summation of hidden vectors; 4) The concatenation of the original hidden vectors and new vectors, and the di erence and dot product of them are put into another BiLSTM; 5) We use both max and average pooling and concatenate all these vectors to represent the input sentences; 6) We feed the concatenation of sentence semantic vectors into the MLP classi er with sigmoid function in the output layer. The following experiments show that our re nedESIM achieves great performance in the question matching task. 2.3
In addition to the neural network, there are many features that can re ect the similarity between question pairs in a way. We roughly summarize these useful features into three categories: character level features such as character overlap, Levenshtein distance, and longest common subsequence (LCS), word level features such as word overlap (TF-IDF weighted), word mover's distance (WMD), and sentence level features such as topic models, average of word embedding. We concatenate all these features including the output of deep models mentioned above and try putting them into a random forest which is a typical ensemble learning method to combine semantic sentence models with additional features. 2.4
As shown in Figure 1, we respectively train a character-level R-ESIM model, a word-level R-ESIM model, and a character-level BiLSTM model based on 10-fold cross-validation. Thus there are 30 models in total. Word-level and characterlevel models have the same structure, but the input of them are word and character embeddings. We try to combine these methods to get a more comprehensive model. Voting and averaging are the two simple but e ective ensemble methods. In this paper, we average the output of 30 models as the nal prediction.
Our proposed models was implemented in Keras2 with TensorFlow backend. We used a TITAN X GPU device to train the model. We pre-trained 100-dimensional word embedding and character embedding by gensim as the initial weight of the
embedding layer which will be updated during the training process. We employed
binary cross-entropy as the loss function. The optimizer we utilized was Adam
[
We evaluated the performance of our models described in Section 2. Table 1 shows the experimental results of these models on the validation set in detail.
We can nd that character-level models are better than word-level models and 10-fold cross-validation can e ectively improve the performance. The ensemble model of character and word level R-ESIM models and character level BiLSTM models based on cross-validation achieves the best result. In other words, our best model is the average of the output of 22 models. We only use the rst two folds of the BiLSTM model due to the time limit. Moreover, we incorporate additional features with R-ESIM models, and it can slightly improve the performance. The method is time-consuming and not better than the ensemble method of simple averaging, so we give up them in the nal evaluation. 3.3
In the nal evaluation of test data, we submitted the result of our best model which contains character and word level R-ESIM models (10 folds) and character level BiLSTM models ( rst 2 folds). Table 2 shows the nal results of our submission and other competitors'.
We achieve the F1 score of 0.85085 and rank the second. We do not make any post-processing for the output of our model. The distribution of positive and negative samples is di erent between the validation set and the test set, but our model is stable and achieves corresponding performance on both datasets which can prove that our model has strong generalization ability.
Model R-ESIM word + char (cv) + BiLSTM char (cv) ThunderUp DST Laiye-Suda GDUFSER 0.8524
F1
In this paper, we describe an ensemble method of various deep neural networks based on 10-fold cross-validation to solve the WeBank question matching task. And the model can be migrated to other similar tasks easily. Furthermore, there are many models that we haven't tried because of time constraint. Further work will focus on combining the 10-fold cross-validation of other deep models and nding a more appropriate method to add additional features.
Acknowledgments This work was supported by the National Basic Research Program of China (2014CB340404), the National Natural Science Foundation of China (71571136), and the Project of Science and Technology Commission of Shanghai Municipality (16JC1403000).