Traditional semantic similarity computing methods mostly regard the text as a set of words, by calculating the number of words occurred in the text to build the feature vector, then using the metrics such as cosine distance between the vectors to calculate the text similarity. However, these methods only consider the word level of the sentence, not the semantic level, which may ignore many important information, including syntax and word order. This paper proposes a new deep learning method, which combines the attention mechanism with BiLSTM based on Siamese network to achieve the semantic similarity matching for given question pairs. Experimental results show that our models can make full use of the semantic information of the text, and the F1 value in the dataset provided by the CCKS2018 question-intention matching task is 0.84586, achieving fourth place in the final test.
The goal of CCKS2018 question-intention matching (QIM) task is to match the question pairs against Chinese real-world customer service data. In the task, we should design models and algorithms to determine which of the provided pairs of questions contain two questions with the same intention. For example, given a sentence pair: “一般 几天能通过审核” and “一般审核通过要多久”, we need to determine whether they convey similar meaning (label 1) or not (label 0).
Sentence matching, also called semantic equivalence identification or short text semantic similarity computing, is one of the most basic tasks of NLP. It is the basis of various NLP tasks such as automatic question and answer, chat robot, information retrieval, and machine translation. In recent years, with the rapid development of Internet technology, various data information shows explosive growth, and the demand for processing large amounts of similar data is increasing. Although researchers have advanced the semantic similarity task, there are many challenges remained.
If the requirements for question matching are different, the definition of matching will be not the same. For example, in classic sentence retelling task, we need to judge whether the two sentences are merely different in expression, but the meaning is the same. We often face short sentences and the complex structures in the task. Thus, it is difficult to extract informative features purely based on the sentence itself. The data of QIM task comes from the log text of original smart customer service in the field of bank, so there is much dirty data, such as mislabeling, typos, missing words and simplified words, which can easily lead to word segmentation errors, including the identification of the unknown words. In addition, the training of word vectors also lacks corpus data for specific financial fields, which associated with this task.
In this paper, we propose a semantic matching method based on Siamese network [1] for this task. For the similarity computing, instead of using the traditional artificial designed features, we adopt the neural network to learn automatically the information contained in the question pairs. In this way, it can mine the potential syntactic and semantic features in the sentences, which effectively compensates for the shortcomings in the manual extraction of features. 2
The core of text similarity computing is to compare the differences between two given texts, usually measured by a real value between 0 and 1. In different fields, domestic and foreign scholars have proposed many methods of the similarity computing and applied them in practice. At present, these methods can roughly divide into four categories: 1) string-based method, which is also called “literal similarity method” and the typical methods include edit distance, Hamming distance, Euclidean distance, cosine similarity, longest common substring, Jaccard, and so on. Since the string-based method does not consider the semantic information of the text, the effect of the method is limited. 2) corpus-based method, which uses the information obtained from the corpus to calculate the text similarity, and can subdivide into a word-bag-based model approach, a neural network-based approach, and a search engine-based approach. The first two types use the document collection that will compare as the corpus, and the latter is a web-based corpus. 3) world-knowledge-based method, which makes use of a knowledge base with a standardized organizational system to calculate text similarity. 4) other methods, which include syntax analysis and mixing methods [2].
Among them, the neural network-based method can represent the text semantic information more in line with human cognition, and has become the development trend of the task, which solves many drawbacks of traditional methods. For example, the InferSent model proposed by Conneau et al. [3], which based on a Bi-LSTM architecture with max pooling, and the ESIM model proposed by Chen et al. [4], which based on chain LSTMs and attention, both preform well on natural language inference (NLI) , a task similar to semantic similarity computing.
Our work can divide into three parts: data preprocessing, model building and post-processing. 3.1
Because the data comes from the log text of original smart customer service in the field of bank, many sentences are informal and contain much noise. The main issues are listed as follows: Traditional Chinese characters such as “何時會邀請我使用微粒貸” and “为什么 我的微信上还没有出现微米粒”, the first one is written in Traditional Chinese and the second is in Simplified Chinese. If the sentences are used directly to train the neural network without preprocessing, it will affect the accuracy of the prediction to some extent. Typos, such as “预期罚息和利息” and “逾期利息”. “预期” should be “逾期” in the first sentence of the two sentences. Too short, such as “我找” and “你老板呢”, the two sentences are very short and it is not easy to extract useful feature information. Redundant Character, such as “用微信都6年，微信没有微粒贷功能 ” and “4。 号码来微粒贷”, there are both extra numbers, punctuation marks such as spaces in the two sentences, which also affect the performance of the model.
Based on the above problems, we carry out the following specific preprocessing on the data: 1) use the open source OpenCC [5] tool to convert the data from traditional to simplified, 2) remove the extra space characters, Chinese and English punctuation marks and numbers, 3) convert all the letters to lowercase. In view of the fact that the word segmentation tool cannot handle the specific domain data well, and the experimental results show that the char level is obviously better than word level when tested. For the lack of training corpus for word vector, we use the training data provided by the task to train embedding at the char-level. 3.2
To calculate the semantic similarity of the question pair, that is, whether the question pair belongs to label 1 or 0. Obviously, this is a classification problem in the field of NLP, but it is different from the traditional text categorization method.
In order to represent the feature information of two questions at the semantic level and capture the commonalities between them, we designed the following two models based on Siamese network. The Siamese network is a neural network framework with two identical structures and a shared weight subnetwork for evaluating the similarity of two input samples.
Figure 1 and Figure 2 show the entire frameworks of the semantic similarity computing models. It can be found easily from the figures that the structure of the two models is similar, they both contain Bi-LSTM layer, self-attention layer, non-linear layer, matching layer and MLP layer.
Let 1 = [ 1, 2, … , ] and 2 = [ 1, 2, … , ] be the given two sentences with length and respectively, where , ∈ is a char embedding of -dimension vector. The task is to predict a label that indicates the similarity relationship between q1 and q2.
For model-I, we apply bidirectional long short-term memory network (Bi-LSTM)
[6] as the basic component of our model, since LSTM unit can effectively extract the
dependence among long-distance and non-continuous words in sentences, which is
more suitable for the semantic representation of the word sequence in the sentence. And
the bidirectional structure can provide complete past and future context information of
each node in a sentence for the output layer. Firstly, we use two layers of Bi-LSTM to
encode the word vector input for the given two sentences q1 and q2, which combines
the forward and backward outputs to obtain a textual representation of each sentence.
The outputs of the first and second layers of the two sentences as follows:
(
D = BiLSTM(B) = softmax(tanh( ) )
= ∑ =1 ∗ = softmax(tanh( ) ) where , ∈ ∗2 and , ∈ ∗2 are the outputs of the two layers of Bi-LSTM respectively. Moreover, the two questions share the parameters of Bi-LSTM.
Secondly, we employ the self-attention mechanism [7] to focus on the key parts of
each output of the first layer with Equation (
= ∑ =1 ∗
(
Thirdly, we utilize the attention mechanism to select the more important parts of each
for the output of the second layer with the following Equation (
= softmax( ) where and are attention weights, and are the attention outputs.
Then, we concatenate the two attention outputs 1 = [ ; ] and 2 = [ ; ], and feed them to a nonlinear layer, so that we can get the deeper and more abstract semantic representation of the two sentences.
In order to represent more intuitively the similarity between the two questions, we calculate the difference: 1 − 2 and elementwise product: 1 ∗ 2 , which denote the distance and angle information respectively between the two questions [8].
Finally, we feed the question representations, distance and angle information to a multi-layer perceptron (MLP) layer for classification.
For model-II, what it differs from model-I is that we adopt the word-alignment
attention to obtain the inter-relationship between each word of two questions. Equations
(
= tanh( [ ; ] + ), ∈ [1, ], ∈ [1, ] = ∑ =1 ∑ exp( ) =1 exp( ) , ∈ [1, ] = ∑ =1 ∑ exp( ) =1 exp( ) , ∈ [1, ] where is the attention score as the similarity between the words of the question pair, and represents the extracted relevant information of while represents the extracted relevant information of .
Then, we calculate the distance and angle information of the alignment output of the two questions respectively with A and B, as the input of the second Bi-LSTM layer. = BiLSTM([ , = BiLSTM([ , , − , − , ∗ , ∗ ]) ]) Finally, the self-attention is used to calculate the key parts of each question. = softmax(tanh( ) ) information provided by the training data, we use K-fold cross-validation to train each model, and then obtain the final outputs by voting. 4
Datasets and evaluation metrics. The training set consists of 100,000 question pairs,
which include labels and where the ratio of positive and negative samples is close to
1:1. The development set consists of 10,000 unlabeled question pairs. And the last test
set consists of 110,000 unlabeled question pairs. Submissions are finally evaluated on
the test set. Four metrics Precision, Recall, F1-score and Accuracy are used for the task.
(
The research work is supported by the National Natural Science Foundation of China (61572338) and the Natural Science Foundation of the Jiangsu Higher Education Institutions (Grant No. 16KJA520001).