In this paper, a bidirectional matching model is proposed to identify whether two Chinese sentences are paraphrases of each other. The model is adapted from the well-known BiMPM model on two main aspects. On the one hand, it exploits a deep contextualized model named ELMo to generate the input word embedding. On the other hand, three out of four bidirectional matching mechanisms in BiMPM are carefully selected to model interaction between two sentences. The proposed model is evaluated on a dataset about Chinese sentence pairs from CCKS 2018. Experimental results show that the model achieves 86.2% F1-score on the validation set and 84.6% F1-score on the test set.
Modeling two natural language sentences is a fundamental task in many natural
language processing (NLP) tasks, such as paraphrase identification (PI) [
In recent years, neural network models have been widely used in modeling sentence
pairs. Two advanced frameworks have been proposed in previous work. The first
framework usually implements two weight sharing sentence encoders, such as
Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), to represent a
sentence pair as two low-dimensional real-value vectors u1, u2 and then makes a
prediction based on the two vectors. This framework usually constructs a feature
vector, such as (u1, u2, |u1 − u2|, ∗ uu21), feeding it into a fully-connected network
followed by a softmax layer to make final prediction. Some typical methods in this
framework include BCNN [
All the above approaches use word embedding as input. Word embedding aims to
represent the tokens from textual documents as low-dimensional real-value vectors.
As known that, word embedding has been widely used in a broad range of NLP tasks,
such as named entity extraction (NER), part-of-speech tagging (POS Tagging),
question answering (QA), textual entailment (TE), machine comprehension (MC), etc. The
most famous word embedding models are Word2vec [
In this study, our model is evaluated on the dataset about Chinese sentence pairs from CCKS 2018. Experimental results show that the model achieves 86.2% F1-score on the validation set as well as 84.6% F1-score on the test set. 2
There are lots of studies for modeling sentence pairs. In this section, we only make a
review on previous deep learning methods. We refer the interested reader to [
... … ... … …… Pr(y|P, Q) Softmax Matching Layer
Full-Matching(→)
Full-Matching(←)
AttentiveMatching(→)
AttentiveMatching(←)
Max-AttentiveMatching(→)
Max-Attentive
Matching(←) ... … ……
Prediction Layer
Aggregating Layer
Context Representation
Layer
Highway Network Layer
Methods in this framework employ two weight-sharing classical encoders, suach as
CNN or RNN, to generate two vector representations for the two input sentences.
BCNN [
Our proposed model is shown in Figure 1. The input of our model has two parts for
each sentence. The first part is the word embedding generated by ELMo. The second
part is the character embedding created by a bidirectional LSTM (BiLSTM) network
on randomly instantiated character embedding. The concatenated vector from these
two parts are fed into a Highway network [
This layer generates a d-dimensional vector for each word within the experimental sentences. There are two parts in this layer. The first part is ELMo generated word embedding. We train an ELMo model on Chinese Wikipedia articles corpus1 and use it to generate word vectors. The second part is the character embedding. We initialize fixed dimensional vectors randomly for each character within a word. They are fed into a BiLSTM network to compose word vectors. We pick the last hidden state of the BiLSTM network as the representation of each word. We feed the concatenated vectors from these two parts into a Highway network to generate the final word vectors. 3.2
This layer generates the context representation of two sentences by using two BiLSTM networks. The weights in these two networks are shared during training. 3.3
This layer calculates the interactive information between two sentences. We apply three out of four bidirectional matching mechanisms in BiMPM, including the fullmatching mechanism, the attentive-matching mechanism and the max-attentivematching mechanism. For details, we use function to calculate the relevance between two contextual representations.
In eq(1), v1 and v2 are the hidden states of the two BiLSTM networks in contextual representation layer. Both v1 and v2 are d-dimensional vectors. W∈Rl×d is a trainable parameter and l is a hyperparameter that means the perspective of the interactive features. For each element mk∈m, k means the k-th dimension of the interactive vector. They are calculated by a cosine similarity function where is the element-wise multiplication and Wk is the k-th row in W.
Further, we apply three bidirectional matching mechanisms to calculate the interac
tive features of each time-step of sentence against all time-steps of the other sentence:
Full-Matching. This matching mechanism calculates the interactive features between each contextual representation ⃗ (or ⃖⃗ ) and the last time-step of the contextual representation of the other sentence ⃗ (or ⃖⃗ ).
⃗⃗ ⃐⃗⃗ ( ⃗ ( ⃖⃗ ⃗ ⃖⃗ ) )
Attentive-Matching. This matching mechanism calculates interactive feature between each contextual representation and the weighted summing contextual representation of the other sentence. Firstly, we calculate the similarity between two contextual representations.
Then, we use (or ⃖ ) as the weight of ⃗ (or ⃖⃗ ). We generate an attentive contextual representation by weighted summing all time-steps of the contextual representations of the other sentence. ( ⃗ ( ⃖⃗ ⃗ )) ⃖⃗ )) ) ) ) ( ⃗ ( ⃖⃗ ⃗ ) ⃖⃗ ) ∑ ∑ ∑ ∑ ⃗ ⃖⃗ ⃗ ⃖⃗ ⃗ ⃖⃗ ⃗ ⃖⃗ ( ⃗ ( ⃖⃗ ( ( (⃗
⃗ ⃗ ⃖⃗ ⃗⃗ ⃐⃗⃗ ⃗⃗ ⃖ ⃗ ⃖⃗ (3) (4) (5) (6) (7)
Finally, we calculate the interactive features between each contextual representation ⃗ (or ⃖⃗ ) and the attentive contextual representation ⃗ (or ⃖⃗ )..
Max-Attentive-Matching. This matching mechanism uses the most similar contextual representation as the attentive representation with max cosine similarity.
Function generates attentive representation ⃗ (or ⃖⃗ highest cosine similarity between the two contextual representations. ) by picking the (8)
We calculate the interactive feature between each contextual representation ⃗ (or ⃖⃗ ) and the max-attentive contextual representation ⃗ (or ⃖⃗ ). 3.4
This layer employs two BiLSTM networks to generate the feature vector individually. The four last hidden states of the BiLSTM networks are used to compose the feature vector. 3.5
This layer employs a two-layer feed-forward network and a softmax transformation function to calculate the probability distribution Pr(y|P, Q). 4 4.1
In the CCKS 2018 challenge, the organizers provided 100, 000 labeled Chinese sentence pairs for the training set, 10, 000 unlabeled sentence pairs for the validation set and 110, 000 unlabeled sentence pairs for the test set.
All the evaluation results are calculated by an official evaluation system for the CCKS 2018 challenge. The evaluation system computes four metrices including micro-average precision (Prec.), recall (Rec.), F1-score (F1) and accuracy (Acc.) on the validation set and the test set. 4.2
We train a ELMo model on 3.3GB Chinese Wikipedia corpus. Both the corpus and the dataset are processed by Jieba2 tool for Chinese word segmentation. We use the ELMo generated word vectors to initialize the word embedding layer and do not update them during training. We initialize the 20-dimensional character vectors randomly. We utilize a 1-layer Highway network to generate the final word representation. We set the hidden size as 100 for all BiLSTM networks. We employ a dropout for each layer in Figure 1 and set the dropout ratio as 0.5. We set the learning rate as 0.0005 for Adam optimizer and 3 for Adadelta optimizer. We generate three results by applying Adadelta optimizer twice and Adam optimizer once. We apply a vote mechanism on the three result to generate the final prediction.
Table 1 shows the performances of some prior methods on validation set. We evaluate seven state-of-the-art models as baseline. We can see that models in the sentence pair matching framework have a better performance than those in the sentence encod
ing-based framework. All the baseline models use word2vec embedding as input. In classical encoding framework, we apply four sentence encoders, including CNN, Hierarchical Pooling, Transformer and BiLSTM. Transformer and BiLSTM have a better performance, achieving 82.9% and 82.6% F1-scores. In the attention based encoding framework, we evaluate four baseline models, including ABCNN-2, ABCNN-2 (Multi-Perspective), ESIM and BiMPM. ABCNN-2 (Multi-Perspective) is the implement of ABCNN-2 model with different kernel sizes. We can see that ESIM and BiMPM achieve better performances than ABCNN. Our model achieves highest performance in single model with 85.0% F1-score. Finally, we employ a vote mechanism to merge different results of our model. It achieves the best performance in the validation set with 86.2% F1-score and achieve 84.6% F1-score in the test set. 5
In this study, we have proposed a model adapted from BiMPM. The model implements three out of four bidirectional matching mechanisms in BiMPM and exploits ELMo to generate word embedding. The final prediction of our adapted model is given by voting three results of our model with different hyperparameters. We evaluate our model on the dataset about Chinese sentence pairs from CCKS 2018. Experimental results reveal that the model achieves 86.2% F1-score on the validation set and 84.6% F1-score on the test set, ranking the fifth in this challenge.
This work was partly supported by National Natural Science Foundation of China (61375056), Science and Technology Program of Guangzhou (201804010496), and Scientific Research Innovation Team in Department of Education of Guangdong Province (2017KCXTD013).