This paper describes a simple neural network system for Semantic Textual Similarity (STS) task. The basic type of the system took part in the STS task of SemEval 2017 and ranked 3rd in the primary track. More variant neural network structures and experiments are explored in this paper. Semantic similarity score between two sentences is calculated by comparing their semantic vectors in our system. Semantic vector of every sentence is generated by max pooling over every dimension of their word vectors. There are mainly two trick points in our system. One is that we trained a convolutional neural network (CNN) to transfer GloVe word vectors to a more proper form for STS task before pooling. Another is that we trained a fully-connected neural network (FCNN) to transfer difference of two semantic vectors to the probability distribution over similarity scores. In spite of the simplicity of our neural network system, the best variant neural network achieved a Pearson correlation coefficient result of 0.7930 on the STS benchmark test dataset and ranked 3rd1.
Semantic Textual Similarity (STS) is a task of deciding a score that estimating the degree of semantic similarity between two sentences. STS task is a building block of many Natural Language Processing (NLP) applications. Therefore, it has received a lot of attentions in recent years. STS tasks in SemEval have been held from 2012 to 2017 [Cer et al., 2017]. In order to provide Copyright c by the paper’s authors. Copying permitted for private and academic purposes.
InI:n:AP.roEcdeietodri,ngBs. oCfoIeJCdiAtoIr W(eodrsk.)s:hoPproocneeSdeimngasntoifc tMheacXhYinZe LWeoarknsinhgop, Location,(SCMouLnt2r0y,1D7)D,A-MugM1M9-Y25Y2Y0Y1,7p,uMbleilsbhoeudranteh,tAtpu:s//tcreaulira-.ws.org 1As of May 26, 2017 a standard benchmark to compare among meaning representation systems in future years, the organizers of STS tasks created a benchmark dataset in 2017. STS Benchmark2 comprises a selection of the English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017 [Agirre et al., 2012; 2013; 2014; 2015; 2016; Cer et al., 2017]. The selection of datasets include text from image captions, news headlines and user forums. Estimating the degree of semantic similarity of two sentences requires a very deep understanding of both sentences. Therefore, methods developed for STS tasks could also be used for a lot of other natural language understanding tasks, such as ”Paraphrasing” tasks, ”Entailment” tasks, ”Answer Sentence Selection” tasks, ”Hypothesis Evidencing” tasks, etc..
Measuring sentence similarity is challenging mainly because of two reasons. One is the variability of linguistic expression and the other is the limited amount of annotated training data. Therefore, conventional NLP approaches, such as sparse, hand-crafted features are difficult to use. However, neural network systems [He et al., 2015a; He and Lin, 2016] can alleviate data sparseness with pretraining and distributed representations. We propose a simple neural network system with 5 components: 1) Enhance GloVe word vectors in every sentence by adding hand-crafted features. 2) Transfer the enhanced word vectors to a more proper form by convolutional neural network (CNN). 3) Max pooling over every dimension of all word vectors to generate semantic vector. 4) Generate semantic difference vector by concatenating the element-wise absolute difference and the elementwise multiplication of two semantic vectors. 5) Transfer the semantic difference vector to the probability distribution over similarity scores by fully-connected neural network (FCNN). 2
Figure 1 provides an overview of our system. The two sentences to be semantically compared are first pre2http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark processed as described in subsection 2.1. Then the CNN described in subsection 2.2 transfers the word vectors to a more proper form for each sentence. After that, the processes introduced in subsection 2.3 is adopted to calculate semantic vector and semantic difference vector from the transferred word vectors. Then, an FCNN described in subsection 2.4 transfers the semantic difference vector to a probability distribution over similarity scores. We implemented our neural network system by using Keras3 [Chollet, 2015] and TensorFlow4 [Abadi et al., 2016]. 2.1
Several text preprocessing operations were performed before feature engineering: 1) All punctuations are removed. 2) All words are lower-cased. 3) All sentences are tokenized by Natural Language
Toolkit (NLTK) [Bird et al., 2009]. 4) All words are replaced by pre-trained GloVe word vectors (Common Crawl, 840B tokens) [Pennington et al., 2014]. Words that do not exist in the pre-trained word vectors are set to the zero vector. 5) All sentences are padded to a static length l = 30 with zero vectors [He et al., 2015a].
One hand-crafted feature is added to enhance the GloVe word vectors: 1) If a word appears in both sentences, add a TRUE flag to the word vector, otherwise, add a FALSE flag. 2.2
The number of our CNN layers is l. Every layer consists of kl one dimensional filters. The length of filters are set to be same as the dimension of enhanced word vectors. The activation function of convolutional neural is set to be tanh. We did not use any regularization or drop out. Early stopping triggered by model performance on validation data was used to avoid overfitting. We used the same model weights to transfer each of the words in a sentence. 3http://github.com/fchollet/keras 4http://github.com/tensorflow/tensorflow Sentence pad length Dimension of GloVe vectors Number of CNN layers l Number of CNN filters in layer1 k1 Activation function of CNN Initial function of CNN Number of FCNN layers n Dimension of input layer Dimension of layer1 m1 Dimension of output layer Activation of layers except output Activation of output layer Initial function of layers Optimizer Batch size Max epoch Run times 30 300 1 300 tanh he unif orm 2 600 300 6 tanh sof tmax he unif orm
ADAM 1500 25 8 2.3
The semantic vector of sentence is calculated by max pooling [Scherer et al., 2010] over every dimension of the CNN transferred word vector. To calculate the semantic similarity score of two sentences, we generate a semantic difference vector by concatenating the element-wise absolute difference and the element-wise multiplication of two semantic vectors. The calculation equation is
SD~ V = (|S~V 1
S~V 2|, S~V 1
S~V 2) (1) Here,SD~ V is the semantic difference vector between two sentences, S~V 1 and S~V 2 are the semantic vectors of two sentences, is Hadamard product which generate the element-wise multiplication of two semantic vectors. 2.4
An FCNN is used to transfer the semantic difference vector to a probability distribution over the six similarity labels used by STS. The number of layers is n. The dimension of every layer is mn. The activation function of every layer except the last one is tanh. The activation function of the last layer is sof tmax. We train without using regularization or drop out. 3
The basic type of our neural network system took part in the STS task of SemEval 2017 and ranked 3rd in the primary track [Shao, 2017]. The hyper parameters used in our basic type system were empirically decided for the STS task and shown in Table 1. Our objective function is the Pearson correlation coefficient. ADAM [P.Kingma and Ba, 2015] was used as the gradient descent optimization method. All parameters of the optimizer are set to be followed with the original paper. The learning rate is 0.001, 1 is 0.9, 2 is 0.999, ✏ is 1e-08. he unif orm [He et al., 2015c] was used as the initial function of all layers. The basic model achieved a Pearson correlation coefficient result of 0.778679 ± 0.003508 and ranked 4th on the STS benchmark5. We explore more variant neural network structures and experiments in this section. 3.1
We run the experiment using more FCNN dimensions in this subsection. The hyper parameters used in this subsection are same with the basic type system in Table 1 except the dimension of FCNN layer1 m1. The dimensions of FCNN layer1 m1 and the Pearson correlation coefficient results are shown in Table 2. Figure 2 shows the average results in every epoch with standard deviation error bar. The highest curve is the Pearson correlation coefficient results on the training data. The curve in the middle is the results on the validation data. The lowest curve is the results on the test data. 3.2
We run the experiment using more CNN filters in this subsection. The hyper parameters used in this subsection are same with the basic type system in Table 1 except the number of CNN filters in layer1 k1. The number of CNN filters in layer1 k1 and the Pearson correlation coefficient results are shown in Table 3. Figure 3 shows the average results in every epoch with standard deviation error bar. 3.3
We run the experiment using more FCNN layers in this subsection. The hyper parameters used in this subsection are same with the basic type system in Table 1 except the number of FCNN layers n and the dimension of FCNN layers mn. The number of FCNN layers n is set to be 3 in this subsection. The dimensions of FCNN layern mn and the Pearson correlation coefficient results are shown in Table 4. The number of filters in CNN layer1 k1 is set to be 1800 based on the previous experiments. Figure 4 shows the average results in every epoch with standard deviation error bar. 3.4
We run the experiment using more CNN layers in this subsection. The hyper parameters used in this subsection are same with the basic type system in Table 1 except the number of CNN layers l and the number of filters in CNN layers kl. The number of CNN layers l is set to be 2 in this subsection. The number of filters in CNN layerl kl and the Pearson correlation coefficient results are shown in Table 5. The dimensions of FCNN layer1 m1 is set to be 1800 based on the previous experiments. Figure 5 shows the average results in every epoch with standard deviation error bar. 3.5
We run the experiment using 2 CNN layers in this subsection. We add a shortcut [He et al., 2015b] between input layer and the second layer. The hyper parameters used in this subsection are same with the basic type system in Table 1 except the number of CNN layers l and the number of CNN filters in layers kl. The number of CNN layers l is set to be 2. The number of CNN filters in layer2 k2 is set to be 301, same with the dimensions of expanded GloVe word vectors. The number of filters in CNN layersl k1 and the Pearson correlation coefficient results are shown in Table 6. The dimensions of FCNN layer1 m1 is set to be 1800 based on the previous experiments. Figure 6 shows the average results in every epoch with standard deviation error bar. 3.6
We run the experiment using 3 CNN layers in this subsection. We add a shortcut [He et al., 2015b] between the first layer and the third layer. The hyper parameters used in this subsection are same with the basic type system in Table 1 except the number of CNN layers l and the number of CNN filters in layers kl. The number of CNN layers l is set to be 3. The number of filters in CNN layer3 k3 is set to be same with the number of filters in CNN layer1 k1. The number of CNN filters in layers kl and the Pearson correlation coefficient results are shown in Table 7. The dimensions of FCNN layer1 m1 is set to be 1800 based on the previous experiments. Figure 7 shows the average results in every epoch with standard deviation error bar. For this experiment, we also tried the model that without the handcrafted feature. The purely sentence representation system achieved an accuracy of 0.788154 ± 0.003412. 4
From the results of experiment 1, we can find that increasing the dimensions of FCNN does not have remarkable effect on the accuracy of evaluations. The curves in Figure 2 are almost coincidental. From the results of experiment 2, we can find that increasing the number of filters in CNN layer can improve the accuracy of evaluations. Although the size of training data (5749 records) is not very large, abstracting more features still benefits the evaluation results. By increasing the number of filters in CNN layer, we achieved a Pearson correlation coefficient result of 0.792580 ± 0.002613 and improve the rank from 4th to 3rd.
From the results of experiment 3, we can find that increasing the layer of FCNN is harmful to the evaluation results. But increasing the dimensions of FCNN layer has little effect on the accuracy of evaluations. From the results of experiment 4, we can find that increasing the layer of CNN could significantly pull down the accuracy of evaluations. However, changing the number of filters in CNN layers only changes the learning speed, has little effect on the final accuracy. The structure with smaller number of filters can learn faster.
From the results of experiment 5, we can find that adding a shortcut between input layer and the second CNN layer can slightly improve the accuracy of evaluations. From the results of experiment 6, we can find that adding a shortcut between the first CNN layer and the third CNN layer can get a result that close to the model that has only one CNN layer. Smaller number of filters in the second CNN layer can achieve better accuracy of evaluations. Comparing with the structure that has only one CNN layer, 3 CNN layers with shortcut structure can learn faster. 3 CNN layers with shortcut structure achieved a Pearson correlation coefficient result of 0.793013 ± 0.002325 and that is the best result in all of the variant neural networks. 5
We investigated a simple neural network system for the STS task. All variant models used convolutional neural network to transfer hand-crafted feature enhanced GloVe word vectors to a proper form. Then, the models calculated semantic vectors of sentences by max pooling over every dimension of their transferred word vectors. After that, semantic difference vector between two sentences is generated by concatenating the element-wise absolute difference and element-wise multiplication of their semantic vectors. At last, a fully-connected neural network was used to transfer the semantic difference vector to the probability distribution over similarity scores.
In spite of the simplicity of our neural network system, the basic type ranked 3rd in the primary track of the STS task of SemEval 2017. On the STS benchmark test dataset, the basic model achieved a Pearson correlation coefficient result of 0.778679 ± 0.003508 and ranked 4th. By investigating several variant neural networks in this research, we found that 3 CNN layers with shortcut between the first layer and the third layer structure achieved the best result, a result of 0.793013 ± 0.002325 improved our rank from 4th to 3rd. We also tried purely sentence representation system for this model and the result is 0.788154 ± 0.003412, also ranked 3rd.