Text morphing is a Natural Language Processing (NLP) task which aims at generating sequences of fluent and smooth intermediate sentences between two input sentences, the start and end sentences. In this paper, we show how to use sentence analogies to augment data for this task. We rely on the notion of analogy to produce sequences of sentences exhibiting step-by-step transitions. We use these sequences to fine-tune a large-scale pre-trained language model that is used for text generation. The performance is evaluated by two criteria: fluency and transition smoothness on both the semantic and formal levels. Compared to a variational autoencoder generative model, our model is shown to generate smoother transitions, although the generated sentences are slightly less fluent.
he was silent for a long moment . S1 S2 S3 S4 he was silent for a moment . it was quiet for a moment . it was dark and cold . there was a pause .
it was my turn .
S1 S2 S3 S4
obtained through variational latent space are smoother and the fluency of the generated sentences is higher. By searching paths through the latent space, it can generate coherent new sentences which interpolate between two already known sentences. Table 1 shows an example of a text morphing sequence generated by Sentence-VAE.
[
Building on the previous work, [
In nowadays NLP, it has become classical to fine-tune a large-scale pre-trained language model to perform a given downstream task, as this has been proven to be eficient in many cases. To perform fine-tuning in a supervised way, implies the use of a data set for the task in question. In our case, this means a data set of text morphing sequences.
In this paper, we show how to create text morphing sequences by exploiting the notion of analogy between sentences. This point is the original point in our proposed method.
Our method thus consists of the following two steps. Firstly and most importantly, we use the notion of sentence analogy (Subsection 2.2) to create a data set of text morphing sequences (Subsection 2.1). Secondly, we use this data set to fine-tune a large-scale pre-trained language model (Subsection( 3.4) on the task of text morphing.
Below, we detail the original point in our method, i.e., the creation of a data set of text morphing sequences using the notion of analogies between sentences. We also explain how we solve sentence analogies. 2.1. Creating a Data Set of Text Morphing Sequences We construct a data set of text morphing sequences by solving sequences of analogies between sentences. We start with a sentence analogy : :: 0 : , where , and 0 are sentences extracted from a data set of sentence analogies and is unknown.1 olving the equation delivers a sentence = 1. We recursively apply the process by replacing 0 with 1, etc., leaving and unchanged. In this way, we obtain a sequence of sentences 0, 1, . . . , . It is a text morphing sequence where 0 and are the start and end sentences and 1, ..., − 1 are the intermediate sentences.2 See Figure 1.
Since we are constantly replacing − 1 with the next sentence predicted by sentence analogy, the direction of changes in the entire sequence is given by the direction between and . Now, as the variation is, by definition of the analogy : :: − 1 : , limited by the variation between and , the transitions should be smooth, if and are not too distant. The tool used to solve the sentence analogies should be responsible for the fluency of the generated sentences .
To summarize, in this process, the sentence 0 is transformed slowly step by step into , along the direction defined by and . Notice that we give the start sentence 0, but that we do not know in advance the end sentence . 2.2. Solving Sentence Analogies The previous process requires a tool to solve sentence analogies. Sentence analogies are more dificult to solve than word analogies ( is to as is to or is to as is to ℎ ). The syntactic structure and semantic complexity of sentences makes the dificulty. 1http://lepage-lab.ips.waseda.ac.jp/en/projects/kakenhi-kiban-c-18k11447/ See Experimental Results. 2In our experiments, we set to have 1 to 5 intermediate sentences. When becomes larger, we observe that the same sentence may be repeated in the sequence of sentences. : :: 0 : ⇒ = 1 ⎫⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ : :: 1 : ⇒ = 2 ⎪⎪⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ : :: 2 : ⇒ = 3 ⎬ ⎪ ⎪ ⎪ ... : ... :: ... : ⇒ = ... ⎪⎪⎪⎪⎪⎪⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ : :: − 1 : ⇒ = ⎭ =⇒ (0, 1, 2, . . . , )
It ’s really not that interes- : It ’s really not that :: It ’s not that : ting . hot . bad . fyhYrooeoruume?’ar,erounanordet : iYnoguh’erree ,naorte ystoauy?- :: ayYooguuai’?nre,coanrfeusne’dt : ⇒ = It ’s not that cold .
You ’re disap⇒ = pointed , are n’t you ?
In early proposals to solve sentence analogies [
[
Here, we solve sentence analogies using yet another method described elsewhere [
3.1. Data Used
The previously mentioned data set of semantico-formal analogies is used to create text morphing
sequences that will be used to train a GPT-2 model for the task of text morphing. This data set
was created from sentences extracted from the English part of the Tatoeba resource3.
3.2. Assessment of Text Morphing Sequences
Assessment of text morphing is done according to two dimensions. Firstly, by the smoothness
of the transitions between the sentences in the morphing sequences: two consecutive sentences
should not difer by too much for the entire sequence to be considered smooth. Secondly, by the
quality of each individual intermediate sentence: all sentences generated should sound natural,
lfuent, grammatical, in a word, it should be reasonable.
3.2.1. Transition Smoothness
We define it as the average of all edit distances between consecutive sentences in the morphing
sequence. The edit distance between two strings gives the number of edit operations needed to
transform a given string into another one. It is thus particularly well suited for our purpose.
Here we use the Levenshtein distance [
Analogies
As mentioned at the end of Section 2.2, to solve sentence analogies, we fine-tune a pre-trained
language model, GPT-2 [
To create a data set of text morphing sequences, we then use each semantico-formal sentence analogy as a starting point as described in Section 2.1 and illustrated in Figure 1.
We asses the quality of the created text morphing sequences with the metrics introduced in
Section 3.2, but, in addition, we compare with an existing model.
3https://tatoeba.org/
4We use the a 3-gram language model, trained on the Tatoeba corpus, with the KenLM toolkit https://github.com/
kpu/kenlm to compute the perplexity of the sentences.
[
In this way, we can compare the transition smoothness and the fluency of two comparable sets of text morphing sequences, created from the same start and end sentences, by two methods, the Sentence-VAE model, and our proposed method. 3.4. Fine-Tuning a Pre-Trained Model with the Data Set of Text Morphing
Sequences Created Using Sentence Analogies We fine-tune the pre-trained GPT-2 model using the sentence sequences generated in the previous section. For comparison, as in the previous section, we still use the Sentence-VAE model as a baseline model. Due to limitations in memory, we choose the medium-sized GPT-2 model (345M). The GPT-2 fine-tuning parameters are shown in Table 3. For the baseline model, we trained the Sentence-VAE model using the Tatoeba corpus dataset which consists of 110,000 English sentences.
GPT-2 [
GPT-2 is a large model based on transformer training on a very large dataset with a large scale, and GPT-2 has a good performance in text generation, both in terms of contextual coherence and sentiment expression.
4.1. Results for the Creation of the Data Set of Text Morphing Sequences
The quality of the created text morphing sequences, that will be used afterwards to train a
large-scale language model for the task of text morphing, is shown in Table 4. Our proposed
approach delivers smoother sentences which are semantically relatively correct, in comparison
with the Sentence-VAE model proposed in [
Table 4 shows that the transition smoothness (average of edit distance between consecutive sentences) of Sentence-VAE is 3.52, while it is 1.31 with our proposed method. This means that for each transition, the Sentence-VAE model changes on average three and a half words on average, while our proposed method changes 1.3 words only, less than half in comparison. The average number of words per sentence being 6.7, the baseline method changes half the sentence at each transition. Our method makes more subtle and smoother changes.
The fluency, as measured by perplexity, is 1.35 in the method using Sentence-VAE, while it is 1.57 in our method (the scores are small because the sentences are short). According to these numbers, the sentences generated by the Sentence-VAE model are more reasonable, but whether there is a real diference may be disputable. We conclude that, in comparison with the Sentence-VAE model, our proposed method delivers smoother sentences that are relatively lfuent.
The following Table 5 shows basic statistics of our created dataset. An example of generated morphing sentences is given in Table 6 below. 4.2. Results for the Text Morphing Task The quality of text morphing is shown in the same table as before, Table 4. The results of this experiment are similar to those obtained in the previous section when creating a data set of text morphing sequences. This indicates that our trained model can deliver smoother sentences which are semantically relatively correct, in comparison with the Sentence-VAE model for the task of text morphing.
The transition smoothness of the Sentence-VAE model is 3.73, while it is 0.72 with our proposed method. The previous remarks made above apply similarly here for this model. It is not a surprise as we use it here in the same way as before. Our proposed method shows improvement in transition smoothness relatively to the creation of text morphing sequences: the average edit distance between two consecutive sentences has been almost divided by two.
The perplexity of the method using the Sentence-VAE model is 1.36, while the perplexity with our proposed method is 1.75. Again, there is no diference between the scores in the data creation step and the text morphing tesk for the Sentence-VAE model because we use it in the same way in both cases. Our proposed method generates sentences with a slightly worse perplexity in the text morphing task compared with the creation of text morphing sequences using sentence analogy. However, again, we can conclude that our proposed fine-tuned model delivers sentences which are relatively fluent, but smoother, in comparison with the Sentence-VAE model. 4.3. Discussion When creating text morphing sequences, we observed that, sometimes the same sentences were generated repeatedly or several sentences were generated alternately. We explain these phenomena by the relative shortness of the sentences used. The sentences contained in our data set are less than 10 words long. Shorter sentences allow for fewer options for changes when the text is morphed, and sometimes repetition occurs, which induces no change.
We proposed to perform text morphing by fine-tuning a large-scale pre-trained language model on the task, as is classical nowadays in NLP. But for that, data was needed. We relied on analogies to create text morphing sequences. We proposed an original method which consists in starting with an analogical equation and in letting the solver perform changes in the direction defined by the two terms on the left of the analogical equation. Variations are obtained step by step and this results in text morphing sequences.
The performance of the fine-tuned model was evaluated with transition smoothness and lfuency. Our model achieved more than three times smoother transitions than the baseline we considered, the Sentence Variational Autoencoder generative model. However, the baseline was shown to generate slightlymore fluent sentences than our proposed model.