The story creation by providing automatic support to writers is a challenging and crucial task. Recently, in the field of story generation and understanding, story completion has been proposed as an approach of generating missing parts of an incomplete story. Despite the usefulness of this approach in providing creative support, its applicability is limited. This limitation is owing to the prior knowledge requirement to the user regarding the missing part of the story. To overcome this limitation, we proposed a novel approach called “Missing Position Prediction” [MYMH20]. It is necessary to predict the position of the missing part in an incomplete story. Through our study, we found that the estimation accuracy when the first or last sentence of a short story was missing was higher than when the middle sentences were missing. In this study, we further analyze this result using sentence similarity and part-of-speech tagging.
Jennifer has a big exam tomorrow. She got so stressed, she pulled an all-nighter. She went into class the next day, weary as can be. Jennifer felt bittersweet about it.
Jennifer has a big exam tomorrow. She got so stressed, she pulled an all-nighter. She went into class the next day, weary as can be. _______________. Jennifer felt bittersweet about it.
Jennifer has a big exam tomorrow. She got so stressed, she pulled an all-nighter. She went into class the next day, weary as can be. Her teacher stated that the test is postponed for next week. Jennifer felt bittersweet about it.
We define S = {s1, s2, ..., sn} as a story comprising n sentences. si(i = 1, 2, ..., n) represents a sentence. An input for MPP is an incomplete story comprising n 1 sentences S0 = {s1, ..., sk 1, sk+1, ..., sn}, where one sentence sk is dropped from the complete story S. The vital thing in this task is that no information about k is provided. Our objective is to predict k from the input, i.e. remaining sentences S0. The model is trained to maximize the probability p(missing = k|S0). Note that the order of the sentences is kept. Specifically, sk 1 and sk+1 are treated as continuous sentences. 2.2
Hierarchical approaches have demonstrated e↵ectiveness in story generation [FLD18, RWM +18]. Most likely, the reason is that a typical story has a hierarchical structure. Referring to them, we proposed a method with a twostep encoder. The first encoder, called Sentence Encoder, receives S0 and outputs the sentence embeddings. As the sentence encoder, We apply Sentence-BERT (SBERT) [RG19] in each sentence. Next, the second encoder, called Context Encoder, receives the sentence embeddings and generates a distributed representation of the entire context vcontext. We hypothesize that the input should be treated as a sequence because the order of the original sentences is preserved. Our experiment showed that a gated recurrent unit (GRU) [CvMG+14] is useful as the main part of the context encoder. The output of the GRU is input into a linear layer and a batch normalization layer [IS15]. Then, we input vcontext into a linear layer and obtain a five-unit output to obtain the MPP result.
Although the stories we used for experiments in [MYMH20] are limited to five-sentence short stories, a story with five sentences is suciently long to have minimal context [MCH +16]. Furthermore, the task design of MPP is not limited to the case of n = 5 and can be extended for even longer stories as well. Hence, we considered a hierarchical approach is suitable for performing this task. 3
For our experiments, we used the ROCStories dataset [MCH+16], a collection of 98,161 non-fictional daily-life stories. As shown in Table 1, we split the dataset in a ratio of 8:1:1 and created train/validation/test sets. For each story, one sentence was randomly excluded to create an incomplete story S0. The missing position k was randomly determined based on a discrete uniform distribution. For the training set, we retained the original five-sentence story in the dataset and removed a sentence randomly when reading the data during training. As a result, a di↵erent sentence could be removed from the same story, with a di↵erent k value, thus acting as data augmentation. For the validation (development) set and the test set, the removal procedure was performed when creating the dataset to improve reproducibility. 3.2
We trained a model for 30 epochs. The validation loss for every epoch was calculated, and the state with the smallest validation loss was used for further tests. Among the trained SBERTs, we used “bert-base-nli-meantokens,” where the output dimension was 768. The Context Encoder consists of a GRU with 256 hidden units, and a linear layer with 256 dimensions for both the input and output. The weights of the linear layer were initialized from a normal distribution with mean = 0 and std = 0.01. To obtain the five-class prediction, we use another linear layer to receive the Context Encoder’s output with 256 dimensions and subsequently outputs five dimensions. We used the Adam optimizer with a learning rate of 0.001, 1 = 0.9, 2 = 0.999, and a weight decay of 0. Gradient clipping with a value of five was used. The batch size was set to 256. 3.3
The prediction accuracy of each position is shown in Fig. 2 When the first or fifth sentence was missing, the accuracy was higher than when the second, third, or fourth sentence was missing. In other words, the beginning or the ending of a story can be easily predicted when they are the lost sentence. This appears to be related to how ROCStories was collected: “the story should read like a coherent story, with a specific beginning and ending, where something happens in between.” [MCH+16] Thus, it is likely that if the beginning or the ending is missing, our method can interpret it as unnatural.
In this study, we analyze this fact further. First, we hypothesize that the second to fourth sentences in a five-sentence story are represented by a similar sentence embedding. For each story in the training set, we considered the sentence embeddings of five sentences and calculated the cosine similarity between them. Fig. 3 shows the average cosine similarity matrix of all stories in the train set. Contrary to our hypothesis, the sentence embeddings of the second to fourth sentences were not more similar to each other than to the first and fifth sentences.
Next, we conducted an analysis using part-of-speech (POS) tagging. POS tagging of words in each sentence was performed for all stories in the training set. For each sentence number, we took the average number of times each tag appeared in that sentence. Further, we used spaCy [HMVLB20] to obtain the universal POS tags. Fig. 4 shows the result. The focus is on whether there is anything special about the distribution of POS tags in the first and last sentences. In the first sentence, the appearance of PROPN (proper noun) is remarkable, and in the fifth sentence, although not as prominent, ADV (adverb) and ADJ (adjective) occur frequently. Whether the prediction models actually pay attention to words with these tags is a subject for future analysis.
To overcome the limitation of the conventional story completion task, we proposed “Missing Position Prediction” to predict the position of the missing part based on the given incomplete story [MYMH20]. We examined the prediction accuracy of our proposed method and found that a prediction is easier if the beginning or the end of a story is missing.
In this study, we summarized what we had proposed and found in [MYMH20] and conducted the further analysis. The analysis suggested that the distribution of part-of-speech tags may play a significant role in prediction accuracy.
This work was supported by JSPS KAKENHI Grant Number JP19H01115, JP20H05556, and Moonshot R&D Grant Number JPMJPS2011. We would like to thank anonymous reviewers for their helpful suggestions and comments. [PGMK18] [RWM+18] [WW19] [YPR+19]
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43–49, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3980–3990, Hong Kong, China, November 2019. Association for Computational Linguistics.
Melissa Roemmele. Writing Stories with Help from Recurrent Neural Networks. In AAAI Conference on Artificial Intelligence; Thirtieth AAAI Conference on Artificial Intelligence, pages 4311 – 4312, Phoenix, AZ, February 2016. AAAI Press.
Hareesh Ravi, Lezi Wang, Carlos Muniz, Leonid Sigal, Dimitris Metaxas, and Mubbasir Kapadia. Show me a story: Towards coherent neural story illustration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2018.
Tianming Wang and Xiaojun Wan. T-CVAE: Transformer-based conditioned variational autoencoder for story completion. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 5233–5239. International Joint Conferences on Artificial Intelligence Organization, July 2019.