Recent approaches to fiction generation make use of generative Large Language Models to create fluent narratives, but they still struggle in other tasks, such as keeping coherence in rather long stories or respecting logical relations between events. Furthermore, most work focuses in English fiction, due to its prevalence in linguistic resources. In this PhD thesis we hypothesise that planning on semantic information may not only help models in such tasks, as stated by previous work, but also allow for fiction generation in other languages as an intermediate representation of stories. To that end, we collected a preliminary version of a dataset of high-quality human-written stories with extensive metadata. This new dataset will be used to test the influence of semantic planning on multilingual fiction generation and improve relevant story attributes such as coherence, logicality or likability in diferent languages.
CEUR ceur-ws.org
Recent advances pushed by the development of generative Large Language Models (LLMs) make
us wonder what are their true capabilities at generating fictional texts in diferent languages.
Thus far, conditioning methods such as control mechanisms or plot-based planning have proven
better coherence along generated stories than just prompting state-to-the-art generative LLMs
[
In this context, we define semantic planning as the creation of plans built upon semantic
information (such as events, semantic roles, named entities, and temporal or causal relations)
that enables guided surface realisation. Thus, for the fiction generation task, semantic planning
should create skeletons based on semantic information to later build narrative texts. Inspired by
recent work on cross-lingual transfer [
The present thesis project addresses the research question How does semantic planning afect multilingual fiction generation? To perform our experiments we created PromptStories1, a large collection of up to 200 k high-quality human-written English prompt-story pairs with extensive metadata. Table 1 shows an example of a prompt-story pair from PromptStories. This dataset will allow us to test the capabilities of generative LLMs at creating narrative texts, study the influence of semantic planning as an intermediate representation of stories and analyse its efect on multilingual fiction generation.
A group of men burst into your house dressed in what looks like Viking armour. In gruf voices, they inform you they are here to serve your dog who they believe is the reincarnation of Fenrir. Your dog is a four pound Chihuahua called Mr Wiggles. 16/3/2019 10318 There was a polite knock at the front door.
Drying my hands, I left the kitchen and slung the towel over my shoulder and opened the large inner front door and pushing the frenzied, barking Mr. Wiggles. On the other side of the screen door stood twenty or so people in strange armor.
A tall man in chainmail, furs, and a rounded skullcap stepped forward. He spoke, but I did not understand a word he said.
Mr. Wiggles jumped into view, resuming his wild, frenzied barking.
They all immediately fell to one knee, crying out a single word in unison. “Fenrir!” I looked to them. Then to Mr. Wiggles, who was still barking. I looked back to them. “I uh. I don’t want any?” I closed the door. [...] 17/3/2019 1704
Story collections are useful resources for training and evaluating fiction generation models.
ROCStories [
1This dataset is still in a preliminary phase. For the moment, a sample of the texts is available at https: //github.com/ixa-ehu/PromptStories.
On the other hand, the STORIUM dataset [
Moreover, XStoryCloze is the only one of the previous datasets to present multilingual parallel stories, but this data is too scarce for our purposes. Although there exist multiple collections of ifctional texts in several languages, these are usually too diverse in style, length and language coverage. PromptStories, on the contrary, contains a large amount of human-written short stories that we plan to translate from English to Spanish and Basque primarily, thus creating a large parallel story dataset.
Plot planning has been used since early attempts to fiction generation, such as the novel writer
from [
[
As an open-ended generation task, the evaluation of human or automatic stories remains
unsolved. Referenced metrics do not capture the complex characteristics of creative generation,
where several outputs may correspond to a single input and many specific attributes could be
considered (fluency, coherence, logicality, creativity, likability, etc.) [
The Main Research Question (MRQ) of this thesis project is the following one: How does semantic planning afect multilingual fiction generation? To articulate the project work, we can divide this MRQ in smaller Research Questions (RQs). RQs present the same experimental structure (prepare the dataset, train models and evaluate them) and focus on diferent evaluation objectives.
Testing the capabilities of current generative LLMs seems to be a proper initial step, given that they are widely used for story generation either in storytelling systems with control mechanisms or to create the surface realisation in those with knowledge-based plot planning. Indeed, RQ1 tackles the fiction generation part in MRQ. To answer this research question, we reduce fiction generation to short story generation (ranging ∼100 to ∼3000 words in length) as it constitutes a more manageable framework than working with larger fiction. On the other hand, we let humans answer the dificult question of what is a good story? by collecting a dataset of stories rated by their own readers. Thus, this dataset will allow us to evaluate the performance of state-of-the-art generative LLMs by comparing their zero-shot and fine-tuned results on a manual evaluation to check whether fine-tuning on stories rated as good by readers improves the story generation capabilities of these models.
Several fiction generation systems use knowledge-based methods to improve diferent attributes like intra-story coherence or logicality between events, among others. RQ2 mostly involves the semantic planning part of MRQ. For this reason, we will test diferent event representation schemes on a small sample of our dataset to find the most appropriate for story semantic planning, and we will use this scheme to annotate our prompt-story pairs. Then, we will prepare a human evaluation to compare stories generated by prompting our fine-tuned model from RQ1 and those from a model fine-tuned on our semantically-annotated dataset. Participants will be asked to evaluate specific story attributes, such as language fluency, intra-story coherence, logicality between events, creativity and relevance based on the prompt or likability from the user perspective.
Finally, RQ3 involves the multilingual part of MRQ. Given the lack of parallel semanticallyannotated corpora of short stories and the weak performance of semantic labelling in other languages (specially for under-resourced ones), we plan to translate our original English promptstory pairs and project their annotations from RQ2 to the target languages. Because of evaluation availability, we wish to apply this translation and projection to Spanish first and, if feasible, to Basque. Similarly to the previous research questions, we will fine-tune multilingual generative LLMs both on raw and semantically annotated stories, and we will compare the human evaluations of these new models with the zero-shot setting. Furthermore, we will also analyse the accumulated error from this annotate-translate-and-project method.
We collected the data in PromptStories from the WritingPrompts subreddit as in [
Before starting experiments on multilingual fiction generation, we need to create a common
frame for all the languages that we wish to study. To that end, we will annotate our English
dataset with semantic information in order to build event plans that represent stories as in [
We wish to test diferent state-of-the-art models for each of these three subtasks by comparing
their performance on a human evaluation of a small sample of prompt-story pairs. These are
the considered models4:
• Event extraction: AllenNLP SRL [
4For very large models such as M2M100 and NLLB200, we will compare the most appropriate versions according to our processing capabilities.
5https://www.deepl.com/
Once we have created our semantically annotated parallel story dataset, we can proceed to study
our specific research questions. We will first test the performance of diferent state-of-the-art
generative LLMs on the RQ1 zero-shot setting and choose the most appropriate one for story
generation. The selected model will be used for analysing RQs 1-3 by fine-tuning it on raw and
annotated prompt-story pairs for each of the studied languages. Following recent progress in
developing generative LLMs, for this task we will consider LLaMA [
We present this thesis project on studying how semantic planning afects multilingual fiction generation. To that end, we propose an analysis frame based on previous work that also explores new tasks such as using planning on semantic information to act as a bridge at creating stories in languages other than English. We explain our future experiments to investigate the main research question and present a new dataset under development for these experiments.
Nayla Escribano is funded by the Basque Government PhD grant “Programa Predoctoral de Formación de Personal Investigador No Doctor del Departamento de Educación del Gobierno Vasco”.