With Generative AI at the forefront of public conversation, the capability of AI systems to tell stories has seen a surge of interest. OpenAI's ChatGPT provides a user-friendly interface, as well as well-documented API access, and is used widely for generative purposes. In this paper we investigate how well it can actually produce narrative text. We present an approach to take a story plan produced by the Glaive narrative planner and turn it into a novella-length text. We then present a preliminary evaluation of the text output and discuss the challenges and limitations of having it actually be read by humans. Crucially, we show that the text is not comparable to human-written text in terms of grammatical complexity, which we posit to be one possible reason for it not being very enjoyable to read. As part of our work we also encountered several particular challenges that led to misspun tales, which we also discuss in detail.
Narrative generation has been a topic of interest for AI
research for many decades. Meehan’s TaleSpin [
While LLMs can produce narrative text, the transformer mechanism that they are based on has one significant limitation: When predicting the continuation of a text, the attention head mechanism employed by the model can only look at a limited context preceding the continuation. This context window thus limits how much information the LLM can even see. Essentially, if one tried to generate an entire novel using an LLM with a limited context window, the model would not be able to take the contents of the first few chapters into account when generating the end of the narrative, which may result in events that contradict earlier changes to the world state. Plan-based narratives, on the other hand, use an explicit (logic-based) world model to represent the state of the story world at each time step, and can ensure that only actions that are actually possible to occur are taken by the characters. However, generating the discourse for such a plan-based story often involves templates or other short story fragments, often resulting in repetitive or terse discourse output.
In this paper we present an approach that utilizes OpenAI’s ChatGPT to generate the discourse for a story generated by a narrative planner. In particular, we focus on generating long-form narratives, that, at present, reach the length of a novella (about 25 000 words), and follow the story as produced by the planner. Our contribution is threefold: First, we present a novel approach to prompt an LLM using the planner output to expand the narrative to the desired length. Second, we have generated several narratives using our approach and show an evaluation of some of their qualities. Third, and arguably most importantly, during the course of our work we have discovered several limitations of this application of LLMs, and we will discuss them in detail.
Our work builds on previous work in narrative generation, combining a logic-based story structure generated by a planner, with text generated by an LLM. We will therefore discuss prior work in both of these areas.
When a story is viewed as a (partially ordered) sequence of
actions taken by the characters, it has striking similarities
to a plan, in the AI planning sense: A sequence of actions
applied to an initial state to achieve a goal condition [
With a story in hand, the next problem is how to convey
this story to an audience, i.e. how to generate the discourse
[
In a more refined proposed model by Barot et al. [
Large Language Models (LLMs) are based on a neural
network architecture called transformers [
Our system is able to generate a long-form narrative consisting of multiple chapters. As input we utilize a plan, as obtained by the Glaive narrative planner, as well as an optional genre descriptor. Our discourse generator works in three steps:
2. Generate chapters from descriptions.
3. Summarize and regenerate each chapter.
In each step, our approach utilizes the diferent roles one can provide when prompting ChatGPT as shown below. To summarize: The system role provides the model with highlevel guidance, the assistant role gives the model context (typically previous model output, but can also be used to demonstrate desirable output to the model), while the user role contains the actual prompt the model should respond to.
The first challenge when generating discourse from story plans is that the planner output is in the form of a plan, in the case of Glaive this comes in a PDDL-like syntax. The ifrst step is therefore to convert these formal representations into descriptions of a story chapter, where each step in the plan corresponds to one chapter (we will discuss the implications of this below). Rather than having the domain author come up with a mapping of plan actions to a description, we utilize ChatGPT itself to make this mapping. Given a plan step , e.g. “(hatch-plan robbie six-shooter brown-horse bank mother-lode)”, we construct the following prompt: • System: You are rephrasing a string of words. • User: Take the following phrase and make it into a coherent sentence: . Provide only the resulting sentence. • Assistant: In a statement like (accept talia rory village), the meaning is: “Talia accepts Rory’s proposal in the village” The example mapping provides the model with guidance how to interpret. For the example step above, the model produces (variations of) the description “Robbie hatched a plan to rob the bank for the mother lode with his six-shooter and brown horse.”. In the next step, we use these steps to generate individual chapters of the story.
Once we have a textual description of each step of the plan, we have the model generate a chapter of the story based on the description. However, when using only the description of the current step, chapters become disconnected, with characters changing frequently between them, needless repetitions, or plain inconsistencies. On the other hand, the context the model can use is also limited, so we cannot provide it with the entire story so far. Instead, we construct the following prompt, incorporating only the text of the immediately preceding chapter: • System: You are a story teller continuing a story. • User: Take the following chapter and make the next chapter, and include dialogue and natural progression: <previous chapter text>. • Assistant: The current chapter is: <i> • Assistant: The current chapter is about: <chapter description> • Assistant: The genre is: <genre description> For the first chapter, the system prompt is changed to “beginning a story”, and no previous chapter text is included. For the last chapter, the model is also explicitly told that it is “ending the story” in the system prompt.
With these first two steps, the system will already produce a discourse following the events of the plan produced by Glaive. However, there will still be noticeable disconnects, as the generation process does not take into account what may follow. Chapters often end with “To be continued...”, or even “The end.”, even though the narrative has not reached its conclusion. We therefore added another processing step to refine the flow of the narrative.
One key limitation with LLMs for our work is their limited context. However, even with larger context size, providing the model with more input does not guarantee more desirable output. Nevertheless, in order to improve the consistency of the narrative, we take each story chapter and have the model rewrite it in context. For each chapter, we ifrst ask the model to provide a summary of the events that happen in it, with the simple prompt “Create a short summary of the following chapter: <chapter text>”. We then ask the model to rewrite each existing chapter with the following prompt: • System: You are a story teller remaking chapters. • User: Rewrite the following chapter: <chapter text> • Assistant: Keep in mind that the <relative position> chapter is: <summary of relevant chapter> Where the assistant-phrase is present up to four times: We take the context of the current chapter to be the two immediately preceding and two immediately following chapters, and include their summaries in the prompt. This causes the model to take the events of these chapters into account when rewriting the current one.
Depending on the input plan, our approach is able to
produce a narrative telling that may reach the length of a short
novella. This makes evaluating the output challenging, as
it requires reading through hundreds of pages of text and
determine its quality. Before we go into more detail about
these challenges, and the evaluation we performed, we will
ifrst discuss the input data we used and then show some
sample output to demonstrate what our approach is
capable of. However, we would be remiss to not also discuss
limitations and challenges that remain, which we will do to
conclude this section.
We use the Glaive narrative planner to produce story plans
for use with our system [
While we did not use the three last domains for our main experiments, we still attempted to use them, and we will elaborate more on some of the problems we encountered below.
Based on the narrative domains discussed above, we generated 10 text outputs for every one of the four domains, using the solution plan included with Glaive (it is, of course, possible, to generate more/diferent solution plans, to produce more varied stories). For our purposes, we were interested in how well the translation of plan steps to chapters would work, what the chapters produced by ChatGPT would look like, and how consistent they are across the narrative. Due to the recent release of ChatGPT 4o we performed our experiments twice, once using ChatGPT 3.5, which has much faster processing times, and once with ChatGPT 4o, to make use of the newest version of the model.
First, despite only being given a single example step, the model is able to translate plan steps into chapter descriptions (open jill general-store) (move-once sally main-street bank) (withdraw-money sally bank dress-money) *(withdraw-money sally bank dress-money) (move-once sally bank main-street) (sell sally anne tomatoes tomato-money main-street) (move-once robbie main-street dark-alley) (hide-in-dark-alley robbie dark-alley) (move-once barney barneys-room saloon) *(move-once barney barneys-room saloon) (pickpocket robbie sally dress-money main-street dark-alley) *(pickpocket robbie sally dress-money main-street dark-alley) very well, as shown in Table 1. In most instances, the descriptions produced by the two model versions were almost identical, with only three steps being described diferently. Notably, the models sometime struggle with distinguishing subject and object of an action, but the resulting descriptions are still serviceable to turn them into chapters.
The second step is to take the chapter descriptions and having the model produce narrative text that expands the description into an entire chapter. At first glance, this produces output that reads like an actual story. For example, one of our produced texts for the heist story starts as follows (as told by ChatGPT 3.5):
In the dusty town of Redwood Creek, the sun beat down mercilessly on the wooden buildings that lined the main street. Robbie stood outside the saloon, wiping the sweat from his brow as he watched the townsfolk go about their business. He had been planning this heist for months, and today was the day he would finally put his plan into action. “Hey Robbie, you lookin’ mighty serious there,” a voice called out from behind him.
Robbie turned to see his old friend, Jake, approaching with a grin on his face. “Hey Jake, just thinking about our little job today,” Robbie replied, trying to keep his voice casual.
We’re gonna hit that bank and make of with enough gold to set us up for life.”
When do we make our move?”
Chapter description Robbie hatched a plan to rob the bank with his trusty six-shooter and brown horse to strike the mother lode.
Jill opened the general store.
Sally moved once on Main Street near the bank.
Sally withdrew money from the bank to pay for her dress. Sally withdrew money from the bank in her dress.
Sally moves once on Main Street.
Sally Anne sells tomatoes for money on Main Street. Robbie moved once from Main Street to a dark alley. Robbie hides in a dark alley.
Barney moves once in Barney’s room at the saloon.
Barney moved once from Barney’s room to the saloon. Robbie and Sally pickpocket dress money on Main Street in a dark alley.
Pickpocket Robbie stole Sally’s money from her dress in a dark alley on Main Street.
The same story, as told by ChatGPT 4o starts like this:
Red Rock, casting long shadows across the wooden storefronts and dirt-packed streets.
The air was thick with the scent of sagebrush and the distant promise of rain that never seemed to come. Robbie McGraw sat on the creaky porch of the Rusty Spur Saloon, his six-shooter resting comfortably in its holster and his brown horse, Whiskey, tethered to the hitching post nearby.
Of particular note is that while the input only mentioned a character “Robbie” by name, both versions of the model ifll in additional names for the town, other characters, or even the horse, as needed. Overall, for the heist story in particular, the models produce text that is very compelling in parts, but falls short of actually being interesting, as we will discuss in the next section.
One key point of our approach is that while we generate chapters one by one, we then have the model rewrite them in the context of the surrounding chapters. The purpose of this is to ensure a higher level of consistency. Even though the model is given the previous chapter when writing the next one, characters are not used continuously, and instead new characters are introduced, or the role of a character is changed between chapters. The rewrite attempts to address this issue by putting the chapter in context of what happens before and after it. One way to show this efect is to determine how often individual characters show up, and how the rewrite afects the output. Table 2 shows some basic statistics of the generated narratives. We used spaCy1 to perform named entity recognition for each chapter, and tracked the diferent character’s occurrences across chapters. We call characters that appear in more than 30% of chapters “main characters”, as plans often include actions performed by other characters, and characters that only show up in one or two chapters “incidental” characters. The 1https://spacy.io/ table shows several efects of the rewrite, as well as some diferences between the two model versions: In every instance, the rewritten narrative is shorter than the original, as the model is able to remove some redundancy. Characters are also utilized slightly diferently, as shown by the changing number of main and incidental characters, but a more detailed analysis of this efect is still an open problem. Also noteworthy is that the newer model is significantly more loquacious than the older version, producing almost double the output given the same input story steps.
As our approach is able to generate narratives consisting
of thousands of words, a more detailed evaluation is very
challenging. Perhaps the most desirable form of evaluating
the produced narratives would be by gathering feedback
from human readers, but we encountered two obstacles
to this: First, the sheer quantity of text the readers would
have to go through is beyond any reasonable compensation
we could ofer, particularly, because second, upon closer
inspection, the writing is not very good. Rather than
subject volunteer participants to what we do not believe to
be good literature, we set out to quantify why the writing
does not seem to be enjoyable. Below we will detail some
perhaps more anecdotal evidence, but we also attempted
to quantify some properties of the writing itself.
Subjectively, the rhythm of the writing seems artificial (which
it is), and we believe this is due to the repetitive sentence
structure. The Frazier score is a measure for syntactical
complexity [
While we believe that the results we show above already constitute a novel contribution, various problem cases we uncovered may also be of interest to future researchers. First, while our approach often results in narratives that follow the given structure, this is not always the case. Since the model is given the previous chapter input, as well as the desired next step, it has to perform a trade-of in how much attention to give to each, and at times the “most likely” continuation ignores the plan steps entirely. In one particularly noteworthy instance, ChatGPT 4o took the Heist narrative, had the bank robbery take place in chapter 3, followed by an escape by sea onto a pirate ship, which then turned into a fantasy story to chase after a powerful artifact, concluding with (as summarized by the model):
town of Port Meridian buzzes with anticipation as the day of the Great Unveiling approaches. Robbie and Talia, along with the committee, work tirelessly to decipher the Heart of the Ancients’ inscriptions. They uncover a crucial passage about the "Convergence of Realms," a moment when the boundaries between their world and the Ancients’ will blur, unlocking immense knowledge but also posing significant dangers. As the committee intensifies their eforts, Robbie and Talia resolve to ensure that this newfound wisdom is used ethically and for the greater good, heralding a new era of unity and potential.
A minor point that this conclusion also demonstrates is the tendency of the model to lead to happy endings, that reassure the reader that whatever power or treasure is obtained will be used ethically. While not “wrong”, the persistent mention of this is not well-placed in all story contexts. We suspect that this is due to some of the “safeguards” OpenAI has integrated into their system, to prevent (some) unethical output. We do not disagree with this choice, but it also shows that controlling LLM output to make it suitable for all applications is challenging.
Generally, control is a major issue when using an LLM. Our approach to rewrite chapters to make characters behave more consistently is not the only thing one could do. We attempted to tell the model outright which characters exist and what their roles are. However, this led to two problems: First, the characters have to come from somewhere. If the domain author is tasked with providing a character list, they will need to foresee which larger cast of the characters the model might need, which may also lead to higher repetitiveness of the generated stories. Our solution to that was to let the model generate an (ideally) varied list of characters to use. However, the model does favor certain names for diferent genres (Robbie’s partner in the heist story is usually called “Hank” or “Jake”, despite neither showing up in the input). The second problem, though, is that the model does not seem to have enough context to work with any character list that is given to it. When instructed to use specific names, the text might initially use the provided names, but they often change back to the names favored by the model for each particular narrative.
Even when the model follows the provided trajectory and uses characters consistently, though, it struggles with keeping a cohesive tone. Step 7 in the heist plan is “(sell sally anne tomatoes tomato-money main-street)”, which consists of a single transaction. In one instance, the model took this idea and just “ran with it”, turning one character selling tomatoes into the three protagonists hosting a tomato festival where they sell produce together with the local farmers in order to finance their travels:
on a mission to gather more tomatoes and secure a venue for the tomato festival. They approached local farmers, explaining their plan and ofering a fair share of the profits.
To their surprise, the farmers were intrigued by the idea and agreed to contribute their tomato harvest.
By itself, the tomato festival is a reasonable interpretation of the given story step, but it stood out in context, as the preceding chapter is titled “The Enigmatic Stranger”, and the following chapter “The Mysterious Stranger”. Overall, the model favored a darker, grittier narrative, which made the tomato festival seem even more out of place. On the other hand, as these other two chapter titles already indicate, encounters with strangers, risk, or danger are all narrative devices the model employs frequently. The narrative in question contains three consecutive chapters titled “A Risky Proposition”, which is then followed by “The Perilous Journey”.
It is already known that ChatGPT often produces
inaccurate information [
away from prying eyes. He signaled the bartender for two whiskeys, hoping the strong drink would sober Barney up enough to have a coherent conversation.
On the flip side, as some of the examples are based on existing media, which OpenAI may have included in the data they used to train the model, the resulting narrative may just reproduce this existing data, rather than producing a new one. This was particularly challenging with the Aladdin story, but for the Indiana Jones-based story another interesting phenomenon occurred: Rather than interpreting “Indiana” as a name (even in context with the Ark and the location of Tanis), ChatGPT would take it as the US state of Indiana, and then either name a town there “Tanis” or turn “Tanis” into a character, as in this example:
horizon, casting a golden hue across the small town of Maplewood, Indiana. The streets were still quiet, with only the occasional chirping of birds breaking the silence.
Tanis stood at the edge of her driveway, her backpack slung over one shoulder and a map of Indiana clutched in her hand. She took a deep breath, feeling the crisp morning air fill her lungs. Today was the day she had been waiting for. “Are you sure about this, Tanis?” Her best friend, Mia, asked as she approached. Mia’s eyes were filled with concern, but there was also a hint of excitement. “It’s a long way to Indianapolis, and you know how unpredictable things can get.”
Overall, using ChatGPT for narrative generation seems to produce reasonable output on the surface, but once one looks at the text more closely, problems keep arising almost fractally, as one digs deeper on a problem another one shows up. The model has a general notion of what a “good” narrative would look like, but no understanding of lfow, composition, coherence, common sense, or purpose. Attempts to rectify this by better prompts are only partially successful, as providing too much instruction to the model makes it more likely to ignore parts of it, while providing too little guidance results in rambling. We will conclude with the entirety of chapter 1 for one iteration of the Ark story (using ChatGPT 4o):
We have presented an approach to using ChatGPT, to produce long-form text for discourse generation – or, more precisely, surface text realization. Our approach takes story steps produced by a narrative planner and tasks the model with first translating the abstract step output to descriptions of individual chapters, and then turn these descriptions into actual chapter text. We perform another pass over the chapters where we ask the model to summarize each chapter, and then rewrite it using the summaries of the preceding and following chapters as additional context. We show several example outputs of our model, and discuss how challenges with an evaluation of rather long texts that are not particularly well written. Crucially, we also investigate why the texts do not appear to read well, and show an analysis of the grammatical complexity of the generated narratives, which tend to be more complex than comparable human-written literature. Finally, we discuss a myriad of other problems we encountered that led to narratives that were ill-formed, illogical or incongruous.
While our work presents a somewhat bleak outlook on the current state of narrative generation using LLMs, we believe these insights are crucial to understanding what makes text seem artificial. The alien-ness of AI text may have been intuitively understood, but our work attempts to quantify it, which may in turn lead to improvements in output, or - perhaps more importantly - highlight the importance of human authorship. Our approach also focused on surface text realization without taking larger questions of discourse generation into account. Some repetitiveness may also be alleviated by not having the model generate an entire chapter for rather trivial move actions.