Generating poetry has become a popular application of LLMs, perhaps especially of OpenAI's widelyused chatbot ChatGPT. What kind of poet is ChatGPT? Does ChatGPT have its own poetic style? Can it successfully produce poems idniferent styles? To answer these questions, we prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 diferent poetic forms and styles, about 40 diferent subjects, and in response to 3 diferent writing prompt templates. We then analyze the resulting 5.7k poems, comparing them to a sample of 3.7k poems from the Poetry Foundation and the Academy of American Poets. We find that the GPT models, especially GPT-4, can successfully produce poems in a range of both common and uncommon English-language forms in superficial yet noteworthy ways, such as by producing poems of appropriate lengths for sonnets (14 lines), villanelles (19 lines), and sestinas (39 lines). But the GPT models also exhibit their own distinct stylistic tendencies, both within and outside of these specific forms. Our results show that GPT poetry is much more constrained and uniform than human poetry, showing a strong penchant for rhyme, quatrains (4-line stanzas), iambic meter, first-person plural perspectives (we, us, our), and specific vocabulary like “heart,” “embrace,” “echo,” and “whisper.” Generating poetry has become a popular application of LLMs, perhaps especially of OpenAI's widely-used ChatGPT. The chatbot's ability to produce poetry was frequently mentioned in early journalistic repor3t6s],[and the question of whether it produced “good” or “bad” poetry quickly became the subject of hot deba1t2e, [5]. While these debates point to the significance of LLM-generated poetry as a topic, in this paper we are less interested in subjectively evaluating the quality of ChatGPT-generated poems and more interested in quantifying and describing key patterns in the poetry produced. What kind of poet is ChatGPT? Does ChatGPT have its own poetic style? Can it successfully produce poems in diferent forms and styles, like sonnets (14 lines with regular rhyme and iambic pentameter) or villanelles (19 lines with alternating ifrst and third lines)? In this paper, we present initial results from a larger study of LLM-generated poetry. We prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 diferent poetic forms or styles, about 40 diferent subjects, and in response to 3 diferent templates.
GPT-4 Prompt: Write a poem about the subject of social Iambic End commentaries in the following form or style: limerick. Meter Rhyme
A world viewed through a smartphone's lens, A Each day a stream of trends begins, A Tales of fortune, tales of woes, B Each one judged as the wind blows. B In the glow of public opinions' glare, Endless chatter fills the air.
Russian bots to kitty memes,
Nothing's truly as it seems. itsraanu BTWheoheriilrndtdlhyotwhueigsshdcotrsmeet,hnoesyr, jtwuwshetoentco,aiinnsest?hpeeyebche?so free, Q s Undiscovered truth, or toys?
As they debate who's wrong or right, Their fingers dance in virtual fight.
From east to west, the judgments fly, Sometimes truth, oftentimes lie.
We see the world in constant chime, A universe of views online, In every tweet and post we glean, A reflection of a world unseen.
A A B B A A B B A A B B A A B B
We then analyze the resulting 5.7k poems, comparing them to a sample of 3.7k poems from the Poetry Foundation and the Academy of American Poets. We find that the GPT models, especially GPT-4, can successfully produce poems in a range of both common and uncommon English-language forms in superficial yet noteworthy ways, such as by producing poems of appropriate lengths for sonnets (14 lines), villanelles (19 lines), and sestinas (39 lines). But we ifnd that the GPT models also exhibit their own distinct stylistic tendencies, both within and outside of these specific forms.
Our results show that the poetry produced by GPT-3.5 and GPT-4 is much more constrained and uniform than human poetry. Unless otherwise prompted (and sometimes when otherwise prompted), both GPT models have a tendency to produce rhymed lines in something like iambic meter—a regular pattern of unstressed and stressed syllables that characterizes the majority of English-language verse before the 20th centu29r]y. [Both models also have a tendency to organize poetic lines into quatrains (4-line stanzas). And they display other distinct signatures, such as a curiously dominant first-person plural perspective and a penchant for words like “heart,” “embrace,” “echoes,” and “whispers.” We release our c1oadned, we plan to share our ChatGPT poetry corpus at a later stage of the project, with the aim of encouraging further analysis through computational and/or more traditional literary studies approaches. 1https://github.com/melaniewalsh/chatgpt_poetry
The history of computational poetry generation dates back to at least the mid-20t1h5]c.entury [
Poets, researchers, and hobbyists have experimented with a range of technical appr9]o,aches [
from rule-based systems19[
While computational poetry has been an active area of inquiry for many d2e0c,a3d0,e1s0[
Training data and memorization are key considerations for LLM-generated poetry from
ethical, legal, and technical perspectives. Models’ ability to produce poetry is intimately tied to
their training data, which partly consists of literary works by both living and dead writers.
Much popular and scholarly attention related to LLMs and literature has focused—rightfully—
on the ethics and legality of such literary trainin2g8d,a17t,a3[
In their work on poetry memorization in ChatGPT specifically, D’Souza and Mi6m]nshoo[w that the most likely factor for a poem’s memorization by the model was its inclusion in the 1983 Norton Anthology of Literature. This finding suggests that canonical poetry is disproportionately represented in the GPT models, which could influence the kind of poetry they produce. In a similar vein, we show in prior wor3k3][that 41% of a curated sample of poems from the Poetry Foundation and the Academy of American Poets (which we also use as a comparison corpus in this study) are likely memorized by GPT-4. Our previous analysis suggests that this memorization may enhance the models’ ability to classify the form of the poems, but the results are not conclusive. More work is needed to evaluate the impact that memorization may have on poetry generation.
To provide a baseline comparison for our ChatGPT-generated poetry and to guide our prompting, we curate a dataset of poems, styles, and subjects from the Poetry Foundation and the Academy of American Poets. Both organizations are well-respected poetry institutions with websites that host tens of thousands of poems spanning hundreds of years, and many of the poems are tagged by style and subject on the websites.
We scrape up to 400 poems from these two sources for 23 diferent poetic forms or styles,
which we also use as prompts for our ChatGPT-generated poetry corpus. Following prior work
[
We manually remove prefatory text—such as dedications, dates, epigraphs, or other contextual information—from human-authored poems with traditionally fixed lengths (e.g., sonnets, villanelles, sestinas) if the poem is within 10 lines of the conventional length. We do not remove prefatory material from other poems; however, based on our qualitative analysis and review, we do not believe prefatory material is extensive in most other poems or significantly impacts results.
While the Poetry Foundation and the Academy of American Poets are among the largest tagged poetry collections available, they are also defined by various kinds of bias that are important to note. They both focus on English-language poetry, and the Academy of American Poets focuses especially on American poetry. Not all of the poems in their collections are tagged, and it is unclear why some poems are tagged and others are not. Neither site hosts a
lore, nature, religion, relationships, social commentaries
funerals, recovery, graduation, gratitude, toasts, weddings
loween, hanukkah, independence day, kwanzaa, memorial day, mother’s day, new year, passover, ramadan, thanksgiving, yom kippur
haiku
egy, visual poetry, ekphrasis, prose poem
verse stanza forms: quatrain, tercet, couplet
or style: Y.
form or style: Y. Do not use the actual word(s) X or Y in the poem.
Specific: Write a poem about the subject of X in the following form or style: Y. Make the poem about something specific. representative collection of poems, in terms of poets’ gender, race, sexuality, and time period (it is also difÏcult to know what a representative collection would be). They also over-represent prestigious and canonical poetry, which may be of particular note in comparison with ChatGPT since the model may be trained on (and perhaps even encouraged to produce) more popular, commercial, and colloquial poetry.
To create our ChatGPT poetry corpus, we proGmPpTt-3.5 Turbo andGPT-4 [
We select styles and subjects from the Poetry Foundation because they ofer an extensive and diverse poetic taxonomy that is developed by an authoritative external source and that is reflective of one of the largest existing collections of human poetry. For our “subjects,” we select the 40 broadest level “topics” from the Poetry Foundation’s tagging schema, which include the subcategories “subjects,” “occasions,” and “holidays” (see T2a)b.lFeor our “styles,” we select the 23 styles and poetic forms described in Sect3.i1o.nWe add the style of “a poem” because we are interested in the models’ responses to the generic idea of a poem without a specified form. These combinations result in 2,880 generated poems per model, with 120 poems per style (per model) and 72 poems per subject (per model) (see Ta1b)l.e
We model the construction of ouwrr3iting prompt templates on popular approaches
demonstrated on social media, in journalistic articles, and by LLM com1p4a,n1i2e,s2[
We include our “figurative“ and “specific“ templates after observing the models’ tendency to repeat the words in the prompts and to be vague. These templates push the model to create more diverse outputs. We believe that prompting significantly impacts the kind of poetry that the GPT models produce, and we reflect on this more in Sect5i.on
We measure the number of lines and the number and kind of stanzas across all the poems by parsing line breaks. We visualize these distributions as boxplots2()Fiagnudreheatmaps (Figure3), revealing the most common lengths and shapes of the poems across styles and forms. These results show that when we prompt the models to generate poems in forms with typically ifxed lengths—such assonnets (14 lines)—they largely adhere to this convention, with notable improvement in GPT-4. While GPT-3.5 and GPT-4 both generate sonnets with a median average length of 14 lines, Figur2esand3 show that there is much more variability in GPT-3.5. As displayed in the boxplot, the upper 75% quartile extends to 32 lines, and the range extends to 55 lines. By contrast, the entire range of GPT-4 sonnets (minus outliers) falls at exactly 14 lines. Line lengths for sestinas (typically 39 lines) and villanelles (typically 19 lines) follow a similar pattern. The median lengths are appropriately 39 and 19 lines for both models, but GPT-4 demonstrates much more consistency, displaying a smaller interquartile range and spread of outliers (the same consistency is also displayed in the heatmap in3F).igure
Interestingly, for these three forms, GPT-4 hues closer to “conventional” lengths than our sample of poems from the Poetry Foundation and the Academy of American Poets. In a small percentage of these human poems, we find that the longer lengths come from explicit or implicit play with or defiance of the forms. For example, Bino A. Realuyo concludes his 15-line poem, “Euler’s Equationw,”ith the line: “a rebellion, the fiteenth line of a sonnet.” More often, in these longer poems, authors include a given form in multiples, such as Algernon Charles Swinburne’s “double sestina” (12 stanzas of 12 lines each“)Tihne Complaint of Lisa.”
An obvious aberration for the GPT models is their atypically long limerick style. Where a traditional limerick is usually about 5 lines long, the median length for both GPT models is 25 lines. Upon closer inspection, it is clear that both models frequently bundle multiple, appropriately-lengthed limericks together. The heatmap in 3Fisghuorwes that the models often produce several limericks in a row. This is also the case for GPT-3.5’s atypically long sonnets, which are usually multiple sonnets packed into one. While these multiples resemble the long poems that we observe in our human poetry sample, we think this tendency more likely suggests that, in certain cases, the GPT models hkonwowto produce a particular kind of poem but don’t knowwhen to stop.
Aside from limericks, the GPT models can broadly produce poems of appropriately diverse lengths for a range of fixed forms, but they demonstrate a penchant for producing poems of an almost “default” size when left to their own devices. For both models, the median average length for a generic “poem” is 36 lines (see Figu2)r,eand the overall median length across all styles is 32 lines.
Another striking feature of the GPT poems is the dominance of 4-line stanqzuaast,raoirns. We find that while just 16.7% of the human-authored stanzas are quatrains, a whopping 66.8% of all GPT-3.5 stanzas and 59.6% of all GPT-4 stanzas are quatrains3()T.aTbhlee heatmaps in Figure4 visually demonstrate how common quatrains are across the GPT-generated poems, showing clear line breaks in regular 4-line intervals, with no such regularity evident in the human poems.
We measure the normalized frequency of pronouns (T4a)bilne each corpus, expressed per 100 words. We find that poems produced by GPT-3.5 and GPT-4 tend to use more first-person plural pronouns (“we,” “us,” “our”) and fewer first-person singular pronouns (“i”, “me,” “myself”) than poems written by humans (see Figur5e). For example, GPT-4 produced the following limerick about Memorial Day in response to ofiguurrative prompt (which specifies not to include the style or subject words in the poem):
In May we stand strong, hearts ablaze, For those who’ve seen war’s smoky haze.
We honor the brave, Who life for us gave,
In silence, we give them our praise.
This limerick continues on for 20 more lines in 5-line, rhyming (AABBA) stanzas, consistent with the results presented in Sect4.i1o.n
Because there are a large number of “holiday” and “occasion” subject prompts like “Memorial Day,” which perhaps encourage meditation on collective experiences, we also show normalized frequency for the GPT-generated poems with these subjects removed (see the dotted lines in Figure5). Without these subjects, the normalized frequency for the first-person plural decreases slightly, and it increases slightly for the third-person. But the curious dominance of the first-person plural is still present. We think this pattern may reflect the models’ preprogrammed attitudes toward inclusivity, as well as its obvious lack of first-person singular experiences, but more work is needed to explore this trend further.
We also analyze the most distinctive opening words and overall words across the poems using
Monroe, Colaresi, and Quin2n3][’s “fightin’ words” algorithm, which uses weighted log-odds
ratios with an informative Dirichlet prior. This method is designed to robustly compare word
usage across unevenly distributed text corpora. We specifically use an implementation by
Hessel [
The most distinctive opening word in both the GPT-3.5- and GPT-4 generated poems is “In” (see Figure6). This preposition is included across a wide range of poems and contexts, such as: In autumn’s blaze of golden hue... (GPT-3.5 — pantoum, Thanksgiving) In the darkest days, a flicker of light...(GPT-3.5 — ars poetica, Hanukkah) In the girth of world-kaleidoscope, we are birthed into living, (GPT-4 — free verse, living) The next most distinctive first word in GPT-4 poems is “Upon,” which also seems to be a frequent way for the model to initiate iambic meter: Upon a stage where shadows nightly reign... (GPT-4 — sonnet, Halloween) Upon this day, we sing the laborer’s song,... (GPT-4 — sonnet, Labor Day)
Upon the chill of winter’s breath descends,... (GPT-4 — blank verse, Hanukkah) The word “upon” is an iamb (the basic unit of iambic meter), meaning it consists of an unstressed syllable followed by a stressed syllable. This is also the case for other distinctive GPT-4 first words, such as “beneath,” “behold,” and “within.” First words in poems from the Poetry Foundation and Academy of American Poets show no such distinctive patterns, mostly consisting of articles and pronouns.
Overall, the distinguishing vocabulary for the GPT models consists of words associated with love (“heart,” “love,” “souls,”), words that rhyme (“grace”, “embrace”), and words that are acoustic (“echo,” “whisper”). For GPT-3.5, words like “embrace,” “grace,” “dance,” and “dreams” are touchstones. At least one of these words shows up in 87% of the GPT-3.5 poems. For GPT-4, either “echo” or “whisper” shows up in 75% of the poems. For example, both words appear in thisdramatic monologue about thearts & sciences:
Look upon me, ageless I stand, the crossing of arts and sciences, In the echoing hallways of knowledge, beneath glimmering frescoed edifices.
I am the whisperer in marbled alcoves, the scribe of thinkers’ existence, Caught in endless dialogue, between creativity and discipline’s persistence.
-GPT-4 (dramatic monologue, arts & sciences) 4.4. Prosody Analysis (Rhyme & Meter) “Prosody” refers to patterns of sound in poetry, encompassing rhyme and meter. Analyzing prosody across a large corpus poses challenges even in human-authored poems because it relies on the pronunciation of particular words in relation to each other. The text of a poem does not provide direct access to its prosody because the same word may have diferent pronunciations in diferent forms of English, and even with the same pronunciation a word might be stressed or unstressed depending on its context. For example, in Alfred, Lord Tennyson“’sUplyosesmes,” the word “I” is unstressed at the beginning of a line and stressed toward the end: “I cannot rest from travel: I will drink.” Analyzing prosody across GPT-generated poems poses additional challenges because unlike human-authored poems, which are often either clearly free verse or clearly aimed at embodying a particular metrical pattern, the meter of GPT outputs can be less precise and harder to define with a single metrical label.
To measure prosody, we thus conduct both a quantitaantdivqeualitative analysis. For our manual analysis, we take a random sample of poems in each form and, drawing on our domain expertise, hand-annotate various prosodic elements where they are discernible, including dominant meter, line-length (in terms of poetic feet—tetrameter, pentameter, etc.), rhyme scheme, and stanza patterns. We evaluate 144 poems produced by GPT-3.5 (6 in each form) and 144 poems produced by GPT-4 (6 in each form) for just over 5% of the GPT-generated corpus. We also analyze 138 poems from the human-authored corpus (6 poems in 23 forms), making up just over 3.7% of the human-authored corpus.
Over 80% of the GPT-generated poems in our random sample contain patterns of end rhyme, as compared with around 50% of the human-authored poems. Over 60% of the GPT-authored poems had a dominant iambic meter, compared to just under 40% of poems from the human corpus. When we break these results down between the GPT-3.5 and GPT 4 models, it appears that the dominance of iambic meter is lessening somewhat in the newer model. Only around 53% of GPT-4-authored poems had a dominant iambic meter compared to almost 74% of GPT3.5-authored poems. Rather than indicating a shift in the model’s default tendencies in relation to poetry, we think this change may reflect GPT-4’s increased abilitnyottporoduce iambic meter when it is prompted to produce poems in forms that do not traditionally include regular meter—for example haiku, prose poetry, or free verse.
We also conduct a quantitative analysis of rhyme across all poems using P2a6r]’rsish [ Python implementation of the CMU Pronouncing Dictio3nUarsiyn. g this dictionary, we measure how many lines end with AA, ABAB, ABBA, or ABCB rhymes. Around 90% of the poems generated by both the GPT-3.5 and GPT-4 models contained at least one end rhyme, compared to around 65% of our human-authored corpus. Moreover, for certain forsmonsnleitkseand ballads, the modelsalways produced poems with at least one end rhyme. This was also the case for unfixed forms likeaubade orpastoral, which do not typically require specific patterns of rhyme. The percentage of rhyming lines for all styles and sources is displayed in8.Figure
Taken together, our analyses suggest that there is a sort of default poetic mode in GPT models, which favors quatrains, iambic meter, and end rhyme. The models can be prompted to produce writing in other styles, but sometimes the persistent iambic/quatrain/end rhyme style still breaks through. For example, although prompted to prosdocuiacelcaommentary-related limerick, which would typically be a five-line stanza with anapestic meter and AABBA rhyme, the GPT-4 model produced the following five quatrains with iambic meter and AABB rhyme: A world viewed through a smartphone’s lens, Worldly wisdom, or just noise? Each day a stream of trends begins, Undiscovered truth, or toys? Tales of fortune, tales of woes, Each one judged as the wind blows.
In the glow of public opinions’ glare, Endless chatter fills the air.
Russian bots to kitty memes, Nothing’s truly as it seems.
We see the world in constant chime,
A universe of views online, Their thoughts they tweet, in speech so free, In every tweet and post we glean, Behind the screens, who can they be? A reflection of a world unseen.
As they debate who’s wrong or right, Their fingers dance in virtual fight.
From east to west, the judgments fly,
Sometimes truth, oftentimes lie.
While this poem is a funny and topical take on a social commentary (“Russian bots to kitty memes / Nothing’s truly as it seems”), it also highlights some of the model’s limitations. Even when instructed to produce more specific or varied forms, the model’s default poetic mode— quatrains, iambic meter, and end rhyme—often resurfaces, favoring structured patterns over more diverse styles.
By showing that GPT-generated poetry contains clear stylistic quirks and characteristics, we
build upon emerging research on the style of LLM-generated texts. For example, after
analyzing millions of biomedical article abstracts, Kobak, González-Márquez, Horvát, and Lause
[
The results of our poetry prompting experiments highlight both advancements and notable limitations with GPT-generated poetry. The models’ ability to produce poems of appropriate lengths for a wide variety of forms and styles—without any fine-tuning—marks a significant development in automatic poetry generation, especially since the models manage to do so while incorporating rhyme and meter and maintaining general clarity. Yet, overall, the models also exhibit far less variation, diversity, and creativity than the human-authored poems. However, we want to flag thatprompting—what the user asks the model to generate and how that ask is constructed—plays a major role in shaping the poetry that the models produce. In related experiments, when we prompted the models with specific author names, our results seemed to shift and become more complex. In this specific study, our goal was not to produce the most creative and interesting poetry possible, but rather to understand the broad contours of the models and their outputs. If we wanted to produce more interesting poetry, we would likely use diferent prompts.
We prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 diferent poetic forms or styles, about 40 diferent subjects, and in response to 3 diferent templates. We compare these GPT-generated poems to a sample of poems from the Poetry Foundation and the Academy of American Poets, showing that the GPT models are much more formulaic and constrained than the human-authored poetry. We argue that the GPT models have a “default” poetic mode, characterized by quatrains with rhymed lines in iambic meter; first-person plural perspectives; and the repetition of words like “heart,” “embrace,” “echoes,” and “whispers.” This default mode sometimes breaks through even when otherwise prompted. We share the code that we used to conduct this anal4yasinsd, we share all the public domain human-authored poems and form/style annotations from prior w5oWrke. plan to share our ChatGPT poetry corpus at a later stage of the project. In future work, we plan to explore a wider range of prompts (potentially including author names) and models, and to study the poems more closely with traditional literary studies approaches.
We would like to thank Zoe LeBlanc, John Ladd, Matt Lavin, and Gabi Kirillof for early feedback on this project. This research was partly supported by the NEH-funded ”AI for Humanists” project and by an ”AI, Creativity, and Humanities” collaboration grant from the Simpson Center for the Humanities at the University of Washington. [31] C. Veltman. “AI is contentious among authors. So why are some feeding it their own writing?” InN:pr (2024). url: https://www.npr.org/2024/04/30/1246686825/authors-usi ng-ai-artificial-intelligence-to-.write [32] G. D. Vynck. “AI learned from their work. Now they want compensatioWn.”aIsnh:ington Post (2023). url: https://www.washingtonpost.com/technology/2023/07/16/ai-program s-training-lawsuits-fair- u.se/