Generating poetry has become a popular application of LLMs, perhaps especially of OpenAI's widely-used ChatGPT. The chatbot's ability to produce poetry was frequently mentioned in early journalistic reports [36], and the question of whether it produced "good" or "bad" poetry quickly became the subject of hot debate [12,5]. While these debates point to the significance of LLM-generated poetry as a topic, in this paper we are less interested in subjectively evaluating the quality of ChatGPT-generated poems and more interested in quantifying and describing key patterns in the poetry produced. What kind of poet is ChatGPT? Does ChatGPT have its own poetic style? Can it successfully produce poems in different forms and styles, like sonnets (14 lines with regular rhyme and iambic pentameter) or villanelles (19 lines with alternating first and third lines)?

In this paper, we present initial results from a larger study of LLM-generated poetry. We prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 different poetic forms or styles, about 40 different subjects, and in response to 3 different templates. Figure 1: "Write a poem about the subject of social commentaries in the following form or style: limerick." An example poetry generation prompt and response by GPT-4, representing common tendencies of the model. While GPT-4 presents a comedic, topical take on a social commentary, it does not produce a typical limerick (usually 5 lines, anapestic meter, AABBA rhyme) but instead produces five quatrains with mostly iambic meter and AABB rhyme, what we suggest is its "default" mode.

We then analyze the resulting 5.7k poems, comparing them to a sample of 3.7k poems from the Poetry Foundation and the Academy of American Poets. We find that the GPT models, especially GPT-4, can successfully produce poems in a range of both common and uncommon English-language forms in superficial yet noteworthy ways, such as by producing poems of appropriate lengths for sonnets (14 lines), villanelles (19 lines), and sestinas (39 lines). But we find that the GPT models also exhibit their own distinct stylistic tendencies, both within and outside of these specific forms.

Our results show that the poetry produced by GPT-3.5 and GPT-4 is much more constrained and uniform than human poetry. Unless otherwise prompted (and sometimes when otherwise prompted), both GPT models have a tendency to produce rhymed lines in something like iambic meter-a regular pattern of unstressed and stressed syllables that characterizes the majority of English-language verse before the 20th century [29]. Both models also have a tendency to organize poetic lines into quatrains (4-line stanzas). And they display other distinct signatures, such as a curiously dominant first-person plural perspective and a penchant for words like "heart, " "embrace, " "echoes, " and "whispers. " We release our code, 1 and we plan to share our ChatGPT poetry corpus at a later stage of the project, with the aim of encouraging further analysis through computational and/or more traditional literary studies approaches.

The history of computational poetry generation dates back to at least the mid-20th century [15]. Poets, researchers, and hobbyists have experimented with a range of technical approaches [9], from rule-based systems [19,8], to Markov chains [2], to most recently neural networks and LLMs [34,35,18,13,1,3,25].

While computational poetry has been an active area of inquiry for many decades [20,30,10,7,27,21], it has arguably remained the purview of specialists until recently. But since the release of ChatGPT in November 2022, hundreds of millions of people have used and experimented with LLMs, opening up computational poetry generation to a broader public.

Training data and memorization are key considerations for LLM-generated poetry from ethical, legal, and technical perspectives. Models' ability to produce poetry is intimately tied to their training data, which partly consists of literary works by both living and dead writers. Much popular and scholarly attention related to LLMs and literature has focused-rightfullyon the ethics and legality of such literary training data [28,17,31,32]. We believe these concerns are vitally important to examinations of ChatGPT's style, which is built from the words of other writers. We also think that it is valuable to ask questions about LLMs' poetic capacities because it can help inform debates about LLMs and creativity while also advancing our understanding of how poetry is being used and propagated in the contemporary world.

In their work on poetry memorization in ChatGPT specifically, D'Souza and Mimno [6] show that the most likely factor for a poem's memorization by the model was its inclusion in the 1983 Norton Anthology of Literature. This finding suggests that canonical poetry is disproportionately represented in the GPT models, which could influence the kind of poetry they produce. In a similar vein, we show in prior work [33] that 41% of a curated sample of poems from the Poetry Foundation and the Academy of American Poets (which we also use as a comparison corpus in this study) are likely memorized by GPT-4. Our previous analysis suggests that this memorization may enhance the models' ability to classify the form of the poems, but the results are not conclusive. More work is needed to evaluate the impact that memorization may have on poetry generation.

To provide a baseline comparison for our ChatGPT-generated poetry and to guide our prompting, we curate a dataset of poems, styles, and subjects from the Poetry Foundation and the Academy of American Poets. Both organizations are well-respected poetry institutions with websites that host tens of thousands of poems spanning hundreds of years, and many of the poems are tagged by style and subject on the websites.

We scrape up to 400 poems from these two sources for 23 different poetic forms or styles, which we also use as prompts for our ChatGPT-generated poetry corpus. Following prior work [33], we select poems in the following categories: fixed forms, unfixed forms, and formal elements (which consists of both meters and stanza forms) (see Table 1). In total, the sample includes 3,874 poem/style pairs, or 3,692 unique poems. We manually remove prefatory text-such as dedications, dates, epigraphs, or other contextual information-from human-authored poems with traditionally fixed lengths (e.g., sonnets, villanelles, sestinas) if the poem is within 10 lines of the conventional length. We do not remove prefatory material from other poems; however, based on our qualitative analysis and review, we do not believe prefatory material is extensive in most other poems or significantly impacts results.

While the Poetry Foundation and the Academy of American Poets are among the largest tagged poetry collections available, they are also defined by various kinds of bias that are important to note. They both focus on English-language poetry, and the Academy of American Poets focuses especially on American poetry. Not all of the poems in their collections are tagged, and it is unclear why some poems are tagged and others are not. Neither site hosts a representative collection of poems, in terms of poets' gender, race, sexuality, and time period (it is also difÏcult to know what a representative collection would be). They also over-represent prestigious and canonical poetry, which may be of particular note in comparison with ChatGPT since the model may be trained on (and perhaps even encouraged to produce) more popular, commercial, and colloquial poetry.

To create our ChatGPT poetry corpus, we prompt GPT-3.5 Turbo and GPT-4 [24] to generate poems in response to 3 different writing prompt templates, in 24 different styles/forms, and about 40 different subjects. The styles and subjects are selected from the tagging schema on the Poetry Foundation's website. We use zero-shot prompts (i.e., prompts that do not provide desired example outputs) because we are interested in testing the model's "out-of-the-box" capabilities in a mostly unmediated form. We select styles and subjects from the Poetry Foundation because they offer an extensive and diverse poetic taxonomy that is developed by an authoritative external source and that is reflective of one of the largest existing collections of human poetry. For our "subjects, " we select the 40 broadest level "topics" from the Poetry Foundation's tagging schema, which include the subcategories "subjects, " "occasions, " and "holidays" (see Table 2). For our "styles, " we select the 23 styles and poetic forms described in Section 3.1. We add the style of "a poem" because we are interested in the models' responses to the generic idea of a poem without a specified form. These combinations result in 2,880 generated poems per model, with 120 poems per style (per model) and 72 poems per subject (per model) (see Table 1).

We model the construction of our 3 writing prompt templates on popular approaches demonstrated on social media, in journalistic articles, and by LLM companies [14,12,22]:

1. General: Write a poem about the subject of X in the following form or style: Y.

2. Figurative: Write a poem about the subject of X in the following form or style: Y. Do not use the actual word(s) X or Y in the poem.

3. Specific: Write a poem about the subject of X in the following form or style: Y. Make the poem about something specific.

We include our "figurative" and "specific" templates after observing the models' tendency to repeat the words in the prompts and to be vague. These templates push the model to create more diverse outputs. We believe that prompting significantly impacts the kind of poetry that the GPT models produce, and we reflect on this more in Section 5.

We measure the number of lines and the number and kind of stanzas across all the poems by parsing line breaks. We visualize these distributions as boxplots (Figure 2) and heatmaps (Figure 3), revealing the most common lengths and shapes of the poems across styles and forms. These results show that when we prompt the models to generate poems in forms with typically fixed lengths-such as sonnets (14 lines)-they largely adhere to this convention, with notable improvement in GPT-4. While GPT-3.5 and GPT-4 both generate sonnets with a median average length of 14 lines, Figures 2 and 3 show that there is much more variability in GPT-3.5. As displayed in the boxplot, the upper 75% quartile extends to 32 lines, and the range extends to 55 lines. By contrast, the entire range of GPT-4 sonnets (minus outliers) falls at exactly 14 lines. Line lengths for sestinas (typically 39 lines) and villanelles (typically 19 lines) follow a similar pattern. The median lengths are appropriately 39 and 19 lines for both models, but GPT-4 demonstrates much more consistency, displaying a smaller interquartile range and spread of outliers (the same consistency is also displayed in the heatmap in Figure 3). Interestingly, for these three forms, GPT-4 hues closer to "conventional" lengths than our sample of poems from the Poetry Foundation and the Academy of American Poets. In a small percentage of these human poems, we find that the longer lengths come from explicit or implicit play with or defiance of the forms. For example, Bino A. Realuyo concludes his 15-line poem, "Euler's Equation, " with the line: "a rebellion, the fifteenth line of a sonnet. " More often, in these longer poems, authors include a given form in multiples, such as Algernon Charles Swinburne's "double sestina" (12 stanzas of 12 lines each) in "The Complaint of Lisa. " The GPT models were also prompted with the generic style of "a poem"; to provide a comparison for the human poems, we include an aggregation of all poems from the sample. The boxes show the "interquartile range" (25% quartile-75% quartile) with a thicker line indicating the median average; the whiskers extend beyond the boxes by 1.5 times the IQR; the outliers are values that fall beyond the whiskers. The dotted red line indicates the expected number of lines for each form, e.g., a sonnet typically has 14 lines.

An obvious aberration for the GPT models is their atypically long limerick style. Where a traditional limerick is usually about 5 lines long, the median length for both GPT models is 25 lines. Upon closer inspection, it is clear that both models frequently bundle multiple, appropriately-lengthed limericks together. The heatmap in Figure 3 shows that the models often produce several limericks in a row. This is also the case for GPT-3.5's atypically long sonnets, which are usually multiple sonnets packed into one. While these multiples resemble the long poems that we observe in our human poetry sample, we think this tendency more likely suggests that, in certain cases, the GPT models know how to produce a particular kind of poem but don't know when to stop. Aside from limericks, the GPT models can broadly produce poems of appropriately diverse lengths for a range of fixed forms, but they demonstrate a penchant for producing poems of an almost "default" size when left to their own devices. For both models, the median average length for a generic "poem" is 36 lines (see Figure 2), and the overall median length across all Darker squares represent a higher concentration of words and lines in specific positions across the poems; lighter squares represent a higher concentration of white space and line breaks. The GPT models are also prompted with the generic style of "a poem"; to provide a comparison for the human poems, we include an aggregation of all poems from the sample.

styles is 32 lines.

Another striking feature of the GPT poems is the dominance of 4-line stanzas, or quatrains. We find that while just 16.7% of the human-authored stanzas are quatrains, a whopping 66.8% of all GPT-3.5 stanzas and 59.6% of all GPT-4 stanzas are quatrains (Table 3). The heatmaps in Figure 4 visually demonstrate how common quatrains are across the GPT-generated poems, showing clear line breaks in regular 4-line intervals, with no such regularity evident in the human poems.

We measure the normalized frequency of pronouns (Table 4) in each corpus, expressed per 100 words. We find that poems produced by GPT-3.5 and GPT-4 tend to use more first-person plural pronouns ("we, " "us, " "our") and fewer first-person singular pronouns ("i", "me, " "myself") than poems written by humans (see Figure 5). For example, GPT-4 produced the following limerick about Memorial Day in response to our figurative prompt (which specifies not to include the style or subject words in the poem):

In May we stand strong, hearts ablaze, For those who've seen war's smoky haze. We honor the brave, Who life for us gave, In silence, we give them our praise.

This limerick continues on for 20 more lines in 5-line, rhyming (AABBA) stanzas, consistent with the results presented in Section 4.1.

Because there are a large number of "holiday" and "occasion" subject prompts like "Memorial Day, " which perhaps encourage meditation on collective experiences, we also show normalized frequency for the GPT-generated poems with these subjects removed (see the dotted lines in Figure 5). Without these subjects, the normalized frequency for the first-person plural decreases slightly, and it increases slightly for the third-person. But the curious dominance of the first-person plural is still present. We think this pattern may reflect the models' preprogrammed attitudes toward inclusivity, as well as its obvious lack of first-person singular experiences, but more work is needed to explore this trend further.

We also analyze the most distinctive opening words and overall words across the poems using Monroe, Colaresi, and Quinn [23]'s "fightin' words" algorithm, which uses weighted log-odds ratios with an informative Dirichlet prior. This method is designed to robustly compare word usage across unevenly distributed text corpora. We specifically use an implementation by Hessel [11] 2 and restrict the vocabulary to words that appear in a minimum of 10 poems. We remove stopwords for our overall word analysis but not for our first word analysis. In Figures 6 and 7, we display words with the highest Z-scores for each category, representing the most distinctive words. The most distinctive opening word in both the GPT-3.5-and GPT-4 generated poems is "In" (see Figure 6). This preposition is included across a wide range of poems and contexts, such as: The next most distinctive first word in GPT-4 poems is "Upon, " which also seems to be a frequent way for the model to initiate iambic meter:

Upon a stage where shadows nightly reign... (GPT-4 -sonnet, Halloween) Upon this day, we sing the laborer's song,... (GPT-4 -sonnet, Labor Day) Upon the chill of winter's breath descends,..

The word "upon" is an iamb (the basic unit of iambic meter), meaning it consists of an unstressed syllable followed by a stressed syllable. This is also the case for other distinctive GPT-4 first words, such as "beneath, " "behold, " and "within. " First words in poems from the Poetry Foundation and Academy of American Poets show no such distinctive patterns, mostly consisting of articles and pronouns. Overall, the distinguishing vocabulary for the GPT models consists of words associated with love ("heart, " "love, " "souls, "), words that rhyme ("grace", "embrace"), and words that are acoustic ("echo, " "whisper"). For GPT-3.5, words like "embrace, " "grace, " "dance, " and "dreams" are touchstones. At least one of these words shows up in 87% of the GPT-3.5 poems. For GPT-4, either "echo" or "whisper" shows up in 75% of the poems. For example, both words appear in this dramatic monologue about the arts & sciences: Look upon me, ageless I stand, the crossing of arts and sciences, In the echoing hallways of knowledge, beneath glimmering frescoed edifices. I am the whisperer in marbled alcoves, the scribe of thinkers' existence, Caught in endless dialogue, between creativity and discipline's persistence.

-GPT-4 (dramatic monologue, arts & sciences)

"Prosody" refers to patterns of sound in poetry, encompassing rhyme and meter. Analyzing prosody across a large corpus poses challenges even in human-authored poems because it relies on the pronunciation of particular words in relation to each other. The text of a poem does not provide direct access to its prosody because the same word may have different pronunciations in different forms of English, and even with the same pronunciation a word might be stressed or unstressed depending on its context. For example, in Alfred, Lord Tennyson's poem "Ulysses, " the word "I" is unstressed at the beginning of a line and stressed toward the end: "I cannot rest from travel: I will drink. " Analyzing prosody across GPT-generated poems poses additional challenges because unlike human-authored poems, which are often either clearly free verse or clearly aimed at embodying a particular metrical pattern, the meter of GPT outputs can be less precise and harder to define with a single metrical label. To measure prosody, we thus conduct both a quantitative and qualitative analysis. For our manual analysis, we take a random sample of poems in each form and, drawing on our domain expertise, hand-annotate various prosodic elements where they are discernible, including dominant meter, line-length (in terms of poetic feet-tetrameter, pentameter, etc.), rhyme scheme, and stanza patterns. We evaluate 144 poems produced by GPT-3.5 (6 in each form) and 144 poems produced by GPT-4 (6 in each form) for just over 5% of the GPT-generated corpus. We also analyze 138 poems from the human-authored corpus (6 poems in 23 forms), making up just over 3.7% of the human-authored corpus.

Over 80% of the GPT-generated poems in our random sample contain patterns of end rhyme, as compared with around 50% of the human-authored poems. Over 60% of the GPT-authored poems had a dominant iambic meter, compared to just under 40% of poems from the human corpus. When we break these results down between the GPT-3.5 and GPT 4 models, it appears that the dominance of iambic meter is lessening somewhat in the newer model. Only around 53% of GPT-4-authored poems had a dominant iambic meter compared to almost 74% of GPT-3.5-authored poems. Rather than indicating a shift in the model's default tendencies in relation to poetry, we think this change may reflect GPT-4's increased ability to not produce iambic when instructed to produce more specific or varied forms, the model's default poetic modequatrains, iambic meter, and end rhyme-often resurfaces, favoring structured patterns over more diverse styles.

By showing that GPT-generated poetry contains clear stylistic quirks and characteristics, we build upon emerging research on the style of LLM-generated texts. For example, after analyzing millions of biomedical article abstracts, Kobak, González-Márquez, Horvát, and Lause [16] show that "hundreds of words have abruptly increased their frequency after ChatGPT became available. " They reveal particular spikes in the use of "style-affecting verbs and adjectives that ChatGPT-like LLMs prefer, " such as "delve, " "significant, " and "crucial. " Although Kobak, González-Márquez, Horvát, and Lause [16] are primarily interested in LLMs' impact on academic research, their findings contribute to growing knowledge about LLM style in specific genres. Other related research in this area has focused on analyzing bias and stereotypes present in LLM-generated texts [4]. We believe there are exciting opportunities for digital humanities scholars and language experts to study more of the artistic dimensions of LLMgenerated texts.

The results of our poetry prompting experiments highlight both advancements and notable limitations with GPT-generated poetry. The models' ability to produce poems of appropriate lengths for a wide variety of forms and styles-without any fine-tuning-marks a significant development in automatic poetry generation, especially since the models manage to do so while incorporating rhyme and meter and maintaining general clarity. Yet, overall, the models also exhibit far less variation, diversity, and creativity than the human-authored poems. However, we want to flag that prompting-what the user asks the model to generate and how that ask is constructed-plays a major role in shaping the poetry that the models produce. In related experiments, when we prompted the models with specific author names, our results seemed to shift and become more complex. In this specific study, our goal was not to produce the most creative and interesting poetry possible, but rather to understand the broad contours of the models and their outputs. If we wanted to produce more interesting poetry, we would likely use different prompts.

We prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 different poetic forms or styles, about 40 different subjects, and in response to 3 different templates. We compare these GPT-generated poems to a sample of poems from the Poetry Foundation and the Academy of American Poets, showing that the GPT models are much more formulaic and constrained than the human-authored poetry. We argue that the GPT models have a "default" poetic mode, characterized by quatrains with rhymed lines in iambic meter; first-person plural perspectives; and the repetition of words like "heart, " "embrace, " "echoes, " and "whispers. " This default mode sometimes breaks through even when otherwise prompted. We share the code that we used to conduct this analysis, 4 and we share all the public domain human-authored poems and form/style annotations from prior work. 5 We plan to share our ChatGPT poetry corpus at a later stage of the project. In future work, we plan to explore a wider range of prompts (potentially including author names) and models, and to study the poems more closely with traditional literary studies approaches.