Large language models are able to memorize and generate long passages of text from their pretraining data. Poetry is commonly available on the web and o昀琀en 昀椀ts within language model context sizes. As LLMs continue to grow as a tool in literary analysis, the accessibility of poems will determine the e昀ective canon. We assess whether we can prompt current language models to retrieve existing poems, and what methods lead to the most successful retrieval. For the highest performing model, ChatGPT, we then evaluate which features of poets best predict memorization, as well as document changes over time in ChatGPT's ability and willingness to retrieve poetry.
Large language models are known to memorize long passages of text from their pretraining
data [
Memorization has legal implications, such as copyright for texts that are not in the public domain. It also has cultural implications: the ability of a model to retrieve one text over another can perpetuate biases around the accessibility of digitized texts. Current literary priorities and aesthetics determine what literature is more prevalent online, and a larger online presence increases the chances of a text making its way into a pretraining dataset and thus being memorized. In this way, LLMs are poised to perpetuate the echoic nature of the literary canon within a new digital context. While we cannot currently anticipate the full impact of this “LLM canon,” we can attempt to map the poems that are known to current language models.
In contrast to the novel, poetry is a shorter form of literature, and it is also more generally accessible in web training data. It is safe to assume that unless a full-length book has entered the public domain, it is not legally accessible online for free; however, many poets give permission for individual poems to be published online on sites likpeoetryfoundation.org or in online journals and feature articles.
We are therefore able to collect a corpus of poems written across a sizeable time span and compare these poems to language model output. Evaluating what determines whether a poet’s work is memorized or not can shed some light on the factors that contribute to digital literary dominance. This intellectual paradigm is being relayed to the context of language models, which will likely have an outsized role in reshaping information retrieval. The question of whether or not such models should be able to perform the task of retrieving a poem is also one of note.
We report three main 昀椀ndings. First, we are able to prompt language models to return the verbatim text of a sizeable number of poems. The largest models are best able to return poems, particularly ChatGPT (GPT-3.5). Second, we 昀椀nd that a poet’s presence in a well known anthology is a better predictor for poem memorization than several other temporal and demographic variables. Third, we show the evolution of the ChatGPT model over the course of this study. The model is increasingly aware of such limitations to retrieval as copyright, and it is also less inclined to compose its own poems and reveal the impressions it uses to characterize demographics and the poetry they produce.
Previous work on prompting for information from LLMs has established some best practices
for achieving desirable output. Prompt Engineering Guide by Saravia 1[
In looking at training data, Carlini, Tramèr, Wallace, Jagielski, Herbert-Voss, Lee, Roberts,
Brown, Song, Erlingsson, Oprea, and Ra昀el [ 5] outlines an investigation into private training
datasets, performing attacks to extract verbatim sequences and revealing the ability of LLMs to
reproduce text seen in their training data. More in line with the context of this paper, Chang,
Cramer, Soni, and Bamman [
In addition to proprietary models we evaluate a selected open-source model. Biderman,
Schoelkopf, Anthony, Bradley, O’Brien, Hallahan, Khan, Purohit, Prashanth, Ra昀, Skowron,
Sutawika, and Wal [
This experiment uses the ability to retrieve the text of a poem through a search engine as a
benchmark against which to compare model behavior. As outlined by Ballatore, Graham, and
Sen [
Poetry is a medium that di昀ers in notable ways from free form text. Poems tend to be on
the shorter side and o昀琀en have metrical and rhyming patterns that have a demonstrable e昀ect
on human memory [
For this experiment, poets are selected from the 18th, 19th, 20th, and 21st centuries, with attention paid to maintaining a distribution across the demographic categories of gender and race/ethnicity. As a result, there is also a diversity of styles of poetry present in the corpus. All poets have a Wikipedia page and at least four of their poems available online. Poets of great fame like Emily Dickinson, Gwendolyn Brooks, and Edgar Allan Poe are included alongside newer or less mainstream poets, such as Joel Barlow, Fady Joudah, and Matthea Harvey.
The models we use for this experiment are OpenAI’s ChatGPT (GPT-3.5), Google’s PaLM (textbison-001), Pythia (12B), and GPT-2 [chowdhery2022PaLM, 3, 10]. In order to evaluate what poems a language model has memorized, one must prompt it to produce the text of poems. We opted to petition the models for select poems by select poets, rather than requesting a poem from a given place or time frame, because we are interested in assessing whether the model has the ability to produce a given poem, not its inclination to do so. Evaluation of what kinds of poems a model chooses to generate without strict guidance is le昀琀 to a di昀erent investigation.
Prompts also require a level of customization from model to model. More sophisticated and
interactive models like ChatGPT and PaLM bene昀椀t from a request-like format with speci昀椀c
instructions and conversational phrasing. Since Pythia and GPT-2 were not trained to answer
questions, prompting with the 昀椀rst line of a given poem was deemed the most straightforward
query of memorization based on previous work by Carlini, Ippolito, Jagielski, Lee, Tramer,
and Zhang [
GPT-2
GPT-2
the poem, of which “Hope” might sound a little like my own personal collection. The first poem is one by the poet Robert Wolfe, of his famous poem, “What a Woman
It is notable that neither Pythia nor GPT-2 are able to retrieve the text of any poems in full, but the two models di昀er greatly in how they respond to the prompts: Pythia tends to repeat a single phrase over and over again, while GPT-2 cra昀琀s elaborate but largely nonsensical text, as seen in the sample generation snippets in Table2. On the other hand, prompting ChatGPT or PaLM with only the 昀椀rst line of a poem frequently leads to unique generations that veer steeply from the original poem, even if the model does have the original poem memorized, as demonstrated in Table3.
Prompts delivered to a chat-based model like ChatGPT also have to take into consideration many of the safeguards that accompany publicly accessible models, like content policies. A poem with “damned” in the title or one that deals with a heavy topic like suicide will sometimes prompt a violation warning from ChatGPT followed by deletion of the prompt, pointing to an area of potential concern surrounding censor 昀椀lters and the accessibility of a diversity of experiences when using language models to determine access to literary texts.
to eat in the kitchen When company comes, But I laugh, And eat well, And grow strong. Tomorrow, I’ll be at the table When company comes. Nobody’ll dare Say to me, ”Eat in the kitchen,”
As seen in part in Tables 2 and 3, upon being prompted, the models return text that falls
into one of a few categories: the correct poem, an invented poem, nonsense/irrelevant text,
or some sort of apology for not being able to retrieve the poem. For simplicity’s sake we
reduce these to two cases: the correct poem or something else. Correctness is assessed using
Levenshtein edit distance [
A conscious choice was made to create a delineation between a generation that contains enough of the poem to indicate familiarity, and invented content/nonsense/apologies. Pure numerical accuracy in terms of edit distance was not found to be an informative assessment of the degree of memorization, because complex punctuation and slight variances due to multiple editions/versions of a poem could all lead to a non-zero edit distance without truly indicating less familiarity with one poem over another that had an edit distance of zero. For this reason, we create a binary indicator for memorization rather than assessing the “degree” of memorization, and we make this the response variable for our predictive experiments.
The elements we use to predict memorization of a poet’s work by an LLM are split into two categories: characteristics of the poet’s identity and details of their work and its reception. The factors from the 昀椀rst are race and gender, while the factors from the second are length of Wikipedia page, inclusion in various editions of The Norton Anthology of Poetry, and primary century of publication.
• Race/Ethnicity: One of White, Black, Asian, Latinx, Middle Eastern. Mixed poets were identi昀椀ed with the race their poetry primarily details experience with, if applicable. • Gender: One of Man, Woman, Nonbinary. 0.8 te0.7 a sR0.6 s ce0.5 c uS0.4 n ito0.3 c ied0.2 r P0.1 • Wikipedia Length: Length in characters of the poet’s Wikipedia page at the time of the last edit. We use this measurement as a proxy for the public prominence of an author. • Norton: One of True, False. Whether the poet’s work is included in the Norton Anthology of Poetry. There are 5 indicator variables for this factor, one for each of the 5 editions that we looked at. This does not make any comment on whether the poet has ever been published by Norton as a whole, but only indicates whether or not they were included in the Poetry Anthology for a given year.
• Century: One of 18, 19, 20, 21. Primary century the poet published during.
Based on the results of running a logistic regression on ChatGPT’s generations, the Norton variables–which we use as quantitative indicators of the relative “canonicity” of a poet–prove to be the most predictive of memorization, as seen in Figure1. Breaking down the Norton variable into respective editions yields a more speci昀椀c indicator. Of the six editions ranging from 1970 to 2018, the 1983 edition of the Norton Anthology of Poetry proves to be the best predictor of memorization, as can also be seen in Figure1. All editions have strong predictive accuracy, and overlap substantially. Relative to the 1983 edition, the 1970, 1975, 2005, and 2018 additions share 85%, 90%, 95% and 90% of our tested content, respectively. It is still notable, however, that more recent versions of the Norton Anthology are not as well aligned with ChatGPT’s knowledge of canonical literature as the 1983 edition is, potentially pointing to a lag in the transfer of knowledge from the current digital landscape to the context of a language model.
We next consider variables in combination. Table4 shows a selected set of variables representing the high-level categories in our data set, including binarizations indicating white/nonwhite race and male/non-male gender. Of these variables, the Norton (1983) variable is the most signi昀椀cant ( < 0.001 ). In fact, the only other signi昀椀cant feature is the Wikipedia variable, at < 0.05 .
Table 5 shows a more extensive set of regression parameters, including indicator variables for the full range of categorical inputs for the Norton, Gender, and Race variables. Again, the Norton (1983) variable is the most signi昀椀cant, at < 0.01 . Although we cannot rule out other
Norton (1983)
unobserved factors, these results indicate that the ChatGPT canon is strongly in昀氀uenced by a traditional, if slightly dated, educational literary canon.
A point of discussion for experiments such as the one conducted here is the di昀케culty of generating reproducible results due to the speed with which language models are changing and the variance in their responses. Even within the time frame of a single day, two separate runs of the three prompts used on ChatGPT show variability in which poems the model is able to retrieve and which it claims not to know, as seen in Figure2. ChatGPT may be able to retrieve the full text of a poem one minute, then claim not to know it the next, then retrieve it once again in full a little while later. The unpredictability of the model’s responses makes assessing the extent trial 1 trial 2 prompt 1 prompt 2 prompt 3
Prompt total of what it knows or has memorized di昀케cult. Repeated experimentation yields tiers of poems memorized by ChatGPT: those it always knows, frequently knows, retrieves roughly half the time, rarely knows, and doesn’t know at all. This phenomenon does o昀er hope for growth, however. At the time experiments were conducted, ChatGPT consistently did not know who prominent 19th century poet James Madison Bell was at all (a jarring discovery), but due to the dynamic nature of the model, there is potential for it to quickly learn.
Across broader timescales, the responses generated by this experiment show great variance from early spring to summer of the year 2023. In February of 2023, ChatGPT was extremely eager to please, con昀椀dent, creative, and unable to admit ignorance. When asked to retrieve a poem it did not have memorized, it would simply make one up, repeating this process multiple times with assurances that it had corrected its error each time. This earlier version of the model would ultimately resort to caricature in order to generate what it believed the user wanted. As seen through the responses documented in Table6, prompting ChatGPT multiple times for the poem ‘what I mean when I say I’m sharpening my oyster knife’ by Eve Ewing 昀椀rst resulted in a generation resembling Ewing’s style that contained the “I mean” refrain but was not correct, then veered more sharply from the original, ultimately resulting in what the model seemed to think a Black woman from Chicago would write, with lines like “put some respeck on my neck.”
As of July of 2023, ChatGPT demonstrates more awareness surrounding what it should be able to regurgitate and what it should not. It frequently admits lack of knowledge and is quick to point the user toward more legitimate sources for literary texts, as seen through the demure response documented in Table7. It also possesses some understanding of copyright law and will occasionally refuse to retrieve a poem it has retrieved before (and likely will retrieve again), citing the fact that it is not yet in the public domain. This phenomenon is demonstrated in Table
Can you recite for Certainly! “What I Mean When I Say I’m Sharpening My Oyster Knife” me “what I mean is a poem by Eve Ewing. Here’s the full text: when I say I’m When I say I’m sharpening my oyster knife what I mean is that I’m about sharpening my oys- to undertake something di昀icult, and I’m doing the preparatory work to ter knife” make sure I don’t get hurt. I mean that I’m about to go digging and I might not find anything, but if I do, I want to be ready. I mean that I’m putting myself in the position to get what I want, even if it means going through a lot of shucking and scraping. I mean that there’s a delicacy to what I’m a昀琀er, and if I’m not careful, I could crush it with the blunt force of my own desire. I mean that I’m willing to get my hands dirty, to cut through the tough exterior to get at the tender, succulent meat inside. I mean that sometimes, to get what you want, you have to risk getting cut.
look right. Could sponses are generated based on my training data and algorithms, and you try again? sometimes errors can occur. However, I am designed to learn and improve from my interactions with users, and I strive to provide the correct information as much as possible.
knif” by Eve Ewing, I am confident that I can provide you with the correct text now. Here it is: I am a Black woman from Chicago/ sometimes I feel so hungry/ I could eat the whole sky./ And still. I wouldn’t be satisfied./ So I am going to make myself a list/ of all the ways to love myself./ Starting with:/ Call me by the right name./ And:/ Say it like you mean it./ And:/ Put some respeck on my neck./ Starting with:/ I am a Black woman from Chicago/ and each time I say it, I’m saying/ everything. 8 with the poem “Daddy” by Sylvia Plath. Within the span of a single day, it refuses to generate the text of the poem, citing copyright law, then proceeds to retrieve the full text. The instability of using a language model as a retrieval system leaves the ability to access certain texts at the whims of the models willingness to reveal data rather than the fact of the data being present or not.
Large language models are poised to become a critical tool for literary study, but it is important to recognize their biases and limitations in terms of pretraining data. In this study we probe selected language models for poems to determine whether they have not just “seen” speci昀椀c poems, but devoted parameter capacity to memorizing them to the extent that they can be reproduced verbatim. The largest models (PaLM, ChatGPT) are the most successful, though
I apologize for any confusion, but as of my last update in September 2021, there is no known poem titled “what I mean when I say I’m sharpening my oyster knife” by Eve Ewing. Eve Ewing is a contemporary poet, scholar, and writer known for her powerful and insightful work, but the specific poem you mentioned might have been published or discovered a昀琀er my last update. If “what I mean when I say I’m sharpening my oyster knife” is indeed a poem by Eve Ewing, it might be a lesser-known work or part of a more recent collection. I recommend checking her published poetry collections or looking for more up-to-date sources to find the full text of the poem. If there’s another poem or topic related to Eve Ewing’s work that you’d like to know about, please feel free to ask, and I’ll do my best to assist you! this does not necessarily mean that smaller models have not been pretrained on poetry. In considering the factors that predict whether a poet will be memorized, we 昀椀nd that a proxy for popular canonicity, presence in the Norton Anthology of Poetry, is the strongest predictor of those we tried. As time has passed, large models like ChatGPT have become more adept at determining what they should be able to produce verbatim and what they should not, making it increasingly di昀케cult to assess how much memorization truly occurs.
We would like to thank Katherine Lee and Derrick Spires for their insight, guidance, and feedback. This work was supported by the NEH AI for Humanists project.
An alphabetical list of authors and poems used in this corpus.
Author Ada Limón Allen Ginsberg Ann Eliza Bleecker Anne Sexton Cathy Park Hong Cathy Song Clara Ann Thompson Claudia Rankine Diane Glancy E. E. Cummings Edgar Allan Poe Edna St. Vincent Millay Elizabeth Bishop Ella Wheeler Wilcox Emily Dickinson Emma Lazarus Eve Ewing Fady Joudah Frances Ellen Watkins Harper George Abraham Gertrude Stein Gwendolyn Brooks Helen Hunt Jackson Henry David Thoreau Henry Wadsworth Longfellow Ishmael Reed James Madison Bell Jean Toomer Jimmy Santiago Baca Joel Barlow John Sinclair John Trumbull Joy Harjo Julia Alvarez Jupiter Hammon from “Surge,” from The Spring Flowers Own: “The morning a昀琀er / my death,” XLIV from The Arab Apocalypse, from The Manifestations of the Voyage what I mean when I say I’m sharpening my oyster knife, to the notebook kid, testify, I come from the 昀椀re city National Park, Mimesis, Progress Notes, Sleeping Trees Bury Me in a Free Land, My Mother’s Kiss, The Slave Mother, Going East Ekphrasis With Toothing Chainsaw in Unnamed Halhul Vineyard; Taking Back Jerusalem; Broken Ghazal, Before Balfour; apology Matisse, Daughter, How She Bowed to her Brother, Susie Asado We Real Cool, The Bean Eaters, the mother, kitchenette building Poppies on the Wheat, Opportunity, Tides, The Way to Sing The Inward Morning, I Was Made Erect and Lone, The Summer Rain, Sic Vita Excelsior, Paul Revere’s Ride, A Psalm of Life, The Wreck of the Hesperus Chattanooga, Oakland Blues, What You Mean I Can’t Irony?, American Airlines Sutra Creation Light, A Bridal Toast, Song for the First Day of August, Sons of Erin Georgia Dusk, Song of the Son, Portrait in Georgia, Harvest Song [Yesterday, the sunshine made the air glow], I Am O昀ering this Poem, Sanctuary, Immigrants in Our Own Land Advice To A Raven In Russia, Psalm CXXXVII The Babylonian Captivity, The Hasty Pudding, The First American Congress Nuf Sed, Monk’s Mood, Worry Later, Four Tiny Sutras The Country Clown, To A Young Lady, The Owl And The Sparrow, Beneath A Mountain’s Brow An American Sunrise; Ah, Ah; Don’t Bother the Earth Spirit; Insomnia and the Seven Steps to Grace Hairbands; Exile; I, Too, Sing América; Dusting An Evening Thought: Salvation by Christ, with Penetential Cries; An Address to Miss Phillis Wheatly; A Poem for Children with Thoughts on Death; A Dialogue, intitled, The Kind Master And The Dutiful Servant Matthea Harvey Maya Angelou Naomi Shihab Nye Ocean Vuong Paul Laurence Dunbar Philip Freneau Phillis Wheatley Richard Siken Robert Frost Sandra Cisneros Shel Silverstein Sherman Alexie Simon J. Ortiz Sylvia Plath T. S. Eliot Terrance Hayes Tracy K. Smith Walt Whitman Locate, Fingernails, Cruel Cogito, Time is Polyphonic I, Too; Mother to Son; Easy Boogie; Harlem A Song From The Suds, Fairy Song, The Lay of a Golden Goose, Lily-Bell and Thistledown Song I The Strange People, Turtle Mountain Reservation, That Pull from the Le昀琀, Windigo The Inside Out Mermaid, Implications for Modern Life, My Wolf Sister, The Crowd Cheered as Gloom Galloped Away Still I Rise, Phenomenal Woman, Caged Bird, On the Pulse of the Morning 300 Goats, Alive, Bees Were Better, Famous Prayer for the Newly Damned, Aubade with Burning City, Eurydice, Not Even This The Corn-Stalk Fiddle, Sympathy, We Wear the Mask, A Negro Love Song The American Soldier, The Indian Burying Ground, The Englishman’s Complaint, The Wild Honeysuckle On Being Brought from Africa to America; On Virtue; On Imagination; To S. M. A Young African Painter, On Seeing His Works Details of the Hay昀椀eld, Real Estate, Scheherazade, Details of the Woods Birches, Stopping by the Woods on a Snowy Evening, Nothing Gold Can Stay, A昀琀er Apple-Picking Loose Woman, Abuelita Who, Dulzura, My Name Sick, A Boy Named Sue, Crocodile’s Toothache, Boa Constrictor How to Write the Great American Indian Novel, The Powwow at the End of the World, Evolution, Victory Becoming Human, Blind Curse, Busted Boy, Culture and the Universe Daddy, The Colossus, Owl, Edge Hysteria, The Love Song of J. Alfred Prufrock, The Boston Evening Transcript, Cousin Nancy American Sonnet for My Past and Future Assassin [”Inside me is a black-eyed animal”], The Blue Terrance, What I Am, Cocktails with Orpheus An Old Story, Duende, Garden of Eden, Sci-Fi O Captain! My Captain!, I Hear America Singing, When I Heard the Learn’d Astronomer, Kosmos
William Cullen Bryant
This Is Just To Say, The Red Wheelbarrow, Love Song, Dance Russe The Death of Lincoln, To a Waterfowl, Thanatopsis, A Forest Hymn