One of the core tasks in cultural analytics is the act of classification. We sort discrete thingsbooks, passages, sentences, works of art, songs-into buckets whose boundaries have either been established by long theoretical traditions (such as categories of poetic form [23,45]), or that are newly circumscribed in the course of our research (cf. "technological strangeness" [36]). We carry out this act for several reasons: to test the association of features with those categories in order to better understand the categories themselves; to recreate human judgment at a scale beyond what people are able to carry out alone; and to problematize original category boundaries, leveraging machines to help us poke holes in our current understanding of them. As models, classifiers "return us to the process …through which we construct our knowledge of phenomena that exceed our direct observation" (Piper [29,p. 651]).
Each one of these goals requires different machinery to accomplish it, from interpretable linear models that allow us to isolate the effect of an attribute on a category, to complex models that may sacrifice such interpretability for predictive accuracy in those cases when accurate judgments are all that matter.
As large language models (LLMs) have arisen to become the state of the art for a wide range of tasks in natural language processing, a growing body of work is investigating the tradeoffs they bring for this act of classification in fields for which text is data [45,49,31]. High-quality LLMs are generally far more expensive than alternatives (in compute, environmental, and financial costs), and come burdened with a Pandora's box of pre-training-their source of state-of-theart performance and general-purpose knowledge, but also the vector along which undesirable knowledge is carried. At the same time, they offer hope for dramatically reducing the amount of human effort required for meaningful humanistic inquiry [49]-trading the scale of manual annotation often required by supervised models for more focused injection of human and domain knowledge into prompts.
To shed some light on the contours of this landscape, we investigate the intersection of LLMs, classification and cultural analytics along four dimensions:
• First, we survey recent work published in cultural analytics (CHR, Journal of Cultural Analytics, NLP conferences, etc.) to identify the uses to which researchers are currently employing classification. We group those uses into a typology of six distinct categories. • Second, we identify a set of ten tasks supported by publicly available datasets on which we can examine tradeoffs for some of those uses. These tasks represent a wide range of phenomena of interest to cultural inquiry and provide a sample of the real ways in which researchers now are using classification in their work. We focus on English-language datasets only in this work; our results are therefore limited to the English-language affordances of downstream applications. • Third, we use that suite of datasets to benchmark the performance of different classification models, including bag-of-words linear models, masked language models (such as BERT and RoBERTa) and large language models (both prompting and fine-tuning). We find prompt-based LLMs to be competitive with traditional supervised models for established English-language tasks, but lag behind supervised models for de novo tasks.
While cautioning against the accuracy-maximizing incentives that benchmarks create [28], this suite of datasets allows us to assess the tradeoffs between computation, cost and accuracy for several of those use cases where classification accuracy is paramount. • Fourth, we carry out an exploratory case study bringing together LLMs with this suite of datasets for a goal that does not seek simply to maximize classification accuracy: category sensemaking. While this study can only be suggestive, we find that LLMs provide a mechanism for exploratory data analysis that complements existing methods.
We release data for others to explore these tradeoffs themselves; data and code to support this work can be found at https://github.com/bamman-group/ca-classification-data.
As distinct from mere automation, the use of classification in cultural analytics is often employed for the purpose of sensemaking. At a high level, the act of supervised classification entails learning a function ĥ(𝑥) from an input source 𝑥 (e.g., text) to a choice (or choices) 𝑦 drawn from an enumerable output space 𝒴 (e.g., for genre classification, 𝒴 = {science fiction, western, detective story, …}), though for our purposes, we also include regression problems (where 𝑦 ∈ ℝ). Classification requires the existence of a previously selected set of categories 𝒴 , which is a political choice grounded in a system of knowledge [12]; as Bowker and Star note, "each category valorizes some point of view and silences another" [4, p. 5]. This is an important subject of critique not only when the objects of study are people [9,2] but also in recognizing that classification can only make statements with respect to the categories that are classified (and not to the gaps that exist between them).
For much work in this space, the representation of 𝑥 is critical-e.g., the choice of representing 𝑥 as a vector corresponding to word counts from a defined vocabulary, or other more specialized features (e.g., type-token ratio). To be precise, we can describe this representation as a function of its own (e.g., 𝑓 (𝑥)), which is the direct input to a classification function ( ĥ(𝑓 (𝑥))), though modern neural classifiers convolve both functions into one learned operation.
We can separate out these uses of classification into distinct yet inter-related categories:
1. Category sensemaking. One of the most widely used applications of classification in cultural analytics is understanding the characteristics of a category through the features that are predictive of it. We can view this as a sensemaking exercise, whose main degree of freedom is the choice of representation function 𝑓 (𝑥). This function can vary from a kitchensink approach (throw every feature into a model and learn what is important) to those that define specific, theory-driven feature classes. We see many examples of this kind of sensemaking. Long and So [24] use a featurized model to identify the defining characteristics of stream-of-consciousness form-above all, type-token ratio-and how those defining features both converge and diverge across English and Japanese stream-of-consciousness texts. Simeone, Koundinya, Kumar, and Finn [36] use classification to uncover the characteristics of technological nova-moments in science fiction texts where new technologies are introduced; using a bag-of-words featurization, they find that strangeness is characterized not merely by diction (the choice of words, such as "ship") but also by morphosyntax (presence of determiners and prepositions). Several studies use the machinery of classification to interrogate the boundaries of gender performance in English [17,42,7] and French [43]. Bode [3] uses classification in combination with topic modeling to learn that the theme of "nonmetropolitan colonial spaces" most strongly differentiates Australian fiction from British and American fiction.
A second, but related, use of classification is explicitly theorydriven: articulating a formal test (often of association between a feature and a category) and carrying out that test. We can differentiate this use from exploratory sensemaking by the falsifiability of its claims: a theoretical claim is articulated, and a corresponding null hypothesis is either rejected or not when evaluated on data. For Piper and Bagga [30], theory provides the boundaries of operationalization (in defining the feature space through which a text is represented), and classification provides a test for whether that operationalization is sufÏciently descriptive to differentiate texts that are narrative from those that are not across a variety of contexts. The boundaries between this use case and that of category sensemaking are often fluid-even within this theory-driven choice of feature space, Piper and Bagga [30] also engage in sensemaking by identifying a minimal set of features from predictive models that characterize narrativity and ground that minimal set in a more precise "distant worlds" theory of narrative. Steg, Slot, and Pianzola [37] take a similar theory-driven approach, testing the degree to which measures of reader suspense, surprise and curiosity are predictive of narrativity.
A third use, related to use #1, employs classification to challenge the initial category boundaries of a concept. Long and So [23] is an exemplary use of this; this work uses classification primarily to explore and challenge the category boundaries of haiku by examining misclassifications-cases where a model classifies a non-haiku poem as haiku and vice versa. They additionally engage in close reading of the model itself, by identifying salient features of texts that are statistically associated with haiku and find that the language of natural imagery and brevity are strong indicators of the form (beyond simple syllabic patterns). Work by Broadwell, Mimno, and Tangherlini [5] likewise trains a folktale type classifier on Danish narratives to identify "liminal stories that shift across existing categories" (23); these stories expose the ambiguity of the initial categories and highlight the ways in which their boundaries are permeable.
4. Category coherence. While use #3 assumes a fixed category object whose boundaries are unknown, a fourth use employs classification accuracy as a proxy for the coherence of a concept itself, under the assumption that statistical regularities could only exist for concepts that are somehow real. Much work in the first three uses carries with it an implicit theoretical test in the coherence of the category; Long and So [24, p. 350] argue that the predictability of stream-of-consciousness texts vs. realist texts provides evidence for "some of degree of formal unification"; Thompson and Mimno [39] use classifier accuracy at recognizing images from the Dada art movement as evidence of its coherence as a concept ("there is substance behind the name Dada, " 193) while stressing that such coherence is not absolute: classifier mistakes illustrate "the porous boundaries of the category" (194). In a similar vein, the counterfactual provides information about the dissolution of the construct. Underwood, Bamman, and Lee [42] predict character gender from the attributes and actions associated with characters, and find that decreasing predictive accuracy over time suggests an increasing "instability" in the gender construct.
We might term a fifth category, search, as "classification-assisted close reading. " This line of work uses the machinery of a classifier as a finding aid to zero in on passages that instantiate a target category. Wilson, Ardanuy, Beelen, McGillivray, and Ahnert [46] is an exemplary use of this approach. This work explores the concept of "atypical animacy" (machines that are depicted as animate vs. non-animate) in a large collection of historical books, newspapers and journal articles. After designing a method to identify passages of machine animacy in order to make them accessible to close reading, this work uses classification (and especially misclassification) to identify textual sites in which to interrogate child labor, slavery and the trope of "mere machines. " This use of classification is not dissimilar from the interactive use of methods like topic modeling and other unsupervised techniques in the act of close reading (e.g., Klein [20] and Walsh and Antoniak [44]).
Replacing human labeling at scale. A final use case is the application of classification as an instrumental device in downstream sensemaking: recreating human judgments at a scale beyond which people are able to carry out themselves. After establishing the validity and interpretability of a model, Long and So [24] then use that model as a large-scale classifier, applying it to passages from 1700 novels in order to measure the association with nationality and genre with stream-of-consciousness; in this case, classification provides a new dependent variable over a much larger scale in order to test the association of other features.
In considering the ways in which the rise of large language models have the ability to influence this landscape of use, we might characterize those use cases by the degree to which calibrated predictive accuracy is critical for accomplishing the intended goals, vs. the degree to which a model enables interpretative sensemaking internally. All categories rely on baseline levels of both accuracy (to validate that a model has learned from data) and sensemaking (for its use to connect to knowledge about its object of study). But we might see use case #6 (replacing human labeling at scale) to lean more toward predictive accuracy as a goal, while use case #1 (category sensemaking) to lean more towards internal interpretability. We consider both of these axes in turn. First, in such cases where accuracy alone is paramount, when are LLMs needed? When are the tradeoffs worth it, and when do smaller, cheaper models sufÏce? Second, most interpretation-maximizing uses of classification in cultural analytics have generally relied on linear models (e.g., logistic regression, SVMs) to directly measure the influence of a feature on a category choice without the confounding effects of non-linearities that more complex models introduce. How are LLMs able to contribute to this goal? To answer both of these sets of questions, we identify a set of ten datasets used in the classification categories above, and put a range of models to the test to explore this landscape.
We draw datasets from public repositories for the following papers. Most of the datasets were created to test specific hypotheses about the association of features with categories (e.g. typetoken ratio with stream-of-consciousness). Across all tasks, we create standard training/development/test splits (summarized in Table 1) to emphasize test validity, generally decreasing the size of training data used relative to the original work. To make the classification tasks more difÏcult for models, we also stratify across groups (e.g., author, genre) to isolate the target construct, so that items from the same group (e.g., texts by the same author) only appear within a single split. These modifications generally make the results reported here incomparable with the original results.
Atypical animacy. Data from Coll Ardanuy, Nanni, Beelen, Hosseini, Ahnert, Lawrence, McDonough, Tolfo, Wilson, and McGillivray [8] differentiating mentions of machine-related terms (engine, machine, etc.) as animate vs. non-animate. Animate machines are those that are depicted as being alive (including people referred to as machines), while inanimate machine are not. We create train/dev/test splits at random.
Emotion. Data from Kim and Klinger [19], which identifies the emotions that characters are experiencing by tying mentions of characters in a sentence to mentions of a trigger emotion word. We transform this into a multiclass classification task by extracting sentences in which a character experiences a single emotion, and ask a model to predict the emotion for that character. We create train/dev/test splits so that passages from the same work appear in only one partition.
Folktales. Data from Hagedorn and Darányi [14], which assigns the ATU type [40] to texts. We select only ATU types that are attested at least nine times among the text labels, and create train/dev/test splits to maintain the same label distribution across splits (i.e., at least three instances per type across each split).
Genre. We draw inspiration from Sharmaa, Hu, Wu, Shang, Singhal, and Underwood [33] in using Library of Congress subject classification as a proxy for genre, and use a subset of 5 genres that work studied (science fiction, detective and mystery stories, adventure stories, love stories, and westerns). We draw texts from Project Gutenberg, sampling 5 passages (each approximately 500 words) from 150 books for each genre. To enable a multiclass classification problem, we only consider books that are tagged with one subject classification from the set above (so that works that are tagged with both "love stories" and "westerns" are excluded). We create train/dev/test splits so that texts by the same author appear in only one partition, and we select a maximum of 5 books per author.
Haiku. Data from Long and So [23], which contrasts haiku poems with non-haiku poems. We create train/dev/test splits so that poems by the same author appear in only one partition.
Hippocorpus. Data from Sap, Horvitz, Choi, Smith, and Pennebaker [32], which solicits firstperson stories written by workers on Amazon Mechanical Turk in three categories: recalled stories, which narrate real events transpiring within the past six months; imagined stories, fictional narratives on the same topic as a randomly selected recalled story; and retold stories, recalled stories told again 2-3 months later by the same workers. This is the only task where a human does not judge a label by inspection of a pre-existing text; accordingly, it is not possible to articulate the textual boundaries between those categories a priori. We create train/dev/test splits so that texts by the same author appear in only one partition.
Literary time. As our sole regression task, we draw data from Underwood [41], which labels the number of seconds that transpire in a fictional passage of approximately 250 words. We create train/dev/test splits so that texts from the same title appear in only one partition.
Narrativity. Data from Piper and Bagga [30], which contrasts passages from narrative genres (biography, fairy tales, novels) to passages from non-narrative genres (scientific abstracts, book reviews, supreme court proceedings). In this data, the genre of a text fully determines its narrativity status; in order to assess how models reason about narrativity rather than genre per se, we create train/dev/test splits so that texts of the same genre appear in only one partition (texts of the "fairy tale" genre, for instance, only appear in dev data).
Strangeness. Data from Simeone, Koundinya, Kumar, and Finn [36], which contrasts sentences mentioning "descriptions or introductions of technology and novel science" with those that do not, both drawn from Project Gutenberg texts. We create train/dev/test splits at random.
Stream-of-consciousness. Data from Long and So [24], which contrasts stream-ofconsciousness passages with control passages drawn at random from realist novels. The original work sampled control passages of fixed character lengths, leading to passages that break between words; we re-sample passages from Project Gutenberg texts of the same titles breaking only across sentences, sampling passage lengths to reflect the same empirical distribution of lengths in the SoC texts. We create train/dev/test splits so that the passages by the same author appear in only one partition.
In the goals of classification for which maximizing accuracy is important, these ten tasks encode a wide range of different phenomena that shed some light on the affordances of LLMs compared to traditional supervised classifiers. We compare the performance of the following different classes of models on these tasks.
Logistic/Linear regression. We train a regularized logistic regression model for all classification tasks and ridge regression for Literary Time, using a bag-of-words representation of text with a vocabulary size of 100K. We tune the ℓ 2 regularization strength on dev data.
BERT/RoBERTa. We train base English models of BERT [11] and RoBERTa [22] with a maximum 512-token input size, truncating excess text. We tune the learning rate on dev data.
Llama 3 8B. To test the ability of LLMs as supervised classifiers, we fine-tune Llama 3 8B [1] on the training split with a multiclass classification head (i.e., selecting a discrete category as output, not a choice of word) for all classification tasks and a regression head for Literary Time, using low-rank adaptation [15]. We represent a passage as the final layer output of the last word within it. Like BERT and RoBERTa, we use a maximum 512-token input size, truncating excess text. We tune the learning rate on dev data.
For each task, we optimize hyperparameters on development data using the optuna optimization library over 50 optimization trials, training up to 100 epochs with early stopping based on lack of improvement over 10 epochs on dev data. Appendix D illustrates the relationship between learning rate and accuracy on development data across all tasks for BERT, RoBERTa and Llama 3 8B; we see the best learning rate to be highly task-and model-specific, necessitating optimization.
We select three LLMs to evaluate classification through prompting alone. For all models, we provide a description of the task and provide 10 shots as exemplars, with no Chain-of-Thought.
As Appendix A illustrates, we experiment with shot selection and Chain-of-Thought prompting on development data with GPT-4o, finding that while the number of shots generally improves performance (though not always significantly so), using Chain-of-Thought does not lead to any meaningful difference.
GPT-4o. We query GPT-4o through the OpenAI API. Across all tasks, inference over the test set cost $164.49.
Llama 3 70B. We run Llama 3 70B-Instruct locally across 4 L40S GPUs with a total of 192 GB of GPU memory. The Llama 3 context size is 8,192 tokens, which is shorter than the prompt length for the Folktales task (which includes descriptions for the 70 categories), so we do not assess Llama 3 on this task.
We run Mixtral 8x22B-Instruct locally across 4 L40S GPUs with a total of 192 GB of GPU memory. Since Mixtral models are more memory-demanding than alternatives due to their mixture of experts, we load them with 4-bit quantization.
We present results in Table 2, reporting accuracy as a metric for all classification tasks and Spearman 𝜌 for Literary time. For all supervised models, we report the total wall clock training time in minutes across all hyperparameter optimization trials. Tasks are ordered (by left to right and top to bottom) by the difference in metric between best performing prompting method and best supervised method.
Established vs. de novo tasks. The diversity of tasks on which we assess models allows us to characterize disparate performance among them. Tasks measuring concepts that are generally widely known-the animacy of objects, determinations of science fiction vs. westerns, folktale types, character emotional states, and stream-of-consciousness-offer sites where LLMs are able to excel, either beating supervised models or coming close in performance to them. For tasks that measure largely de novo concepts-the passage of time, differentiating stories that are recalled from those that are imagined or retold, "technological strangeness"-the evidence from supervision appears to be important for accurate prediction.
Memorization. The outsized performance of GPT-4o on the folktale identification task raises the question whether that model has memorized that task; as we detail in Appendix B, we assess the influence of memorization on task performance using a membership inference method inspired by Shi, Ajith, Xia, Huang, Liu, Blevins, Chen, and Zettlemoyer [35]. We see little evidence of memorization, and after applying correction for multiple hypothesis correction, no evidence for downstream impact.
Sufficiency of masked LMs. Among models that are optimized for each specific task using supervised data, we do not see a strong difference between base masked language models and Llama 3, which is pre-trained on a much larger amount of data, has 65 times more parameters, and generally takes an order of magnitude longer to train.
The comparative performance we see reported above only speaks to high classification accuracy as a desideratum. This leaves aside the other, perhaps more important, ways in which we see classification being used as a tool for sensemaking in cultural analytics.
If we probe deeper into the ways in which linear classification methods have been used for making sense of categories, both theory-driven approaches and exploratory analysis make use of conceptualization (deciding on specific constructs to measure) and implementation (instantiating those constructs into specific measures), to use the terminology from Piper [29]. A primary difference, however, comes in the role of prior knowledge and theorization guiding those choices. Top-down approaches often design models to test theories; exploratory approaches may also incorporate theory in the design of the representation space 𝑓 (𝑥), but a starting point is often to let the structure of the data guide our understanding of the concepts themselves. Topic modeling is a widely used example of this form; an unsupervised method learns structure from data which then guides the analysis that follows. We see many examples of this approach. Goldstone and Underwood [13] use topic modeling to first learn that "critical theory" is a dominant theme in literary journals; after surfacing this fact, they more formally test its rise and fall over time; Bode [3] likewise uses topic modeling to identify a set of themes organizing a collection of Australian, British and American fiction, then uses close reading and classification to narrow in on a formal analysis of nonmetropolitan colonial spaces as a major organizing principle. Sensemaking as an activity is driven in these cases by the structure found through exploratory models. Indeed, the practice of computational grounded theory [27] is premised on exactly this movement between exploration through models, close reading, and formal testing.
If we consider the ways in which LLMs are able to serve some function in this larger sensemaking process, a natural starting point is their role in exploratory analysis. What structure exists in the data that can lead to formal theory-testing? We experiment with this use case by prompting GPT-4o to reason about the characteristics that differentiate a set of data points Consider the data below, which contains a list of text/label pairs that illustrate a set of categories (0, 1):
[{"text": "Strong called to the remaining ships of his fleet Take formation K. Land and attack the enemy base according to prearranged order. ", "label": 0} {"text": "She dithered for a moment then stepped back toward the kitchen and the stairwell.
GPT-4o output for characterizing the difference between Haiku and Non-Haiku poems through anonymized labels (only presented to the model as class "1" vs. "0"); for brevity, the first 5 characteristics are presented here; the full list can be found in Appendix C.5.
Concise and Lyrical: Texts are often shorter, more concise, and have a lyrical quality.
Texts often tell a story or describe a scene in detail. Nature Imagery: Frequent use of natural imagery, such as flowers, trees, and weather.
Complex Structure: Sentences are often longer and more complex, with multiple clauses. Emotional and Sensory Language: Focus on emotions and sensory experiences.
Frequent references to historical events, mythological figures, or classical literature. Simple Structure: Sentences are generally simpler and more straightforward.
Character Focus: Emphasis on characters, their actions, and their emotions. Personal and Intimate: Often feels personal or intimate, as if sharing a private moment or thought.
attended by anonymized category labels, as illustrated in Figure 1. We include as many data points as the context length allows.
Table 3 presents the output of this experiment for the Haiku task; the full outputs for all ten tasks can be found in Appendix C. Here we see similar affordances for sensemaking as alternative models. Just as Long and So [23] challenge the initial category boundaries of haiku by identifying naturalistic language and brevity as defining characteristics of it, so too does GPT-4o identify "nature imagery" and conciseness as characteristic. An examination of the outputs in Appendix C shows that the sensemaking enabled by this form of LLM prompting intersects with that enabled by alternative methods in these tasks' original papers. The defining features of Animacy identified by GPT-4o stress "the mechanical nature of humans or the human-like qualities of machines, " ( §C.1) including the focus identified by Wilson, Ardanuy, Beelen, McGillivray, and Ahnert [46] on "mere machines. " This method identifies "narrative style" vs. "lack of narrative" ( §C.8) as the defining feature of the Narrativity task but offers alternatives to the distant worlds theory of Piper and Bagga [30]-offering "descriptive language" and "imagery" as additional characteristics. In many ways, prompting LLMs to find the ways in which a set of categories are differentiated from each other within the evidence of a labeled dataset can suggest meaningful directions for more formal interrogation.
At the same time, it is clear that such LLM characterizations are not to be trusted at face value; we know this from prior work on LLM hallucinations [16,47,18] and confabulations [38], but also in examining its points of failure within this set of tasks. For Emotion ( §C.2), we see clear evidence of overfitting (e.g., "references to royalty or nobility" for the anticipation class, "mentions of specific names" for joy); and its description of the Folktale task ( §C.3) fails to meaningfully differentiate categories that it performs well on when given their description. Additionally, one of the risks of using pre-trained LLMs for exploratory data analysis is the degree to which the analysis they drive is a direct result of examining the data given, or a function instead of task knowledge acquired during pre-training. While we are careful to anonymize the category labels in the experiments above and not mention the classification task by name (e.g., "haiku"), it is clear that pre-training data informs the knowledge that models bring to this exploratory task, muddying the inferences we can make about data-in-itself. Understanding the role that pre-training plays both in model performance (e.g., through disparate memorization) and in any biases that result (e.g., through disparate focus on whose language is represented [25]) is an important area of current research on the limits of these models.
As an exploratory exercise, the defining characteristics of categories seen through the lens of a prompted LLM provide suggestions on potentially salient dimensions that only later formal testing can truly assess the significance of. Is it indeed the case that westerns "focus on personal experiences and emotions" more than other genres ( §C.4)? Is sentence length indeed correlated with the duration of elapsed time ( §C.7)? While not providing an answer themselves, they suggest potentially interesting directions to test with more formal means (i.e., put through the stages of conceptualization and implementation). In this sense, we might view LLMs as tools for sensemaking-helping us understand categories and the boundaries that circumscribe them-in the same way that other exploratory methods like topic modeling make possible.
In this work, we outline the major ways in which current work in cultural analytics is employing classification, primarily as a method for sensemaking distinct from mere automation, and detail the ways in which large language models fit (and do not fit) within this landscape. For uses that seek to maximize predictive accuracy, LLMs offer competitive performance through prompting alone for established tasks, while traditional supervised methods excel for newly constructed phenomena (even in scenarios with limited training data). For sensemaking applications that aim to understand category constructs better through labeled data, LLMs may have a role to play in exploratory data analysis by outlining potential characteristics that can then be subjected to more formal testing. By considering a set of ten tasks considered in the space of cultural analytics, we complement an existing body of work that probes model knowledge of culture [21,34]) to focus in particular on the ways in which computational models can shed light on culture itself. Data and code to support this work can be found at https://github.com/bamman-group/ca-classification-data.
We investigate the relationship between number of shots, Chain-of-Thought reasoning and accuracy for GPT-4o in Table 4 below.
GPT-4o dev performance by number of shots/chain of thought. Increasing the number of shots (from 1 to 10) largely helps performance (e.g., for animacy, genre, literary time, and strangeness), even if not significantly so. CoT generally does not improve performance. We do not test CoT on Hippocorpus since it does not originate as a human labeling task (as noted in §3 above).
The outsized performance of GPT-4o on the folktale identification task raises the question whether that model has memorized that task; indeed, we can see that the text itself is often discoverable on Google attended with its label, as illustrated in Figure 2 below. Prior work in cultural analytics has assessed the impact of memorization on downstream tasks, with work demonstrating memorization influence on encyclopedic knowledge (predicting date of publication [6]) but not on text-internal qualities like poetic form [45].
We assess instance-level memorization in prompt-based LLMs using a membership inference method similar to Shi, Ajith, Xia, Huang, Liu, Blevins, Chen, and Zettlemoyer [35]; while that work identifies membership through the log probabilities of the 𝑘 least probable tokens in a text (necessitating probability assessments for every token), we identify first the 𝑘 = 5 least probable tokens (relative to the complete data for that task) and ask a model to predict them given a fixed context length of 25 tokens, as in the example below: My wife and I have been living in the same family home for the last [MASK] years . It was our first family home and → ten We judge a passage to be memorized if at least four of the five masked terms are able to be predicted. We measure the influence of memorization on the task through the Pearson correlation between an instance's memorization and whether the model correctly predicted its label (in the case of regression for Literary Time, the mean absolute error between true and predicted label); for correlation between two binary variables, this is equivalent to the 𝜙 coefÏcient [48]. We select this method over others due to the specific nature of these datasets, which do not always contain names (required by Chang, Cramer, Soni, and Bamman [6]) and do not always have titles or authors notable enough to condition on (required by D'Souza and Mimno [10]).
As Table 5 shows, while GPT-4o on Folktales, hippocorpus and literary time all rank highest on memorization, there is no measurable effect on downstream tasks for any model/task combination when correcting for multiple hypothesis tests.
This assessment of memorization, however, only considers memorization of the text alone, and not the association of text with a label. While past work has examined this form of label memorization in controlled settings where the training data is known [26], it remains a difficult task for membership inference-and in fact may be impossible to disentangle from the understanding of the task itself.
Dataset memorization by prompting models. With Bonferroni correction for the 𝑛 = 29 hypothesis tests carried out, no model shows significant correlation at 𝛼 = 0.05 between memorization and task performance. We do not assess Folktales on Llama 3 70B since the prompt for that task exceeds the Llama context size (as noted above). For cases with 0 memorization, Pearson's 𝑟 is undefined (denoted by blank values).
Llama 1. Themes of supernatural intervention: Supernatural beings like trolls or witches play a significant role. 2. Rural and pastoral settings: The stories often take place in rural settings, involving common folk and simple tasks.
Category 4 (sc. 700) Characteristics:
1. Themes of cleverness and resourcefulness: The protagonist uses cleverness to outwit others, such as a tinker finding treasure. 2. Elements of folklore and superstition: The stories often involve elements of folklore, such as dreams leading to treasure.
Category 5 (sc. 175) Characteristics:
1. Themes of trickery and cunning: Characters like Brer Rabbit use cunning to outwit stronger opponents. 2. Animal characters with human traits: The stories often feature animals with humanlike intelligence and behavior.
1. Frequent use of dialogue and conversational tone: Texts often include direct speech, with characters interacting in a casual or informal manner. 2. Presence of conflict or tension: Many passages involve arguments, disagreements, or confrontations between characters. 3. Focus on mundane or everyday events: The content often revolves around ordinary activities, such as meals, conversations, or minor incidents. 4. Detailed descriptions of characters' actions and emotions: There is a strong emphasis on what characters are doing and feeling in the moment. 5. Use of colloquial language and idiomatic expressions: The language tends to be more relaxed and may include slang or regional dialects. 6. Shorter, more fragmented sentences: The writing style often includes brief, choppy sentences that reflect the immediacy of the scene. 7. Frequent shifts in focus or perspective: The narrative may jump between different characters or scenes, creating a sense of movement and dynamism.
Pole 2 (sc. longer time) Characteristics:
1. Formal and elaborate language: The text often employs a more sophisticated and refined vocabulary, with complex sentence structures.
There is a greater emphasis on setting the scene, providing background information, and describing the environment in detail.
The content may include introspective or contemplative passages, exploring deeper themes and ideas. 4. Historical or literary references: The text may allude to historical events, classical literature, or other cultural touchstones. 5. Focus on significant or dramatic events: The narrative often centers around major occurrences, such as battles, political intrigues, or life-changing decisions. 6. Longer, more complex sentences: The writing style tends to be more fluid and continuous, with fewer breaks and a more measured pace. 7. Consistent narrative perspective: The text usually maintains a steady point of view, providing a cohesive and unified narrative voice.
Category 0 (sc. narrative) Characteristics:
1. Narrative Style: The texts often have a narrative or storytelling style, focusing on characters, events, and dialogues. 2. Descriptive Language: Rich in descriptive language, often detailing settings, emotions, and actions. 3. Literary Elements: Use of literary elements such as metaphors, similes, and personification. 4. Character Interaction: Frequent interactions between characters, including dialogues and internal monologues.
5. Emotional Tone: Often convey emotions and personal experiences, creating an immersive atmosphere. 6. Historical or Fictional Context: Many texts are set in historical or fictional contexts, providing a backdrop for the narrative. 7. Plot Development: Presence of a clear plot or storyline, with events unfolding over time. 8. Imagery: Use of vivid imagery to paint scenes and evoke sensory experiences.
Category 1 (sc. non-narrative) Characteristics:
1. Formal and Technical Language: Use of formal, technical, and legal language, often related to business, contracts, or regulations. 2. Structured Format: Texts are often structured in a formal format, including sections, clauses, and bullet points. 3. Objective Tone: An objective, impersonal tone, focusing on facts, procedures, and instructions. 4. Specific Terminology: Use of specific terminology related to business, law, or technical fields. 5. Lack of Narrative: Absence of a narrative or storytelling style; instead, the focus is on conveying information or instructions. 6. Contractual and Legal Content: Frequent references to agreements, obligations, rights, and legal terms. 7. Professional Context: Texts are often set in a professional or corporate context, dealing with business operations, legal matters, or technical specifications. 8. Precision and Clarity: Emphasis on precision and clarity, ensuring that the information is unambiguous and easily understood.
Category 0 (sc. strange) Characteristics:
1. Technical or Scientific Language: Texts often include technical or scientific terms, such as "proton and neutron structure, " "viewer magnification, " "synthetic gravity generator, " and "molecular started biting at the fort's ray screen. " 2. Space and Technology Themes: Many texts reference space, ships, and technology, e.g., "space navigator, " "rocket ships, " "air ship, " "space speed, " and "space tug. " 3. Descriptive of Actions and Objects: Texts frequently describe actions or objects in a detailed manner, such as "The ship plunged planetward, " "The boats slid into space, " and "The hull of the _Enterprise_ bulked huge in the guidance screen. " 4. Formal and Objective Tone: The tone is often formal and objective, focusing on factual descriptions and technical details, e.g., "The machine replenishes the energy because it is set to maintain a certain energy level in the field. " 5. Lack of Personal Interaction: There is minimal personal interaction or dialogue between characters, with a focus on events and descriptions, e.g., "The car roared through the gates of the spaceport and sped across the hard surface of the field. "
Figures 4, 5 and 6 illustrate the range of learning rates considered during hyperparameter optimization, along with the ensuing accuracy on development data; while we consider 50 optimization trials of each learning rate, optuna discards trials that are unlikely to outperform learning rates seen so far (so each task/model may have fewer than 50 points illustrated). Across all supervised methods-BERT (fig. 4), RoBERTa (fig. 5) and Llama 8B (fig. 6)-the choice of learning rate is very dependent on the task, necessitating task-specific optimization.
We thank the original authors of the tasks we study for openly releasing the data on which this current work stands. The research reported in this article was supported by funding from the National Science Foundation (IIS-1942591) and the National Endowment for the Humanities (HAA-271654-20), with computing resources provided by Microsoft Azure (Accelerating Foundation Models Research) and the Accelerating Computing for Emerging Sciences (ACES) high-performance computing cluster at Texas A&M University, funded by the National Science Foundation (NAIRR-240114).
Category 6 (sc. 980) Characteristics:
1. Themes of family and respect: The story involves family dynamics and the importance of respecting elders.
The story teaches a lesson about treating family members with kindness and respect.
Category 7 (sc. 726) Characteristics:
1. Themes of supernatural and magical intervention: Supernatural beings like trolls or witches play a significant role. 2. Rural and pastoral settings: The stories often take place in rural settings, involving common folk and simple tasks.
Category 8 (sc. 1430) Characteristics:
1. Themes of laziness and cleverness: The protagonist, Heinz, is initially lazy but uses cleverness to achieve his goals. 2. Transformation and success: Heinz transforms from a lazy boy to a successful individual through a series of clever actions.
Category 9 (sc. 2250) Characteristics:
1. Themes of brevity and simplicity: The story is very short and simple, often involving a single event or action. 2. Lack of detailed plot: The story lacks a detailed plot or character development.
Category 0 (sc. science fiction) Characteristics:
1. Frequent use of dialogue and conversational tone: The texts often include characters speaking to each other, which makes the narrative dynamic and engaging.
The texts often describe characters performing actions, moving from one place to another, or engaging in some form of physical activity. 3. Descriptive language for settings and events: The texts provide detailed descriptions of the environment, events, and actions, creating a vivid picture for the reader. 4. Suspense and tension: The texts often build suspense and tension, keeping the reader engaged and curious about what will happen next.
Category 1 (sc. western stories) Characteristics:
1. Focus on personal experiences and emotions: The texts often delve into the personal thoughts, feelings, and experiences of the characters.
The narrative often includes characters reflecting on their past actions, decisions, and emotions.
The texts provide in-depth descriptions of characters' personalities, backgrounds, and motivations.
4. Use of first-person perspective: Some texts are written from a first-person point of view, giving a direct insight into the character's mind.
Category 2 (sc. detective and mystery stories) Characteristics:
1. Mystery and investigation elements: The texts often involve characters trying to solve a mystery or uncover hidden information. 2. Presence of law enforcement or detective figures: Characters such as detectives, police ofÏcers, or investigators are commonly featured. 3. Clues and evidence: The narrative includes details about clues, evidence, and logical deductions made by the characters. 4. Formal and procedural language: The texts often use formal language and describe procedural actions related to investigations or legal matters.
Category 3 (sc. adventure stories) Characteristics:
1. Historical or period settings: The texts are often set in historical or period-specific contexts, with detailed descriptions of the time period. 2. Focus on social and cultural aspects: The narrative often explores social norms, cultural practices, and historical events relevant to the time period. 3. Descriptive language for attire and environment: The texts provide detailed descriptions of clothing, architecture, and other elements specific to the historical setting. 4. Formal and elaborate language: The language used is often more formal and elaborate, reflecting the historical context.
Category 4 (sc. love stories) Characteristics:
1. Romantic and emotional themes: The texts often focus on romantic relationships, emotional experiences, and personal connections between characters. 2. Descriptive language for emotions and interactions: The narrative includes detailed descriptions of characters' emotions, facial expressions, and interactions. 3. Use of dialogue to convey feelings: Dialogue is often used to express characters' feelings, thoughts, and relationship dynamics. 4. Focus on personal and intimate moments: The texts highlight personal and intimate moments between characters, emphasizing their emotional significance.
Category 0 (sc. non-haiku) Characteristics:
1. Narrative or Descriptive Style: Texts often tell a story or describe a scene in detail.
2. Complex Structure: Sentences are often longer and more complex, with multiple clauses. 3. Historical or Mythological References: Frequent references to historical events, mythological figures, or classical literature. 4. Character Focus: Emphasis on characters, their actions, and their emotions. 5. Dialogue and Monologue: Presence of dialogue or internal monologue.
While there is some emotional content, it is not as intense or dramatic as in Category 2.
Category 2 (sc. recalled) Characteristics:
1. Emotional depth and intensity: Entries in this category are characterized by a high level of emotional content, often describing feelings of sadness, joy, anxiety, or relief. 2. Personal and introspective: These entries often include personal reflections and introspection, exploring the emotional impact of events on the writer. 3. Detailed and elaborate: The descriptions are often more elaborate and detailed, providing a comprehensive view of the events and their emotional significance. 4. Significant life events: Entries frequently describe major life events, such as births, deaths, illnesses, or significant personal challenges. 5. Narrative style: The writing style is often more narrative, telling a story with a clear beginning, middle, and end, and focusing on the emotional journey.
[Note for this regression task we used the following prompt.]
Consider the data below, which contains a list of text/label pairs that illustrate a regression problem:
[10pt] [{"text": "Carlyle disappeared. Barbara had taken off her own warm night-gown then, and put it upon William in place of his cold one-had struck a light and was busily dressing herself. …", "label": -4.454347296253507}
{"text": "At first he lived up above entirely, reading a good deal, and strumming upon an old harp …", "label": 4.033719940405666} …]
Using this data, and this data alone, what are the textual characteristics that differentiate the two ends of this scale (let us call those ends "Pole 1" and the "Pole 2")? The label "-1" is closer to Pole 1 than the label "1" is; the label "1", likewise, is closer to Pole 2 than the label "-1" is. Provide a list of bullet points of those textual features in the following format: