Large language models have shown breakthrough potential in many NLP domains. Here we consider their use for stylometry, speci昀椀cally authorship identi昀椀cation in Early Modern English drama. We 昀椀nd both promising and concerning results; LLMs are able to accurately predict the author of surprisingly short passages but are also prone to con昀椀dently misattribute texts to speci昀椀c authors. A 昀椀ne-tunedt5large model outperforms all tested baselines, including logistic regression, SVM with a linear kernel, and cosine delta, at attributing small passages. However, we see indications that the presence of certain authors in the model's pre-training data a昀ects predictive results in ways that are di昀케cult to assess.
English that it may be challenging for LLMs primarily trained on modern text to parse. Additionally, the culture of co-authorship and collaboration among writers during the Early Modern era o昀琀en makes it di昀케cult to distinguish stylistic delineations between individuals.
Despite its challenges, the attribution of Early Modern drama is a well-studied 昀椀eld, and
techniques like cosine delta2[
We have three primary 昀椀ndings. First, for short texts the 昀椀ne-tuned LLM outperforms all tested baselines, including logistic regression, a support vector machine (SVM) with a linear kernel, and cosine delta. Accuracy varies by author and is not fully explained by the number of plays by the author in the 昀椀ne-tuning set. Second, LLMs are more prone than cosine delta to con昀椀dently misattribute texts to speci昀椀c authors. These “scapegoat” authors o昀琀en have large vocabularies and word use similar to the corpus average. Third, trained LLMs may be able to quantify “style”. When we apply the model trained on Early Modern drama to “attribute” excerpts of plays written between the 1500s and 1900s, we see an increasing proportion attributed to Shakespeare, possibly suggesting a quanti昀椀cation of his lasting in昀氀uence.
Many di昀erent methods have been used to perform authorship attribution tasks with Early
Modern drama. These include function word adjacency network8s],[multi-view learning7[],
clustering algorithms1[
Attempts to attribute shorter passages in Early Modern drama have been controversial.
These studies include the attribution of 63 words froMmacbeth [
Work has also been done on the attribution of short texts in di昀erent 昀椀elds. Cosine similarity
is e昀ective at attributing 500 word excerpts from blogs1[
Some researchers have begun testing the feasibility of using LLMs for attribution. These
studies used the embedding output of LLMs to train custom attribution models using LSTMs
[
We use a collection of Early Modern English drama—plays written in the 1500s and 1600s— gathered from two sources: the Folger Digital Anthology of Early Modern English Drama (EMED) [5] and the Shakespeare His Contemporaries corpus (SHC1)3[]. We 昀椀rst gathered 367 plays from the EMED corpus and then added the 181 remaining plays from S H1 CIn. order to remove features that may distinguish 昀椀les from di昀erent corpora, we stripped all non-accent non-ASCII characters from the play texts and replaced them with standardized alternatives where appropriate. Each XML 昀椀le o昀ered regularized spellings of non-standard words in the play. In creating our corpus, we used the regularized spellings from the EMED corpus that agreed with the greatest number of other sources when possible and the SHC regularizations otherwise. We chose to use the regularized text for two reasons. First, we did not want the model to be able to distinguish between authors based on spelling choices. Although di昀erences in spelling may help the model identify authors, they are not indicative of the kinds of stylistic di昀erence we are interested in studying. Second, we hypothesized that standardizing the play texts would make them appear more similar to modern text and thus improve the model’s ability to accurately tokenize the input. Finally, we removed all line breaks from the texts as the di昀erent corpora do not consistently mark them.
We then split each play into speaker utterances to create a challenging but coherent identi昀椀cation problem. We separated any utterance longer than 450 words into multiple samples by splitting directly a昀琀er every 450th word, regardless of sentence or line breaks. We then removed utterances with fewer than 5 words. Because authors sometimes develop distinctive speaker voices within a play, we hypothesize that separating the texts by speaker utterance adds an extra layer of di昀케culty to the attribution task.
We further reduced the training and testing corpora to maximize validity and statistical reliability. We removed all plays with fewer than 300 remaining utterances, plays by multiple authors, and plays by authors with fewer than three works in the corpus. Plays that were mislabeled as by a single author, but were actually of disputed (co-)authorship were placed into a separate subcorpus. We were thus le昀琀 with 253 plays by 23 authors in the primary corpus and 23 plays in the subcorpus. Further details about the corpora are listed in the appendix.
We used these corpora to assess the capability of several di昀erent authorship attribution
methods to label short texts. Speci昀椀cally, we tested logistic regression, SVMs with a linear
ker1Because the original Shakespeare His Contemporaries corpus is no longer publicly available, we have drawn these
sources from a port of the original Github linked in the citation.
nal, cosine delta 2[
One play from each author in our corpus was Number of Samples per Dataset withheld from the training dataset. From the re- Train maining − 1 plays by each author, we included Validation 235 random samples in ts the training dataset and tsaae Test 15 samples in the valida- D tion dataset used for pa- Test Disputed rameter tuning. We included another 50 sam- Test Comparative ples from each of these plays in the 昀椀nal test 0 10000 20000# Samp3le0s000 40000 50000 dataset for which we report results. Thus, we Figure 1: The size of each dataset used in the experiment by numdraw 300 distinct samples ber of samples. from each play withheld from the training dataset. To the test dataset, we added 200 randomly selected samples from each of the plays withheld from training. We then created a separate test dataset containing 200 samples from each of the 23 plays in the disputed authorship corpus. Figu1rsehows the relative size of the training, validation, and test datasets, as well as the held-out disputed set and a set of post-Early Modern plays used in Sectio7n.
We 昀椀ne-tuned a small, base, and large version of T5 on the train and validation datasets, using batch sizes of 16, 8, and 4 respectively and running for 10 epochs. The additional 昀椀ne-tuning hyperparameters are reported in Section A of the appendix. We then asked each model to predict labels for every sample in the primary test dataset. Finally, we used the best performing model,t5-large, to predict labels for the test dataset of disputed authorship plays. We also experimented with 昀椀ne-tuning two comparable decoder-only generative LLMs: Pythia with 1 billion parameters and Falcon with 1 billion parameters. The input and output strings described above were edited for these experiments so that thAeUTHOR tag was placed at the end of each string. However, both models hallucinated extensively; Pythia produced 10,221 unique strings as author names and Falcon produced 9,180. Even when the 昀椀rst two words of each produced string, stripped of punctuation, were used as the predicted author name Pythia and Falcon still performed considerably worse thatn5-large. We thus omit a further analysis of these models from the paper.
For our baseline comparisons we used only the original quotation withouAtUthTeHOR pre昀椀x
and T5 tags. The correct authors were included as labels. We ran two logistic regression
models and two SVM models with linear kernels: one version of each used TF-IDF weighted word
counts as features and the other used plain word counts. Each of these baselines was
implemented using the sklearn package. Cosine delta [
Results are shown in Table2 for the per-sample accuracy of each attribution method and the accuracy of the “majority vote” predicted author of each play. In order to display the e昀ect of play-speci昀椀c language such as character names and settings, we show predictive results for both held-outsections of plays and fully held-out plays.
We begin by establishing that accurate author attribution is possible for this dataset using only the available information. It is known that authorship attribution is more reliable for longer samples. To establish an upper bound on expected performance, we thus apply a cosine delta model to the full held-out text of each play rather than the short samples we use for all other experiments. This setting increases the length of attributed samples by a factor of 50 for plays in the training set and 200 for fully held-out plays. Cosine delta accurately attributes 94.9% of the long samples, performing better on plays in the training set than those fully heldout.
The 昀椀ne-tuned t5-large model correctly attributes more short samples than any other method tested. It accurately labels 52.7% of held-out samples from plays included in the training dataset and 33.2% of samples from plays fully withheld from trainitn5g-.large performs substantially worse on the individual sample level than the cosine delta upper bound, but it only falls seven plays short of the upper bound when attributing plays to the most-predicted author. Although it is not surprising that results are better for partially-seen plays, the accuracy of both subsets exceeded our expectations. Because the text excerpts we use are very short, they frequently contain no named entities, and we thus conclude that attribution was not performed solely using this information.
Longer samples were more accurately attributed. The average length of correctly attributed samples in our primary test dataset was 36.7 words whereas the average length of misattributed samples was 20.7 words. Figure2 shows the distribution of sample lengths and the accuracy for each range. Accuracy exceeds 50% with only 20 words (random≈i5s%). Model scale also e昀ects accuracy. The t5-large model performed better than the smaller models we compare it to, t5-base and t5-small. We observe that t5-large does 30.3% better on samples from plays included in training, 21.4% better on samples from plays withheld from training, and 63.4% better at attributing plays by majority vote thatn5-small. This e昀ect may be due to the larger model’s greater capacity to 昀椀t the particulars of author-speci昀椀c language in 昀椀ne-tuning, a greater capacity to represent linguistic variation in pre-training, or some combination of both.
It appears that the reason for the large improvement in play attribution accuracy with model size is a signi昀椀cant reduction in the assignment of large numbers of samples to 2–3 speci昀椀c (incorrect) authors, which we calslcapegoating. t5-small assigns 60.5% of misattributed segments to the top two scapegoated authors (Thomas Heywood and William Shakespearte5),base assigns 32.8% of misattributed samples to two authors (Heywood and James Shirley), and t5-large only attributes 25.6% of misattributed samples to two authors (also Heywood and Shirley). Because misattributions both occur less frequently and are spread more evenly between authors in the larger models, it is more likely that the author of a play will have the majority of samples assigned to them.
Correct
Mistake
Logistic regression and linear SVM prove to be strong baselines. Howevte5r-,large performs 4.7% better on samples from plays included in training and 9.9% better on samples from withheld plays than linear SVM with TF-IDF values, the highest performing of these baselines. Since these models have access to the same data, the di昀erence must either come from the LLM’s ability to use arbitrary combinations of non-sequitive words or its access to patterns from pre-training. It is important to note that we do not know what data T5 saw during pretraining. However, because thet5-small model performs worse than logistic regression, it is unlikely that this is the sole source of improvement. Logistic regression and linear SVM are also prone to scapegoating: all models assign over 35% of misattributed samples to two primary authors (Shakespeare and Shirley). Linear regression with TF-IDF values is a particularly egregious scapegoater, assigning over 50% of samples to Shakespeare and Shirley, over double the number thatt5-large assigns to Shirley and Heywood.
In addition to the “merged samples” upper bound, we apply cosine delta to the short samples. This approach performs worse than all methods btu5t-small and the simple baselines. However, cosine delta achieves high performance at the play level. Evte5n-large only attributes 6 more of the 253 plays in the original corpus correctly. This, again, appears to be related to scapegoating. Cosine delta assigns the samples it misattributes relatively evenly between authors, only assigning 12.4% to the two most scapegoated authors (Richard Brome and Thomas Middleton). Thus, while cosine delta may be less accurate overall than T5, the way in which it fails is less skewed. Compared to T5, SVMs, and logistic regression, it is less likely to con昀椀dently misattribute a play.
For the best-performing modelt,5-large, accuracy varies considerably by author for both withheld plays and those included in training (Tab3l)e. Authors with more plays in the training set are more accurately predicted for the held-out set; the Pearson correlation coe昀케cient between these values is 0.65, with < 10 −3.
The t5-large model performs well on samples from many of the well-represented authors in our corpus. For 9 of the 23 authors, the model accurately attributes more than 50% of samples from plays included in training, well above random. The four authors for whom the model performs best on samples from included plays are Shakespeare (79.0%), Margaret Cavendish (74.9%), Shirley (61.7%), and John Lyly (60.6%). The model also accurately attributes many samples from the withheld plays by these authors: 72.0% of samples from Shakespe2araere correctly attributed, 68.5% of samples from Cavend3is,h58.5% of samples from Shirle4y, and
60.0% of samples from Lyl5y.
The reasons that the model attributes samples from these four authors with such high accuracy di昀er. Shirley and Shakespeare are the authors with the most and second-most plays in the dataset, with 31 and 30 plays in the corpus respectively. But Cavendish (8) and Lyly (12) are close to the average. Authors comparable to Cavendish in representation, such as George Chapman (11), Philip Massinger (13), and Thomas Middleton (13), all have accuracies below 50% for plays included in training. Similarly, authors comparable to Lyly such as John Ford (7) and John Marston (7) both have accuracies below 35% on included plays. Therefore, there is likely something distinctive about these two authors that makes them easier for the model to identify. Note that Cavendish is the only female author in the corpus (we were unable to include others), so we are not able to determine if her plays are distinctive because she has an individual style or if women authors of the period wrote di昀erently from men.
To further explore the cause of Cavendish and Lyly’s distinctiveness, we compare each author’s usage of the 100 most frequent words in the corpus. We 昀椀rst calculate z-scores comparing the frequency with which an author used each word to the mean frequency of that word’s usage for all authors in the dataset. The frequencies are normalized by author so that no single author skews the distribution and we ensure that the set of 100 most frequent words contains no named entities. We then sum the absolute values of each author’s z-scores to create a ‘uniqueness’ metric. For a further exploration and validation of this metric, please see Section B of the appendix. The summed z-scores ranged from 47.5 to 138.2. The author with the most unique usage of common words by this metric is Cavendish, with a score of 138.2. The authors with comparable play counts to Cavendish each have considerably lower scores (Chapman: 50.57, Massinger: 69.9, Middleton: 67.7). The second most distinctive author is Thomas Killigrew, with a summed z-score of 108.7. Indeed, Killigrew has a very high accuracy on samples from included plays (52.0%) considering only four of his works are in the corpus. Lyly also has a relatively high summed z-score of 99.4, which is the fourth largest in the dataset. Again, this is higher than the scores of comparably represented authors (Ford: 68.3, Marston: 70.8), but not by as much. Notably, both Shirley and Shakespeare have low uniqueness scores by this metric. Shakespeare’s is the lowest (47.5) and Shirley’s is the 16th lowest (67.8). In addition, both authors have large vocabularies; Shakespeare has the largest vocabulary and Shirley the thirdlargest of all authors in the dataset. Both of these trends are likely related to their prominence within the training dataset, but they may still be meaningful. It is possible that Shakespeare and Shirley’s uniqueness comes from using words that the other authors do not, instead of using common words uniquely. Overall, it seems that an author’s usage of common words does a昀ect how well the model can identify their writing. But it does not explain all of the variation seen in the dataset. 5.1. Quote Misattribution There is also considerable variation in how the 昀椀ne-tunetd5-large model misattributes quotes (Figure 3). Instead of assigning the misattributed quotes to authors randomly, it scapegoats two primary authors, Heywood and Shirley, and assigns them a disproportionate number. A
William Shakespeare Margaret Cavendish
James Shirley
John Lyly Thomas May
John Fletcher Christopher Marlowe
Thomas Killigrew
Robert Greene
Ben Jonson trshuo ThPohRmiilciaphsaMHradesByswrinoogmoeedr A Thomas Nabbes
George Chapman William Davenant Thomas Middleton
John Marston
John Ford Thomas Dekker Henry Glapthorne
Robert Wilson John Webster
% Misattributions Assigned by Author 0 2 4
6 8 % of Misattributions 10 12 14 confusion matrix depicting who quotes are misattributed to by original author demonstrates that the scapegoating phenomenon is not caused by confusion between speci昀椀c pairs of authors (Figure 4). Instead, the misattributions to Heywood and Shirley are spread throughout the dataset. Again, it appears that contribution to the corpus is one factor that a昀ects who samples are misattributed to. The Pearson’s R correlation between the number of plays by an author in the dataset and the percentage of misattributed samples assigned to them is 0.86 with< 10 −6. The outliers from this relationship appear to be Heywood, Shakespeare, Cavendish, and Ben Jonson (Figure5).
Examining authors’ scores for the summed z-score metric again provides an indication of why some are scapegoated. Cavendish’s high uniqueness score likely means it is more di昀케cult for the model to mistake a given quote for hers. In contrast, Heywood, who has the most samples misattributed to him, has the second-lowest uniqueness score in the corpus, 49.4. He also has the second-largest vocabulary. The combination of these factors may help explain why he is so frequently scapegoated. Given a random quote from the test dataset, Heywood is more likely than most authors to have all of the words in the sample in his vocabulary. Even if he doesn’t, the model could have learned that he is more likely to use a broad range of words than other authors. In addition, common word usage in the average corpus sample is likely to resemble Heywood’s. Shirley, who has the second-most misattributed quotes assigned to him, has the third-largest vocabulary and the 16th lowest uniqueness score. Thus, it appears that vocabulary size and common word usage are factors that a昀ect to whom the model’s misattributes quotes. Wiliam Shakespeare Margaret Cavendish
James Shirley
John Lyly Thomas May
John Fletcher Christopher Marlowe
Thomas Kiligrew Robert Greene
Ben Jonson Philip Massinger Thomas Heywood
Richard Brome Thomas Nabbes George Chapman Wiliam Davenant Thomas Middleton
John Marston
John Ford Thomas Dekker Henry Glapthorne
Robert Wilson John Webster 12 s10 n o i tu 8 b ittr isa 6 M %4 2 80 60 40 20 0 % Misattributions by Number of Plays
Thomas Heywood ilirkseeepahaaSmW itrrvseadnheagaCM lirsyJheeaSm lJyynohL syaahoTmM ltrcJeheonhF litrrrsaoehpeohwCM ilirsegaohTKwm trreeenoebRG sJnonnoeB iilirssanghpePM yseooodhaTHwm irrcodaheBRm ssababeohTNm rphagnaeoeCGm ilitveanaanDmW litsoeddnahoTmM trsJanoonhM rJonohdF rkkseeahoTDm ltrryonhapeenHG litrsonbeoRW trsJeebhonW Figure 4: Confusion matrix demonstrating how frequently samples from row authors were misattributed to column authors. Each matrix row sums to 100%. Prolific authors Heywood, Shakespeare, and Shirley are most commonly guessed.
5 10 15 20 25
30 # Plays
However, there are two major outliers which indicate that these three factors—number of plays, common word usage, and vocabulary size—cannot be the only ones a昀ecting scapegoating. These are Jonson and Shakespeare. Shakespeare has both the largest vocabulary and lowest uniqueness score of any author in the corpus, and yet samples are less likely to be misattributed to him than would be expected given his contribution to the dataset. Similarly, Jonson has the fourth-largest vocabulary and the 19th lowest uniqueness score, yet he also stands out as an outlier to whom fewer samples are misattributed than expected. We hypothesize that these outliers are caused by the model’s pre-training. Of the authors included in our corpus, Shakespeare and Jonson are among the best-known today. The model is likely to have seen the writing of these authors during pre-training, and may therefore be more likely to correctly label data from these authors than would be expected given only the 昀椀ne-tuning process.
Interesting patterns and outliers emerge when we examine the model’s play-by-play accuracy at attributing samples. There are several authors, like Cavendish, for whom the proportion of correctly attributed samples is largely consistent across all plays and some, like Brome, for whom there is considerable variation in the play-level accuracy but for whom there are no noticeable outliers. When there is an outlier among an author’s plays, there is usually (though not always) an identi昀椀able reason for why that play stylistically di昀ers from the rest of the author’s work.
The New Inn The Staple of News The Devil Is an Ass
Bartholomew Fair
Catiline His Conspiracy Epicoene, or The Silent Woman s y a l P
The Alchemist Sejanus His Fall
Volpone
Poetaster Cynthia's Revels
A representative example of this can be seen in Ben Jonson’s plays. Samples from all but
two of Jonson’s plays are correctly attributed more than 40% of the time, including those from
the withheld play (Figur6e). However, only 18% of samples fromEvery Man Out of His Humour
and 26% of samples fromThe Case is Altered are attributed to Jonson, causingEvery Man Out
of His Humour to be misattributed by the model. Both of these plays di昀er from Jonson’s
typical work. AlthoughEvery Man Out of His Humour was advertised as a sequel to the
wellreceivedEvery Man in His Humour, it is very di昀erent from the original play1[
The Unfortunate Mother
The Bride lysa Hannibal and Scipio P
Tottenham Court Covent Garden
Correct Guesses by Play Thomas Nabbes
Another outlier isCovent Garden by Thomas Nabbes. Although there is some play-level variation in Nabbes’ attribution accuracyC,ovent Garden is the only play for whom the model correctly attributes less than 20% of samples and the only one it misattributes, assigning Brome 23% of samples (Figure7). While this variation may be in part becauseCovent Garden was withheld from training, the underlying reason for the model’s confusion is likely that Nabbes’ Covent Garden was written as a direct response to Richard Brome’s plTahye Weeding of Covent Garden, which is also included in the dataset. There are likely named entities that cross-over between these two works and there may even be stylistic similarities. Sample-level cosine delta correctly attributeCsovent Garden to Nabbes, but with only 13% of samples. It assigns 10% of samples to Brome.
We also see that the model performs poorly on the withheld plays of almost all authors with only three works in the corpus. For four of these 昀椀ve authors, less than 5% of samples from withheld plays are correctly attributed. The only deviation from this pattern is Robert Wilson; 19% of samples from the withheld Wilson play are correctly attributed. However, the withheld Wilson play is a prequel to one included in the training set. Thus, the model has more knowledge of this play than it would otherwise. It appears that including two plays by an author, or 470 samples, in the training data is not su昀케cient for the model to learn to extrapolate an author’s style to an unseen text. It thus suggests a boundary for how much data may be needed for LLMs to be used for authorship attribution. 6.1. Disputed and Co-Authorship We also asked thet5-large model predict the author of samples from 23 plays which are of disputed authorship or which are believed to be co-authored, although they were labeled as written by a single author in the corpora we drew from. We determined which plays were coauthored or of disputed authorship using the Oxford National Dictionary of Biography, which provides a detailed biography for each author in this corpus. these plays to Heywood. However, since there is no evidence that Heywood helped to author these plays, it is likely that this is an artifact of scapegoating. This trend also means that it is di昀케cult to attribute plays to Heywood. 22% of samples fromThe Fair Maid of the Exchange, which Heywood is suspected to have co-authored, are attributed to him. However, a comparable proportion of samples are attributed to Heywood for multiple other plays in this dataset, meaning that we cannot use this as evidence for his authorship. This is a clear example of a case in which the model’s misattribution patterns detrimentally a昀ect its usability. The results are confusing even for plays from whom all of the suspected contributors are in the dataset, such as The Laws of Candy.
Thus, the results for non-Shakespearian plays provide little evidence for or against certain writers’ authorship. While sample-level cosine delta appears to have no clear advantage over t5-large in attributing samples from these plays, the two methods attribute samples in very di昀erent ways. In some cases, t5-large more strongly attributes a play to its suspected author, and in others sample-level cosine delta does. O昀琀en the models attributed samples to di昀erent subsets of authors.
A very interesting pattern emerges when we look at the plays co-authored by Shakespeare in this test corpus. Over 50% of samples from each of the eight plays that Shakespeare contributed to are attributed to him, with little to no samples attributed to those who he supposedly coauthored the plays with, even if they are in the dataset. The most signi昀椀cant indication we see of another author’s contribution to one of these plays is fTohre Two Noble Kinsmen. Here, only 54% of samples are attributed to Shakespeare and 8.5% are attributed to Fletcher, with whom he wrote the play. However, this is still not a strong signal of Fletcher’s involvement. This pattern again suggests that thet5-large model recognizes Shakespeare from pre-training. If the model had seen these plays attributed solely to Shakespeare during pre-training, as is likely, it may help explain why it assigns them so con昀椀dently to Shakespeare despite the in昀氀uence of other authors. In contrast, sample-level cosine delta never assigns more than 25% of samples from any of these plays to Shakespeare, and the presence of his theorized co-authors is much more prominent in the results.
In addition to the narrower task of author attribution, a measure of stylometric similarity can also be used to quantify authors’ in昀氀uence. To study shi昀琀s in dramatic style over time, we created a comparative corpus of plays written between the 14th and 18th centuries. In this corpus, we included 74 plays gathered from the EMED and SHC corpora not written by authors in our training dataset. To these, we added 67 additional plays from Project Gutenberg (see Table 4). We performed the same utterance separation and splitting with these plays as with the original corpus and formatted the input and output pairs identically. Further details can be found in the appendix. Thet5-large model 昀椀ne-tuned on the original dataset was asked to predict authors for 200 samples from each of these plays. The percentages we report in Figure 9 are averaged by the original text author instead of by play; for example, we calculate the percentage of samples attributed to Heywood from each author writing in the 1500s and then average those percentages to reach the depicted value. This was to prevent any single writer whose style may somehow mimic that of an author in our original dataset from skewing the results.
In the 1500s and 1600s, the greatest proportion of samples are assigned to Thomas Heywood. This aligns with the scapegoating trends we saw in the original corpus. However, starting in the 1700s the greatest proportion of samples are assigned to Shakespeare, and this value increases in the 1800s and 1900s (Figure9), for which nearly half of the samples from each author were attributed to Shakespeare. In addition, if we attribute plays to an author by majority vote, no plays in the 1500s are assigned to Shakespeare, but 97% of plays are attributed to him by the 1900s. This result does not imply that 20th century plays are similar to Shakespeare, only that of the Early Modern authors known to the model, Shakespeare is both distinct and increasingly more similar to more recent plays than any other Early Modern author.
Average % Samples Assigned by Century William Shakespeare
Thomas Heywood d e in40 g s s A ls30 e p m a20 S % e g10 a r e v A 1500s 1600s
1700s Century 1800s 1900s
Generative large language models provide a promising tool for stylometry. While simpler methods such as cosine delta remain more accurate for larger text segments, we 昀椀nd that LLMs, particularly at larger scales, are remarkably e昀ective at predicting the author of a di昀케cult corpus of short 5–450 word text segments, which are more aligned with LLMs’ shorter input windows.
In addition to quantitative power, LLM-based stylometric analysis provides evidence for a range of interpretive arguments both when it succeeds (such as with Margaret Cavendish) as well as when it fails (both in scapegoating and in the stylistic di昀erences in the work of Ben Jonson). Because T5 demonstrates an ability to recognize style, it may prove useful in other situations where recognizing implicit signals is key such as tracking genre di昀erences and stylistic movements. There are also substantial practical advantages to using 昀椀ne-tuned LLMs: despite their complexity and computational intensity, generative LLMs provide a remarkably simple text-in/text-out user interaction that requires no specialized so昀琀ware.
However, there are several disadvantages to using pre-trained LLMs for authorship attribution. They are more computationally intensive than more traditional methods of authorship attribution and the content and e昀ect of pre-training corpora are di昀케cult to assess. In addition, the ways in which the model con昀椀dently misattributes texts means that it is more likely to produce misleading results than traditional attribution methods. Given the di昀erences that emerged between the performance of cosine delta and the 昀椀ne-tuned LLM, using the two methods in conjunction may provide more accurate results than using either method separately. Due to the weaknesses we have observed, however, we recommend against using LLMs for authorship attribution in forensic or legal settings.
We would like to thank Federica Bologna, Katherine Lee, Noam Ringach, Rosamond Thalken, Andrea Wang, Matthew Wilkens, and Gregory Yauney for their thoughtful feedback. This work was supported by the NEH project AI for Humanists and Cornell University’s Hopcro昀琀 Fellowship.
M. Brown, M. Poston, and E. WilliamsonA. Digital Anthology of Early Modern English Drama. Online corpus. N/a. url:https://emed.folger.edu. [20] P. Plecháč. “Relative contributions of Shakespeare and Fletcher in Henry VIII: An analysis based on most frequent words and most frequent rhythmic patterns”. IDni:gital Scholarship in the Humanities 36.2 (2021), pp. 430–438. doi: 10.1093/llc/fqaa032.
Parameter Evaluation Strategy Learning Rate Weight Decay Save Total Limit Value Epoch 2x10−5
To explore the validity of our uniqueness metric, we ran 1,000 synthesized trials to examine what the expected correlation between dataset contribution and the uniqueness metric would be given randomly assigned plays. Concretely, in each trial we randomly assigned plays to synthetic authors in the same proportions they are assigned to authors in our true dataset. We then calculated the Spearman’s rho correlation between number of plays and uniqueness values for each trial. We plot the binned synthetic correlations and the true correlation from our dataset in Figure10. The true correlation from our dataset, depicted with the vertical red line, is 0.2 away from any value reached in our synthesized trial. Thus, it seems that there are some notable deviations from the expected trend in our dataset.
To further explore this relationship, we averaged the uniqueness values for each synthetic author over all trials and plotted these values as well as the true values in Fi1g1u.rIet is
Correlation between # Plays and Uniqueness 350 300 250 t200 n u o C150 100 50 Simulated vs. Real Relationship between # Plays and Uniqueness 140 Margaret Cavendish Average Simulated Value True Value
John Lyly 15 # Plays 5 10 20 25 30 clear that the true uniqueness values frequently deviate from the expected relationship between uniqueness and number of plays. In particular, Margaret Cavendish and John Lyly have much higher uniqueness values than expected given the number of plays they contribute to the training dataset. Because of this, we believe that this metric represents a valuable measure of uniqueness and does not simply reemphasize the impact of contribution to the training corpus.
All plays in our original training and test corpora by author. The withheld plays are bolded and italicized.
The Northern Lass, The City Wit or The Woman Wears the Breeches, The Queen’s Exchange (The Royal Exchange), The Weeding of Covent Garden or The Middlesex Justice of Peace, The Novella, The Queen and Concubine, The New Academy or The New Exchange, The Sparagus Garden (Tom Hoydon o’ Tanton Deane),The English Moor or The Mock Marriage, The Antipodes, The Damoiselle or The New Ordinary, A Mad Couple Well Matched, The Lovesick Court or The Ambitious Politic, The Court Beggar, A Jovial Crew or The Merry Beggars The Lady — Part 1, The Lady — Part 2, The Unnatural Tragedy, Wit’s Cabal — Part 1, Wit’s Cabal — Part 2, Love’s Adventures — Part 1, Love’s Adventures — Part 2, Several Wits, The Matrimonial Trouble — Part 1, The Matrimonial Trouble — Part 2, The ReligiouTsh, e Wooers
The Blind Beggar of Alexandria, A Humorous Day’s MirAthl,l Fools, The Gentleman Usher, May Day, The Widow’s Tears, Bussy D’Ambois, Monsieur D’Olive, Caesar and Pompey (The Wars of Caesar and Pompey), The Tragedy of Charles Duke of Byron, The Revenge of Bussy
D’Ambois William Davenant The Cruel Brother, Albovine King of the Lombards, The Just Italian,
The Wits John Fletcher John Ford Robert Greene
Old Fortunatus, Satiromastix or The Untrussing of the Humorous Poet, The Honest Whore — Part 2, Match Me in London, If It Be Not Good the Devil Is in It The Faithful Shepherdess, The Woman’s Prize or The Tamer Tamed, Bonduca, Valentinian, The Mad Lover, The Chances, The Loyal Subject, The Humorous Lieutenant (Generous Enemies, Demetrius and Enanthe), Women Pleased, The Island Princess, The Wild Goose Chase, The Pilgrim, Rule a Wife and Have a Wife, A Wife for a Month The Lover’s Melancholy, The Broken Heart, ’Tis Pity She’s a Whore, Love’s Sacri昀椀ce , Perkin Warbeck, The Fancies Chaste and Noble Friar Bacon and Friar BongayT,he Scottish History of James the
Fourth, Orlando Furioso Henry Glapthorne The Hollander, Ladies’ PrivilegWe, it in a Constable Thomas Heywood The Four Prentices of London, Edward IV — Part 1, Edward I — Part 2, The Royal King and the Loyal Subject, How a Man May Choose a Good Wife from a Bad, A Woman Killed with Kindness, If You Know Me Not You Know Nobody or The Troubles of Queen Elizabeth — Part 1, If You Know Me Not You Know Nobody or The Troubles of Queen Elizabeth — Part 2, The Fair Maid of the West or A Girl Worth Gold — Part 1, The Wise Woman of Hogsdon, The Rape of Lucrece, The Golden Age or The Lives of Jupiter and Saturn, The Brazen Age, The Iron Age — Part 1, The Iron Age — Part 2, The English Traveller, Love’s Mistress,
A Challenge for Beauty Ben Jonson Thomas Killigrew The Prisoners, The Princess, The Parson’s WeddingC,laricilla Christopher Marlowe John Marston Philip Massinger Thomas May
Sappho and Phao, Campaspe (Alexander, Campaspe, and Diogenes), Gallathea, Endymion, Midas, Love’s Metamorphosis, Mother Bombie, The Woman in the Moon Tamburlaine the Great — Part 1, Tamburlaine the Great — Part 2, The Jew of Malta, Doctor Faustus, Edward the Second, THe Massacre at Paris Antonio and Mellida, Antonio’s Revenge, Jack Drum’s Entertainment, What You WillT,he Malcontent, Parasitaster or The Fawn, The Dutch Courtesan The City Madam, The Duke of Milan, The Maid of Honour, The Bondman, The Unnatural Combat, The Renegado or The Gentleman of Venice, A New Way to Pay Old Debts, The Roman Actor, The Great Duke of Florence, The Picture, The Emperor of the East, The Guardian, The Bashful Lover The Heir, Cleopatra — Queen of Egypt, Julia Agrippina — Empress of
Rome Thomas Middleton The Phoenix, Michaelmas Term, A Trick to Catch the Old One, A Mad World My Masters, The Puritain or The Widow of Watling StreYeotu,r Five Gallants, The Widow, The Mayor of Quinborough, A Chaste Maid in Cheapside, More Dissemblers Beside Women, Women Beware
Women, A Game at Chess Thomas Nabbes William Shakespeare
Covent Garden, Tottenham Court, Hannibal and Scipio, The Bride, The Unfortunate Mother The Comedy of Errors, Richard III, The Taming of the Shrew, The Two Gentlemen of Verona, Romeo and Juliet, Richard II, King John, The Merchant of Venice, Henry IV — Part 1, Henry IV — Part 2, Much Ado About Nothing, Henry V, Julius Caesar, As You Like It, Twel昀琀h Night, Hamlet, Merry Wives of Windsor, Troilus and Cressida, Othello, Measure for Measure, Macbeth, King LearA,ntony and Cleopatra, Coriolanus, Cymbeline, The Tempest John Webster Robert Wilson
The School of Compliment, The Maid’s Revenge, The Wedding, The Witty Fair One, The Grateful Servant, The Humorous Courtier, Love’s Cruelty, The Ball, The Traitor, Hyde Park, Changes or Love in a Maze, The Bird in a Cage (The Beauties), The Young Admiral, The Gamester, The Opportunity, The Example, The Lady of Pleasure, The Coronation, The Duke’s Mistress, The Royal Master, The Doubtful Heir, The Constant Maid, The Gentleman of Venice, Saint Patrick for Ireland — Part 1, The Politician, The Arcadia, The ImposterT, he Sisters, The Cardinal, The Brothers, The Court Secret The White Devil (Vittoria Corombona)T, he Duchess of Mal昀椀 , The Devil’s Law Case (When Women Go to Law the Devil is Full of Business) The Three Ladies of London, The Three Ladies of London, The Cobbler’s Prophecy
All plays in the disputed and co-authored corpus by the author they were attributed to in the original corpora.
Labeled Author
Plays
Sir Giles Goosecap, Two Wise Men and All the Rest Fools
Patient Grissel, The Wonder of a Kingdom Henry Glapthorne Revenge for Honor (The Parricide)
George a Green the Pinner of Wake昀椀eld Thomas Heywood The Fair Maid of the Exchange John Marston
Histriomastix or The Player Whipped, The Insatiate Countess Thomas Middleton Anything for a Quiet Life, The Family of Love William Shakespeare
Henry VI — Part 1, Henry VI — Part 2, Henry VI — Part 3, Henry VIII, Pericles — Prince of Tyre, Timon of Athens, Titus Andronicus, The Two
Noble Kinsmen John Webster
Appius and Virginia, The Thracian Wonder
All plays in the comparative corpus by author. Plays that were attributed to Shakespeare by the model are bolded and italicized. Francis Beaumont The Knight of the Burning Pestle
Hans Beer-Pot (See Me and See Me Not) William Berkeley The Lost Lady Hugh Henry Brackenridge
The Battle of Bunkers Hill Alexander Brome The Cunning Lovers
A Blot in the Scutcheon
Landgartha Lodowick Carlell The Deserving Favorite Richard Claude Carton
Lady Huntworth’s Experiment
The Country Captain, The Variety Susanna Centlivre The Busie Body, The Perjur’d Husband George Coleman
John Bull Abraham Cowley
Love’s Riddle Aleister Crowley Robert Daborne
A Christian Turned Turk John Denham Thomas Drue William Dunlap Lord Dusany Nathan Field Jasper Fisher The Duchess of Su昀olk Andre If Sicelides Imperiale Amends for Ladies, A Woman is a Weathercock Fuimus Troes (The True Trojans) A Bit O’ Love, The Eldest Son, A Family Man, The First and the Last, The Foundations, The Fugitive, Joy, Justice, The Little
Dream, The Little Man, Loyalties, The Mob, The Skin Game, Strife
The Prince of Parthia Johann Wolfgang Faust von Goethe John Gough Alaham
Adrasta William Kemp
A Knack to Know a Knave
The Seven Champions of Christendom The Love Chase Soliman and Perseda, The Spanish Tragedy (Hieronimo is Mad Again) Andria William Habing- The Queen of Aragon ton Samuel Harding Sicily and Naples Joseph Harris The City Bride William Haughton Englishmen for My Money Peter Hausted The Rival Friends William Hawkins Apollo Shroving EGdomrgoensd Howard The Female Famester Henrik Ibsen A Doll’s House, Hedda Gabler Elizabeth Inchbald Such Things Are, The Widow’s Vow Jerome K. Jerome Fanny and the Servant Problem, Woodbarrow Farm AHrenthruyr Jones Dolly Reforming Herself , Michael and His Lost Angel D. H. Lawrencee Touch and Go John Leacock The Fall of British Tyranny Thomas Lodge The Wounds of Civil War Samuel Low The Politician Out-Witted Sir William Lower The Phoenix in Her Flames Thomas Lupton All for Money James Mabbe The Spanish Bawd (Calisto and Meliboea) Charles Macklin The Covent Garden Theatre Gervase Markham The Dumb Knight, Herod and Antipater Shakerley A Fine Companion, Holland’s Leaguer Marmion John Mason The Turk Jasper Mayne The City Match Edward Moore The Gamester Thomas Morton Speed the Plough Arthur Murphy The Grecian Daughter
The Andrian Woman (Andria), The Eunuch Mordecai Manuel She Would Be a Soldier Noah John O’Kee昀e
Wild Oats Henry Nevil Payne The Fatal Jealousie Arthur Pinero Henry Porter
The Big Drum, The ’Mind the Paint’ Girl
The Two Angry Women of Abingdon Thomas Randolph The Jealous Lovers Messalina — The Roman Empress Ponteach: The Savages of America Cyrano de Bergerac The Noble Spanish Soldier (The Noble Soldier or A Contract Broken
Justly Revenged), When You See Me You Know Me (Henry the Eighth) Edward Sharpham Cupid’s Whirligig, The Fleer George Bernard Shaw
Arms, The Devil’s Disciple, Fanny’s First Play, Man and Superman William Stevenson Gammer Gurton’s Needle Algernon Charles The Duke of Gandia, Erechtheus, Rosamund Swinburne Albumazar, Lingua or The Combat of the Tongue and the Five Senses of Superiority Royall Tyler The Contrast Nicolas Udall Ralph Roister Doister George Wapull The Tide Tarrieth No Man Oscar Wilde Vera, A Woman of No Importance George Wilkins The Miseries of Enforced Marriage Nathaniel Woodes The Con昀氀ict of Conscience Robert Yarington Two Lamentable Tragedies Richard Zouch The Sophister