Introduction

Over the past several years, the application of transformer-based neural networks [51] has led to significant advancements in many NLP tasks related to historical languages [44,32,43]. However, unlike in the case of modern languages, where fine-tuning pre-trained transformers for linguistic forensics is very common [14,48,19,1], the application of such models for authorship attribution tasks in historical languages remains relatively underexplored, although some excellent seminal studies and surveys have been recently published [45,15,40,41]. The availability of state-of-the-art pre-trained language models [2,32,39,54] excelling in multiple downstream tasks suggests that the situation with authorship analysis can be different as well.

Yamshchikov, Tikhonov, Pantis, Schubert, and Jost [54] obtained a pre-trained model for Ancient Greek by fine-tuning a Modern Greek BERT model [23]. The resulting model subsequently served as the backbone for a classifier and proved effective for authorship attribution of the so-called Pseudo-Plutarch corpus. Interestingly, despite being fine-tuned on a limited amount of Ancient Greek data, the model obtained through transfer learning showed results comparable to those of the models trained from scratch on significantly larger corpora, as reported by Singh, Rutten, and Lefever [39] and Riemenschneider and Frank [32]. Drawing inspiration from Yamshchikov, Tikhonov, Pantis, Schubert, and Jost [54], this study experiments with a similar approach focusing on the works of late Greek rhetoricians.

Greek prose on rhetorical theory from the period known as the Second Sophistic serves as a crucial source, documenting the cultural and intellectual framework of Greek thought and literature in the first centuries AD [9,6,20,5]. However, the study of this extensive corpus of texts, collectively referred to under the broad concept of Rhetores Graeci [52,42], is significantly complicated by endless controversies surrounding authorship, dating, and contextual factors [21].

In this paper, we explore the potential of a transformer-based models, fine-tuned for sequence classification task, to provide further insights into the debate.

The focal point of our study is the text conventionally referred to as the Ars Rhetorica (Art of Rhetoric, hereafter ars). This work has long been attributed to, and frequently published under the name of, the rhetorician and historian Dionysius of Halicarnassus (ca. 60-7 BC). However, Sadée [36], followed by Usener [49] and Usener and Radermacher [50], demonstrated that the ars most likely circulated anonymously, with its association to Dionysius emerging from a much later conjecture. This conjecture appears to have been based on an overinterpretation of a scholion (a marginal commentary) on chapter 10 of the text.

Ars Rhetorica

Several aspects of the ars must be discussed in the context of statistical modelling of its writing style.

Not one, but multiple works

The text has a complex structure. In Parisinus Graecus 1741 [30], the only manuscript that preserves all the material associated with the ars (ff. , the text is divided into 11 chapters. However, these chapters do not constitute a homogeneous work, as the text is generally understood to consist of two [18], three [49,50,35,33], or even four [38] distinct parts.

The first part, covering ch. 1-7, provides concise instruction on ceremonial (epideictic) oratory, addressing seven epideictic genres. These chapters are connected by cross-references and recurring addresses to the author's former pupil, Echecrates, to whom the text is presented as a wedding gift. The remainder of the text, ch. 8-11, may be interpreted as a combination of two or three distinct works on separate topics. Ch. 8-9 explore the so-called "figured speeches", i.e., speeches intended to convey a hidden meaning that may conflict with the literal content and stated purpose of the speech, while ch. 10-11 focus on the criticism of declamation.

Authorship

Ch. 1-7 exhibit a consistent compositional pattern and a recognizable writing style, suggesting they were authored by the same rhetorician. However, whether these chapters form a coherent and complete treatise is a matter of debate. This portion of the ars has been interpreted as a collection of distinct letters or essays [38,18], as remnants or excerpts from a much longer work [4,49,50], or as a unified treatise [53,47,22]. For ch. 8-11, the situation is even more ambiguous. Usener [49] speculated that ch. 8-9 were written by two different disciples attending separate lectures of the same teacher. Penndorf [28] and Schöpsdau [37] rejected the idea that ch. 8-9 had a single author, suggesting instead that these texts drew from various sources. Similarly, ch. 10-11 have been attributed either to the same author as ch. 8-9 (with Heath [18] tentatively identifying him as Sarapion Aelius, a 2nd-century Alexandrian rhetorician whose entire corpus is lost) or to two different authors unrelated to the rest of the ars. Table 1 summarizes the content and authorship hypotheses for the various sections of the ars.

Ars Rhetorica, Menanderian Corpus, and Pseudo-Hermogenes' On Method

Since the early days of scholarship on the Ars Rhetorica, it has been noted that the rhetorical instruction provided in ch. 1-7 and ch. 8-11 shows a clear methodological afÏnity with, respectively, the treatises ascribed to Menander Rhetor (particularly the second one) and Pseudo-Hermogenes' On Method. The parallels with the second treatise attributed to Menander are especially noteworthy. In both works:

• the occasion -rather than the subject, as in the first treatise attributed to Menanderdetermines the genre; • a very similar selection of genres is discussed (of the seven genres mentioned in the ars, only two are absent from Menander's purported work; see Table 7); • the author addresses a former disciple throughout the text. This afÏnity led Heath [18] to describe the ars as "comparable to, though less sophisticated than" Menander's work.

The numerous parallels between ch. 8-11 of the ars and Pseudo-Hermogenes' On Method [29,18] have led scholars to hypothesize either a shared source [29] or a closer, albeit indirect, connection [35,18].

Dates

The dates of the texts constituting the ars have been assessed differently. For ch. 1-7, a mention of the 2nd-century sophist Nicostratus (ch. 2, par. 9, p. 266, l. 14), along with the considerable focus on speeches addressing Roman magistrates, suggests a composition date no earlier than the High Empire [35,22]. Race [31] posits that the first part of the ars is roughly contemporary with the corpus attributed to Menander Rhetor, which is datable to the late 3rd century AD. In contrast, ch. 8-11 may be a century earlier [18], i.e., 2nd century AD.

Aims

The hypotheses concerning the authorship of different parts of the ars have multiplied, as have suggestions regarding its potential relationship with other texts. However, the evidence presented in the scholarship so far is drawn almost exclusively from close reading and remains inconclusive. Additionally, unlike the case with ch. 8-11, no efforts have been made to identify the author responsible for ch. 1-7.

The aim of our investigation, therefore, is to apply modern natural language processing techniques to this rich textual material in order to gather new evidence about the structure of the ars and gain further insights into its authorship. The arguments formulated through language modeling could provide a novel and valuable contribution to the debate, particularly when considered alongside the accumulated philological evidence and existing codicological indications.

The main contributions of this work can be summarized as follows:

• We further fine-tune two pre-trained models for Ancient Greek and one model for Modern Greek on a corpus of Greek rhetoricians. We subsequently use the resulting models to train "open set" 10-class classifiers capable of attributing short fragments of text to different authors of the Second Sophistic period; • Analyzing in more details the results provided by two best-performing models, we shed light on the history of the Pseudo-Dionysian ars, suggesting that:

-Ch. 1-7 of the ars could have been authored by an individual from the same school as the author(s) of the Menandrian treatises; -Ch. 8-11 not only differ in authorship from ch. 1-7, but may have been written by two distinct individuals, one responsible for ch. 8-9 and another for ch. 10-11.

Corpus

The primary focus of our study is a corpus comprising at least 18 rhetores Graeci from the 1st-4th centuries AD and Dionysius of Halicarnassus. We only retained the authors whose teachings are relatively well-preserved, excluding those known only through fragmentary or indirect evidence. Importantly, we focus exclusively on rhetorical theory, i.e., works with a theoretical or pedagogical intent. A significant limitation of this dataset is that many of the rhetorical corpora within it have notorious attribution problems of their own. In particular, there is a compelling case for the heterogeneity of the Hermogenean corpus (see Section 6). Similarly, the question of whether both treatises attributed to Menander were authored by the same person remains unresolved [34,17,31,8]. Other corpora raise similar questions, too [21]. Being aware of this and currently working on a follow-up authorship verification study of these corpora (the importance of which was also insightfully emphasized by the reviewers of this work), for simplicity, we continue to group the studied texts by authorship as categorized in the Thesaurus Linguae Graecae (TLG), where our dataset stems from. 1 We deem this simplification legitimate. In most cases, these questionable attributions are rooted in long-standing traditions that date back to the early stages of textual transmission. For example, the Hermogenean corpus has been consistently attributed to Hermogenes of Tarsus since as early as the 5th century (for more details see Section 6). Therefore, with all necessary reservations, these conventional groupings can be considered to represent at least some kind of connection. Even if they do not link texts written by the same individual, they may still group works originating from the same school. After all, this is why such simplification is commonly used in scholarship.

<UNK> category

The literature on oratory theory was undoubtedly much richer than what has been preserved. To account for this, we created an "open set" scenario. For this purpose, we set aside 9 smaller authorial corpora -those with a number of sentences below the dataset's median value of 517 (marked with * in Table 2). These texts were excluded from the dataset before our conventional 80/10/10 split and later added to the the test set. The idea is straightforward. If at the test stage the model encounters a text that does not belong to any of the authorial classes learned during the training, it is likely that the calibrated probability associated with the top prediction will be relatively low. If it falls below a certain threshold, the model is programmed to abstain from making a decision and assign an <UNK> label to the text in question. The samples with <UNK> label in the test split are necessary to monitor the model's capability to do.

An overview of the classification dataset is presented in the Table 2.

Methodology

Base Transformers

To train our classifiers, we used three different pre-trained transformers as starting points:

(1) RoBERTa-sized GreBerta presented by Riemenschneider and Frank [32],

(2) Ancient Greek BERT trained by Singh, Rutten, and Lefever [39] (3) Modern Greek BERT published by Koutsikakis, Chalkidis, Malakasiotis, and Androutsopoulos [23].

Masked Language Modeling Fine-Tuning

Fine-tuning pre-trained models on domain-specific corpora prior to further tuning them for a downstream task at hand is a common practice in NLP. It allows the model to adapt better to the unique linguistic features of the target domain. This intermediate step may enhance the model's ability to capture specific syntactical patterns and vocabulary, which in turn improves the performance on the final downstream task, such as classification. For this reason, before training classifiers for authorship attribution, we ran training with a masked language modeling objective. BERT-sized models were trained for 3 epochs with a learning rate 1 × 10 −5 and warmup during the first 10% of training steps. RoBERTa-based model was trained for 1 epoch only with a learning rate 1 × 10 −4 and without warmup steps. In both scenarios, the learning rate was decreasing linearly.

Sequence Classification

Authorship classifiers were trained on both out-of-the-box models and their MLM-fine-tuned versions. We employed a sliding window technique to segment the texts into chunks. The process was as follows:

1. Tokenization: We used the bowphs/GreBerta tokenizer to convert the entire corpus into tokens.2 2. Chunking: The tokenized corpus was then divided into chunks of 64 tokens, respecting the boundaries of works (and even chapters -in the case of the ars); 3. Overlap: To ensure continuity and capture context that might span chunk boundaries, we implemented an overlap between chunks. Each chunk overlapped with its adjacent chunks by 32 tokens (half of the chunk length). 4. Decoding: Finally, we decoded these token chunks back into text, resulting in our training data segments. By using a single tokenizer to chunk the entire corpus beforehand instead of splitting the texts with a tokenizer of the corresponding model, we ensured that all models were trained on the same segments of text.

The training was carried out for 700 steps by sampling batches containing 4 chunks per authorial class. Validation set was checked each 350 steps, i.e., twice during the training. Test set including <UNK>-labelled samples was checked upon the end of training. We report the results obtained on the test set.

Results and Discussion

General Performance

Table 3 summarizes the overall performance of the classifiers. Notably, additional MLM training proved beneficial only for the RoBERTa-sized bowphs/GreBerta model. For BERT-sized models, however, the inclusion of new data was detrimental. bowphs/GreBerta appears to be more stable, behaving more like general-purpose language models trained for well-resourced modern languages. This stands to reason: out of the three models with which we experimented, bowphs/GreBerta [32] is the largest and was trained on the riches and highest-quality Ancient Greek corpus.

Authorship Attribution of the ars

The aim of this study was to get some fresh evidence about the authorship of the pseudo-Dionysian ars, a precious witness to the development of rhetorical theory during the High to Late Roman empire. Based on the status quaestionis surveyed in the section 2, we set up 3 research questions:

Table 3

Performance metrics on the test split with the <UNK> category (not represented in the training data). The models were configured to assign <UNK> to samples with a calibrated top probability below 80%. (R) denotes models fine-tuned with an MLM objective on the same data that was used to train the classifiers.

Model

F1 Score Accuracy 1. Can we further comfort or challenge the existing consensus opinion, according to which the attribution to Dionysius of Halicarnassus is incorrect? 2. How many works are discernible in the ars in the form we know it? 3. Can the model convincingly suggest an alternative attribution for the ars or any of its parts?

To address these questions, we applied the trained classifier to individual chapters of the ars, split into chunks following the described procedure. Table 4 summarizes the predictions made by the best-performing BERT-sized and RoBERTa-sized models. 3 For each chapter, we report the "majority vote" (i.e., the number of chunks in the chapter attributed to a given author), the author's "share" (i.e., the proportion of chunks assigned to that author in the total number of chapter chunks), and the mean probability of the author across the chunks of the chapter. In the "majority vote", the attribution is defined by the top probability even if it falls below 80%.

No trace of Dionysius of Halicarnassus

In line with previous scholarship, although the name of Dionysius of Halicarnassus appears among the attributions, its weight is insignificant. Therefore, with regard to the first of the research questions, the evidence is overwhelming: stylistic afÏnity with Dionysius of Halicarnassus's writings is scarce, and the attribution to him cannot be supported by any of the two models.

ars's association to the Menandrean corpus further strengthened

Apart from this rather predictable conclusion, our classifiers yield new insights into more complicated questions concerning the inner structure of the ars and the authorship of the texts, which constitute it. As clear from the Table 4, the attribution profiles for ch. 1-7 and 8-11 are drastically different. Even when the probability is not high enough, Menander Rhetor is the top-ranked candidate in ch. 1-7. The signal is less clear in ch. 8-11. This difference goes in line with the communis opinio that the work is composite: a nearly identical attribution profile of ch. 1-7 being yet another argument in favour of its unity.

What does the model learn?

For the sake of explainability, DH specialists still widely use the bag-of-words model and corpus-specific manual feature engineering for various tasks involving writing style analysis, such as authorship attribution, authorship and self-authorship verification, clustering, etc. [13,12,24,3]. Since deep learning methods lack this level of transparency, understanding exactly what our classifier learned is crucial. A thorough investigation of this matter will be the subject of a separate study, using explainable AI techniques such as integrated gradients and token attribution. Here, we limit our discussion to one insightful example, which seems to illustrate how the model works.

As previously mentioned in Section 2, all the genres addressed in ch. 2-5 are also discussed in the second treatise attributed to Menander Rhetor. Only the most prestigious of the epideictic genres, the panegyric -focused on in ch. 1 and 7 -does not correspond to any section in Menander's works. However, ch. 1, which provides introductory notes on panegyrics, often echoes the examples and some wording of the first treatise by Menander. Ch. 1 offers guidelines on how to appropriately praise gods ("leaders and name-givers of any festival"), cities where the festivals take place, and emperors who organize and preside over the festivals. All these topics are covered in Menander's first treatise.

Considering only ch. 2-5 or the fragments of ch. 1 that have clear parallels in Menander's work, one might argue that the classifier's decision was biased due to the significant content and semantic overlap, especially since such a tendency has been reported about the BERT-based classifiers [7]. However, the consistency of the attribution profile across the chapters by both models is reassuring, as it suggests that they capture more than just semantics.

Menandrean association appears all the stronger when the values for the logical subdivisions of the ars, ch. 1-7, are calculated. As Korenjak [22] has shown, in its current form, the order of the chapters is disorganized, and it is possible that the author intended to arrange them as follows: chapters 1 and 7 (panegyrics or appraisal speeches), chapters 2-4 (speeches related to

Table 7

Content overlap between the ars and the second treatise attributed to Menander. Chapter division for Menander's treatise follows Race [31].

Menander Rhetor

Treatise II ars 5, 6 2, 4 7 3 9 5 8, 10, 15 6 family life occasions), and chapters 5-6 (speeches addressed to ofÏcials and epitaphs). In each of these sections, Menander maintains a stable leadership (Tables 5 and 6).

ars, ch. 8-11: multiple authors? The discrepancy between the attribution profiles of ch. 8-9 and ch. 10-11 might suggest a division, albeit a less distinct one, than ch. 1-7 versus ch. 8-11. This result aligns with the assessment made by Usener [49], although it does not provide any further hint at the identity of the possible author. However, the opposite hypothesis should still be considered seriously. In ch. 1-7, top two single attributions (i.e., Menander Rhetor and another author) in terms of "share" would cover at least 0.58-0.68 of the attributed chunks (ch. 7). In contrast, the top two attributions in ch. 8-11 provide, at best, 0.58-0.62 of the attributions (ch. 11 and 10), the attributions are more evenly distributed. Apparently, among the author classes present in our dataset, none is stylistically similar enough to the text of ch.8-11. This can be explained in two ways. Texts written in a comparable style are either completely absent from the dataset or are not appropriately distributed among author groups, making it challenging for the model to learn the features of this particular writing style. Keeping in mind the existing hypothesis about the relationship between the so-called Hermogenean canon and the works ascribed to Apsines, with extreme caution, we incline to the latter explanation.

Two works, which are part of the Hermogenean canon, On Invention and On Method, were already in Late Antiquity associated with the name of Hermogenes. In our dataset, therefore, following the TLG, we reproduce this conventional attribution. Yet, both are most likely inauthentic [10,11,25,26,27]. If the argumentation presented by Heath [16,17] proves correct and these two texts can securely be ascribed to Apsines, the "new" writing style they would represent might possibly demonstrate a more pronounced afÏnity with the style of ch. 8-11. This and similar possibilities should be thoroughly checked in further experiments.

The scope of the much-needed detailed follow-up study becomes evident. A systematic and critical reassessment of attribution problems within the corpus of the Rhetores Graeci is necessary. Beyond merely reflecting on the attributions of individual works, it is important to establish the homogeneity of different rhetorical corpora within the framework of a pairwise authorship verification study.

But if we set aside the obscure case of ch. 8-11, should we conclude that ch. 1-7 were written by Menander Rhetor? Given the aforementioned limitations of our dataset, we would not go that far. However, our results suggest that the connection between the first part of the Pseudo-Dionysian ars and the Menandrean corpus likely extends beyond a theoretical afÏnity. Despite the obvious terminological discrepancies between the texts and their different levels of elaboration, the possibility of multiple authorship within the same school, or even common authorship, should be considered with all seriousness. The divergence between the ars, ch. 1-7, and the Menandrean corpus can also be explained, apart from the natural evolution of personal style and preferences, by the likelihood that those presenting complex rhetorical theory would probably follow the advice formulated by the author of ch. 11. The art of rhetoric involves presenting material in a way that convinces the audience. Thus, orators are similar to doctors who must not only select the right medication but also administer it in a manner acceptable to the patient [50, ch. 11, par. 9, p. 385, ll. [7][8][9][10][11][12]. In other words, multiple contextual factors influenced the style of the presentation, and, in the cases when the stylistic afÏnity is clear, one should not probably overinterpret isolated differences.

Conclusion

This study uses transformer-based models to analyze ancient rhetorical texts for authorship attribution in classical philology. First, we adapted these models to handle the linguistic nuances of Ancient Greek texts from the 1st to the 4th century AD using masked language modeling. We then apply the fine-tuned models to identify authorship markers in Ars Rhetorica, a text possibly written by multiple ancient writers. This application not only reminds of benefits of modern AI techniques to classical studies but also deepens our understanding of ancient literary compositions through modern computational methods.

The results of BERT and RoBERTa classifiers do not support connection of the ars to Dionysius of Halicarnassus, going in line with the previous studies that question his authorship. They also strengthen the link of ars to the Menandrean corpus, particularly evident in the distinct attribution profiles between chapters 1-7 and 8-11, which suggests a composite nature of the work.

Despite the lack of transparency of MLM techniques compared to conventional methods, which prioritize human-interpretable features, the effectiveness and relevance of machine learning methods is noteworthy.

While neural networks are often criticized in digital humanities for their black-box nature [12], their ability to detect writing styles make them a valuable tool in the field of digital humanities. The use of these models promises significant advancements in authorship attribution and our understanding of ancient literary works.

Limitations

This study has several limitations that should be considered when interpreting the results.

Firstly, the issue of disputed authorship within the dataset is a significant challenge. For instance, the Hermogenean corpus and Menandrean treatises, both central to our analysis, have long-standing debates regarding their true authorship, see Section 4. These uncertainties could affect the attribution accuracy. We are currently working on a study intended to solve this issue, adopting an authorship verification approach.

Secondly, the use of transformer-based models like BERT and RoBERTa, come with limitations related to their opaque nature. The lack of interpretability in these models means that understanding the specific features and patterns the models use to make attributions is challenging. This limits our ability to provide a transparent rationale for the models' decisions, which is often critical in digital humanities research. Yet, the attempts were made to find way to make the results of pre-trained language models more interpretable, e.g., by means of the so-called integrated gradients [46]. These methods can perhaps be adapted for cases similar to ours.

Despite achieving notable accuracies with relatively short chunks (64 tokens), the models' performance still leaves room for improvement, particularly in terms of handling unbalanced corpora and downplaying the influence of the thematic clues. Nevertheless, their performance, comparable to state-of-the-art results for modern languages, demonstrates an ability to capture writing style. There clearly are instances where the models are overly confident, leading to incorrect authorship attribution. These errors could arise from factors such as the models' sensitivity to stylistic nuances and the complexity of the texts. Embracing more sophisticated methodologies for uncertainty-aware training would be an interesting avenue for further exploration.

Another potential avenue for future research is the development of chronological and regional classifiers. Texts from different regions and periods may exhibit unique linguistic and stylistic features that are not captured by a generalized model. Developing classifiers specific to historical periods or geographical (and cultural) areas could enhance attribution accuracy and offer more detailed insights into the ars and many other texts.