Multilingual Stylometry: The Influence of Language on the Performance of Authorship Attribution using Corpora from the European Literary Text Collection (ELTeC) Christof Schöch1 , Julia Dudar1 , Evgeniia Fileva1 and Artjoms Šeļa2,3 1 Trier Center for Digital Humanities, Trier University, Trier, Germany 2 Institute of Polish Language, Polish Academy of Sciences, Kraków, Poland 3 Institute of Czech Literature, Czech Academy of Sciences, Prague, Czech Republic Abstract Stylometric authorship attribution is concerned with the task of assigning texts of unknown, pseudony- mous or disputed authorship to their most likely author, often based on a comparison of the frequency of a selected set of features that represent the texts. The parameters of the analysis, such as feature selec- tion and the choice of similarity measure or classification algorithm, have received significant attention in the past. Two additional key factors for the performance and reliability of stylometric methods, how- ever, have so far received less attention, namely corpus composition and corpus language. As a first step, the aim of this study is to investigate the influence of language on the performance of stylometric authorship attribution. We address this question using four different corpora derived from the European Literary Text Collection (ELTeC). We use machine-translation to obtain each corpus in the other three languages. We find that, as expected, the attribution accuracy varies between language-based corpora, and that translated corpora, on average, display a lower attribution accuracy compared to their counter- parts in the original language. Overall, our study contributes to a better understanding of stylometric methods of authorship attribution. Keywords Stylometry, Authorship Attribution, Multilingualism, ELTeC, Translation 1. Introduction Stylometric authorship attribution is the task of assigning texts of unknown, pseudonymous or disputed authorship to their most likely author, often based on a comparison of the frequency of a selected set of features that represent texts [14, 26, 6, 23]. Traditionally, the way stylometric methods approach authorship attribution is to use the fre- quencies of a large number of simple features, such as words, lemmas or character sequences, for a determination of the degree of similarity between texts. These similarities, in turn, are in- CHR 2024: Computational Humanities Research Conference, December 4–6, 2024, Aarhus, Denmark £ schoech@uni-trier.de (C. Schöch); dudar@uni-trier.de (J. Dudar); fileva@uni-trier.de (E. Fileva); sela@ucl.cas.cz (A. Šeļa) ç https://christof-schoech.de/en (C. Schöch); https://artjomsh.github.io/web/ (A. Šeļa) ȉ 0000-0002-4557-2753 (C. Schöch); 0000-0001-5545-9562 (J. Dudar); 0009-0006-4555-7824 (E. Fileva); 0000-0002-2272-2077 (A. Šeļa) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 386 CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings terpreted as an indicator of the likelihood for two texts to have been written by the same author: the more similar the feature vectors, the more likely is identical authorship. The parameters of the analysis, such as feature selection and the choice of similarity measure or classification al- gorithm, have received significant attention in the past (e.g. [1, 25, 27, 11]). Two additional key factors for the performance and reliability of stylometric methods, however, are corpus compo- sition and corpus language. They are relevant not only for the results in a specific case, but also for the overall performance and reliability of stylometric methods of authorship attribution. Therefore, the aim of ongoing research by our group is to disentangle the influence of corpus composition and language on the performance of stylometric authorship attribution: To what extent do the attribution accuracy and robustness of such approaches depend on the language of the materials? To what extent do they depend on the corpus composition, which can present more or less challenging constellations of authorship and style? How do these two factors interact with each other, and how do they interact with feature selection? This paper tackles one part of this issue, that of language, by investigating four distinct but broadly comparable corpora in a classification scenario. The corpora are all derived from the European Literary Text Collection (ELTeC) and contain at least three novels each by 8-10 different undisputed authors. These corpora can therefore be used as benchmarking datasets, where the true authorship is known for all novels under investigation. Specifically, corpora in English, French, Hungarian and Ukrainian are included. While the corpora contain similar texts (fictional narrative prose from a similar time period, i.e., 1840–1920), they differ both in terms of their exact composition, which can affect overall difÏculty of attribution and therefore attribution performance, and in terms of their language, which can likewise affect attribution performance. In order to investigate the role of language independently of corpus composition, all four corpora were automatically translated into the other three languages using the DeepL machine translation system (Pro version, used in the time period July to September 2023). This allows us to vary language while keeping corpus composition stable, and in this manner, tease out effects of both in terms of attribution performance (measured as classification accuracy). In addition, and in order not to disadvantage any one corpus or language by selecting fea- tures that may not be suitable to them, a number of further parameters have been varied, namely: the type of feature considered (word forms, lemmas, part of speech or characters); the length of the feature sequence considered as the unit of analysis (unigrams, bigrams, 3- grams, 4-grams or 5-grams); the total number of different features considered (from 50 to 2000 in several increments); and the length of contiguous textual segments considered (from 5000 words to 10000 words as well as using entire novels). Testing all of these various parameters, corpora and languages in all of their possible com- binations results in a large number of individual results. In order to make them accessible to inspection and analysis, we have developed an interactive visualization that displays the attri- bution quality (i.e. the classification accuracy) in a heatmap as a function of the parameters described above. While the heatmap displays the accuracy for the entire range of features and segment lengths at the same time, other parameters can be selected by the users in order for the heatmap to be updated with the corresponding accuracy values. Two such heatmaps are provided next to each other to enable convenient comparison of results between any two configurations of parameters, whether corpora, languages or other settings. 387 The present paper can be understood as a background publication to these heatmaps pub- lished as a publicly available, interactive online showcase that is available at https://showca ses.clsinfra.io/stylometry. The target audience of this showcase are scholars and students interested in a key methodological aspect of stylometry-based authorship attribution as well as those interested in cross-lingual approaches within the digital humanities. The showcase is intended both as a convenient way to document a specific research output and as a pedagogical tool that could be used in workshop or classroom settings alike. Readers are encouraged to use the showcase online, where much more detailed results can be explored, beyond the high-level summary of results we present here. The detailed results are discussed below, and contribute to our general understanding of the influence of language and corpus composition on stylometric results. Broadly speaking, we can show that corpora of different language and composition lead to different attribution accuracy levels and different best-performing features, an expected result. We can also show that trans- lated corpora (at least when all texts have been translated by the same machine translation system) usually lead to a lower attribution accuracy, overall, compared to their counterpart in the original language. In the following, we first discuss relevant prior work evaluating stylometric performance in the context of multilingual corpora (section 2). We then outline our research questions and objectives (section 3), before describing our data and methods (section 4 and section 5). We discuss the main outcomes with respect to attribution performance, detailing corpus-based and language-based variations in results (section 6). By way of a conclusion, we discuss the consequences of our findings, document some limitations of this research and outline ongoing research into the second aspect of this investigation, namely corpus composition. 2. Relevant prior work There has been a significant body of work investigating the methodological and theoretical foundations of stylometric authorship attribution. Generally speaking, it seems fair to say that most attention has been devoted to feature selection and the choice of distance measures or classification algorithms. While much research is focused on English-language materials, a limited amount of attention has been given to the question of variation across languages. Only rarely, however, has the influence of corpus composition on attribution performance been assessed beyond general statements about the advantages of generically-homogeneous corpora. In the following, we discuss some of this research in more detail. With respect to investigations of feature selection, John Burrows [5, 4] proposes multiple measures each focused on a different part of the frequency spectrum: Delta, Iota and Zeta. Smith and Aldridge [25] use word forms in order to focus on the best setting for most frequent words and word vector onset and have found that “a word frequency vector of between 200 and 300 words gives the most accurate results” when using Burrows’ Delta. They also show that shifting the feature vector onset, effectively skipping the most frequent features, is detrimental to attribution accuracy. In addition to varying the number of most frequent words, examples of using features other than words can also be found in the research literature. In particular, Stamatatos [27] focuses 388 on the effectiveness of using character n-grams. He has conducted tests of 3-grams and fre- quent words on texts of different genres and topics. The researcher views as an advantage of n-grams that they capture linguistic patterns and may therefore be suitable to represent an author’s style beyond word choice. He has obtained the best results with a “model based on character 3-grams in combination with an SVM classifier with linear kernel”, and suggests using this as a baseline for authorship attribution. Building on the conventional use of word frequencies, Halteren et al. [13] investigate the application of syntactic features in stylometry. Their research highlights how syntactic patterns, based on word class tags, complement lexical features in finding what they call the “human stylome”, i.e. individual measurable linguistic features that help distinguishing between authors. The study shows that although 100% ef- ficiency results could not be achieved, lexical features perform slightly more efÏciently than syntactic features and that in general, a greater number and/or variety of features improves the results. In a similar vein, Gorman [12] uses syntactic information for the attribution of short texts. With respect to distance or similarity measures, Argamon [1] proposes a geometric interpre- tation of Burrows’ Delta and derived several alternatives to this particular distance measure, among them Quadratic Delta. Smith and Aldridge [25] propose to move beyond distance-based comparisons and suggest the adoption of the Cosine similarity instead, where the directions of the vectors are compared without being influenced by their length. Evert et al. [11] follow this important new direction and propose Cosine Delta, which combines z-score normalization with the Cosine measure. They have confirmed the finding by Smith and Aldridge [25], namely that performance is more stable for longer feature vectors when Cosine Delta is used. In addi- tion, they show that this effect holds across several languages, at least for long fiction narratives (novels). In several cases, interactions between various factors (in particular between feature vector length, distance measure and language) have been observed. With respect to languages, many methodological investigations focus on English corpora only, as is the case with Burrows [5] or Smith and Aldridge [25]. Rybicki and Eder [19], how- ever, show that results vary widely as a function of language when they investigated feature vector length and onset for multiple corpora in different languages and covering various liter- ary genres. Evert et al. [11] test all their hypotheses on English, German and French corpora consisting of novels, with only minor differences between languages. However, the extent to which the differences in performance can be explained by language, on the one hand, and by the inevitable differences between corpora with respect to their genre and/or composition, is not within the scope of their research. Stylometric research into translated works does exist, but is relatively rare. Rybicki and Heydel [22] show that in the case where several different translators translate the works of a given author, their stylistic signature can be detected by stylometric methods. In other cases, especially in corpora involving multiple authors and multiple translators, the authorial signal appears to be stronger than the translator’s signal [21]. In a recent study, Rybicki [20] focuses on network analysis, using an extensive collection of translated texts from different genres and languages into Polish. He observes that the translated and original texts reveal stylistic similarities, from which it can be concluded that the translations are an extension of native literary traditions. Rybicki also notes that the distinctiveness of genres becomes visible during stylometric analysis. 389 There are also several relevant studies that use machine translated texts. This approach underlies Cross Language Authorship Attribution, a concept introduced by Bogdanova and Lazaridou [2]. Although the quality of automatically translated texts may distort an author’s style, translation is one way of bringing texts written in different languages “into one space”. Machine translation in this study is combined with a focus on lexical and higher-level features. The best results were achieved by combining machine translation and k-Nearest Neighbors. Based on this study, Mikros and Boumparis [17] apply the machine translation method for the language pair Greek-English. They conclude that when trained and tested on machine- translated texts, attribution accuracy is not strongly affected, compared to training and testing on original texts, but that the relevant features differ between the two languages, making cross- linguistic authorship attribution unreliable. As mentioned above, the influence of corpus composition on attribution performance has so far not been investigated in any considerable detail. Smith and Aldridge [25] observe that a chronologically more diverse corpus leads to lower attribution accuracy but do not go into much detail. Kestemont et al. [15] show that cross-genre attribution is more challenging than within-genre attribution. A challenge for this kind of investigation is the availability of mean- ingful, high-quality metadata to assess corpus composition. Based on this review of relevant research, we can conclude that we should expect some variation in attribution accuracy depending on language. Also, there is a clear need to investi- gate the role of corpus composition for the accuracy and robustness of stylometric authorship attribution. 3. Research questions and objectives The overarching research question that ongoing work of our team addresses is to what ex- tent, how, and under what conditions, both language and corpus influence the performance of stylometric methods of authorship attribution. The objective of our research, therefore, is to provide detailed data evaluating the accuracy and robustness of stylometric methods for cor- pora of different compositions and in different languages. As a first step, the present paper aims to determine how the results of stylometry are influenced by the language of the texts, when the corpus composition remains the same, that is, using corpora translated into multiple languages. How does accuracy vary across languages? How does it change when a given cor- pus is translated into a different language? What kind of interaction can we observe between feature selection and language, as well as between sample size? In a second step, not covered by the present paper, we aim to determine how corpus compo- sition affects attribution accuracy. In order to address this question, a formalization of corpus composition needs to be designed that quantifies the level of difÏculty, for an attribution task, of a given corpus. How does such a measure need to be designed in order to adequately repre- sent corpus difÏculty in terms of authorship attribution? What types of metadata, for example regarding subgenre, time period or formal aspects, are most strongly correlated to differences in attribution accuracy? Does a higher corpus difÏculty as measured based on metadata cor- relate with lower attribution accuracy? What is the effect of translation on this relationship? Our preliminary work on this issue is discussed in subsection 7.3 of this paper. 390 4. Data The dataset was derived from the European Literary Text Collection (ELTeC; see [24, 3]) in order to build corpora in four languages: French, English, Hungarian, and Ukrainian. The choice of languages is determined by our desire to cover a number of different language groups: Ro- mance, Germanic, Finno-Ugric, and Slavic languages. In this way, one can illustrate the effec- tiveness of the stylometric method and see how language affects the analysis results, for the four language groups. The original ELTeC corpora each include one hundred novels. For our research, a selection of novels was made, retaining novels by 8 to 10 authors, each represented with three novels from ELTeC. This resulted in the following collections of texts: the English and the French corpora include each 30 novels, the Hungarian corpus 27 novels and the Ukrainian corpus consists of 24 novels (see key information on the corpora in Table 1). Table 1 Key information on the four corpora used in this study (limited to the corpora in their original lan- guages; we expect numbers of types and tokens to vary in the translations.) fra eng hun ukr number of authors 10 10 9 8 number of novels 30 30 27 24 total number of types 68,783 96,603 256,867 99,870 total number of tokens 2,414,226 4,791,903 2,635,112 824,130 median number of tokens 75,218.5 148,004.5 76,646.0 20,008.5 shortest novel, in tokens 29,460 23,459 14,590 9,517 longest novel, in tokens 149,810 348,793 285,818 102,859 earliest novel, year 1840 1844 1842 1841 latest novel, year 1900 1912 1908 1919 Each of the corpora was translated into the other three languages, thus the entire corpus includes 16 sub-corpora. The dataset also includes a metadata table for each corpus, with in- formation about each of the novels. The metadata table includes information about the author, year of publication of digital and print editions, language, number of words in the novel, as well as subgenre (social, historical, adventure, detective or sentimental novel, bildungsroman or other) and narrative perspective of the novel (heterodiegetic, homodiegetic, epistolary, di- alogue or mixed). This metadata was collected from experts in each of the relevant literary traditions who based their information on a reading of both the novels and of relevant sec- ondary literature. Data preparation involved two stages: translation of texts into the other languages and their linguistic annotation. The texts were translated automatically using DeepL Pro. For subse- quent analysis, extracting lemmas and POS (part of speech) tags was necessary, a task we accomplished using the SpaCy library (version 3.7) for both original and translated texts. Ad- ditionally, unigrams and n-grams (from 2 to 5) were extracted using the stylo package [10]. 391 5. Method(s) Fundamentally, we used an authorship attribution classification task with leave-one-out cross- validation. Inspired by Rybicki and Eder [19], we used grid search over the features space to assess the general performance of authorship attribution and quantified the attribution per- formance in terms of its accuracy (mean attribution accuracy score for each condition as well as Cohen’s Kappa for each condition). The goal was not to optimize for performance, but to cover the space of reasonable approaches that work for different languages and to understand the nature of differences in performance across languages and corpora. 5.1. Features While in many scenarios, simple word form frequencies still prove effective, other kinds of features can in some settings be advantageous, for example for short texts or agglutinative languages [18]. Therefore, we did want to vary the feature types to some extent, in order not to miss conditions which work well in one or the other of the less-frequently investigated languages. There are three main levels of variation: types of features, sample size and feature vector length. In terms of types of features, we used frequencies of word forms, lemmas, part-of-speech (POS) tags and characters. Each feature was cut to n-grams of different size: 1-3 for words and lemmata, 2-5 for character and POS n-grams (because the number of POS and character unigrams is too limited to be useful). Each n-gram length was tested independently. With respect to sample size, frequency-based approaches to authorship attribution naturally depend on the available size of the text. There is a considerable variation in text sizes in EL- TeC (as shown in Table 1); to mitigate this, we draw a random sample of consecutive tokens (a “chunk”) for each text based on the shortest text across all corpora (10,000 words). We use test sizes of 5,000 to 10,000 tokens for word-based features, and 10,000 to 50,000 for character-based ones. The number of available tokens per text differs dramatically between words and charac- ters; so sample sizes mean different things for n-grams based on these features. To account for the variability that is introduced with taking only one limited sample out of sometimes very large novels, at each step we take a random consecutive sample out of all available ‘chunks’ and record performance 100 times. As the last step, we perform classification on full-length texts. In terms of feature vector length, vectors that represent texts are constructed based on most frequent feature length cutoffs. We used 50 to 2000 features with incrementally increasing step sizes. 5.2. Classification task For author-based text classification, we use the Support Vector Machine classifier and perform leave-one-out cross-validation. This means that in each step, one text is removed from the data, the model is then trained on the remaining texts, and then the authorship of the left-out text is predicted and the result is recorded. This process continues until all texts have been left out once. For each language combination, we run 100 iterations of leave-one-out cross validation 392 classification, taking a sample from each text (chunks of consecutive sequences of size n) at random. Additionally, we run a single full-text analysis for each corpus. We report two performance measures: simple accuracy (proportion of correctly predicted au- thors) and Cohen’s kappa (typically used for measuring inter-annotator agreement [7]). The latter measure, while highly correlated to accuracy, allows us to partly offset the different amount of classes across corpora, since it provides an inter-rater agreement score adjusted for random classification. 5.3. Visualization To create the interactive visualizations of the showcase, we used Bokeh, a Python library of- fering extensive capabilities in crafting interactive and dynamic data visualizations. The visu- alization – which is available online at https://showcases.clsinfra.io/stylometry – consists of two heatmaps, enabling users to compare and contrast two sets of results simultaneously. In this article, we focus on the results for the Hungarian corpus, which are presented in subsec- tion 6.2. For results from other corpora, please see section 8 (appendix) and/or visit the online showcase. The x-axis represents various settings of the most frequent features (MFF) used in a specific analysis, while the y-axis denotes distinct sample sizes, ranging from shorter text snippets to entire novels (”full novel”). Words and characters are treated differently and analyzed across diverse sample sizes (see subsection 5.1 above). Each heatmap cell correlates MFF and sam- ple size, with the color intensity indicating accuracy levels, offering a visual approach to the accuracy obtained in the analysis. A mouseover provides further information, such as the fea- tures used as well as numerical indications of accuracy and Cohen’s Kappa. When there is no data corresponding to selected feature combinations, the plots display a ”No data available for these selectors” message. The range of MFF values depends on the features; in some cases, as for character and POS bigrams, the theoretical maximum of features is limited. Users can engage with the data through several selectors, enhancing the exploration and analysis process. The available choices include: 1. Corpus selector: Enables the selection from a range of available corpora, facilitating comparative studies; 2. Feature level selector: Permits users to toggle between ”words” and ”chars” (characters) feature types; 3. Feature type selector: Allows for the choice between plain text, lemmas, or POS tags; 4. Ngram size selector: Offers options to choose n-gram sizes from 1 to 5, allowing for detailed linguistic patterns examination. A color scale on the right side of each graph delineates the accuracy levels, ranging from 0 to 1. Here, cooler tones like greens, blues and purples signify lower values, whereas warmer hues, like red and orange, indicate higher values. This color coding serves as a primary indicator of stylometric effectiveness across various parameter settings, vividly illustrating the success rate of stylometric analysis as a function of the chosen corpus and the selected criteria. 393 6. Results In relation to the main outcomes, we aim to present a concise overview of some of our key findings. First, we focus on evaluating the overall performance of authorship attribution, based on mean accuracy scores across multiple settings, for all corpora. Second, we provide a detailed description of the main outcomes for the Hungarian corpus and its translations. We focus on one language for reasons of space, but readers can find a detailed description of the performance of the other three languages in the appendix (section 8) to this paper. 6.1. Attribution performance across all corpora Regarding overall performance, we discovered several parameters that consistently yield the best results across all corpora and translations. At the word level, the following parameters showed the highest scores: unigram for both feature types (plain word forms and lemmas), with only minimal differences. Meanwhile, at the character level, very high performance was observed using 5-grams based on both plain word forms and lemmas as feature types and using full novels as sample size. In Figure 1 and Figure 2, the average classification accuracy at the word level is illustrated, employing unigram plain word forms, with sample sizes ranging from 7,000 to 9,000 and from 10,000 to full novel size, respectively. As observed in the figures, the clear winner among the original texts is the Hungarian corpus. Translations from Hungarian into three other languages also demonstrate very high performance, slightly decreasing to 0.6 and 0.7 from 0.86 compared to original texts when full texts and sample size of 10000 are taken into account. Concerning smaller sample sizes, the decrease is more noticeable, achieving an accuracy of 0.4-0.5, while the accuracy of original texts is 0.8. The second-best performance comes from the original Ukrainian corpus; however, translations from Ukrainian have low performance, falling to 0.5 and 0.4 from the original scores of 0.76 and 0.65 for large and smaller sample sizes respectively, indicating that translations from Ukrainian are particularly challenging for attribution tasks. The best performance among translations from Ukrainian, surprisingly, is shown by the Hun- garian language. The French and English corpora have almost similar performance when the original texts are analyzed. However, translations from English at the word level perform lower compared to the original texts, and the difference in performance is more pronounced with smaller sample sizes, falling from 0.5 to 0.3-0.4. On the other hand, at the character level, translations into Hungarian outperform original texts with the following settings: 5-gram based on both lemmas and plain word forms. Remarkably, the smallest difference in performance between translated texts and the origi- nal is observed in the French corpus, where the difference is smaller than 0.1 for large sample sizes and only slightly greater than 0.1 for smaller sample sizes. It appears that the authors’ style is best preserved in translations from French. All three translations from French demon- strate similar performance; however, translations into English slightly outperform the other translations. 394 Figure 1: Mean accuracy across each corpus with the following parameters: word forms, word level, ngram = 1, sample size = 7000, 8000, 9000. Number of most frequent features: 100–1000. Figure 2: Mean accuracy across each corpus with the following parameters: word forms, word level, ngram = 1, sample size = 10000 and full novels. Number of most frequent features: 100–1000. 395 6.2. Hungarian corpus attribution performance As already discussed earlier, the Hungarian corpus (Figure 3) demonstrates the highest accuracy scores overall, reaching approximately 80–95% across many sample sizes; starting with samples of 8000 words, results are reliably very accurate. This achievement is realized by employing the following settings: the number (N) of the most frequent words ranges from 50 to 700, with the analysis based on unigram plain word forms. Although slightly less effective, classification based on unigram lemmas still produces high scores. Regarding the analysis using character n-grams as features, the performance remains remarkably high, fluctuating between 70-93% for sample sizes starting from 4000, with the most remarkable performance observed when 5-gram character sequences are used with full novels (Figure 4). In this scenario, the attribution based on characters, derived from word forms with the number of MFF from 50 to 500, maintains a constant accuracy rate of over 90% for full novels. When evaluating the accuracy of translations from Hungarian into other languages, it be- comes evident that the performance is noticeably lower compared to original novels. One of the few configurations displaying high accuracy (over 90%) is observed in translations into Ukrainian utilizing 5-gram characters based on plain word forms. For lemmas, the Ukrainian translation, in some cases, outperforms the original texts (Figure 5). As for the attribution at the word level, high accuracy is observable only for full novel sample sizes. Translations into English and Ukrainian exhibit very similar performance, whereas French displays lower accuracy. 7. Conclusion In conclusion, we would like to summarize some of the key findings of this study, but also discuss a number of limitations and next steps for research in this area. 7.1. Findings We can discern consistent patterns across all languages and translations when analyzing their performance. These trends remain mostly consistent whether we are examining analyses based on words, lemmas or character n-grams. Optimal performance consistently emerges when the analysis is based on entire novels rather than shorter samples. When using samples, results tend to improve as the size of the analyzed samples increases. This finding is consistent with previous studies [4, 8, 9]. Conversely, increasing the number of features used for attribution beyond a certain value tends to lead to a deterioration in classification scores in most cases. The decline in perfor- mance becomes noticeable at full novels and word unigrams beyond roughly 300-800 most frequent words for French, English, and Hungarian (but this is not the case for Ukrainian). Furthermore, the impact is considerably less pronounced for word bigrams and trigrams. This finding is consistent with results from distance-based attribution studies when using Burrows’ Delta, but not when using Cosine or Cosine Delta [25, 11]. This phenomenon may be caused either by an increasing proportion of features driven by topic rather than authorship, or by an increased amount of noise in the signal, when using higher numbers of features. 396 Figure 3: Mean accuracy scores for the Hungarian corpus in the original language, for different settings of MFF and sample size. Parameters: Plain word form unigrams. Performance for corpora is always better when they are analyzed in their original language, rather than in a translation into another language. However, this might be due to the fact that we have used a unique translator, as it were, this fact being a major limitation of our study (see below). 7.2. Limitations The finding, mentioned above, that the translated corpora always perform less well, in our ex- periments, than the original corpora, may very well be an artefact of our method of translating these texts. Given that we used DeepL for all translations, the translations could be said to have all been produced by the same ‘agent’, leading to a certain homogenization of their style and lexicon, which we would expect to be a challenge for stylometric authorship attribution. In contrast, translations of literary texts are usually performed by a range of different translators, some of which maintain privileged relationships with certain authors, in a constellation where authorial and translatorial signal may reinforce each other. At the same time, the question of the (in)visibility of the translators in stylometry has been addressed several times [21, 16]. Another limitation with respect to the corpora is that for each language, we have only one particular corpus available. This means that, given how important the genre of a corpus as well as the corpus composition are likely to be, we cannot generalize our results based on these particular corpora to the languages in general. 397 Figure 4: Mean accuracy scores for the Hungarian corpus in the original language, for different settings of MFF and sample size. Parameters: Character 5-grams of lemmas. 7.3. Next steps As mentioned above, the research presented here only covers one of two aspects that we con- sider relevant to our overall research question. Our corpora differ between each other both in language and in composition. Therefore, the investigation of the influence of language and of translation described here needs to be complemented by an analogous investigation into the influence of corpus composition on the results. We have already made significant progress in this investigation, notably by collecting rel- evant metadata about all four corpora and by computationally modeling the relationship be- tween corpus composition and attribution performance. With respect to the metadata, we have collected information about the year of publication, the narrative form and the subgenre for each novel in each corpus. To enable cross-corpus application of the subgenre labels, they come from a small, closed set of eight different, relatively broad categories, including one category called ’other’. With respect to the computational model of the relation between corpus com- position and attribution accuracy, we have fitted a multiple generalized Bayesian linear model that takes into account within-author and between-author variability and allows us to assess the positive or negative influence of metadata differences on textual distinctiveness (and, there- fore, stylometric performance). However, further work on this issue is needed and presenting this work warrants a separate publication. Apart from this, our research opens up several avenues for future research, in order to at- tempt to close some of the gaps just described in the section on limitations. For instance, the same kind of analysis could be performed using several corpora of different genres, size and composition for each language, in order to assess the degree of within-language variation and 398 Figure 5: Mean accuracy scores for the Hungarian corpus translated into Ukrainian, for different settings of MFF and sample size Parameters: Character 5-grams of lemmas. the robustness of differences between languages that we have observed in this study. In addi- tion, a study could be conceived that varies corpus composition deliberately and systematically, based on a significantly larger corpus with document-level metadata (at least) on subgenre and narrative perspective. 8. Data and code Data and code are available online at https://gitlab.clsinfra.io/cls-infra/d33 (DOI: https: //doi.org/10.5281/zenodo.11080205). Acknowledgments The research presented here as been performed as part of CLS INFRA, a project supported by funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101004984. References [1] S. Argamon. “Interpreting Burrows’s Delta: Geometric and Probabilistic Foundations”. In: Literary and Linguistic Computing 23.2 (2008), pp. 131–147. doi: 10.1093/llc/fqn003. 399 [2] D. Bogdanova and A. Lazaridou. “Cross-Language Authorship Attribution”. In: Pro- ceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). Reykjavik: Elra, 2014, pp. 2015–2020. url: http : / / www . lrec - conf . org / pro ceedings/lrec2014/pdf/145%5C%5FPaper.pdf. [3] L. Burnard, C. Odebrecht, and C. Schöch. “In Search of Comity: TEI for Distant Reading”. In: Journal of the Text Encoding Initiative 14 (2021). doi: 10.4000/jtei.3500. [4] J. Burrows. “All the Way Through: Testing for Authorship in Different Frequency Strata”. In: Literary and Linguistic Computing 22.1 (2007), pp. 27–47. doi: 10.1093/llc/fqi067. [5] J. Burrows. “The Englishing of Juvenal: Computational Stylistics and Translated Texts”. In: Style 36.4 (2002). url: https://www.jstor.org/stable/10.5325/style.36.4.677. [6] J. Byszuk. “Analysis in authorship attribution”. In: Survey of Methods in Computational Literary Studies. Ed. by C. Schöch, J. Dudar, and E. Fileva. Trier: Cls Infra, 2023. doi: 10.5281/zenodo.7782363. [7] J. Cohen. “A CoefÏcient of Agreement for Nominal Scales”. In: Educational and Psycho- logical Measurement 20.1 (1960), pp. 37–46. doi: 10.1177/001316446002000104. [8] M. Eder. “Does size matter? Authorship attribution, small samples, big problem”. In: Dig- ital Scholarship in the Humanities 30.2 (2015), pp. 167–182. doi: 10.1093/llc/fqt066. [9] M. Eder. “Short Samples in Authorship Attribution: A New Approach”. In: Book of Ab- stracts of the Digital Humanities Conference 2017). Montréal: Adho, 2017. url: https://dh 2017.adho.org/abstracts/341/341.pdf. [10] M. Eder, J. Rybicki, and M. Kestemont. “Stylometry with R: A Package for Computational Text Analysis”. In: The R Journal 8.1 (2016), p. 107. doi: 10.32614/rj-2016-007. [11] S. Evert, T. Proisl, F. Jannidis, I. Reger, S. Pielström, C. Schöch, and T. Vitt. “Understanding and explaining Delta measures for authorship attribution”. In: Digital Scholarship in the Humanities 32.2 (2017). doi: 10.1093/llc/fqx023. [12] R. Gorman. “Universal Dependencies and Author Attribution of Short Texts with Syntax Alone”. In: Digital Humanities Quarterly 16.2 (2022). url: https://www.digitalhumanitie s.org/dhq/vol/16/2/000606/000606.html. [13] H. v. Halteren, H. Baayen, F. Tweedie, M. Haverkort, and A. Neijt. “New Machine Learn- ing Methods Demonstrate the Existence of a Human Stylome”. In: Journal of Quantitative Linguistics 12.1 (2005), pp. 65–77. doi: 10.1080/09296170500055350. [14] D. I. Holmes. “The Evolution of Stylometry in Humanities Scholarship”. In: Literary and Linguistic Computing 13.3 (1998), pp. 111–117. doi: 10.1093/llc/13.3.111. [15] M. Kestemont, K. Luyckx, W. Daelemans, and T. Crombez. “Cross-Genre Authorship Verification Using Unmasking”. In: English Studies 93.3 (2012), pp. 340–356. doi: 10.1080 /0013838x.2012.668793. [16] C. Lee. “Do language combinations affect translators’ stylistic visibility in translated texts?” In: Digital Scholarship in the Humanities 33.3 (2018), pp. 592–603. doi: 10.1093/ll c/fqx056. 400 [17] G. Mikros and D. Boumparis. “Cross-linguistic Authorship Attribution and Author Pro- filing. Machine translation as a method for bridging the language gap”. In: Digital Schol- arship in the Humanities 39.3 (2024), pp. 954–967. doi: 10.1093/llc/fqae028. [18] A. Nini. A Theory of Linguistic Individuality for Authorship Analysis. Cambridge Univer- sity Press, 2023. doi: 10.1017/9781108974851. [19] J. Rybicki and M. Eder. “Deeper Delta across genres and languages: do we really need the most frequent words?” In: Literary and Linguistic Computing 26.3 (2011), pp. 315–321. doi: 10.1093/llc/fqr031. [20] J. Rybicki. “A Third Glance at a Stylometric Map of Native and Translated Literature in Polish”. In: Retracing the History of Literary Translation in Poland. New York: Routledge, 2021, pp. 247–261. doi: 10.4324/9780429325366-20. [21] J. Rybicki. “The great mystery of the (almost) invisible translator: Stylometry in transla- tion”. In: Studies in Corpus Linguistics. Ed. by M. P. Oakes and M. Ji. Vol. 51. Amsterdam: John Benjamins, 2012, pp. 231–248. doi: 10.1075/scl.51.09ryb. [22] J. Rybicki and M. Heydel. “The stylistics and stylometry of collaborative translation: Woolf’s Night and Day in Polish”. In: Literary and Linguistic Computing 28.4 (2013), pp. 708–717. doi: 10.1093/llc/fqt027. [23] C. Schöch, J. Dudar, and E. Fileva, eds. Survey of Methods in Computational Literary Stud- ies. Cls Infra, 2023. doi: 10.5281/zenodo.7892112. [24] C. Schöch, R. Patraș, D. Santos, and T. Erjavec. “Creating the European Literary Text Collection (ELTeC): Challenges and Perspectives”. In: Modern Languages Open (2021). doi: 10.3828/mlo.v0i0.364. [25] P. W. H. Smith and W. Aldridge. “Improving Authorship Attribution: Optimizing Bur- rows’ Delta Method”. In: Journal of Quantitative Linguistics 18.1 (2011), pp. 63–88. doi: 10.1080/09296174.2011.533591. [26] E. Stamatatos. “A survey of modern authorship attribution methods”. In: Journal of the American Society for Information Science and Technology 60.3 (2009), pp. 538–556. doi: 10.1002/asi.21001. [27] E. Stamatatos. “On the robustness of authorship attribution based on character n-gram features”. In: Journal of Law and Policy 21.2 (2013). url: https://brooklynworks.brookla w.edu/jlp/vol21/iss2/7/. 401 Figure 6: Ukrainian corpus, results for word form unigrams. Appendix Note that all result data can be inspected online in our interactive visualization at https://show cases.clsinfra.io/stylometry. A selection of findings is documented here for convenience. A. Ukrainian The Ukrainian corpus demonstrates lower performance compared to the Hungarian original corpora, achieving accuracies ranging from 50-80% when using unigram lemmas. Analyses based on word forms yields higher scores (Figure 6, 60-85%), particularly when the number of features is not higher than 400-500. An interesting observation is that variation in accuracy is a bit less across parameters in the Ukrainian original texts than for other corpora. When evaluating the accuracy of character-level analysis, high scores (70-85%) are observed using 3-5-gram characters with full novel sample size in word forms and lemmas settings (Figure 7). Translations of the Ukrainian corpus into other languages demonstrate significantly lower results than the original texts, with accuracies surpassing 50% only for full novel sizes in word- level analysis with unigram word forms and lemmas. Among the translations from Ukrainian, the English translation shows the highest results, reaching up to 75% for full novel sample size (Figure 8). This observation holds true for analyses based on full novel sample size using character n-grams on translations as well. In most cases, the performance of translation into English achieves 80% accuracy for 5-gram characters in both word forms and lemmas. Con- cerning the mean performance, the translation into Hungarian has the highest mean accuracy in most settings. 402 Figure 7: Ukrainian corpus, results for character 5-grams. B. French The French corpus shows a bit lower average performance compared to Ukrainian and Hungar- ian, with accuracy ranging from 40-70%, and in some cases, up to 80-95% for original texts based on unigram word forms and lemmas with N features up to 550 and a sample size starting from 8000 (Figure 9). Notably, the mean accuracy of original texts based on unigram word forms with sample size of 10000 and full novels is very high and reaches 83%, which approximates the performance of the Hungarian original corpus. When discussing character n-gram features, we observe high scores (up to 80-95%) exclu- sively for full novels, with n-gram sizes ranging from 3-5 (Figure 10). This observation holds true for both lemmas and plain word forms. Translations from French show a moderate decrease in performance compared to the original texts in both lemma- and word-form-based analyses, maintaining high accuracy only for the full novel sample size. Regarding the character n-grams, the scores of translations are almost identical to those of the original novels, with high performance achieved only for the full novel sample size. C. English The English corpus (Figure 11) yields the lowest performance among all corpora for original texts, with accuracy occasionally reaching 93% only on the full novel sample size in unigram lemma-based analysis at the word level. Apart from these instances, the corpus’s performance displays high variability, including the lowest results around 20% for both lemma-based and 403 Figure 8: English translations of Ukrainian texts, performance for word form unigram. plain word form-based analyses at the word level. Regarding the classification performance when using character n-grams, we observe a simi- lar trend as seen in the case of the French original corpus, where high accuracy is evident only for full novels. However, for the English corpus, the performance is slightly lower, reaching 86% in the best cases (Figure 12). Importantly, the Hungarian translation of the English corpus (Figure 13) approximates the performance of the original novels, and in some cases even outperforms the original novels. The highest accuracy of up to 96% (lemma-based analysis) and 90% (word forms-based) is achieved for full novel sample size at the word level. However, the performance decreases rapidly with an increase in the number of features and decrease in sample sizes. The translation into Ukrainian (Figure 14) also demonstrates high performance, up to 86% for full novels, although the average accuracy is lower compared to the original corpus and the translation into Hungarian (Figure 13). On the other hand, the translation into French achieves best performances at around 84% for lemma-based analysis, with most cases hovering around 20-30% (Figure 15). Regarding the analysis of translations at the character level, Hungarian translation outperforms the original corpus, in full novels sample size setting, while French translations exhibit lower accuracy even for full novels, reaching a score of 80% only for the 1500 and 2000 features setting for word forms and lemma-based analysis. 404 Figure 9: Results of French original novels, based on lemma unigrams. Figure 10: Results of French original novels, based on character 5-grams. 405 Figure 11: Results of the English original corpus, based on lemma unigrams. Figure 12: Results of the English original corpus, based on character 5-grams. 406 Figure 13: Performance of translations from English into Hungarian, lemma unigrams. Figure 14: Performance of translations from English into Ukrainian, lemma unigrams. 407 Figure 15: Performance of translations from English into French, lemma unigrams. 408