With this paper we aim to assess whether Early Modern book titles can be exploited to track two aspects of multilingualism in book publishing: publications featuring multiple languages and the distinction between editions of works in their original language and in translation. To this scope we leverage the manually annotated language information available in two book catalogs: the Collectio Academica Antiqua, recording publications of scholars of the Old University of Leuven (1425-1797) and a subset of the Eighteenth Century Collections Online, namely publications of Ancient Greek and Latin works. We evaluate three diferent approaches: we train a simple tf-idf based support vector classifier, we ifne-tune a multilingual transformer model (BERT) and we use a few-shot approach with a pre-trained sentence transformer model. In order to get a better understanding of the results, we make use of SHAP, a library for explaining the output of any machine Learning model. We conclude that while the few-shot prediction is not currently usable for this task, the tf-idf approach and BERT fine-tuning are comparable and both usable. BERT shows better results for the task of identifying translations and when generalizing across diferent datasets.
catalogues of Early Modern books are highly informative regarding the linguistic form of the book’s content: they may mention the translator, the language in which the text is printed, and the language from which the text is translated. A typical is example is provided by the title ‘A poetical translation of the works of Horace: with the original text, and critical notes collected from his best Latin and French commentators. By the Rev. d Mr. Philip Francis. In four volumes.’. This paper aims to answer three research questions: • RQ1: Do the titles recorded in catalogues of Early Modern books contain sufÏcient information to predict if they were multilingual or monolingual, and printed in the original language or translated? • RQ2: Which approach yields the best results: a simple tf-idf classifier, training a Large
Language Model, or adopting a few-shot approach? • RQ3: Given the heterogeneity of Early Modern publications, can models trained on one dataset yield satisfactory results on others? Does the diversification of training data improve the results on the datasets analyzed?
The work is structured as follows: in Sectio2n,we discuss the importance of multilingualism for Early Modern studies and the current possibilities for automatic language information extraction. Section3 introduces the two datasets used in this experimen3t. In Section 4, we describe the tasks (Section4.1) and models (Section4.2) employed. Finally, Section5s and 6 present the results and discuss the potential of this approach.
Early Modern Europe was marked by multilingualism. As Latin’s dominance as ltinhgeua
franca waned, vernacular languages began to emerge in scientific and literary production. This
shift influenced various practices in the printed press, drawing interest from linguistics, book
history, literary studies, and translation studie2s].[A key focus is the reception of classical
texts. During Humanism and the Renaissance, Ancient Greek and Latin gained prominence,
and, on the one hand, reading original works became central to humanistic educati2o2n, 1[
This study examines two datasets reflecting aspects of Early Modern multilingualism: the
diverse linguistic environment of the Low Countries and the evolving practice of printing
classical authors in England. The Low Countries, a multilingual hub due to their political situation
[
Our work utilizes long titles of Early Modern books to annotate their linguistic
characteristics. Book titles have been leveraged for metadata enrichment and large-scale analysis in
several studies: from the decline of the average length of modern British novel ti2tl5e]s, t[o
genre classification [
We aim to achieve two objectives: label a work as multilingual or monolingual and
identify whether it is printed in the original language or translated. These tasks, while related to
language identification [
The present study relies on two datasets: the CA5Afrom KU Leuven, and a version of Eighteenth Century Collections Online (ECC O6)manually enriched by a group of students. The CAA is curated by the Special Collections of KU Leuven Libraries and comprises books related to the Old University of Leuven (1425-1797), mostly of scholars that, at a certain point of their career, were afÏliated to this university. The CAA version used for this study (exported on 28 July 2023) comprises 3660 holdings, each of them described in MARC XML records. ECCO is a digital database assembled by Gale and stores the (OCRed) full text of a collection of 184,536 titles published in the eighteenth century. Within this collection, we identified the set of classical publications, as those authored by Ancient Greek or Latin authors living before the sixth cen4Enriching metadata based on book titles is also of interest to GLAM institutions, as demonstrated by a recent experiment on British Library datah,ttps://living-with-machines.github.io/genre-classification/01_BL_fiction_no n_fiction.html 5https://dial.uclouvain.be/digitization/en/digital-collection/old-academic-colle.ction 6https://www.gale.com/primary-sources/eighteenth-century-collections-on.line language pair # CAA language pair #ECCO lat|grc fre|lat lat|heb ita | lat dut|fre tury.7 The total number of classical editions amounts to 5237 rows. We refer to this dataset as ECCO-classics. These two datasets are chosen because of their meticulous language annotation, their partial chronological overlap, the shared presence of classics (several classical works were printed in Early Modern Flanders, and feature in the CA8Ab)u,t also clear diferences in terms of languages included and cultural and geographical background: these characteristics make them useful sets for comparing the capacities of generalization of the diferent approaches.
Both datasets have been manually annotated with respect to language. The MARC21 metadata schema includes a specific code for language annotation (041), further specified by several subfields, two of which are used in the CAA: ‘a’ indicating the language of the record, and ‘h’, indicating the original language. Hencem, ultilingual works are those including several ‘a’ codes, regardless of the presence of a ‘h’ codeM.onolingual works include only one ‘a’ code. Within the monolingual works, some also include an ‘h’ code, which is noted when the original is diferent from the language of the edition. We speak omfonolingual edition if no ‘h’ code is recorded, andmonolingual translation if it is recorded (and is consequently diferent from the ‘a’ code). In fact, monolingual translations are usually works translated into a single target language and published without the original text. We include only monolingual works for identifying translations, because for multilingual works it is hard to single out the function of the diferent target languages and be sure that one of them is used for translation.
An example of multilingual work in the CAA is represented by ‘Les dialogvesde Iean Loys Vives, traduits de Latin en François pour l’exercice des deux langues .../Les dialogues de Jean Loys Vives’, which is labeled as French and Latin. Tabl1e lists the most frequently attested language combinations for multilingual works in the CAA.
The ‘Histoire de Notre-Dame de Hale,par Juste Lipse ... Traduit du latin, & augmentée de plusieurs merveilles, venues en lumière depuis la mort de l’auteur’ is the title of a work labeled as monolingual translation. Tabl2eshows the most frequent pairs of original and target languages in the CAA. As both Table1 and 2 demonstrate, translation of the classical languages (Ancient Greek and Latin) plays a central role in the multilingualism of the academic production. 7More information on the identification of classical authors is provided i9n].[ 8We haven’t counted the exact number of classical works in the CAA, but, as an example, there are at least five editions of Homer, more than 10 editions of Cicero, etc. source-target languages # CAA source-target languages # ECCO lat-dut lat-fre fre-dutch lat-ger monolingual ed. monolingual transl.
CAA balanced CAA monolingual balanced CAA translation
ECCO-classics combined 3466
200 not used 550 7020 194
194 not used 1765 1877 3291 not used
350 1156 4513 175 not used
175 609 2507
The same schema was used to label the books in ECCO-classics, and the most frequently attested language-combinations are shown in Tabl1eand 2. An example of multilingual work is for instance ‘Phaedri Augusti liberti Fabularum aesopiarum libri quinque. Or, a correct latin edition of the Fables of Phaedrus: with a new literal English translation, and a copious parsingindex; Whereby young Beginners may easily and speedily attain the Knowledge of the Latin Tongue. By a gentleman of the University of Cambridge. For the Use of Schools’, while an example of monolingual translation is given by ‘The iliad of Homer. Translated by Alexander Pope, Esq.’.
4.1. Tasks As mentioned above, we aim at classifying the titles following two criteria, namely whether the edition is monolingual or multilingual (multilingual task henceforth), and whether, in case it is monolingual, it contains a work in its original language or in translation (monolingual translation task henceforth). We work with four combinations of the datasets, as listed in Table 3: the CAA, ECCO-classics, balanced CAA9, and ECCO and CAA combined. The datasets were split in 80-20 for training and test.
Multilingual and translated works are proportionately more frequent in the ECCO-classics dataset, because printing multilingual editions (i.e. the original text + a commentary or a translation in a modern language) was common practice for the circulation of classical works. When testing the diferent models, we evaluate the option of training on each dataset separately and 9We kept double the number of monolingual editions compared to monolingual translations in order to still achieve enough critical mass in the number of examples. testing on each dataset separately, or training with the union of the two and testing on the datasets separately and combining them. In this way, we want to assess both the capacity of the separate models to generalize, and whether more increasing and diversifying the training data improved the final results (RQ3).
In order to answer RQ 2, we have tested three diferent approaches: (1) a simple tf-idf model
with Linear Support Vector classification 3[
For the BERT approach, we fine-tuned the base model bert-base-multilingual-cased7][,
using the HuggingFace API and packages. We used the model hyperparameters set out in the
HuggingFace documentation for fine-tuning BERT for text classification [
For the few-shot experiment the aim was to provide a small number of examples which were as representative as possible with respect to each task. Separate sets were made for the multilingual and translation tasks. For the multilingual task, the final training set contains 5 examples from each of the languages or language pairs, and an equal number of monolingual and multilingual titles, from both the ECCO and CAA datasets, resulting in about 80 examples in the train set. The train set for the translation task was constructed in a similar way but with an even number of original language and translated works. These were then evaluated using the same test sets as above.
To perform the few-shot classification, the SetFit library was used. SetFit fine-tunes a
pretrained SentenceTransformers model29[] using a contrastive training approach.
SentenceTransformers is a form of Transformer-based Large Language Model which can be trained to
generate embedding representations at the sentence, paragraph, or document level (rather than
at the word-level as a regular LLM). These embeddings are then generally used for tasks such as
semantic textual similarity or semantic search. SetFit is a framework for few-shot fine-tuning
SentenceTransformers models. Setfit has shown to have performance comparable to a
LLMbased approach on tasks such as text classification, but with far fewer data and training time
[
For each set of results we recorded the accuracy, as well as the precision, recall and f1 scores
separately for each class. We include tables comparing the results of the two main tasks, plus
the full tables as an appendix. Moreover, we used the SHAP (SHapley Additive exPlanations)
library 2[
Below are shown some of the most relevant results, for the full set see the Appendix. Table 4 summarises the performance of the models trained on the ‘combined’ dataset and tested on both the individual and combined datasets. We report on the class-wise f-scores because the classes are very unevenly distributed, particularly for the CAA, and so the accuracy score is not a good indication of performance. Table5sand 6 give direct comparisons between the models on the multilingual and translation tasks, listing a diference simply by subtracting the score of the BERT model from the ML model (negative numbers mean the BERT model performed worse). Tables7 to 10 in the Appendix provide the details of precision, recall and f1 for the ML and Bert models, on each task, for each class.
As can be seen from Table4, both BERT and the ML method gave quite comparable results across both tasks and all datasets. The SetFit method performed noticeably worse in most cases, except when tested on the combined CAA and ECCO dataset. Overall, results can be considered satisfactory which leads to the conclusion that titles can be used to this scope (RQ1), however the task requires an extended set of labeled training data to be provided.
and the smaller ‘balanced’ set, as well as the other way around. For this task, BERT was able to generalize much better than the ML method when testing on a diferent dataset than the one on which it was trained.
The performance of the setfit method (see the Appendix, Table11) had a comparable pattern to the BERT models. It similarly had low recall and precision for the 0 class (multilingual works), but performed well with most tests on the translated works task, with just 40 examples of each class, across multiple languages.
In general, the ML and Bert models, when trained on examples froamcross datasets, are able to perform reasonably well - meaning that a training set made from a combined dataset of ECCO and the CAA gives satisfactory results. Both the ML method and the BERT fine-tuned model give very similar results.
Both models perform very well at identifying monolingual/multilingual works when trained and tested on ECCO. Models trained and tested on ECCO fared better in general, while still underperforming when applied to the CAA test dataset.
The results from models trained on one dataset and tested on the other are much worse. In particular, models trained on the CAA and tested on ECCO perform very badly at both recall and precision of the multilingual class. Again, there is little diference between the ML and BERT models, though the BERT model performs marginally better. The ‘CAA balanced’ model, trained on a sample of the CAA containing an equal number of monolingual/multilingual titles, balanced across the various target languages, did not perform significantly better than the CAA model, though it was marginally better and much quicker to train. However, the very small number of records might represent a limitation.
Since for the Setfit method we used a mix of examples coming from both datasets, RQ3 does not apply to this model.
To understand qualitatively what parts of the text caused the classification, we use SHAP explanations, and looked at a range of true positive, true negative, false positive and false negative predictions. Here, we focus on the BERT model trained on the CAA and tested on both CAA and ECCO for the prediction of multilingual texts (a particularly ‘difÏcult’ combination).
When the models wrongly label a title as monolingual when it is multilingual, in general, these phenomena seem to occur: • There is no trace of multilingualism in the title (e.g. the Latin title ‘Specimen doctrine traditae ab anno MDCXCI.usque ad annum MDCXCVI. inclusive.’ doesn’t contain any mention of parts in a diferent language). • Most of these titles, despite containing hints of multilingualism, are fully in Latin. The wrong prediction might be due to the fact that the CAA contains a lot of Latin monolingual titles, and hence Latin context is considered monolingual despite possible multilingual records. Figur2eshows a very long title in Latin with an explicit mention of a Train
ML combined combined combined
combined combined combined
Acc
caa caa caa caa ecco ecco ecco ecco combined combined combined combined caa_balanced caa_balanced caa_balanced caa ecco combined caa_balanced ecco caa combined caa_balanced combined caa ecco caa_balanced caa_balanced caa ecco 0.01 -0.04 0.00 0.01 0.00 0.21 0.17 0.09 -0.01 0.01 0.03 0.01 -0.04 -0.02 -0.13
translated bit (‘cum latina interpretatione’) being entirely assigned to monolingual (blue) by the model.
Another recurrent trend in both false and true prediction is the role of Greek: the word ‘Greek’ (orGracae, in Graecam linguam) is always used as a predictor of multilingualism, even when the work is monolingual (either in the original language or in translation). Fi4gure and 3 show an example of two monolingual works whose titles contain the word ‘Greek’. In both cases, the word Greek heavily impacts the ‘multilingual’ component, despite the fact that the output is diferent for the two predictions. This might be due to the fact that in the CAA Ancient Greek texts usually come with translations/notes in a modern language. Text in the Greek alphabet also seems to be used to make identifications of multilingual texts. This raises the issue of the dependency of the models on these specific dataset features. Furthermore, the model in some cases uses the text which we would read as making it likely to be multilingual as an output pointing to monolingual. For example things like ‘original subjoined’ or ‘notes at the end’, ‘on the opposite page’... One example of this can be seen in Figure5. This is because these phrases are not found in the CAA titles for multilingual works. The ‘combined’ model doesn’t have this bias, in this case, words relating to notes or annotations contribute to a positive prediction of a work as multilingual, as one might expect.
Words like ‘translated’, or ‘lexicon’ across languages increase the output of the model in identifying multilingual works, which is close to what we would expect.
Overall, these experiments suggest it is a difÏcult problem to solve using machine learning methods. In particular, the approaches do not seem to generalise well, even using multilingual LLMs which we hoped might mean that diferent styles of title would be recognised if they were in some way semantically similar. This is perhaps because the way that multilingual and translated works are signified in a title is varied and changes over time and across languages. Despite these reservations, when trained on examples across both datasets, the performance of both traditional machine learning and LLM methods was at a level which we deem usable in real-world applications.
The multilingual fine-tuned BERT has some advantages over traditional ML approaches in identifying translated works but performs worse when distinguishing multilingual works. This seems to be because the signifiers for translated works are more descriptive and straightforward (e.g. ‘translated from’ or ‘made English by’). The multilingual approach means that these kinds of phrases tend be be picked up by the model in diferent languages.
The few-shot method using SetFit shows some promise in a number of tasks, but does not, from our experiments, seem to be a ‘silver bullet’ for low-resource metadata enrichment of this kind. However, perhaps with a very well thought-out and diverse set of examples, it may be possible to build a model which can be trained and used for inferences on real-world data. An ideal real-world scenario for metadata enrichment may involve collecting a small number of examples from a specific dataset or collection, fine-tuning a bespoke but small model, and applying it only to that collection. However, as of yet, from our experiments, it does not seem that the multilingual capabilities of SetFit or SentenceTransformers are enough to get highquality results on this task without at least some annotation of the target dataset.
Automatically enriched metadata has significant value to heritage collections catalogue data, potentially helping to increase the accuracy and findability of records. If the purpose is to get enriched metadata, our experiments show some promise and could potentially be operationalised in the future. In fact, traditional ML methods may be enough in many cases, particularly for identifying multilingual works, and have big advantages in terms of ease of use and use of resources. In some cases, methods such as keyword search or regular expressions might also provide acceptable results, though when using multilingual datasets, machine learning methods should have an advantage.
Furthermore, we suggest that certain evaluation metrics are more important than others, particularly with library catalogue data, which is likely to be very unevenly distributed with regards to language and classes. This is of course dependant on the particular task and usecase. If the purpose is to improve catalogue metadata for example, the recall of the multilingual or translated classes may be particularly important, as it may be better to find additional false positives which can then be checked manually afterwards, rather than aiming for precision but missing some relevant works. If the information is not necessarily intended to be ‘fed back’ to a catalogue but used for bibliographic data science at scale, it may be more important to focus on the overall f-scores to get a broad, albeit imperfect, accuracy.
We want to express our gratitute to the STUDIUM.AI team, particular to Violet Soen, whose eforts enabled this research. In addition, we would like to thank the KU Leuven Libraries staf, in particular the metadata and digitization services for sharing the CAA metadata and the relative documentation. Finally, we would like to thank the Computational History group of Helsinki, for providing the framework and infrastructure for annotating the ECCO training data. N. Constantinidou. “Printers of the Greek Classics and Market Distribution in the Sixteenth Century: The Case of France and the Low Countries”. InS:pecialist Markets in the Early Modern Book World 40 (2015). Ed. by R. Kirwan and S. Mullins, pp. 273–93.
A.3. Translation Task: BERT r (0) p (0) r (1) p (1) A.2. Multilingual/Monolingual Task: TFIDF/SVM r (0) p (0) r (1) p (1)