In this work we introduce the REVERINO dataset, a collection of 4533 pairs of Latin regesta with their respective full text medieval pontifical document extracted from two collections, Epistolae saeculi XIII e regestis pontificum Romanorum selectae. (1216-1268) and Les Registres de Gregoire IX (1227/41). We describe the pipeline used to extract the text from the images of the printed pages and we make high level analysis of the corpus. After developing REVERINO we use it as a benchmark to test the ability of Large Language Models (LLMs) to generate the regestum of a given Latin text. We test 3 LLMs among the best performing ones, GPT-4o, Llama 3.1 70b and Llama 3.1 405b and find that GPT-4o is the best at generating text in Latin. Interestingly, we also find that for Llama models it can be beneficial to first generate a text in English and then translate it in Latin to write better regesta.
ITSERR [
A regestum, the Latin word for list, enumeration, specification 1, is a summary of a document made for the use of stakeholders and scholars, making the document content readily available without the need to consult it in its entirety. Each regestum contains some essential information, namely, 1) the name of the author (i.e. the Pope); 2) the name of the recipient; 3) an abstract of the content (with the object and the operative verb); 4) the date (calculated from the year of pontificate) and 5) the place of production of the document. A regestum always has a reference full text document, of which it is the “summary” and both come together with an apparatus, a formal text indicating the collection and the manuscript where the regestum is found.
While the three components, regestum, full text and apparatus, are conceptually close, collecting
them together can be challenging. Indeed, there are three main issues when trying to retrieve both a
regestum and its full text from a collection or creating a new one: a) the regestum and the corresponding
document are often not collected in the same volume (as in the Potthast collection [
In the Natural Language Processing (NLP) literature, automatic text summarization is the task of rewriting the content of a text passage into a shorter form while retaining the relevant information without involving a writer in the process. Regesta fit well in this framework since they are summaries of longer texts meant to expose specific information in an easy to consult form. Therefore, REVERINO is well suited to be both an easy to inspect dataset of Latin regesta, as well as a training and testing benchmark for Latin text summarization.
There is an extensive number of printed collections of regesta in Latin, edited from several scholars during the 19th and 20th century, however only a few examples that are digitally available, generally as sets of high quality digital images, and only very rarely with a full text, machine readable version (and often performed with old or bad quality OCRs). One of the most relevant fields in which regesta have been produced is the corpus of the pontifical acts and letters issued by the popes and the papal chancery during the Middle Ages. This corpus guarantees the presence of large collections of regesta and extended texts that were already edited and published in printed versions during the 19th and 20th centuries, such as Jafé and Potthast’s Regesta Pontificum Romanorum ; the collection published by the Bibliothèque des Écoles françaises d’Athènes et de Rome (BEF); the editorial series of the Monumenta Germaniae Historica (MGH).
Regesta collections are only available as images of full pages, this poses a first obstacle to the creation of a large scale corpus of these documents. Each manuscript has diferent pagination, layout, writing font, image quality, format, etc., and thus requires a custom pipeline for the extraction of the text into a machine readable format. Nevertheless, manuscripts from one collection undergo a similar digitization procedure and therefore can be processed together through a single pipeline, to extract the content of all the documents.
Given the need for a custom approach for each corpus, we choose to limit our collection to two sets
of printed collections, specifically, we identify 2 main sources:
1. MGH: Epistolae saeculi XIII e regestis pontificum Romanorum selectae. (1216-1268) [
While conceptually similar these two collections are formally diferent: MGH is written in a single column format while Auvray in two columns, the first has apparati visually separated from the original document, while for the second they are part of the regestum. The diferent collections have thus several smaller visual diferences linked to the layout and the font used.
Finally, from a qualitative perspective they collect and summarize the documents related to two diferent medieval popes: Gregory IX and Honorius III. In particular, Auvray collects only the documents related to pope Gregory IX, and MGH collects the documents issued by both Gregory IX and Honorius III. These collections were chosen as starting corpus because they allow for the collection of diferent types of regesta. Indeed, although the creation of a regestum is based on specific rules shared in the research domain, diferent scholars inevitably produce diferent regesta from the same document. It is therefore important to consider regesta produced by diferent scholars in diferent periods.
The pipeline leading from a collection of images of printed pages to the REVERINO corpus is composed of 4 steps: Annotation, Training, Extraction and Post-processing.
Annotation We manually annotate a selected set of pages from each collection of regesta to use as a
training dataset, this is done on a local instance of the eScriptorium platform [
The models in eScriptorium ingest annotations with two kinds of information: 1. areas isolating the parts of a page that contain text, and 2. lines identifying the text of a line and its position in the page. Thus, each page is annotated in two steps, first the relevant areas are circled and then each line is colored.
Training To adapt models to the outline of a manuscript, we start from a working segmentation
model provided by eScriptorium, catmus print large [
Extraction Once the model has been trained, we use it to process all the pages in each collection, obtaining text lines that the model is able to identify along with their position on the page, and thus giving us a clean continuous stream of text spanning the full document.
Post-Processing The last step consists in using a series of heuristics based on the content and the position information in the page of each text line extracted, to separate each regestum from the longer text it summarizes and from the apparatus.
From a quantitative perspective, MGH is composed of a total of 2283 regesta and full text pairs and Auvray of 3983. However, for Auvray, several of the full texts extracted were only short passages, often quotations or incipits that don’t contain the information needed to generate a regestum, therefore we drop them. After this cleaning there 60 are 2250 regesta left in Auvray. MGH
While MGH and Auvray are similar, – Auvray they are both collections of regesta –, they 40 show several diferences: they collect documents written by diferent popes and 2 20 they are edited by diferent scholars, and ten on top of this they are diferent as printed on publications. Indeed, due to the layout opm 0 opfostehde otwfsoincgolleleccotilounmsn, apsagMeGsHwhiislecoAmu-- -tESCN 20 vray is composed of two columns pages.
Also, the quality of the second dataset is generally lower, due to minor errors in 40 OCR quality, few characters and numbers are wrongly transcribed by our custom 60 model. Therefore we keep two separate 40 20 0 20 40 splits of the dataset. t-SNE Component 1
To provide a qualitative understanding
of the diference, we use t-SNE [
Text summarization is a long standing task in Natural Language processing [
To evaluate text summarization the most used metrics are based on text overlap, the most spread
one is Rouge [
An alternative metric, Bleu [
To understand how well LLMs can summarize text in Latin, we measure the performance of three
powerful LLMs, Llama 3.1 70b, Llama 3.1 405b and GPT-4o. The first two are openly available language
models released from Meta [
Each setting, format and backtranslate, is identified by the prompt we provide the LLM to make it generate the regesta, Table 1 shows the prompt used for each setting as well as a Shared Prompt, added next to the setting specific one, where we request the model to at least add the key elements of a regestum, as mentioned in Section 1.1: the author, the recipient, the summary, the date and the place. Finally, to facilitate the model during generation we add two full texts with their respective regesta.
We let the models generate up to 8048 tokens and we use greedy decoding, i.e. we pick the most likely word and avoid any form of sampling during inference since the regestum is meant to be a short and detailed summary, we will ablate diferent sampling techniques in future works.
To evaluate model performance we use both a quantitative and a qualitative analysis: first, the quantitative analysis is based on Rouge and Bleu measuring the similarity between synthetic regesta generated by an LLM and the original ones from the REVERINO dataset summarizing the same text, second, the qualitative analysis is based on inspecting in detail a subset of the machine generated regesta to understand which of the 5 key properties they lack.
Table 2 shows Rouge and Bleu achieved by the three models we test: Llama 3.1 70b, Llama 3.1 405b and GPT-4o. The first finding is that no model can generate regesta proficiently, this appears from the fact that none of those we tested can achieve a Rouge higher than 0.40 and a Bleu above 0.15. We can also see that GPT-4o strongly outperforms both Llama models. The highest Rouge-1 is 0.38, achieved by GPT-4o on MGH, which also has the higher Rouge-1 on Auvray, although at a significantly lower value, 0.28, which we attribute to the lower quality of the Auvray dataset.
The wide gap between Rouge-1 and Rouge-2 shows how generally LLMs output texts that share the general context, higher value of 1-grams overlap, but they find it harder to generate actually similar texts, lower value for 2-grams overlap.
The two versions of Llama, Llama 3.1 70b and Llama 3.1 405b, show a small performance gap, indicating that it is not useful to use the larger and more costly Llama 3.1 405b, comparing format and backtranslate the first is the best setting for the GPT-4o model, while the opposite is true for Llama models, which show higher performance when asked to translate in English before writing in Latin. We have performed a limited prompt tuning that resulted in the choice of the format and backtranslate settings and we will further explore this aspect in future works.
Finally, we notice how in rare cases, between 2 and 6 for MGH and between 2 and 3 for Auvray, the guardrails preventing GPT-4o to answer questions involving violence make it refuse to generate a regestum, thus the lower values in the N. Samples column, while Llama models do not incur in this issue.
For a more in-depth analysis of the model abilities as seen in the quantitative results, we identify a corpus of 20 pairs of extended regesta (10 from MGH and 10 from Auvray) to conduct a qualitative analysis of the results. The regesta are humanly checked by a domain expert in their diferent versions produced by GPT-4o, Llama 3.1 70b and Llama 3.1 405b, for a total of 120 artificially created regesta reviewed and compared with their original versions. Through a manual inspection it is possible to identify the reasons for the results presented in Table 2 and possibly take corrective actions in the future.
In agreement with Table 2, also from a qualitative analysis GPT-4o performs better than both Llama models. This mainly concerns the generation of Latin text and thus the summarization of the document content in the regestum form. Indeed, Llama models encounter more problems in text generation, as shown by the fact that the best results are obtained when the summarization is created in English and then translated into Latin (i.e., backtranslate). More broadly, it can also be observed that the systems perform better in the case of MGH, but as already mentioned, this can be traced back to how the dataset is constructed.
Finally, the qualitative analysis reveals the most critical failures in automatic summarization (i.e., automatic regesta creation). One of the problems identified concerns the recognition of documents’ author, namely the Pope. Indeed, in the case of MGH, which collects documents from multiple popes, both systems show dificulties in recognizing the correct author. In fact, GPT-4o correctly recognizes the Pope in 11 cases out on 20 taken into account. Another critical element concerns dating, which in these medieval texts is based on the year of pontificate and not on the modern dating system. Although Llama 3.1 70b, Llama 3.1 405b and GPT-4o recognize and identify the dating system used in the extended text of the medieval document and show that they have the tools to accomplish the conversion, both systems show dificulty in providing correct dating (either because they do not ofer it or because they miscalculate it). Out of 20 manually inspected records generated by GPT-4o, only 3 cases correctly report the date of the document. The result is improved in the case of the recognition of the document recipient (often reported in the first line of the extended text), which in the same sample is recognized correctly by GPT-4o in 15 out of 20 cases.
Finally, it should be noted that in a few cases, since our prompt requests the data topica (the place) to be extracted, the systems correctly extract it even when the original regestum does not report this information. Thus lead to a lower score in the table, but to a qualitatively better result in regesta generation.
In this work we have developed the REVERINO dataset, a dataset of 4533 pairs of regesta with their respective full text (and apparatus). The texts in this dataset come from two collections of regesta, Epistolae saeculi XIII e regestis pontificum Romanorum selectae. (1216-1268) (MGH) and Les Registres de Gregoire IX (1227/41) (Auvray), to collect the dataset we have followed a pipeline composed of 4 steps: annotation, training, extraction and post-processing.
Despite containing more than 4000 samples, REVERINO is too small to be used as a training set for a language model that automatically generates regesta, however it can be used as a benchmark to test the ability of existing LLMs to do summarization in Latin and thus to develop better tools and methodologies in the future.
We have tested 3 LLMs among the best performing ones, our general finding is that these models can’t be used as-is to summarize texts in Latin. More precisely, we find that GPT-4o is the best and that models from the Llama family are less able to generate text in Latin. Interestingly, for both Llama 3.1 70b and Llama 3.1 405b we find that initially translating to English is an efective technique to generate better regesta.
We also want to underline the limitations of our work, the samples in our dataset are automatically extracted, and therefore a share of them contain transcription errors and imperfections. However, we use the dataset only as a benchmark and it is still too small to serve as a training dataset for a text-summarization model.
Despite these limitations, we hope that REVERINO will foster future works on the development of Language Models proficient in Latin and we will continue improving on by extending it to grow larger than 10k samples, and by using it to train a custom Language Model specifically tailored to the generation of regesta in Latin. This work was supported by project "Italian Strengthening of ESFRI RI RESILIENCE" (ITSERR) funded by the European Union under the NextGenerationEU funding scheme (CUP:B53C22001770006).