Until the mid-15th century, illuminated manuscripts were the main vehicle for the circulation and preservation of knowledge among the Western upper classes, surviving even beyond Johannes Gutenberg's introduction of movable type printing. This typology of cultural artifacts, difficult to access due to their materials' fragility, has found new life thanks to digitization which has made available to scholars and the common public many little-known works, rarely released from deposits.
Each complete codex comprises hundreds of pages, which do not only contain text, but are also richly decorated with miniatures that share subjects and iconography with other artistic expressions like paintings and sculptures. Although illuminations occupy only a limited number of sheets, they are also the components that attract the most non-academic audience, making the often-incomprehensible written content partly accessible.
To enhance the thousands of volumes made available online we are developing an immersive visualization interface that allows users to move easily between the digitized pages, searching for figurative miniatures and capable of relating reproductions of different works with the same subject. To make this possible, a deep neural network system is being trained on a specially constructed dataset containing reproductions of Medieval and Renaissance volumes together with artworks. Our objective is to obtain a model that can recognize elements in the page layout and identify handwritten sheets within heterogeneous collections of items. Given that deep neural networks require massive datasets with high-quality annotations that are labor-intensive, we implemented an Interactive Machine Learning approach to rapidly compile increasingly larger databases, in which a domain expert and an AI expert cooperate to train a model with domain-specific knowledge using transfer learning [1,2].
Section 2 outlines the state of the art while Section 3 proceeds to present the dataset by describing its composition and image classification system before exposing the technique adopted for training the model. Finally, in Section 4 we discuss conclusions.
Most datasets concerning digitized handwritten and early printed texts are composed of materials merged according to a shared language, creation period and resource-distributing institution [3]. Often, they are small if compared with newspapers [4] and museum collections [5], containing frequently only a subset of pages from the volumes or few complete codices [6], such as the HBA dataset [7] with images that originate from 11 books of the French digital library Gallica. In several cases, the dataset needs to be augmented with new items and an example is [8] which uses generative adversarial networks (GANs) to create synthetic pages to train the model.
The utilization of machine learning and specifically neural networks to facilitate the study of historical documents is an established field of research [9]. The main investigated task has been text recognition [10], but the focus on graphic elements such as layout analysis [8], drop caps for letter extraction [11], figure gestures classification through template-based detectors [12] and miniature retrieval [13] have gradually shifting attention from text to artistic aspects. Also attempts with bounding boxes have been made for illumination detection [14] and iconographic recognition [15] with good results.
In the historical-artistic field, among the several neural networks architectures ResNet [16] was used individually [17] or within more complex pipelines [18], mostly, because it performs well on labeling and detection tasks and is easily trainable, given that its implementation of residual functions allows better error propagation. Often, transfer learning is used to train ResNet architectures given small datasets [18,19].
For an in-depth review of "human-in-the-loop" approaches refer to [20]. Specifically, [1,2] discuss Interactive Machine Learning. Label Studio is a flexible labeling tool that implements active learning [21]. As well, ilastik [22] or Cellpose [23] are machine learning tools that offer Interactive Machine Learning capabilities.
The collective interest in illuminated manuscripts is still lower than the attraction exerted by works of art, almost entirely coming from relatively few expert researchers due to language barriers and the complexity of interpreting handwritten sheets. In addition to the textual part, which simultaneously serves as both the main vehicle and greatest obstacle to content communication, the miniatures that captivate the viewer can be used to build accessible experiential paths based on a visual language understood regardless of one's country of origin or historical knowledge. The first step in creating an interface that will allow the user to explore the manuscript pages through its more intuitive graphical contents was to automate their recognition.
To develop the dataset required for deep neural network transfer learning, we implemented an iterative incremental training cycle, composed of four main phases. First, a domain expert annotates or corrects the labels of a smaller dataset. Second, an AI expert uses transfer learning and hyperparameter optimization to train a deep neural network. Third, the model predicts the larger dataset labels. Thus, the cycle restarts from the first phase with the domain expert correcting the annotations predicted for the larger dataset. Labels were created using Label Studio [21].
For training our deep neural network to identify the different visual components inside the pages and distinguish any miniatures present also from ornaments or illuminated initials, we have thus formed an original training dataset. It contains images from digitized manuscript volumes, mainly dating back to the Medieval and Renaissance periods, together with reproductions of incunabula and later printed copies but also works of art and objects normally present in digital museum catalogs often featuring figurative subjects with a sacred theme as decoration.
Files come exclusively from institutional databases like the US Library of Congress, The Metropolitan Museum of Art of New York and the J. Paul Getty Museum Collection of Los Angeles but also from European ones such as the Staatsbibliothek of Berlin and the Koninklijke Bibliotheek of the Netherlands. We preferred images from public domain or released under creative common license CC-BY. Furthermore, by using different sources we tried to obtain a certain variety of scanning and shooting conditions as well as the quality of reproductions, selecting both overall views and detail shots.
Each element was classified using an alphanumeric code that allowed to quickly trace the origin and provide information about the depicted subject, the century of production, and the specific author or, if unknown, at least its geographical area. As the number of elements increased, a prefix was also added to each series to distinguish the type of physical object digitized, the associated category and subclass. An example of the result thus obtained is "P.01_S(An)_14BMGetty" where "P.01" indicates the category and the subclass it belongs to, "S(An)" identifies the subject as Saint Anne while "14" is the century of creation, "BM" the author's initials and "Getty" is the provenance institution. At the same time, this system has kept low the risk of inserting multiple copies of a work, especially in cases such as engravings, and associating images with the same themes already in the early stage of research to possibly increase the examples of a specific subject when poorly represented.
In total, the dataset consists of 45.798 items divided into two macro-categories, volumes (P) and art (A), and into ten subgroups: manuscript sheets (P.01), printed pages (P.02), paintings (A.01), engravings (A.02), drawings (A.03), sculptures (A.04), stained glass windows (A.05), tapestries (A.06), art prints (A.07) and other objects (A.08). Since the project is focused on manuscripts, the number of images dedicated to these is higher than all other typologies and constitute 68.25% of the total, chosen to cover a wide variety of layouts and styles. They chiefly come from books of hours, psalters, missals, breviaries, and bibles selected primarily for the presence of illustrations. The miniatures in the chosen manuscripts are both full page and inserted in the text.
Always privileging the figurative element, the sheets without illustrations have been classified according to the presence of decorations and, only in the absence of these, on text or musical scores considering the dominant component. Four summary partitions were thus obtained in the main category, first useful to balance the composition of the dataset, avoiding the insertion of too many text pages compared to the other elements, and afterwards for the subdivision into three training groups progressively larger in size of 400, 4.000 and 41.398.
The smallest group consists of 330 pages and 70 artworks, a proportion that remains constant in the second and undergoes slight variations in the third set where the category "Pages" contains a total of 34.447 items to which are added 6.951 images of the other one. In the preprocessing steps we uniformed the dataset through resizing to 1280 × 1280 pixels maintaining proportions.
To limit human effort in the annotation phase, we manually labeled only the first group to report the presence or co-presence in the pages of figurative miniatures (Fig), decorations (Deco), text (Text), music scores (Mus) and illuminated initials (Let) with a minimum of one and a maximum of four tags for a total of 818. This set, split into 70% training and 30% test, was used to initially train our deep neural network to distinguish between pages and art, manuscript and printed pages as to detect the layout elements.
We use a transfer learning approach based on a ResNet architecture [16] pretrained on ImageNet dataset [24]. We planned to test ResNet-50 and 101 as well as ResNet-18. Since we had a relatively small dataset, we started with ResNet-18 because models with fewer parameters are less likely to be in an over-parametrization regime. We obtained unexpectedly good results for both the training and the test metrics. In future, we will test the larger networks.
In the training phase, we augmented the dataset by transforming the images passed by the dataloader through the following sequence of transformations. First, the dataloader resized the images to 256×256, then it randomly cropped and resized the pictures to 256 × 256, and finally, it normalized the images according to ResNet's suggested parameters. Initially a distinct group of deep neural networks was trained on the first group without the category "Art" and this allowed us to test that art images do not negatively impact the prediction accuracy.
We considered two gradient descent training methods: Stochastic Gradient Descent (SGD) and Adam. For SGD we used an exponential learning rate scheduler with a starting learning rate 0.001, momentum 0.9, learning rate scheduler step size 7, and gamma 0.1. For Adam, we settled on a learning rate 0.001.
We also compared two types of transfer learning: fine-tuning on all weights or keeping the features in the training phase while fine-tuning only the last layer.
Figure 1 shows the accuracy level achieved: in almost all cases, the fine-tuning of all weights and SGD gave the best results, even if the difference is not very large. The accuracy rate is almost always above 90%, particularly high in recognizing pages from art and detecting text or music, while the lowest value was recorded for illuminated initials, mostly caused by the inherent ambiguity between simple ornamental letters and historiated initials.
Considering the promising results, the models with the highest accuracy level for each target were used to automatically predict the labels for the 4000 images of the second set. Then, the domain expert carefully checked to identify specific model biases and corrected the assigned labels. These models demonstrated their ability to recognize with high precision the presence of decorations and illustrations on the pages, especially when there are borders and the scene occupies a large portion of space. They are however less precise in distinguishing between the miniatures inserted in the text and the illuminated initials, often confusing the two elements and labeling some of the former as "Let". They also incorrectly classified blank pages as "Text" and adopted the same tag for some sculptures together with the right "Fig" . Only in ten cases the prediction was completely wrong: out of 4.000 images only 1.549 required manual intervention, saving several hours of work to reach an overall result of 7.205 annotations. Comparing the automatically applied tags with the correct ones, the most significant decrease was observed for "Text", which went from 3.174 to 2.854, followed by "Let" reduced from 966 to 936. On the other hand, the "Fig" and "Deco" classes increased, going from 1.495 to 1.765 and from 1.214 to 1.292 respectively.
The accuracy of the predictions was overall considered satisfactory with a rate of 0.98% for the recognition of figurative miniatures, 0.88% for text, 0.86% for decorations and 0.74% for illuminated initials. However, a particular criticality has been observed with musical scores printed after the 15th century which are not recognized as such, significantly impacting the number of labels assigned by the system, only 251 out of the exact 358.
The initial recognition tests on the layout elements gave us very encouraging results as accuracy levels higher than 80% are not often obtained with such small datasets. Our approach allows for a gradual and efficient labeling process and in this paper we stop at the end of the first step of the incremental learning cycle. In future work, we plan to progressively grow the annotated items' number to verify if similar results will be maintained also for the third larger group.
Once this phase is completed, a further step will be taken to improve the dataset with more complex label tasks such as instance semantic segmentation. Specifically, we will gradually move from image classification to the detection of specific components using the activation zones as a guide to automatically place boxes and then perform the semantic segmentation of any miniatures present.
The masks thus obtained will be used to enrich an interactive and immersive web3D system based on an open-source framework [25], that will be investigated in the upcoming stages of the research. This will give the user the possibility to inspect and query large amounts of pages in a 3D space, juxtapose similar images, and comparing them with art reproductions.
In doing so, we intend to demonstrate how, through the use of new technologies, figurative language can be used to structure narratives capable of making artifacts, originally prerogative of a few, available to all by freeing a delicate cultural property from its physical and conservative limits and exploiting iconography as an interpretative system that can be understood independently of the spoken language.