"MIDRASH: Migrations of Textual and Scribal Traditions via Large-Scale Computational Analysis of Medieval Manuscripts in Hebrew Script, " supported by an ERC Synergy grant, is an international effort to develop a revolutionary, computational approach to manuscript studies. Among other aspects, it combines traditional, digital, and computational paleographic methods to refine and potentially rewrite our understanding of Hebrew scripts, particularly their geographical variation in scribal practices [8]. The project is led by Daniel Stökl Ben Ezra (École pratique des hautes études [EPHE], Paris Sciences-Lettres University), Judith Olszowy-Schlanger (École pratique des hautes études, Paris Sciences-Lettres University and Oxford University), Nachum Dershowitz (Tel Aviv University [TAU]), and Avi Shmidman (Bar-Ilan University [BIU]), with the participation of the National Library of Israel (NLI) and Haifa University.
The main goal of the project is to develop new methodologies for studying medieval Hebrew manuscripts. In addition to employing handwriting text recognition to extract text from images, We will analyze these manuscripts using computational tools from paleographic, codicological, linguistic, and literary perspectives. This analysis will contribute to the understanding of the manuscripts' materiality, textuality, transmission, and the historical and intellectual context of their creation and readership. By combining traditional philology with machine learning, computer vision, and computational linguistics, we will process large amounts of textual and paleographical data that traditional philology cannot handle. See Figure 1. The principal aims include:
1. Develop optical character recognition (OCR) algorithms to convert manuscript images into searchable text. 2. Implement text mining algorithms to compare a large corpus of texts and identify quotations, paraphrases, borrowings, allusions, and other intertextual relationships. 3. Train machine learning models to perform handwriting analysis and predict each manuscript's geographical and temporal origins. 4. Design natural language processing (NLP) algorithms to extract and analyze linguistic features for improved textual searches and historical context placement. 5. Integrate traditional and computational methodologies for paleographic, philological, and textual analysis.
Accessing manuscripts' textual and non-textual information is valuable only if we can understand the texts in their specific context of place and time. Out of all the medieval Hebrew manuscripts, only about 3,500 are dated and have colophons (scribes' notes) or other identifying marks. Paleography (the study of handwriting) and codicology (the study of the physical aspects of books) are the primary methods used to determine the provenance of manuscripts. SfarData (sfardata.nli.org.il) is the only existing database that focuses primarily on codicology for dated medieval Hebrew manuscripts. However, reliance on paleography is essential for a project, like ours, studying document images. The MiDRASH traditional paleography team aims to make precise regional and chronological classifications. They scan well-defined manuscript samples to find correlations between their textual features and scripts. We use HebrewPal (hebrewpalaeography.com), an ongoing effort to build a comprehensive database of Hebrew paleography. Processing this data involves synergetic collaboration between the traditional and computational paleography teams.
As the computational paleography team, we are currently working on solving the problem of finding subgroups among the Ashkenazi square script documents. Paleographers describe medieval Hebrew manuscripts according to their script mode (square, cursive, and semicursive) and geographical type. The six geographical types are Oriental (Egypt, Palestine, Syria, Lebanon, Iraq, Iran, Uzbekistan, and Bukhara, Eastern Turkey), Sephardic (the Iberian Peninsula, Provence and Languedoc, North Africa, and Sicily), Italian, Ashkenazi (France and England, the Holy Roman Empire, Central and Eastern Europe), Byzantine (Greece, the Balkans, Western Asia Minor, and regions surrounding the Black Sea), and Yemenite (Figure 2). This level of codicological classification for Hebrew manuscripts and initial automatic dating has already been successfully performed using computational means [9,10,5]. Within certain script type-modes, there are distinct subclusters. Only in rare cases have these subclusters been relatively well-studied. This is, for example, the case of the Ashkenazi square script, which has been well-studied and clustered [1,4,3,6]. However, even the most experienced paleographers are most familiar with the manuscripts they work with frequently, and no human memory can retain thousands of script examples. Moreover, the variations within some script type-modes are very subtle. Therefore, we are working to develop computational methods to identify clusters, and subclusters within different script types that have yet to be discovered (or those that are not discoverable) by paleographers. This work will contribute to identifying the place of copying for manuscripts of unknown provenance, more exactly than the current results [2].
The Ktiv project at the National Library of Israel has led a significant digitization campaign of Hebrew-character manuscripts from collections worldwide. It has accumulated more than 80% of extant manuscripts, making tens of thousands of manuscripts accessible via a unified catalog. The Friedberg Genizah Project has contributed images and metadata of approximately 350,000 fragments from medieval book and document depositories, known as genizot (geniza or genizah in the singular. This digital corpus serves as the source material for our project. It includes relatively well-preserved scrolls and codices, as well as hundreds of thousands of fragments. For the clustering task, we used high-resolution pages from well-preserved manuscripts.
We are using a dataset of images built for us by Judith Olszowy-Schlanger specifically for the Ashkenazi-square clustering problem, a style for which she is the leading expert. She challenged us (as part of the MiDRASH project) with the task of automatically clustering within this specific type-mode, and potentially revealing additional subclusters as yet uncategorized by traditional Hebrew paleography. This "ASC" dataset, publicly available online at github.com/TAU-CH/midrash_ASC_dataset, contains 206 images, each depicting part of a page from 59 manuscripts, with approximately four pages from each manuscript (Table 1). It also includes an annotation file for the bounding boxes of the main text regions and text lines. Samples are unlabeled, but it is known that 17 manuscripts are from Germany and 11 are from France, while the origins of the remaining 31 manuscripts are currently unknown (Figure 3). All the manuscripts are written in Ashkenazi square script, and we aim to discover potential subclusters within these manuscripts based on their script types, which have slight variations.
Our preliminary work focused on clustering medieval manuscripts written in Ashkenazi square script using the ASC dataset. Conventional computational methods, such as the bag-of-words approach, struggle to identify the intricate features necessary for effective paleographic clustering, as the frequency of occurrence of paleographical features varies even within the same script type. To address this, we had expert paleographers identify ten critical features that they use in their analyses of this script type. The ten features identified in this way are vocalization marks, left (end of line) justification, vertical stretch, strings, short descenders, fishtails, left slant, biting, nesting, and shading (Figure 4).
We trained a multi-label VGG-19 network [7] model to predict the presence of these features on a given page image. Treating this as a multi-label problem allowed us to account for the coexistence of multiple labels, as their spatial location and frequency of occurrence are not crucial for paleographical definition.
In order to prevent overfitting, we utilized the regularization approach known as early stopping. This technique stops the training process when the model's performance on the validation set stops improving, thus preventing the model from simply memorizing the training data.
As a result, we ended the training when the validation loss reached 0.12, resulting in the model achieving its best validation performance (see Figure 5).
To evaluate model performance on unseen data, we split the dataset at the manuscript level (Figure 6). Splitting at this level ensures that entire manuscripts, rather than individual pages, were held out for testing. Unseen testing involves evaluating the model on data that contains patterns not seen during training. This is important for ensuring that the model can generalize well to unseen data, like in the real-world scenarios where new manuscripts are encountered.
The prediction performance on the unseen test set, as shown in the bar graph, demonstrates that our model can effectively automate the tasks performed by a paleographer, achieving accuracy levels of 98%. The performance graph (Figure 7) shows three types of F1 scores: macro During training, we monitored the prediction accuracy for each of the ten labels to identify the ease or difÏculty of learning specific features (Figure 8). For instance, we observed that the "left slanted" feature took longer to learn due to its non-binary nature. It eventually achieved high accuracy because of its frequent occurrence in a single-page image. On the other hand, features such as "nesting, " "shading, " and "string" also took longer to learn due to their gradual values and finally resulted in lower accuracies due to their less frequent appearances. The "left slanted" feature, despite taking longer to learn, eventually achieved high accuracy, while features such as "nesting, " "shading, " and "string" features exhibited lower accuracies due to their gradual values and less frequent occurrences.
To identify the subclusters, we performed a brute-force search to find the feature combinations that lead to the most cohesive subclusters. Principal component analysis (PCA) was used to visualize the samples in 2D and identify potential clusters (Figure 9). In Figure 10, you can see a sample page labeled for its regional origin by a colored frame in each cluster. We systematically tested all features or selected features and found that 𝜒 2 feature selection led to visible clusters. This feature selection process highlighted visible clusters based on the selected features (strings, left slanted, vertical stretch, and nesting), addressing the challenge faced by paleographers who can quickly identify individual features on a single page but struggle to simultaneously remember and analyze these features across multiple pages to discern grouping patterns. The clustering algorithm mainly successfully grouped manuscripts of known provenance and suggested some meaningful grouping of other manuscripts. One of the main challenges in computational paleography is the time and effort needed to build an initial dataset. We plan to enlarge the existing dataset and experiment on other known clusters (Oriental square and nonsquare; Sephardic square, non-square, and cursive, etc.) and cluster the lesser studied script types such as Yemenite. We expect this to improve the results and to significantly advance our knowledge of both human and computational paleography. Each dot is labeled with the identifier of the corresponding manuscript.
Our approach tackles the challenge encountered by paleographers. They can quickly identify individual features on a single page but find it difÏcult to remember and analyze them across multiple pages to identify grouping patterns. We identified some clusters using this method and provided insights into the paleographic features that drive these formations. Hence, we automated some of the constraints of traditional paleographic analysis.
In future work, we aim to explore methods to discover discriminative features besides those defined by paleographers. Assuming that a script type 𝑆 ′ possesses 𝑛 distinct paleographical features that are absent in a baseline script type 𝑆 (Ashkenazi square script, in our case), we will train a multi-label CNN to predict the presence of all 𝑛 features in images of script 𝑆 ′ , while predicting the absence of these features in images of the baseline 𝑆. We can visualize the spatial locations of these 𝑛 features within the images of 𝑆 ′ using gradient-weighted class activation mapping (Grad-CAM). This approach enables us to identify characteristics that may not be immediately apparent to human experts, furthering our understanding of these script types.
We plan to incorporate another deep learning architecture to further enhance the representation of handwriting style features. For instance, we will train a sequence-generating recurrent neural network (RNN) on the ordered sequence of contour tip points from letter strokes. The hidden state vectors from the RNN will then be used as embedding vectors, which are expected to capture stylistic features of the handwriting.
Funded by the European Union (ERC, MiDRASH, Project No. 101071829). Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.