Introduction

The digitization wave of the past two decades has significantly increased online access to historical manuscripts. Despite this progress, a substantial number of these documents are available only as images, lacking machine-readable text. Handwritten Text Recognition (HTR) has emerged as a vital tool for converting these images into text, facilitating the analysis of vast historical collections such as Camps et al. 's work [2]. Consequently, multiple large datasets have emerged in recent years [21,16,18,17]. However, most of these datasets are mono-or bilingual, with relatively limited geographical, temporal, scribal, and generic diversity. While this does not affect the quality of the datasets per se, it limits the generalization of models derived from them. Specifically, such models may face vocabulary limitations in the case of language or generic unicity (e.g., corpora composed solely of biblical content [7]), and graphical interpretation issues due to the lack of scribal, temporal, or geographical variation.

The Middle Ages, spanning approximately ten centuries, encompass a period of immense linguistic and cultural diversity. This era witnessed the evolution of numerous languages and dialects, each with distinct characteristics and scripts. From Old English and Latin to Old High German and Old French, the linguistic landscape of the medieval period was dynamic and continually evolving. This diversity poses both challenges and opportunities for HTR, as models must be capable of handling a wide array of scripts and languages that changed significantly over time. Addressing these challenges requires datasets that reflect the rich and varied nature of medieval manuscripts, incorporating a broad spectrum of geographical, temporal, and scribal variations to enhance the robustness and generalizability of HTR models.

In late 2023 and early 2024, the publication of the CATMuS Kraken model [13] and subsequently the CATMuS Medieval dataset [3] has opened up new opportunities for training and evaluating generic models across a vast diversity of categories and traits. With 200 manuscripts in their initial release in January 2024, and 250 in their 1.5.0 July release, encompassing 10 languages and 6 other metadata fields, these resources provide a robust framework for developing generalizable models that account for these specific features. However, in their initial study, Pinche et al. [13] indicate that the new generalizing models, trained on the comprehensive dataset, exhibited a drop in performance compared to earlier, more language-specific models. This finding seems to contradict the intended benefit of large, intercompatible datasets 1 .

One promising approach to mitigate these issues is the enrichment of handwritten text datasets with metadata. Metadata provides contextual information that can enhance model training and improve recognition accuracy. For instance, metadata on the century of production, language, script, and genre can help models better understand and adapt to the specific characteristics of the text they are processing.

This paper explores the potential need for metadata-enriched handwritten text datasets. We hypothesize that incorporating detailed metadata can improve HTR performance, particularly for complex historical texts. By analyzing the performance of current models on metadataenriched versus non-enriched datasets, we aim to demonstrate the benefits of this approach and propose a framework for its implementation.

Background and Related Work

Automatic text recognition in general and in particular the processing of historical typewritten and machine-printed material has seen a stellar rise in recent years. This advancement has had a profound impact on scholarly work, especially in the field of historical research. The retrodigitization and accurate transcription of most types of historical documents, which were once laborious tasks, can now be accomplished with relative ease and sufÏcient precision to enable a multitude of novel investigations.

Metadata and domain knowledge have long played important roles in the design of automatic text recognition systems (ATR). In fact, the limitations of early ATR methods, principally utilized for the processing of documents in tightly constrained domains, necessitated incorporating both to restrict the search space and boost accuracy to acceptable levels. Examples of these are systems designed to aid in automatic letter sorting where the vocabulary is effectively closed but also general-purpose ATR software such as Tesseract [15] utilizing extensive dictionaries and other means of language modelling.

Unfortunately traditional techniques to incorporate metadata have strong normalizing tendencies which are problematic for the recognition of historical documents which often have diverse language use, orthography, and multilingualism. While modern ATR software with its more powerful text recognition methods dispenses with many of these accuracy-boosting techniques, this is doubly true for software designed for historical document digitization like [9] which in most cases go to great lengths to eliminate them as far as possible.

Automatic Text Recognition

The principal paradigms employed in typical Automatic Text Recognition text recognizers have been stable for more than a decade although considerable research has resulted in recognition methods that are significantly more powerful, with higher accuracy, better generalization, and increased ease-of-training than the basic algorithm proposed in [6]. These recognizers are placed at the end of a pipeline of interconnected processes. A rudimentary but fairly standard ATR pipeline will ingest a digital scan of a page image at a time, perform any necessary pre-processing, e.g. rectification, dewarping, or binarization, find individual lines on the page image in a step called layout analysis, and feed the identified lines individually through the text recognizer. In a final step, the recognition results of the individual lines are reassembled into a paginated text by concatenation and serialization into raw text files or combined with data from the layout analysis to produce a digital facsimile, most frequently in standardized formats like ALTO or PageXML.

The most important feature of these ATR systems is that they implement text recognition as a sequence to sequence modelling task where the input sequence is typically a line image and the desired output sequence a string of characters. There are multiple ways to construct such a sequence-mapping text recognizer albeit the most popular way is with Connectionist Temporal Classification loss (CTC) [6] which permits the model to learn without requiring an explicit alignment between input and output. Further, these methods have multiple other advantages, some especially pertinent for historical document retrodigitization: training data creation is typically much faster than with older character-based ATR methods as line-wise annotation is generally more efÏcient, a lack of explicit character segmentation markedly improves error rates on cursive writing and connected scripts, and the ability of the recognizer to take contextual information into account boosts accuracy of characters that are difÏcult to recognize in isolation, e.g. in the case of degraded writing.

Style-aware HTR and other metadata-enriched architectures While interventions contributing domain knowledge into ATR systems at a general language level, e.g. with dictionaries or language modelling, are widespread, approaches explicitly leveraging other metadata that might be known about the text to be recognized have rarely been described in the liter-ature. Minor exceptions include a method described in [20], similar to the semantic context token in section 3.2, for the processing of standardized European Accident Statements, achieving a 10% reduction in CER with an architecture concatenating a metadata vector to the encoder features in a standard CNN-LSTM trained with CTC.

[1] describes a metadata-aware handwritten text recognition method albeit for a very different use case. A 𝑘-shot learning algorithm for style-aware HTR based on meta-learning, a base model is first trained from a text recognition training set enriched with writer labels where each meta-learning task corresponds to writing produced by a single writer. During inference on writing produced by a previously unknown individual scribe, an update of the model weights with a low number of labelled samples results in an adapted model for this particular scribe. This approach boost accuracy by around 5-7 percentage points in comparison to naive fine-tuning.

Automatic Text Recognition (ATR) datasets for historical, and specifically medieval, manuscripts likely began with Latin script datasets from the Historical Databases of IAM, notably the Partzival [5] and St. Gall [4] subsets. These datasets, which remain widely used for benchmarking new ATR engines, are relatively small (1,000 and 4,000 lines respectively), derive from single source documents, and are fundamentally incompatible due to differing annotation guidelines.

Late 2010s datasets, such as those developed by D. Stutzmann and the company Teklia [19,17,18] 2 , have taken a more focused generic approach (e.g., cartularies, books of psalms) and provided a significantly larger number of lines (more than 120,000 for HIMANIS). However, these datasets are limited by their generic and language unicity, and their use of annotation guidelines that resolve abbreviations restricts their reusability in multilingual settings. This is due to genre-or language-specific abbreviations and normalizations, which pose challenges for contextual-dependent abbreviation resolution [21].

The CATMuS dataset offers an innovative framework to address these limitations, enabling testing of ATR models across diachronic (8th-16th century), diageneric (from practical documents to poetry), and multilingual (10 languages) variations. With a consistent annotation approach, the CATMuS dataset [3] allows for the development and evaluation of single models capable of handling the rich diversity of medieval manuscripts.

Proposed method

We propose two basic approaches to evaluate the impact of metadata on recognition performance at different points of a text recognition method and evaluate it against a baseline of an advanced attentional text recognizer based on the Conformer architecture [8] and the default hybrid convolutional and recurrent neural recognizer of the kraken OCR engine. Although our experiments are run on an adaptation of fairly complex Conformer models the fundamental idea can be employed in almost any type of text recognizer based on neural networks.

Text Recognition with Transformers

The baseline system consists of an adapted Conformer, a Transformer-style [22] neural network augmented with convolutional layers, currently the dominant neural network architecture in automatic speech recognition (ASR). While ASR and ATR share many of the same features, e.g. relatively low-dimensional inputs and a prevalent sequence-to-sequence paradigm, there is no reported use of them in the ATR domain as of yet.

While the fundamental architecture requires no adaptation for text recognition, the size of even very large text recognition datasets is significantly smaller than the corpora of spoken speech typically used in ASR research which necessitates downscaling the network for reliable convergence (encoder_dim = 144, encoder_layers = 16, num_attention_heads = 4). In addition, we adopt the computationally more efÏcient depthwise-convolution downsampling schema (conv_channels = 32, subsampling_factor = 4) from [14] which roughly doubles inference speed without accuracy losses.

Our baseline recognizer consists of this down-sized Conformer encoder followed by a single fully connected layer as a decoder. Like most text recognition methods it is trained with CTC loss.

Semantic Context Token

Our first proposed method explicitly supplies the text recognizer with contextual information of the line to be recognized during training and inference. Given an input image of a line 𝐼 ∈ ℝ 𝑤×ℎ×𝑐 with height ℎ, width 𝑤, 𝑐 channels to the recognizer and a vector ⃗ 𝑡 ∈ {0, 1} containing the encoded metadata, which we will call the semantic context token, we simply expand the token to size 𝑤 × | ⃗ 𝑡| and concatenate it to the input resulting in a new input to the network 𝐼 ′ ∈ ℝ 𝑤+| ⃗ 𝑡|×ℎ×𝑐 . The neural network is then trained as usual with CTC loss.

The chosen metadata is encoded into semantic context token ⃗ 𝑡 through a simple multi-hot encoding, suitable for a wide-range of tag-type metadata, placing a high value at a particular position in the vector to indicate the presence of a tag. Classes are dealt with through expansion, e.g. for a language metadata field and possible values 𝐿 = {Castilian, Venetian, Latin} we would be converted into a semantic context token | ⃗ 𝑡 𝐿 | = 3. An obvious drawback of this method is that the text recognizer needs to be supplied the same array of metadata during both training and inference, i.e. it can only effectively recognize unknown text lines when the same metadata using during training is known.

Auxiliary Loss

In contrast to the first approach which is intended to induce the recognition model to context switch based on explicitly provided information during inference, our second method relies on an auxiliary loss during training to aid the network in learning the structure of the input data without requiring a semantic context token during inference.

Instead, the network is trained to reconstruct the semantic context token as the output of a side-branch of the text recognition network. This side branch, situated just after the Conformer encoder, consists of a simple adaptive max pooling and fully connected layer and operates on the totality of the encoder features. For a context token 𝑡 and a prediction of the side branch t of size 𝑛 the auxiliary loss 𝐿 aux is computed using binary cross-entropy (BCE):

𝐿 aux (𝑡, t ) = − ∑{𝑙 1 , … , 𝑙 𝑛 } ⊤(1)

where

𝑙 𝑛 = − [ t 𝑛 ⋅ log 𝑡 𝑛 + (1 − t 𝑛 ) ⋅ log(1 − 𝑡 𝑛 )](2)

The overall training objective thus becomes:

𝐿 = (1 − 𝑤) ⋅ 𝐿 CTC + 𝑤 ⋅ 𝐿 aux (3)

where 𝑤 is an additional hyperparameter of the training process that determines the proportion between the main CTC and auxiliary BCE loss. In line with common practice and confirmed with preliminary experiments we chose to put a relatively low weight (𝑤 ∈ (0.1, 0.3)) on the auxiliary loss during training. benefit from the expanded and more varied test set, enhancing the robustness of our evaluation without compromising the integrity of our initial training and validation processes.

Representing the diversity, or lack thereof, in the CATMuS dataset is challenging due to the various metrics (lines, characters, pages, or documents) and numerous features to consider (genre, language, script, century, etc.). Language can be seen as a super-category, which is then refined by genre if we view genre as primarily limiting vocabulary. In our dataset description, we focus on script (which can serve as a proxy for century), language, and use lines as the metric of choice. Lines are ultimately the unit used for training (sample and batch size) and offer a compromise between document and character count. However, it is important to note that some documents are heavily represented in terms of lines, while others have much longer lines (particularly in the context of prose vs. poetry), affecting the overall representation.

CATMuS 1.0.1 and 1.5.0 are heavily uneven across categories. In Table 2, we identify four particularly challenging "couples" in the test set: 156 lines of Castilian in Humanistica script, 273 lines of French in Semihybrida, 736 lines of Navarese, and 147 lines of Venetian in Textualis script. Each of these scripts has representatives in the training and development sets in other languages, but Venetian has only two documents in CATMuS (1 in train and 1 in test since CATMuS 1.0.0) and Navarese has only one document overall, and only in the test set. However, the Textualis script, which represents these languages, is the most common script in the training and development sets (see Table 1). We anticipate these test lines to be the most difÏcult for the model to predict. Latin is the most represented language across scripts, missing

Experiments

We perform experiments on the latest 2024 version of the CATMuS Medieval dataset. While this dataset is sufÏcient in size to train a Conformer model from scratch, the models in our experiments were fine-tuned from a base model trained on around 2.5 million text lines in a large number of scripts and languages in order to reduce the time and computational resources expended.

Implementation Details

All experiments are performed using the same hyperparameters and identical initial seeds. The model architecture follows section 3.1. Line images are scaled to a fixed height of 96 pixels and padded on both sides with 16 pixels. The batch size is set to 32, the maximum supported by our Nvidia A40 GP under BFloat16 mixed precision.

Models are trained using the AdamW optimizer [12] for 100 epochs with a cosine learning rate schedule with linear warmup over 35000 iterations, equivalent to slightly more than 8 epochs on our dataset and batch size. Initial learning rate after warmup is 3𝑒 − 4 decaying to 3𝑒 −5 by the end of the schedule. The network is regularized with weight decay (1𝑒 −5), dropout (0.1), and augmentation with random blurring, scaling, rotation, and elastic transforms5

Experimental Setup

We chose to evaluate our methods on a subset, shown in Table 3, of the line-level metadata provided by the CATMuS dataset. To determine the impact of each metadata field and potential synergistic effects on recognition accuracy, both methods were trained with language, script type, and age fields both separately and jointly. For the auxiliary loss weight an upper limit was determined empirically, from below which the values {0.1, 0.2, 0.3} were sampled for evaluation.

All models are evaluated on character accuracy. For comparison, baseline models were trained with both the default configuration of the Kraken OCR engine (CNN+LSTM recognizer) and the unmodified Conformer architecture.

Results

Table 4

Test Results (Character Accuracy): Models or combinations not reported failed to converge, exhibiting a micro-accuracy below 15%. Macro-accuracy represents the mean of document-level accuracies.

Input

Micro General results. Out of the two proposed architectures, only the Conformer model using contextual input tokens with all context tokens (Century, Script, Language) consistently outperforms the other models. Specifically, this model surpasses the baseline Conformer architecture, which itself outperforms the original Kraken baselines (see Table 4). Models that utilized a single category of features, such as Language or Century, ultimately performed worse than the baseline. The auxiliary loss approach yielded unexpected results: out of the 12 configurations (four types of tasks with three types of loss weights), half did not converge and resulted in character accuracies below 15%. Even worse, the observed unstable training behavior seems to be unrelated to the chosen weight 𝑤, which indicates that optimal hyperparameters must be determined for each new dataset and metadata token.

Accuracy dispersion across manuscript. The Contextual Input model consistently outperforms all other models, with the lowest median CER and the lowest variance. For the most challenging manuscript, it achieves over a 2 percentage point increase in accuracy compared to the Conformer baseline (see Table 5). Additionally, the Contextual Input model, without ablation, exhibits the smallest variance among all models (see Figure 3a). Compared to the baseline (see Figure 3b), the model utilizing the contextual token demonstrates superior accuracy, with a median improvement of 0.64 percentage points. It only underperforms on three manuscripts: Paris, BnF, fr. 6447 (baseline: 97.20%, -0.33); Paris, BnF, lat. 17903 (baseline: 80.25%, -0.32); and Paris, BnF, lat. 130 (baseline: 97.31%, -0.08). Ablation study. To evaluate the impact of the contextual token, we present results with null contextual tokens in Table 7. For models utilizing a single category of contextual input, removing the contextual token results in decreased accuracy, with macro-accuracy dropping by up to 3.2 percentage points for the model using the Century metadata and by as little as 0.88 points for the model using scripts. These findings suggest that the models may be overfitting to the contextual token, as evidenced by the baseline Conformer models outperforming them.

However, for the model using all contextual inputs (Context Input All), the removal of the context token leads to a smaller reduction in efÏciency. Despite being less efÏcient with null contextual tokens, the model still leverages learned features during decoding, aligning with our expectations for the Auxiliary Loss training architecture. The minimal variation in accuracy between the zeroed-out context and the full context (< 0.15 percentage points) while still surpassing the baseline may indicate that the model has effectively learned to separate features, even without manually provided context.

Impact of unknown features.

In documents featuring unknown or extremely rare features, such as the Navarrese language (unknown) and Venetian language (represented by only one training sample), our results not only remain stable but also surpass those of the conformer model when utilizing all contextual tokens. Particularly noteworthy are manuscripts BnF 65 and BnF ita. 783 (cf. Table 6), where we observe consistently stronger performance. Even in cases with null semantic tokens, we achieve improvements ranging from +0.2 to +0.4 points in

Conclusion

In this study, we explored the effectiveness of incorporating contextual metadata into Handwritten Text Recognition (HTR) models to enhance the digitization of medieval manuscripts.

Utilizing the CATMuS Medieval dataset, which offers a rich variety of scripts, languages, and centuries, we compared the performance of Conformer models with and without contextual inputs, as well as training these models with auxiliary classification tasks. Our objective was to determine whether adding metadata such as Century, Script, and Language could improve model accuracy and robustness. We tested several configurations, including models with single and multiple contextual tokens, and evaluated them against both the baseline Conformer architecture and the original Kraken baselines. By doing so, we aimed to identify the most effective strategies for leveraging contextual information in HTR tasks.

Our results showed that the Conformer model using all contextual input tokens (Century, Script, Language) consistently outperformed other configurations, including the baseline models. This model achieved higher accuracy, particularly on the most challenging manuscripts, with an improvement of over 2 percentage points in some cases. Moreover, it exhibited the smallest variance in performance, indicating its robustness across different types of manuscripts. The use of multiple contextual tokens enabled the model to effectively learn and utilize diverse features, leading to better generalization. Interestingly, models with single contextual tokens did not perform as well and often fell short of the baseline, suggesting that a more comprehensive approach to metadata integration is necessary. Additionally, the auxiliary loss approach did not yield the expected improvements and frequently resulted in non-converging models, indicating the complexity of effectively balancing multiple training objectives.

While our approach demonstrated significant improvements, there are several areas for future exploration. The current approach relies on multi-hot encoding various categories without embedding these features into a learnable space beforehand. Approaches in natural language processing, such as [10], could potentially allow the model to approximate relationships between scripts and languages that are closely related, such as 'Caroline' and 'Humanistica' scripts. Secondly, the context token method appends information directly onto the image data fed into the encoder, a design choice motivated by the very lightweight FFN decoder which we deemed to be unlikely to effectively make use of the encoder features augmented with the raw context token. Combining a more powerful decoder, e.g. a pre-trained language model like in [11], and injecting metadata after the encoder is an avenue of future research. Such an architecture with a clear separation between the visual and linguistic model would presumably be beneficial for some types of semantic tokens, in particular language and genre, which we consider to be of more importance to the latter than the former.

A. Appendix