Huge amounts of handwritten historical documents are being published by digital libraries world wide. However, for these raw digital images to be really useful, they need to be annotated with informative content. State-of-the-art Handwritten Text Recognition (HTR) approaches require an impressive training e ort by expert paleographers. Our contribution is a scalable, end-to-end transcription work- ow { that we call In Codice Ratio { based on ne-grain segmentation of text elements into characters and symbols, with limited training e ort. We provide a preliminary evaluation of In Codice Ratio over a corpus of letters by pope Honorii III, stored in the Vatican Secret Archive.
Large document collections are sources of important correlations between entities
such as people, events, places, and organizations. Previous studies [
Popular automatic tools for transcribing the text content of digital images
include Optical Character Recognition (OCR) systems, which work great for
typewritten text but are not suitable for handwritten text recognition (HTR).
Since most digitized documents by historical archives are manuscripts, HTR has
recently gained more and more attention by researchers worldwide. Handwritten
text is more challenging to recognize than typewritten one because characters
have less regular shapes, and are often combined into single units known as a
ligatures. Therefore, while OCR systems are trained to recognize individual
typewritten glyphs, most state-of-the-art HTR systems use holistic approaches: all
text elements (sentences, words, and characters) of a single text line are
recognized as a whole, without any prior segmentation of the line into these elements.
So-called segmentation free models [
Training with fragments has two advantages over training with words (as in segmentation free HTR): the number of fragments needed for training is much smaller, because it does not need to deal with the impressive variety of lexicon; fragments can be labeled by volunteers in large transcription projects, with little or no expertise, provided with adequate examples.
To this end, we set up a simple crowd-sourcing application.
Proof of Concept. Our hybrid work- ow takes the best of two worlds: we can handle challenging ligatures as in state-of-the-art HTR, and at the same time we require limited training e ort like typical OCR systems. Our system is not mature enough for a thorough experimental evaluation, but for sake of demonstration, we take into account the \Vatican Registrers" corpus in the Vatican Secret Archives. The Vatican Registers is a huge collection of volumes (more than 18.000 pages) produced in the 13-th century, and containing o cial correspondence of the Roman Curia, such as political letters, opinions on legal questions, and documents addressed to various religious institutes throughout Europe. Such records are of unprecedented historical relevance, have not been transcribed yet, and have a regular writing style (see Figure 1b). For these reasons, we believe that they can motivate our work.
Our crowd-sourcing application enrolled 120 high-school students in the city of Rome, that did the labelling as a part of their work-related learning program. The program is jointly organized by the engineering and humanities departments of Roma Tre University, and includes frontal lessons in a variety of topics, such as paleography, history and machine learning, thus also serving as school guidance. 2
tect macroscopic cultural trends has been discussed
for many years [
Data-driven history. Many studies have been
published over the past few years [
Handwriting text recognition. HTR can be de
ned as the ability to transform handwritten input
represented as graphical marks into symbolic representation as ASCII text.
According to the mode of data acquisition used, HTR can be classi ed into o -line
and on-line. In o -line systems the handwriting is given as an image or scanned
text, without time sequence information. In on-line systems the handwriting is
given as a temporal sequence of coordinates that represents the pen tip
trajectory. For high quality text images, current HTR state-of-the-art prototypes
provide accuracy levels that range from 40 to 80% at the word level [
Crowd-sourcing. Expected users of HTR technology belong mainly to two groups: individual researchers with experience in handwritten documents; volunteers which collaborate in large transcription projects.
Recent HTR project [
Other works. Language modeling for on-line handwriting recognition bears
many similarities with OCR and speech recognition, which often employ
statistical n-gram models on the character or word level [
Our system rst segments every word into small (possibly overlapping)
fragments, then recognizes characters in fragments, and nally the entire word. The
main steps are schematically shown in Figure 2.
1. Pre-processing. In this phase the color image of a page is cropped into
lines and words. The image is also transformed into a bi-chromatic one.
2. Cut-level operations. For each word we guess cut-points for characters
and build the so-called segmentation lattice data structure [
In this section we describe the inner phases of our system. Inner phases are designed for the Carolingian minuscule script, which is used in the manuscript for our proof of concept (see Figure 1b of Section 1).
Cut-level operations. The input for on-line HTR consists of pen up/pen down switching movements, therefore cut-points for character boundaries can be selected at certain positions of each stroke. Since we do not have access to stroke sequences, we design a simple heuristic based on black pixel distribution local minima, as shown in Figure 3a for a sample occurrence of the word \culpam". Then, we consider all the possible segmentation options induced by the cutpoints and let the further phases select the best one. We build a segmentation lattice where each start-end path represents a segmentation option.
Each cut-point corresponds to a node. The leftmost cut-point is the start node, and the rightmost the end node.
Each edge corresponds to the fragment of word bounded by its endpoints. We observed experimentally that edges corresponding to fragments smaller than 8 pixels or larger than 34 pixels can be safely dropped our further consideration. The segmentation lattice for our \culpam" example is shown in Figure 3b. In the gure, edges have labels, as will be clari ed in the next section.
fragment any squared portion of a word that is bounded by two cutpoints, including incomplete characters, combinations of incomplete characters, or multiple characters together.
Each edge of the segmentation lattice corresponds to a di erent fragment, and each path corresponds to a di erent segmentation option of the word into fragments. Our next step is a OCR step, where each fragment/edge of the segmentation lattice is labeled with the result of a character classi
er. There are many principled ways
for the classi cation task at hand. We
decide to use a convolutional Neural
Network (NN), that is one of the most
common and popular approaches [
Since the NN return a score distribution, rather than a unique answer, we add multiple edges between the same two nodes. Some of the fragments are submitted to a crowd-sourcing application, for training the NN. The application guides the workers, providing sample images of characters and highlighting variations in shape and style.
Therefore, workers without experience in transcription can give answers based on their perceived similarity of fragments to sample images. We refer to the resulting data structure as labeled lattice. (a) (b) Fig. 3: Cut-points for the word \culpam" and corresponding lattice. We use green for actual character boundaries, and red otherwise.
Word-level operations. Labeled paths represent candidate transcriptions for the current word. In order nd the best transcription recognition result, we use language models (LM). A statistical LM is a representation of a certain language as a probability distribution over sequences of words or characters. In other words, a LM expresses the likelihood that certain sequences of words or characters appear in texts written in the language under analysis. To this end, we downloaded a large medieval Latin corpus ( 1:5M words) and computed 3-grams frequencies. Then, we select the best path maximizing the corresponding word probability. For instance, for the path \culham" in Figure 3b we have7p($culham^) = p(cj$)p(uj$c)p(ljcu)p(hjul)p(ajlh)p(mjha)p(^jam), where $ and ^ are special symbols denoting the beginning and the end of a word. Every term in the product can be computed directly from language models. 4
mental results, which serves us as a proof of concept for In Codice Ratio. Ideas and methods in the proof have been realized in collaboration with the Vatican Secret Archives, with the aim of demonstrating in principle the practical potential of our system.
Dataset. The \Vatican Registrers" corpus consists of 43 parchment registers, for a total of 18650 pages (i.e., writing facades). All the registers are written with the same script: the so-called Cancelleresca.
Our dataset consists of 30 pages ( 15K characters) of register 12 by Pope Honorii III, that is the only Pope with un-transcribed registers and therefore is of most interest for the VSR. 7 Using 2-nd order Markov assumption. 8 \uiuscemod" and \i" are processed as separate words. future, we plan to repeat the training phase by removing such wrong items. Finally, \culpam" is overthrown by \cullum" because of more serious errors in the OCR process (\p" and \a" are labelled as \l" and \u', respectively). While word probability does not help (3-gram \llu" is more frequent than \lpa" in our LM), we are con dent to correct this kind of error in future works. Furture works include the following.
More sophisticated path selection criteria, for instance, taking into account NN output score (second \l" has lower score than \p").
More advanced language model tools, such as Hidden Markov Models. This is currently being developed with promising result.
Excluding labels that do not t in current line margins, such as excluding an \l" where the lower margin contains black pixels, such as for \p". 5
In Codice Ratio is an automatic transcription work
ow with low training e ort. Our proof of concept is done on a high-resolution digitized copy of the reg- path prob isters by pope Honorii III. Manuscript pages rst culpam culnedaenrgvoersaiosneroiefsthoef ttreaxntsifmoramgea.tiTonhse,nt,hparteperxotcreascsteda cccuuulllllpuuanmmi 318 111000 747 pages are decomposed into fragments containing ba- criminis sic text elements, such as characters and symbols. criminis 8 10 7 Some fragments are labelled by unskilled crowd- ccrriinnuuinuiss 46 1100 88 sourcing workers, which are asked simply to select uiuscemod(i) matching images to template symbols selected by uiufemod 2 10 2 paleographers. Labelled symbols are used to train a uuifemod 2 10 3 Neural Networks, that is in charge of automatically uiiifemod 5 10 4 compute labels for all the un-labelled fragments. Table 1: Path probabilities Automatically computed labels are aggregated at for the words in Figure 4. the word-level into a segmentation lattice, in which all the traversing paths represent candidate transcriptions for the word. Selection of the best path is done based on language models. Our proof of concept suggests that In Codice Ratio can be applied to the large collection of Vatican Registers in the Vatican Secret Archives, subject to speci cs improving, that are matter of ongoing work.
We thank Gaetano Bono glio, Veronica Iovinella e Andrea Salvoni for their work on the OCR neural network. We thank Matteo Mariani for helping with all the pre-processing steps. We also thank Gianlorenzo Didonato for developing the front-end of the In Codice Ratio crowd-sourcing application. Finally, we are indebted to all the teacher and students of Liceo Keplero and Liceo Montale who joined the work-related learning program, and did all the labelling e ort.