Automatic transcription of historical handwritten documents is a challenging research problem, requiring in general expensive transcriptions from expert paleographers. In Codice Ratio is designed to be an end-to-end architecture requiring instead limited labeling effort, whose aim is the automatic transcription of a portion of the Vatican Secret Archives (one of the largest historical libraries in the world). In this paper, we describe in particular the design of our OCR component for Latin characters. To this end, we first annotated a large corpus of Latin characters with a custom crowdsourcing platform. Leveraging over recent progresses in deep learning, we designed and trained a deep convolutional network achieving an overall accuracy of 96% over the entire dataset, which is one of the highest results reported in the literature so far. Our training data are publicly available.
Historical documents are an essential source of knowledge concerning past
cultures and societies [
In Codice Ratio is an interdisciplinary project involving Humanities and
Engineering departments from Roma Tre University, as well as the Vatican Secret
Archives, aiming at the complete transcription of the Vatican Registers, a corpus
of more than 18000 pages contained as part of the Vatican Secret Archives, with
minimal labeling effort. The Vatican Secret Archives is one of the largest
historical libraries in the world, containing more than 85 linear kilometres of shelving.
Interestingly, ‘secret’ does not stand for confidential, but rather denotes them as
private property of the Pope. The corpus is comprised of official correspondence
of the Roman Curia produced in the 13th century, including letters, opinions on
legal questions, addressed from and to kings and sovereigns, as well as to many
political and religious institutions throughout Europe. Never having been
transcribed in the past, these documents are of unprecedented historical relevance,
and could shed light to that crucial historical period. A preliminary description
of the system appeared in [
Our contribution. In this paper, we describe the design of a novel
component for optical character recognition (OCR) of the Latin characters extracted
from the text. Building a corpus for this task is extremely challenging due to
the complexity of segmenting the characters and reading ancient fonts [
Due to the many challenges involved in a fully automatic transcription of
historical handwritten documents, many researchers in the last years have focused on
solving easier sub-problems, most notably keywords spotting [
One of the largest effort to this end was the EU-funded tranScriptorium
project [
The dataset is collected from high-resolution (300 dpi, 2136 × 2697 pixels) scans of 30 pages coming from register 12 of Pope Honorii III. All pages are in the socalled Caroline minuscule script, which spread in Western Europe during Charlemagne’s reign and became a standard under the Holy Roman Empire. Compared to similar fonts, writings in the Caroline minuscule are relatively regular and have fewer ligatures. A sample text is shown in Fig. 1a.
All pages are pre-processed according to the workflow in [
Character classes. We take into account minuscule characters of the latin
alphabet, yielding initially 19 classes (a, b, c, d, e, f, g, h, i, l, m, n, o, p, q, r, s,
t, u) plus one special non-character class ⊗ . Since our dataset includes multiple
3 Segmenting and recognizing a character are two heavily interdependent processes:
this is known as Sayre’s paradox [
Crowdsourcing. To collect annotations on the segmentations of the manuscript words, a custom crowdsourcing platform was developed. We enrolled 120 highschool students in the city of Rome, that did the labeling as a part of a workrelated learning program. The task to perform was simple: having positive and negative examples for a given character, each student was required to select any matching images from a grid appearing on the platform. In Fig. 3, we show a screenshot of a task.
arranged in a grid, each with its own check-box. Every time the check-box is marked, the image receives a vote.
Image labels correspond to the most voted characters, among those with at most 3 votes.4 If there is no such character, the image is labelled with a special non-character class, denoting a wrong segmentation.
Characters with less than 1K
exFig. 3: Sample screen of our platform. amples were augmented to match the
required quantity and balance the
training set. The augmentation process involves slight random rotation, zooming,
shearing and shifting, both vertical and horizontal. Before training, all image
values are normalized in the range [
Our deep CNN takes as input 56 × 56 single-channel images, which are binarized
before training. The input is then propagated through 8 adaptable layers, whose
design is inspired to similar networks having achieved state-of-the-art results in
4 In our experiments, we did not observe any tie.
5 http://www.dia.uniroma3.it/db/icr/.
modern OCR recently [
The output of the ReLU is down-sampled using a max-pooling operation with stride 2 × 2 to reduce the number of adaptable parameters. The previous three operations (convolution, nonlinearity, and max-pooling) are repeated another two times, using 28 filters for the convolutional layer instead of 42. The output of the last convolutional layer is then flattened and fed through a fully connected layer with 100 neurons and ReLU nonlinearities, and a final output layer with a softmax activation function to output a probability distribution over the 23 classes.
In order to prevent overfitting, we apply 50% dropout during training [
N K
J (w) = − X X −yˆi,k log (yi,k) + λkwk2 ,
where N is the number of examples in the training dataset, K = 23 is the
number of classes, yi,k is the correct output of the kth class over the ith input, yˆi,k
is the predicted output of the network, w is the vector of adaptable parameters
of the network, and λ > 0 is a regularization factor. The regularization factor
is selected as λ = 0.001 by doing a grid-search over different values in an
exponential interval and computing the accuracy on a held-out validation set of 2500
examples from the original training set. This validation set is also used to
select a stopping point for the optimization procedure. We minimize (2) using the
Adam algorithm [
Overall accuracy reached 96%, while average precision, recall and F1-measure for each class are reported in Table 5 (support is always 100). The confusion matrix is shown in Fig. 4. Some typical errors are the following. • Characters “f” and “s1” are easily confused, due to their similar shapes.
Specifically, ≈ 8% of “s1” are labelled as “f”, and 14% of “f” as “s1”. • Images not containing any character are sometimes mis-classified as actual characters, mainly as “m” Specifically, ≈ 10% of “not-character” are labelled as “m”, and ≈ 15% of “not-character” are labelled as some other character. For comparison purposes, we report that a simple logistic regression classifier on the same dataset achieves average 80% precision and 79% recall.
Convolution visualization. We show in Fig. 6a the effect of the filters learned by of our network at the first level. Specifically, we show the result of convolution with first layer filters on a sample input image after the activation function (blues are positive values). Visually inspecting activation output is indeed useful for debugging purposes. In the figure, the effect of edge and lighting detection filters is clearly visible.
network, we now perform gradient ascent over the
input image (initially random) and maximize the output
of each filter, separately. This is a common step to
visualize what the network has learnt to recognize [
Intuitively, we generate synthetic images that maximize the activation of the filters of each layer, including the output layer. In deep CNNs, the first layers usually detect simple features, and the features become more complex and abstract as the layers go deeper. The result of this experiment for a sample of our filters is shown in Fig. 6b. The figure suggests that the first layer of our network is in charge of detecting edges, while the second layer exhibits more complex, geometrical patterns. Finally, the third and deepest layer, seems to detect whole character strokes.
Prec. Rec. F1 a 0.98 0.99 0.99 b 0.98 0.97 0.97 c 0.95 1.00 0.98 d1 0.97 0.98 0.98 d2 0.92 0.98 0.95 e 0.99 0.98 0.98 f 0.89 0.85 0.87 g 0.97 0.99 0.98 h 0.96 0.97 0.97 i 0.98 0.96 0.97 l 0.96 0.99 0.98 m 0.91 0.99 0.95 n 0.99 1.00 1.00 o 0.98 0.91 0.94 p 0.98 0.97 0.97 q 1.00 0.99 0.99 r 0.94 0.96 0.95 s1 0.86 0.90 0.88 s2 0.99 0.94 0.96 s3 0.95 1.00 0.98 t 0.94 0.97 0.96 u 1.00 1.00 1.00
0.95 0.74 0.83 ⊗ avg 0.96 0.96 0.96
Fig. 5: Per-class results.
After activation function
1st layer
2nd layer
(b)
3rd layer
In this paper, we have described the collection of a large corpus of annotated
Latin characters, and the design of a novel deep convolutional network for the
classification step. The described system is a key component in the In Codice
Ratio project, whose aim is to fully transcribe a large corpus of documents
contained in the Vatican Secret Archives [
Future work will require the design of a fully differentiable system to
substitute the currently hand-tuned segmentation step. Recently, indeed, some authors
have proposed the use of recurrent networks to process the entire text
sequentially [
We thank Debora Benedetto, Elena Bernardi and Riccardo Cecere for their help with the pre-processing steps and the crowd-sourcing application. Finally, we are indebted to all the teacher and students of Liceo Keplero and Liceo Montale who joined the work-related learning program, and did all the labeling effort.