Historical manuscripts such as 17 ๐กโ century Spanish-American notarial scripts, carry a plethora of information that is highly useful to historians in understanding the social, economical, cultural, and political developments during different time periods. Manually searching through the scripts is time consuming and limits the scope of a historian's research findings. With advances in deep learning, OCR models have become more accurate and efficient. However, the lack of high quality training data on specific handwritten collections restricts applicability of pretrained OCR models. Furthermore, efficient and accurate document retrieval is required as the collections can contain millions of handwritten words.
In this paper, we present a new document retrieval system called KGSAR (KG for Spanish American Notary Records) for a set of hand-written Spanish notary documents from the National Archives of Argentina. KGSAR synergistically combines retrained OCR models and the concept of KG to address the challenges in accessing, reading, and searching within the documents. A KG can provide numerous benefits for information/document retrieval [8]. It can enable semantic search and better understanding of users' queries and documents as well as provide explanations for matched entities and their relationships [8]. In KGSAR, the Resource Description Framework (RDF) and SPARQL are used for efficient representation, indexing, and query processing of data extracted from the documents (e.g., predicted words) via OCR. The KG contains additional facts about the notaries, and is stored and queried using a fast graph database. The UI allows a user to provide additional training data for retraining the OCR models. The design of KGSAR is generic and can be easily adapted to other historical scripts.
Deep learning techniques achieve high accuracy when large, labeled datasets are available [3]. They have also enabled high quality OCR on handwritten documents. To use existing OCR models for specialized collections, we require high quality labeled data from experts.
Alrasheed et.al. [2] showed that after retraining on the Spanish-American notary records, Keras-OCR and YOLO-OCR achieved a better performance as compared to Kraken, Tesseract, and Calamari-OCR. When tested on our collection, the latter systems (which are based on pretrained models for the English language) were only able to detect lines over words and could not recognize any of the characters present in those lines. [12,13,14]. For an image containing 670 manually annotated words, Keras-OCR [11] and YOLO-OCR [9,10] were able to recognize 306 and 146 words respectively, while Kraken, Tesseract and Calamari-OCR were not able to recognize any words in the detected lines.
Shaw et.al. [5] proposed a system for converting handwritten medical prescriptions digitally using electronic writing pad and utilized OCR techniques for character recognition in the digital prescriptions, instead of whole words. Sugarawara et.al. [7] proposed a method for retrieving Japanese keywords using a text query where they first generated an image of the query text using Generative semi-supervised model, and then retrieved regions in documents similar to the generated image by feature matching. Preliminary works such as of Kim et.al. [6] presented an end-to-end system that combined word recognition using segmentation with a matching technique designed to handle the large dimensional feature vectors that represented shape description of characters in a word.
Unlike most prior work that focus on text recognition, KGSAR aims to synergistically combine OCR and knowledge management techniques to facilitate efficient and accurate retrieval of 17 ๐กโ century Spanish-American notarial scripts.
Seventeenth-century Spanish American notarial scripts include multiple handwritings due to a high turn-over rate in the notary office. Interim notaries did not receive extensive training, thus, handwriting in the documents consists of highly irregular scripts. The current implementation of KGSAR stores 20,000 out of 200,000 images that comprise the entire digitized collection.
KGSAR's architecture is illustrated in Figure 1. Component (A) transforms the document scans into grayscale, applies a median filter to soften backgrounds and removes background noise, and applies image binarization to convert the images to black and white as scanned document images contained noise that affected feature extraction and classification [1]. Component (B) contains 83 cleaned images (166 manuscript pages) that were labeled by Spanish-proficient labelers. This yielded a dataset containing 26,482 words for retraining the OCR models. This dataset is from the hand of Baldibia y Brisuela, who by 1650, acted as an interim notary in Buenos Aires, Argentina.
Pretrained Keras-OCR and YOLO-OCR models failed to identify handwritten text as they have been trained on printed English characters. Component (C) represents OCR model training where Keras-OCR recognizer was trained on 21,185 labeled words of 77 images and pretrained detector was used as it was able to accurately draw bounding boxes around the words. YOLO-OCR was trained in a novel way where YOLO was trained as a word localizer to predict only the bounding box coordinates, and convolutional recurrent neural network (CRNN) was trained as a recognizer to identify the text in the bounding boxes.
The retrained models were used to predict on about 20,000 unlabeled images. Component (D) denotes a KG representation built using the predictions. Entities such as the predicted words, bounding box coordinates, image containing the predictions, and the OCR model type that was used were stored as nodes in the KG. These nodes were connected using their respective relations, and serialized into N-triples format. The KG was stored in Blazegraph [4], a popular graph database, as denoted by Component (E). Bulk data loader was used to load all the N-Triples files as an atomic transaction.
Component (F) denotes an intuitive Web UI for a user to pose a keyword query. The word and its n-grams (for word length > 3) are used to construct a SPARQL query, which is executed by Blazegraph. We utilized Blazegraph's FullTextSearch feature to perform exact and partial word matching. Each search result was scored using the cosine distance to the query, allowing words with exact matches and higher match probabilities to be ranked higher. The matching scans were ranked to show the most relevant results. Component (G) denotes the annotation feature where a user can correct the results, delete or annotate more words after a query, to retrain the OCR models with better labeled data.
The UI was developed using HTML5 and AngularJS, and the backend code was developed using Python 3.8. We packaged the entire tool, Blazegraph journal and JAR file into a Docker image to facilitate quick testing and experimentation.
During the demo, a user can interact with KGSAR by posing queries and correcting the bounding boxes as well as labeling new words. We highlight the primary features of KGSAR. Figure 2 shows a screenshot of KGSAR after searching for the word poder 1 . The user will see the bounding boxes of the words matched in the images and can navigate through the images.
Figure 3 shows the annotation feature for the same word. Here, the user can see edit and delete options for the word, as well as the predicted value for the bounding box. The code for KGSAR is available on GitHub at https://github.com/MU-Data-Science/KGSAR.
Acknowledgments: This work was supported by a National Endowment for the Humanities (NEH) Digital Humanities Advancement Grant (HAA-271747-20) and a Research and Creative Works Strategic Investment Tier 3 Award from the University of Missouri System. We would like to thank Ryan Rowland and Adam Sisk for labeling a subset of the notary records.