Project In Codice Ratio is developing tools to extract the knowledge contained in the ancient manuscripts of the Vatican Archives. The scarcity of datasets suitable for our setting has led us to rely on crowdsourcing in all phases of our project. In this paper we discuss our approaches for leveraging inexpensive non-expert workers to fruitfully perform labelling operations on challenging manuscripts. We describe the range of di erent tasks we are devising, as well as the corresponding priority and redundancy policies we are employing. We describe the datasets collected thus far and the corresponding results.
Recent advancements in Knowledge Extraction from unstructured data have opened up new, exciting possibilities in digital humanities. These novel methodologies typically require large amounts of labelled data to reach satisfactory performances; nonetheless, when dealing with speci c tasks in vertical domains, the lack of suitable datasets makes it unfeasible to use such approaches. As a matter of fact, in many scenarios the only available resources are raw and unlabeled. This has led several projects to rely on crowdsourcing techniques, having human operators, or workers, manually process and label part of the available data. By gathering the outputs yielded by the workers it is possible to build new, extensive datasets to train automatic systems on.
Our Digital Humanities project (ICR) In Codice Ratio [
ICR aims at (i) obtaining a textual transcription of the manuscripts in the Archive, and (ii) extracting entities, relations and facts from the obtained text to construct an extensive Knowledge Graph (KG). Performing automatic transcription from the high de nition images of the document pages amounts to perform Handwritten Text Recognition (HTR), while extracting structured knowledge from the resulting transcription is a well known task of Text Mining.
Machine Learning (ML) approaches have been shown to reach state-of-theart performances in both tasks. Since our documents are written in Medieval scripts and languages, no comprehensive datasets could be found for the adequately training ML models. In this paper we discuss how we are leveraging crowdsourcing methodologies to process, label and enrich data, in order to collect suitable datasets, we describe the crowdsourcing tasks we have devised and the assignment policies we have implemented.
Our pool of workers counts over 700 members, and it is entirely constituted by high-school students in the city of Rome, that have joined the project for free as a part of their work-related learning program. In addition to performing tasks, students also attended frontal lessons in a variety of topics, from paleography to machine learning, managed by both the engineering and humanities departments of Roma Tre University. As a consequence, they are provided with an opportunity of personal growth, and they receive guidance for their future studies and careers.
So far we have focused on the transcription phase: with the data labelled by our workers, we were able to successfully generate datasets large enough for training and developing very promising HTR models. These results allow us to start assigning to our workers the rst batches of Text Mining tasks, with the goal of extracting knowledge from the transcribed text.
The rest of this paper is structured as follows. In Section 2 we report the HTR-oriented tasks we have employed so far, and we describe the obtained datasets and the corresponding transcription results. In Section 3 we discuss the typologies of tasks we have devised for text mining and knowledge extraction. We discuss related works in Section 4, and provide concluding remarks in Section 5. 2
The extraction of textual transcriptions from images is nowadays generally performed with automatic HTR models. As already mentioned in the Introduction, the lack of representative labelled datasets for the Papal Registers (i.e., similar script, abbreviations, and layout) makes it di cult to train a full- edged HTR system without dealing with a very expensive data preparation e ort.
Mark the segments that form symbols similar to these:
Next
We have thus employed a crowd-sourcing approach to build our own dataset
with workers manually identifying portions from the original images. The
obtained labelled segments are then gathered into datasets large enough to enable
the training of HTR models. The registers are strikingly hard to decipher, and
the extensive use of abbreviations makes them even less understandable.
Therefore, unlike other transcription projects [
In a preparatory step, we apply to the images of the manuscripts pages (i.e. written facades) a pipeline of computer vision operations. We rst run a custom binarization algorithm to transform them into black and white images. We then use the alternation between black and white sections to perform line segmentation and word segmentation. In each of the obtained word images, we analyse the upper and lower pro le of the writing, and cut the word wherever the stroke is absent or exceedingly thin. This is an over-segmentation, because the obtained segments are ner-grained than the actual characters in the word.
We then use this setting to generate our crowdsourcing tasks for character recognition. In each task, we use as an input the oversegmented image of a word, and we ask our workers if a speci c symbol is present within the word (e.g. symbol \ ", that is a Tironian note meaning \et"). If it is, the worker should highlight the set of consecutive segments that belong to that character. In case the symbol occurs multiple times in the word, the worker can also highlight multiple sets of segments. In order to facilitate the selection, in our UI we display each segment in a di erent color, as shown in Figure 1, in which the worker is asked to identify the symbols for character 'a'.
Our approach to design the task was inspired by [
Speci cally, we use the rectangular policy in [
We have then used the produced datasets to train our HTR model, based on
Deep Convolutional Neural Networks. Our model is described in greater detail
in [
The whole process that generates tasks, assigns them to the workers, and gathers the corresponding results, has been implemented as a Web Application hosted in the engineering department at Roma Tre University. We used a standard stack of technologies, using Java and the Spring Boot Framework, and storing data in a centralized relational DB. 3
After the correct transcription of a manuscript has been obtained, it is possible to extract its entities and relations, and to use them to populate a KG. In modern languages, these tasks are usually tackled by automatic classi ers based on machine learning and NLP approaches. However, similarly to the HTR step, when it comes to Medieval Latin texts the scarcity of labelled datasets makes it hard to train automatic models. Therefore, we are going to approach this phase too with crowd-based techniques. We are currently engineering two main categories of tasks for our workers: Entity Classi cation and Relation Classi cation. Each category may contain multiple types of tasks, with di erent requirements. Entity Classi cation. We have veri ed that, in our manuscripts, it is possible to automatically select almost all named entities by just applying simple heuristics, e.g. searching for capitalized words. Nonetheless, human intervention is still required to classify the selected entities into semantic categories, such as \people", \locations", \organizations", \temporal data" etc.
In a basic Entity Classi cation task, each worker may receive the text of an entire transcribed page, with the named entities to classify already highlighted; the task would consist in labelling each entity with the corresponding class. Note that, once the correct class for an entity has been identi ed, it is possible to propagate it to all the other occurrences of the same entity across the manuscript. For this task, we assume that workers can make mistakes, and manage redundancy with a policy similar to the Rectangular strategy employed for the transcription task, extending it to the case in which multiple classes are available. The order in which tasks will be handed to the workers will depend on their priority value, that is, the number of entities it contains that have not been classi ed yet.
Entity Classi cation can be further optimized by devising automatic labeling rules. For instance, if a named entity lies in close proximity to special keywords, such as the title "episcopus" (meaning bishop), one can infer that it belongs to the \people" category. Modeling these rules requires knowledge of the speci c syntactic patterns that occur most often in the manuscripts under analysis. Our workers, after they have become accustomed to the basic Entity Classi cation task, should ideally possess this kind of knowledge. Therefore, we plan to design advanced Entity Classi cation tasks based on rule building. We will provide students with the basics of rule-based logic, and divide them into groups. Each group will be receive, as an input, sets of several already labelled pages, from which they will be asked to build entity-classifying rules. We will evaluate the resulting clauses on standard metrics such as precision, recall and f-measure, and automatically select the combinations that maximize performances. We will nally apply the best performing mix of rules to pages in which the entities have not been classi ed yet.
Relation Classi cation. In order to build a KG, after extracting entities and their classes it is also necessary to identify the relations linking them. Once again, we will model basic Relation Classi cation tasks as tasks of manual labelling. In each task, the worker will receive as an input a single sentence containing at least two distinct entities, and will be asked to identify the relations occurring among such entities. The relation class can be chosen from a xed vocabulary, that can be manually extended, if necessary. Once again, we will assume that workers can make mistakes, and make the crowd results more robust through redundancy, employing policies similar to those described for the basic Entity Classi cation task.
The Entity Classi cation and Relation Classi cation tasks will yield large
amounts of labelled data that we will use to train automatic classi ers, similarly
to the HTR phase. Once the KG has been populated, further facts can be inferred
by leveraging recent Link Prediction techniques based on KG embedding (see [
Works related to ours can be roughly divided into Text Recognition and KG
Generation works. We summarize these works in the following paragraphs, focusing
on those based on crowdsourcing approaches, and refer the reader to the recent
works [
Our work is fundamentally di erent from these projects as the tasks we propose to workers do not require any transcription skills: each task only requires to identify speci c symbols or characters inside the image of a word and thus it can be solved by non-experts such as high-school students.
KG Generation Projects. Many open KGs nowadays are built in a
collaborative fashion, meaning that they rely on contributions to add missing information
or correct wrong pieces of the current contents. This collaborative e ort can be
seen as a very broad form of crowdsourcing, in which any user can temporarily
become a volunteer worker. In this setting each task has a very loose
formulation, with the worker herself choosing the entities, relations or facts to add or
update. Examples of KGs built this way include FreeBase [
While these approaches are viable for general purpose KGs, they do not work well when handling vertical topics that the workers may not have knowledge of, or when facts must actually be extracted from text. This has led researchers to devise more structured approaches for these scenarios.
In a completely di erent domain (drugs and their side e ects), the work in [
In this paper we have described how crowdsourcing methodologies are allowing us to tackle the extraction of knowledge from Medieval handwritten manuscripts in our In Codice Ratio project.
We have shown that the lack of datasets suitable for our scenario a ects both the transcription step and the text mining step. We have thus discussed how a pool of not expert workers can be put in condition to perform very simple tasks, yielding enough labeled data to train e ective automatic systems. We have described in detail our strategies to assign tasks to speci c workers, including the policies we employ to increase robustness to potential mistakes made by workers. We have reported our very promising results in character-level transcription, as well as our plans for the oncoming KG construction phase.
Among future work, we plan to inject data provenance methods in the whole process of knowledge extraction from ancient manuscripts, with the goal of improving the understanding of the results and simplifying the ability to trace errors back to the root cause. Work funded by Regione Lazio LR 13/08 Project \In Codice Ratio" (14832).