This paper presents a Linked Open Data publishing model for aggregating data from heterogeneous, distributed pre-modern manuscript databases into a global, harmonized data model and service. Our research hypothesis is that on top of the global data service based on ontologies and well-defined semantics, tools and applications can be created for solving novel research problems in manuscript studies using Digital Humanities methods. First results in implementing such a system in the international Mapping Manuscript Migrations project are described with lessons learned discussed in dealing with complex and imperfect historical data.
Thousands of European pre-modern manuscripts have survived until the present day.
As the primary surviving witnesses to the world of pre-modern Europe they provide
crucial evidence for research in many disciplines, including textual and literary studies,
history, cultural heritage, and the fine arts. [
Over the last twenty years there has been a proliferation of digital data relating to these manuscripts, including catalogues, specialist databases, and numerous collections of digital images – many of them IIIF-compliant6. But there is little in the way of coherent, interoperable digital infrastructure, with the result that large-scale discovery and analysis requires the time-consuming exploration of numerous disparate resources.
This paper introduces the Mapping Manuscript Migrations (MMM) project7 [
disparate sources, in order to provide researchers with searchable and browsable access to aggregated evidence about the history of pre-modern manuscripts.
In the following, we first present the publications model underlying the MMM initiative, where data from distributed heterogeneous manuscript data silos is converted and aggregated into a harmonized form and published as the MMM Data Service, based on Linked Data principles and best practices of W3C8. After this, a workbench MMM Portal under construction for studying the manuscripts is described. In conclusion, lessons learned from the work thus far are summarized, and directions for further research are outlined. 2
Publishing Model Figure 1 depicts the publication model of the MMM system. In
its initial phase, the project brings together more than 450 000 records from four
important European and North American datasets, which use very different data models: the
Schoenberg Database of Manuscripts9, Medieval Manuscripts in Oxford Libraries10,
and Bibale11 and Medium12 [
9https://sdbm.library.upenn.edu 10https://medieval.bodleian.ox.ac.uk 11http://bibale.irht.cnrs.fr 12http://medium.irht.cnrs.fr
Harmonizing Data Model As each of the source manuscript datasets involved in
the project have their own preconditions and goals, and thus follow their own data
modeling conventions, a unified data model for harmonizing them is needed. This model –
still under some development and to be published in detail later on – is based on input
from manuscript researchers with the goal of being semantically sufficient for
answering a wide range of research questions. The most essential core classes in the model
are 1) physical manuscript objects with properties such as shelf-mark, owner and
title, and 2) provenance events that describe events related to the manuscripts, such as
production, appearance in an auction, acquisition, etc. The model makes use of the the
CIDOC CRM13 [
The MMM publishing model facilitates the aggregation and concurrent use of
heterogeneous, distributed datasets with shared user interfaces, tools, and SPARQL queries.
In our work, the MMM Portal is implemented on top of the MMM Data Service (the
bottom in the Figure 1), but also other applications (left in the Figure) can make use of
the data in a similar way, including also the original data providing web services. The
MMM Data Service also includes a wide range of tools for managing and using linked
data, such as a linked data browser, SPARQL query interfaces, automatic
documentation tool, and so on. These services are provided by the Linked Data Finland platform14
that is used as the basis [
Data Transformation The data harmonization work was initiated by investigating the source datasets, starting with the Schoenberg Database. In our first demo system, the MMM Data Service contains a version of this dataset that is mapped into the harmonizing data model. Other datasets and transformations will be integrated later on in the system. Integrating the datasets has involved building individual pipelines for transforming the source datasets into simple RDF formats by the data providers. Three of them are customized relational databases, while the fourth dataset – the Bodleian Library’s catalogue – consists of XML documents encoded in accordance with the Text Encoding Initiative (TEI) Manuscript Description guidelines15. A special pipeline has been built by the data provider to extract and convert a selection of elements from these TEI documents into a record-like form more suitable for transformation to RDF. In the case of the Schoenberg Database, a new SPARQL endpoint has also been implemented by the data provider, which is available for general use16.
Matching between the various datasets has initially focused on shared places and persons. Two of the four data sources have now annotated their records with identifiers from the Getty Thesaurus of Geographic Names (TGN)17 and three with those from 13http://cidoc-crm.org 14http://ldf.fi 15http://www.tei-c.org 16https://sdbm.library.upenn.edu/sparql-space 17http://www.getty.edu/research/tools/vocabularies/tgn/index.html 223 the Virtual International Authority File (VIAF)18, as well as references to a range of other vocabularies. These identifiers have been used to match places and persons in the aggregated MMM project data. Matching manuscripts themselves is more problematic, since there is currently no standard for constructing and managing unique identifiers for manuscripts, though an International Standard Manuscript Identifier has been under discussion19. MMM has been testing and evaluating different approaches for assigning identifiers, including the use of ARKs (Archival Resource Keys)20 from the Bodleian Library and in the IRHT’s Medium database. 3
The goal of the MMM Portal is to provide a search and discovery interface for users with or without clearly defined research questions. The portal offers four main application perspectives based on the following classes of aggregated MMM project data: 1) Manuscripts, 2) Places, 3) People, and 4) Organizations. The instances of the core classes can be presented to the user as a paginated table, on a map based on various geographical information of the instances, and as a percentage frequency distribution based on arbitrary properties of the instances.
In each application perspective, the focus is on enabling the user to both explore
and browse the data freely and identify a group of instances of the core classes based
on a combination of criteria. In the Manuscripts perspective, a combination of criteria
could be manuscripts produced in Castile, including Spanish texts, previously owned by
English private collectors, currently owned by an institution in North America. Faceted
search [
At the moment a first version of the Manuscripts application perspective has been implemented. The perspectives for places, people and organizations can be constructed in a similar fashion by re-using the components of the Manuscripts perspective. Figure 2 depicts the Manuscripts perspective with faceted search. By default all manuscripts are shown in a paginated table. The key properties of manuscripts, such as shelf-mark, author or creation date, can be filtered with facet selections. Figure 3 shows an example of a hierarchical facet selection with search functionality. Besides hierarchy, the facet selections need to support further filtering. For example, the author facet must provide an additional filter for the birth date of the authors. All facet selections are connected, so whenever the user makes a selection, the value list of other facets is updated. This way it is impossible to end up with an empty result set by using any combination of the facets.
Moreover, each instance is associated with an information ”home page” with an aggregated description on the instance and how it is related to other instances. For a person instance there could be, e.g., lists of related manuscripts based on different roles such as author, scribe or owner.
18https://viaf.org 19https://www.irht.cnrs.fr/?q=fr/agenda/manuscript-ids-identifiants-des-manuscrits 20https://www.ifla.org/best-practice-for-national-bibliographic-agencies-in-a-digital-age/ node/8793
Result format Facet selections
Pagination
Figure 4 illustrates how the aggregated manuscript data (with optional filters selected by the user) are rendered on a map with clustered map markers, based on the creation places of the manuscripts. The numbers on clusters and markers indicate how many manuscripts were created in the area or specific place in question. When zooming in closer, Figure 5 shows how historical map sheets aligned on modern maps can be used to provide contextual information. However, using historical maps in this way is problematic in many ways and further user interface research is needed in order not give the end user wrong impressions about the data. Here 5 dots are spread over 5 spots in Paris, but there are also lots of dots that are all in one place corresponding to the general annotation ”Paris” that occurs frequently in the data. In general the place annotations can anything between a continent and a specific building, so a method for visualizing the varying granularity of geocoded data is obviously needed.
Furthermore, the times of the maps shown typically do not match with the times of the underlying manuscripts (or other data) that are also usually different from each other, too. For example, the place selected in Figure 5 is in fact the geographical spot of the current building of the Bibliothe`que nationale de France (BnF), which did not exist at the time of the map (neither the BnF as an institution, nor the building). The predecessor of the BnF, the French Royal Library, moved to this spot in the 18th century (well after 1705), but the urban landscape as depicted on the 1705 map has changed since.
The general architecture of the MMM Portal21 is presented in Figure 6. The system consists of a NodeJS22 backend build with Express framework23 (in the middle) and a frontend based on React24 and Redux25 (on the right). The MMM Data Service is shown on the left. An instance26 of MapWarper27 (on the left) can be used for aligning and publishing historical maps. When designing the architecture, the main goal of the backend was to ease the combining of attribute data from multiple SPARQL endpoints and raster data from various spatial data sources into a React frontend.
The data is published on the Linked Data Finland platform, which is powered by a combination of Fuseki SPARQL servers28 running in Docker containers29 for storing the primary data and a Varnish Cache web application accelerator30 for routing URIs and content negotiation.
21https://github.com/SemanticComputing/mmm-web-app 22https://nodejs.org/en/ 23https://expressjs.com 24https://reactjs.org 25https://redux.js.org 26http://mapwarper.onki.fi 27https://mapwarper.net 28https://jena.apache.org/documentation/fuseki2/ 29https://www.docker.com 30https://varnish-cache.org
Map Warper Map and spatial data services Data service Linked Data Finland
WFS
WMS, WMTS
MMM Portal Backend
MMM Portal User interface GIS software, statistical tools, etc. Digging into manuscript data has turned out in many ways more challenging from a data modeling and technical perspective than expected. Defining the very concept of ”the manuscript” itself raised many ontological modeling questions, since manuscripts can be just fragments of a whole, can be separated into parts, copied, annotated, and united to others over time. Also identifying records describing the same manuscript can be very hard, in many cases probably impossible, as they have been described in different contexts in different ways, in terms of different titles, and in different languages. There is no unique identifier scheme for manuscripts, in contrast to printed books, and library shelf-marks are not quoted consistently or accurately. The amount of data is also fairly large, hundreds of thousands of records, which sets efficiency requirements for the technical solutions.
The data are often also incomplete, uncertain, and imprecise in many ways. A
major goal of the project is to map manuscript migrations, i.e., to illustrate and study
manuscripts in spatio-temporal spaces using maps and and timelines, but references to
locations in many cases are missing, the mentions refer to historical places that may not
exist on modern maps or may have changed over hundreds of years of history [
Also the datasets turned out to be fundamentally different in nature. The data models used in the datasets were different in different collections from TEI to relational models and RDF. But most importantly, there are substantial differences in the semantic con228 tents of the datasets: the Schoenberg Database records primarily provenance events and observations of manuscripts at specific points in time, based on, e.g., auction catalogs, and does not focus on manuscripts as unique objects, while in Bibale, Medium, and the Bodleian catalogs the main focus is on describing manuscripts as objects.
The project started by creating a list of Digital Humanities research questions relating to manuscript histories, and continued by trying to figure out what kind of data model and data are needed to solve them. The next step was to find out, given the constraints imposed by the actual data available, what questions can be addressed and under what assumptions on data. Section 3 illustrated the first steps towards this ultimate goal of the project.
Acknowledgements Thanks to Kevin Page, David Lewis and Athanasios Velios for collaborations in developing the unified data model and working on the transformations related to the Bodleian library data. Benjamin Heller developed the transformation from the Schoenberg Database format to raw RDF from which it was transformed into the unified model. Similarly, Guillaume Porte was in charge of the transformation from the Bibale database to raw RDF. Discussions with Pip Willcox, Mitch Fraas, Doug Emery, Emma Cawlfield, Antoine Brix, Synn ve Myking and other members of the project team are acknowledged.
Our work is funded by the Trans-Atlantic Platform under its Digging into Data Challenge31 for 2017–2019. The project is led by the University of Oxford, in partnership with the University of Pennsylvania, Aalto University and Helsinki Centre for Digital Humanities (HELDIG) at the University of Helsinki, and the Institut de recherche et d’histoire des textes (IRHT). The authors wish to acknowledge CSC – IT Center for Science, Finland, for computational resources.