An important part in European cultural identity relies on European cities and in particular on their histories and cultural heritage. Nuremberg, the home of important artists such as Albrecht Durer and Hans Sachs developed into the epitome of German and European culture already during the Middle Ages. Throughout history, the city experienced a number of transformations, especially with its almost complete destruction during World War 2. This position paper presents TRANSRAZ, a project with the goal to recreate Nuremberg by means of an interactive 3D tool to explore the city's architecture and culture ranging from the 17th to the 21st century. The goal of this position paper is to discuss the ongoing work of connecting heterogeneous historical data from various sources previously hidden in archives to the 3D model using knowledge graphs for a scienti cally accurate interactive exploration on the Web.
Preparing cultural heritage collections for exploration by a wide range of users with multidisciplinary backgrounds requires to integrate heterogeneous historical data into modern information systems to structure and curation. Knowledge graphs (KGs) have proven to be a reliable way of structuring data for exploration purposes. They enable to connect data within historical collections by means of ontologies, enrich them with external data sources like Wikidata or national authority les, and identify entities within collections unambiguously by using URIs as persistent identi ers. However, historical collections are challenging to integrate into a KG as the data quality varies and rarely one- ts-all-solutions can be applied.
Goal of this position paper is to introduce the e orts of the ongoing research project TRANSRAZ3 in which heterogeneous historical data collections are Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
connected to an architectural 3D virtual research environment (VRE) using KGs to enable the exploration of the historic city of Nuremberg in di erent time periods ranging from the Middle Ages to the 21st century. Exploring city architectures along with the people living in it, their progress in technology, their craftmanships as well as arts and culture is highly relevant for many domains related to (digital) humanities. Nuremberg was one of the great European metropolises in the Middle Ages and beyond. It was the birthplace of renaissance artist Albrecht Durer, who worked there all his life. The city developed into the epitome of German and European history and culture. Then, during the Second World War, the city was largely destroyed and only few buildings could be reconstructed. However, without a systematic and scienti c reconstruction of the city in di erent time periods, this important part of the European cultural heritage will be forgotten.
This reconstruction of Nuremberg was rst initiated with the TOPORAZ
research project[
The remainder of the paper is structured as follows. Section 2 introduces related work, in Section 3 the TRANSRAZ project as main use case is presented. Section 4 presents an overview of the data, the envisioned work ow and curation challenges, followed by a conclusion in Section 5. 2
The interest in the representation and curation of the social, cultural, and geographical evolution of the past through digital systems has recently taken hold in several initiatives. Especially the exploration of urban spaces and their changes throughout time have been matter of interdisciplinary research projects for a few years. One of the most prominent and largest e orts in this domain was the Time Machine4 project. It aimed to rebuild the history of European cities, not only by digitizing the archives, but also by opening the doors to modern technologies to shape the elements that represent their evolution. As an example, spatial and temporal historical information about places, people, and events of the
city of Amsterdam have been digitized and represented through the Amsterdam
Time Machine5. This platform bridges data ranging from humanities to cultural
heritage, and provides linked open data (LOD) to explore. An important feature
provided by the Amsterdam Time Machine is the interlinking between the LOD
and 3D models, which provides the possibility to explore the city of Amsterdam
in space and time, and supports a better human understanding of its evolution.
The work by [
The Timetraveler Berlin application for smartphones lets users experience
historical multimedia content of the Berlin Wall by means of augmented reality.
The application guides users on a GPS-based tour to historically signi cant
locations in Berlin and displays stories of events that happened in history[
In comparison to previous attempts, TRANSRAZ will focus on automated
systems to transform the data from its unstructured form to meaningful
representations, and will provide data curation opportunities through ontologies and
KGs. The project will carry out research to study methodologies to integrate and
make sense of data coming from heterogeneous sources, and will provide formal
representations of contents for advanced explorations that are not supported
by current systems. Remarkable and timely examples of existing formal
representations are ArCO [
The use case of the presented approach is an exploration tool for the city of Nuremberg, provided in the ongoing project TRANSRAZ by means of an interactive 3D city model. Its goal is to provide scienti cally accurate means of exploration for Nuremberg in various points of time ranging from the 17th to the 21st century. The research of urban spaces and art history usually starts from location-bound objects, e.g. buildings or streets. The 3D reconstruction and research of buildings, creating complexes and entire topographical spaces that no longer exist or have undergone major changes is now an established procedure in the digital humanities. The presented approach will provide a direct
connection of the architectural objects to archival source material along with information on their origin, function and signi cance. These sources can be historical photographs, drawings, graphics, handwritten sources, address books and chronicles. Linking these information directly to the buildings in the 3D environment enables to explore the way of life in Nuremberg directly at the spatiotemporal point of action. Researchers are then able to explore the architectural changes along with the development of the public life. It will be possible to explore the development of craftmanship, arts, and culture to understand the way technological advancements spread throughout the city and to research the dynamics of local industries. In the predecessor project TOPORAZ, preliminary work on the VRE has been completed and is currently extended with a coverage of more than 3.000 buildings, enriched with historical data from several sources. An example of the exploration environment is pictured in gure 1. The user can explore the 3D space and a click on a building in the model triggers the information attached to a building. The 3D environment along with the connected data will be made available for researchers as well as for educational purposes and the general public. Fundamental for this exploration is the connection of accurate historical data of di erent time periods to the interactive 3D surface using an intelligent data model. This will be achieved by a knowledge graph which is currently under development. GLAM institutions will have the possibility to connect their resources to the KG provided in this e ort which widens the research area and allows to draw cross-connections between repositories. In the following section, the data that has to be curated and linked to the 3D environment to allow sophisticated research will be described, as well as the overall envisioned work ow will be presented. 4
An important step to depict the history of Nuremberg in a 3D VRE is to connect architectural objects to the people who lived there or owned them at a certain point in time. This allows researchers to gain a better understanding of the development of the city and its social networks as well as to add depth into a genealogical research of a family. 4.1
In this section, a selection of currently processed data sources is introduced with the goal to integrate the data into the TRANSRAZ KG and to connect it to corresponding objects in the 3D model.
Address Books. One of the rst printed sources that contains information about persons living in Nuremberg are the address books, which are physically stored in Nuremberg City Archives6 and in The Germanisches Nationalmuseum7. The annually published books starting with the year 1792 are provided digitally in as scanned images as shown in Figure 2. To access the contained text information, Optical Character Recognition (OCR) has to be performed. Challenges of the transcription process are manyfold and range from bad paper quality, distortion of pages, and poor inking up to exceptional linguistic features. Antiquated fonts, ligatures, archaic terms, old spelling variants, abbreviations and typos are common characteristics of these historical documents that complicate the recognition process. An additional challenge is to capture semantics encoded in non-alphanumeric symbols and speci c font features. For the sake of correct text segmentation and in order to avoid producing chaotic text blocks, most of the OCR systems are trained to remove any of such content from the image before the recognition. However, in historical documents such characteristics often contain meaning. For example, people whom civil rights were granted are pre xed with a circle ( ) and households with a telephone are denoted with a handset icon.
Nuremberg Artists Lexicon. The \Nurnberger Kunstlerlexicon" (NKL) [
MVGN. The \Mitteilungen des Vereins fur Geschichte der Stadt Nurnberg"8 (Journal of the Association for History of the City of Nuremberg) is a journal that publishes scholarly articles on all areas of the history of Nuremberg. Since 1879 an annual issue contains up to 40 reviews on important events, persons and every day life of the citizens. In cooperation with the Association for the History of the City of Nuremberg9 and the Bavarian State Library10 the MVNG were scanned and are now available online, however not in a textual form. A de ning feature of the articles in the early issues of MVGN is a long compound structure of the sentences lled with a wide range of coordinating conjunctions and descriptive introductory phrases and sentences.
Books of Nuremberg's Twelve Brothers. \Nurnberg Zwolfbruderbucher"11 were rst created in the middle ages as a collection of portraits and biographical data of old Nuremberg craftsmen that decided to retire in an old's people home. The books were previously digitized, transcribed and indexed12. During the transcription process no adjustment of the modern spelling was conducted. While indexing the entries, however, an alternative spelling with the modern alphabet and orthography was provided, and the abbreviations were resolved. The data is semi-structured, i.e., every entry contains information on rst names, last name, professions and di erent spelling variations of the data, the birth and death dates and places, the date when the person was registered in the retirement home and the duration of the stay. Also, descriptions of the person's portrait are mentioned.
To enable the exploration of the described historical data by means of the 3D VRE, the scanned documents will be transferred to machine understandable data. The work ow can be divided into four main steps which will be described below: OCR. Today, OCR systems applied on modern fonts reach such high accuracy that it is considered an almost solved research task. However, such models do not produce equally satisfying results if applied to historical writing. Several
models are trained to recognise historical text (e.g., Tesseract OCR13 or Calamari
OCR14). The address books described above mostly contain proper names and
historical job titles. Therefore, common approaches as e.g., high frequency words
[
Relation extraction. After the named entities are recognized, the relations
within the data have to be de ned and corresponding triples have to be generated.
The sentence structure of articles, e.g., in MVGN, is highly complex and the
information provided there is too extensive (i.e., each article has a main subject
with unstructured natural language text related to it). Therefore, a set of the
most relevant relations is composed together with domain experts. These relations
are used to describe birth and death dates, the date of marriages and divorces
and the date a person has entered or left a working position. Based on this list, a
subset of the textual data will be selected and the extracted named entities will
be linked to the subject of the article (e.g., person, organization) via prede ned
relations. This can be done by leveraging the format of the semi-structured
contents, or by exploiting data mining-driven approaches to relate the content of
texts to the designed ontology properties for the unstructured contents. Finally,
to represent the temporal context of a triple, several methods might be used,
e.g., rei cation [
Knowledge graph integration. Once the data sources have been analyzed,
the extracted information is represented in a KG. A major challenge here is
13 https://github.com/tesseract-ocr
14 https://github.com/Calamari-OCR/calamari
15 https://de.wikipedia.org/wiki/Kategorie:Historischer_Beruf
16 https://nlp.stanford.edu/software/CRF-NER.shtml
17 https://spacy.io/usage/linguistic-features#named-entities
the accurate alignment of entities (people, organizations, events). Furthermore,
the extracted data have to be integrated into the overall platform environment
including the interactive 3D model. As part of the predecessor project TOPORAZ,
a relational database was created linking data with the 3D model. However, this
approach lacks interoperability and, therefore, an ontology is currently under
development to describe and represent the historical city architecture, and connect
the resources described in 4.1 to the 3D model. Existing ontologies and thesauri
are reused for this e ort, including CIDOC CRM 18, ArCo [
In this section, the project's main challenges are presented, how they may be tackled, and how the expected results might support the understanding and curation of historical data.
Information extraction. TRANSRAZ builds on heterogeneous data sources that contain information that can be parsed, structured, and linked for a better curation. However, there are challenges that need to be addressed to correctly integrate them into the exploration environment. To start with, data sources describe Nuremberg's citizens without providing unique identi ers to distinguish them. For example, people who share the same name might be considered the same person, generating incorrect information about the story of the Nuremberg city. To tackle this issue, automated methodologies based on local information about people, places, and events will be studied to provide certainty of the correctness of the knowledge graph links.
needs to be addressed when parsing historical data is the evolution of the language. Obsolete words that are not used anymore, or words that changed their meaning, can lead to grasp an incorrect message from the sources of data and, therefore, to generate knowledge graphs that can mislead the correct interpretation of historical facts. An example is represented by the evolution of names and types of professions throughout the centuries.
Temporal component. Understanding and representing how the interactions between people, places, and events evolved in the city of Nuremberg require the association of temporal components. In fact, there might be relations between historical entities that existed only in a speci c point in time or intervals (for example a person who lived in the city from 1811). This kind of information can be easily interpreted by humans, but it might become challenging when formal representations are required to make the information machine understandable. Therefore, one important goal of the project is to come up with a reasonable representation of time and temporal relations by means of ontologies to provide appropriate means for correct interpretation of the represented facts. 18 http://www.cidoc-crm.org/ 19 https://www.getty.edu/research/tools/vocabularies/aat/ In this position paper, the vision of the project TRANSRAZ is explained in which KGs are created to connect heterogeneous historical data about people, organizations and events in the historic city of Nuremberg to an interactive 3D model. The nature of the data sources provide a number of curation challenges which are introduced in the paper. Most of these historical data are currently only available as scans and are hidden in archives unable to be explored by anyone. Publishing the data using KGs allows to make these sources available for an intuitive exploration on the Web, and enables to connect these resources to further repositories by GLAM institutions for research and education. Acknowledgement. We would like to thank Christiane Stockert, Felix Schonrock and Gerhard Weilandt for their indispensable input as domain experts. This work is funded by the Leibniz Association under project number SAW-2020-FIZ KA-4-Transraz.