This paper is a use case report for the population architecture of a commercial Knowledge Graph. We introduce our pilots and their focus within the MindLab Project, which aims to build Knowledge Graphs with a lifecycle-based approach to enable conversational agents. We describe and evaluate our pipeline for the rst step of the lifecycle, namely Knowledge Creation. Our approach satis es all de ned requirements in terms of provenance tracking, scalability, and usability.
Knowledge Graphs are an important means to provide large-scale integrated
data to intelligent applications like conversational agents. Making a Knowledge
Graph a useful resource requires to complete a set of tasks that comprise the
Knowledge Graph lifecycle [
For each of these tasks, there are various approaches, mainly developed and applied in an ad hoc way. Even for the - seemingly straightforward - Knowledge Creation task, there are bureaucratic and technical challenges due to the variety and heterogeneity of data sources. In this paper, we address this very task by providing a holistic approach and architecture to populate a commercial Knowledge Graph based on heterogeneous data sources. The Knowledge Graph is built in the scope of the MindLab project3, which aims to build industrial Knowledge Graphs to enable conversational agents in domains like e-tourism and beyond. In the remainder of the paper, rst, we describe the use cases and their requirements that our Knowledge Graph must ful ll (Section 2). We describe our technical approach that provides a work ow for creating knowledge from heterogeneous sources and populating a Knowledge Graph (Section 3). We give a brief overview of the related work in Section 5. Last but not least, we evaluate our technical approach in terms of satisfying the requirements (Section 4) and conclude with nal remarks and future directions in Section 6. 2
Our general use case is to import and integrate various heterogeneous data
sources into one coherent data model using a schema that is highly aligned
to the schema.org, a de facto standard for annotations on the web [
In total, we selected three pilots from the tourism domain in our project. The three pilots are 1) Seefeld, 2) Serfaus-Fiss-Ladis, and 3) Mayrhofen.
All pilots have in common that users will interact with a bot in natural language (either via text or voice) and inquire about information related to the tourism domain. The data itself is modeled and stored in the form of a Knowledge Graph, which should be exploited to get i) a better language understanding, ii) more precise answers (result presentation), and iii) data-driven and guided dialogs.
Additionally, all pilots will operate over a set of data sources that contain information about various accommodation providers and local businesses, events, sports activities (e.g., hiking, trails, skiing), and sports areas (e.g., ski resorts) or webcams. The core tourism data is provided by ve di erent data providers. Feratel provides accommodation and event data, General Solutions geospatial data, Intermaps ski resorts and slopes, Outdooractive outdoor activity data (e.g., hiking tours), and Verkehrsauskunft O sterreich transportation data. The data comes in various formats (e.g., JSON and XML) and is either supplied in the form of data dumps or via RESTful API requests.
Some pilots will further connect the tourism data to public and open data sources, which contain additional common knowledge about some entities of the tourism data. For instance, DBpedia4 and Wikidata5 provide additional information about cities used in the pilots or descriptions about certain sports activities. Also, publicly available geodata will be used for more advanced geolocation related conversations (e.g., OpenStreetMap6). A prominent open-source geodata provider is OpenStreetMap7.
The data sources can provide static information (e.g., geolocations, names, categories) but also dynamic information (e.g.,, the current weather, snow level, open ski lifts, hotel rates, or transport information). Our solution needs to handle and represent such dynamic information, either in the form of updates in the Knowledge Graph or by specifying services to derive the current and most upto-date information. 2.2
Data importers The requirements for the data importers are that they have to scale with the input data, be usable and easy expandable, and that they track the provenance for generating statistics and easier debugging of the importer pipeline.
R-DI-prov the importers should track vital provenance and performance metrics that are used to lter subgraphs, generate KPIs, performance analysis, and assist debugging8.
R-DI-scale data importers have to scale w.r.t. the number and size of input sources. As such, the developed solutions should be executable in parallel setup.
R-DI-usability data importers should be designed in a way that they are easy to extend and adapt to new data sources. Ideally, the importing pipeline can be con gured and used without speci c technical knowledge (e.g., programming language agnostic) 3
In this section, we describe our technical approach to tackling the knowledge creation task while addressing the requirements described in Section 2.2. The major features of the architecture are: { Storing data as named graphs by utilizing quads
5 https://www.wikidata.org/wiki/Wikidata:Main Page 6 https://www.openstreetmap.org/ 7 https://www.openstreetmap.org/ 8 Also aligned with the Data on the https://www.w3.org/TR/dwbp/ Web
Best
Practices { Attaching provenance information based on the modeled provenance metadata (Section 3.1) { Scalable mapper for RDF generation from heterogeneous sources { Flexible, declaratively con gurable work ow for managing new data sources to populate the Knowledge Graph
The following explains how these features are conceptualized and implemented. We rst explain our provenance tracking approach, including our metadata. Afterward, we describe our implementation, MindLab Importer, for the overall knowledge graph construction process. 3.1
The MindLab Knowledge Graph must be consumable by applications from
different perspectives such as data from certain providers or a given set of
geospatial areas (R-DI-prov). The MindLab Knowledge Graph will be constructed by
integrating data from heterogeneous sources. In order to satisfy the R-DI-prov
requirement, we use the named graph [
Provenance Metadata The metadata used for describing the provenance is
mainly based on two vocabularies, namely PROV-O [
Listing 1.1 is an excerpt of the provenance information for a named graph. The named graph is connected to the organization (i.e., DMO Seefeld) and the IT solution provider (i.e., feratel) through schema:provider and prov:wasAttributedTo properties. 3.2
In this section, we introduce the architecture of the MindLab Importer and its
core components. Figure 1 provides an overview of the architecture. The central
component of the MindLab Importer is the Import Manager that coordinates
9 A comparison of di erent rei cation approaches can be found in [
Listing 1.1: An example provenance information attached TVB Seefeld and
feratel named graph
the import process. The data sources from which the Knowledge Graph is
populated are registered via a web interface to the Import Manager. When an import
starts, the Import Manager uses the source-speci c information (e.g., con
guration required to access an external source) to retrieve the raw data and passes
it to the mapping service. The service o ers a scalable RML [
register external
data sources
MindLab Import
Manager Collect Raw Data
Map to schema.org with RML
Add provenance information
Store in GraphDB
The import manager is implemented with the Apache NiFi12 data ow management tool. NiFi o ers a UI for de ning data ows in a drag-and-drop manner, alongside an API to create and manage such ows programmatically. It has many features like load balancing, bu ering, guaranteed delivery and scheduling out of the box. It o ers built-in processors for many tasks (e.g., connectors for di erent database solutions) and generic HTTP processors for accessing external services (e.g., our mapper service).
A NiFi work ow consists of connected units called processor. A processor typically receives a ow le, uses or manipulates it in some way and passes a new ow le to another processor. A subset of connected processors can be placed in containers called process groups. The Import Manager contains one process group for each distinct IT Solution provider and a single NiFi processor per organization per IT solution provider that acts as an initiator. This provides exibility and scalability for the creation of new import tasks to some extent. Once a ow for an IT solution provider is created, a new organization using existing IT solution providers can create their own import task by just entering the necessary con guration (e.g., organizations API key) through the registry. The registry interface takes care of the generation of the initiator processor, and its connection to the right work ow. 4
The data importing pipeline is implemented with Apache NiFi and a set of RML Mappers. We ran an evaluation for 29 days between 23.06.2019 and 20.07.2019.
We imported total nine named graphs: [
We calculated the average, median and standard deviation of both size and time elapsed for each import13. The entire graph ( 1.37M statements) is imported in approximately 26 minutes. This number is well within the daily frequency requirement. The largest graphs are provided from feratel, and they take 12 https://ni .apache.org/ 13 see the full statistics at : https://docs.google.com/spreadsheets/d/1C4aoBsN9p16LjPOLy33M9d8Mz0tFeUErrJDvRn approximately 1ms per Triple (Table 2). The overall overhead of API access and authorization has a bigger impact than the actual generation of the data; therefore, the time needed for generating RDF triples is actually signi cantly below 1ms per triple14. Moreover, Apache NiFi allows parallelization and distribution, which we can adopt if necessary. We satisfy the requirement R-DI-scale.
The Apache NiFi instance provides a user
interface for advanced settings and monitoring. Addi- Graph Time/Tripple
tionally, we provide a web interface for the creation [
Depending on the domain, one or a combination of some of the four approaches
is used more dominantly. For example, in the health domain, it is common to
extract knowledge from unstructured health records, whereas in the cultural
heritage domain, several methods need to be combined due to the
heterogeneity of the sources [
A particularly related work to our Knowledge Graph construction pipeline
comes from the automation industry [
in https://sumutcan.github.io/kgb-workshop-presentation/#/8/1
15 We refer readers to Section 4 of Deliverable 2.1 of the MindLab project [
Graphs use RML instead of R2RML. Moreover, we have an additional provenance model, since it is important to compartmentalize the data for di erent customers and applications. 6
In this paper, we described our use case, namely the population of MindLab Knowledge Graph, that is generated based on heterogeneous semi-structured data (e.g., in JSON and XML). Our main focus is the knowledge generation pipeline that populates the MindLab Knowledge Graph. Currently, a big majority of our data comes from external APIs that provide data in JSON and XML format. We implemented a exible and scalable architecture based on a high-performance RML mapper implementation and an open-source data ow management tool, Apache NiFi. We demonstrated that the developed tool satis es the requirements of the MindLab project regarding provenance tracking, scalability, and usability.
The next steps will focus on assessing and improving the quality and on further enriching the knowledge of the MindLab Knowledge Graph. The quality improvement step typically contains cleaning tasks such as handling the detection and correction of errors, which may also be introduced by fusing duplicate instances (e.g., di erent set of opening hours for the same hotel). We want to enrich the knowledge graph by creating new relations between entities. This involves to add new relations between entities (e.g.,, equality statements, missing links, or pre-computed relations such as "inWalkingDistance"), but also new information not available in the data sources. The Knowledge Graph should contain not only facts but also the description of actions and services. For instance, given the hotel domain, we do not only want to represent and search for hotel information but also describe the actions related to hotels, such as booking a hotel room or some services o ered by the hotel.
We would like to thank all participants of the MindLab project.